AI Security and PHI: Why Many Healthcare AI Vendors Fall Short—and the Mistakes Health Systems Pay For

Healthcare organizations are under extraordinary pressure to do more with less, and AI promises relief: faster prior authorizations, stronger appeals, cleaner documentation, and fewer hours lost to administrative work. That promise is real. But the speed of adoption has created a security gap that most organizations have not yet closed.

The evidence is consistent. IBM’s 2025 Cost of a Data Breach report found that 13 percent of organizations reported breaches involving their AI models or applications, and another 8 percent did not know whether they had been compromised in this way. Of those that were breached, 97 percent lacked proper AI access controls. Sixty-three percent of breached organizations either had no AI governance policy or were still developing one, and only 34 percent of those with a policy regularly audited for unsanctioned AI use.

Unsanctioned AI—commonly called shadow AI—is its own source of exposure. One in five organizations in the same study reported a breach tied to shadow AI, and organizations with high levels of it saw an average of $670,000 in additional breach costs. Incidents involving shadow AI compromised personally identifiable information more often than the global average.

For healthcare, the stakes are higher than in any other sector. Healthcare breaches averaged $7.42 million in IBM’s 2025 study, the highest of any industry for the fourteenth consecutive year, and took an average of 279 days to identify and contain. In calendar year 2024, the HHS Office for Civil Rights received 663 reports of breaches affecting 500 or more people, together affecting approximately 242.9 million individuals. The largest single incident, a ransomware attack on a healthcare clearinghouse, ultimately affected an estimated 192.7 million people.

Many AI products marketed to healthcare organizations are, at their core, a user interface and a set of prompts placed in front of a general-purpose language model operated by someone else. That approach can produce an impressive demonstration quickly. It does not, by itself, produce a healthcare-grade system.

General-purpose models were not designed around the obligations that attach to protected health information. When a product simply forwards clinical documents to an external model endpoint, the most consequential security questions are answered by default rather than by design: what data leaves the organization, where it is processed, how long it persists, who can access it, and whether it can be reconstructed later.

The issue is not whether a vendor uses large language models. Most modern AI systems do. The issue is whether the vendor has built the boundaries around the model—de-identification, isolation, access control, retention limits, and auditability—that clinical data requires. A thin wrapper inherits the risk profile of every system it touches while giving the health system little visibility into any of them.

Health systems evaluating AI partners should therefore look past the interface and ask a simpler question: if this vendor’s application were removed, what protections would remain between our patients’ records and the model?

The most common shortfall is the belief that a signed Business Associate Agreement resolves the security question. It does not. A BAA is a legal instrument that allocates responsibility and establishes permitted uses of PHI. It is essential, and it is required. But it is not a technical control.

A BAA does not prevent unnecessary PHI from being transmitted to a model. It does not stop identifiers from being written to logs. It does not limit which employees can see a patient’s record, and it does not detect when a sub-processor’s environment is compromised. When a breach occurs, a BAA determines who notifies whom and who bears liability. It does not undo the exposure.

Regulators are signaling the same distinction. In its December 2024 proposed update to the HIPAA Security Rule—the first major revision since 2013—HHS proposed requiring encryption of electronic PHI at rest and in transit with limited exceptions, multi-factor authentication, and a written technology asset inventory and network map showing how ePHI moves through an organization’s systems. Those are architectural expectations, not contractual ones.

The right posture is to treat the BAA as the floor. The questions that matter come after it is signed: what is the minimum PHI this system needs, where does it go, and what technical controls enforce those answers?

The documents that make healthcare AI valuable are also the most difficult to protect. Consult notes, pathology reports, operative notes, treatment plans, and appeal packets are unstructured narratives. They do not store identifiers in tidy fields that a filter can remove.

A clinical narrative can identify a patient without ever stating a name. A rare diagnosis, a specific treatment sequence, a referral from a named practice in a small town, an unusual occupation, or a precise timeline of events can each narrow the field of possible individuals. In combination, they can point to one person. Pattern-matching redaction that looks only for names, dates, and record numbers routinely misses these indirect identifiers.

This is why de-identification method matters. HIPAA recognizes two approaches: Safe Harbor, which removes eighteen specified identifiers, and Expert Determination, in which a qualified expert evaluates the data and the context of its use and certifies that the risk of re-identification is very small. For unstructured clinical text processed by intelligent systems, the second standard is far better suited to the actual risk.

The most protective design de-identifies information before it ever reaches the reasoning engine, so that the model works on the clinical substance of a case rather than on the patient’s identity. If a vendor cannot explain where de-identification happens, what method it follows, and whether that method has been independently evaluated, the health system should assume that identifiable narratives are reaching the model.

Even organizations that carefully control the primary data flow often overlook the secondary ones. AI systems generate exhaust: prompt logs, response histories, debugging traces, analytics events, error reports, cached documents, vector indexes, and backups. Each can contain the same PHI the main workflow was designed to protect.

This exhaust frequently lives outside the systems the health system governs. A model provider may retain inputs for abuse monitoring. An observability tool may capture full request bodies. An error-tracking service may record the document that caused a failure. A retrieval index may persist embeddings derived from clinical text long after the original case is closed. None of these is visible from the product interface.

IBM’s 2025 research found that the most common AI security incidents occurred in the AI supply chain—through compromised applications, APIs, and plug-ins—and that 60 percent of those incidents led to broad data compromise. Every additional sub-processor that touches PHI expands that supply chain.

Health systems should require a complete data-flow inventory from any AI partner: every system that receives PHI or data derived from it, the retention period for each, whether customer data is used to train or improve any model, and how deletion is verified. A vendor that cannot produce this inventory does not fully know where its customers’ data lives.

Security is not only about keeping data in. It is also about controlling what the system does with it and being able to prove it afterward. AI that produces clinical arguments, coverage determinations, or appeal letters without showing its sources creates a different kind of risk.

When a reviewer cannot see which policy criteria, clinical facts, and evidence a recommendation relied on, errors pass silently into submissions and letters signed by clinicians. When access is not tied to identity and role, staff may see records their work does not require. When actions are not logged, an organization cannot reconstruct what happened during an audit, a dispute, or an incident investigation.

Physicians are already wary of opaque automation in this space. In the AMA’s 2024 prior authorization survey, 61 percent of physicians said they were concerned that AI is increasing or will increase prior authorization denial rates. Health systems that adopt AI on their own side of the process should hold it to a higher standard of transparency than the systems they are working against.

The controls are well established: single sign-on through SAML, automated provisioning and deprovisioning through SCIM, role-based access, a complete audit history of every action and review, and outputs that link each conclusion to the source material behind it. Most importantly, a qualified person should review and approve the work before it leaves the organization.

Just Healthcare Solutions was built for the work where these risks are most acute: prior authorization, appeals, peer-to-peer preparation, and revenue operations that depend on detailed clinical narratives. Our approach starts from the assumption that every layer must protect patient information on its own.

Protected health information is de-identified locally in the browser before a case reaches our clinical reasoning engine, so the intelligence works on the clinical substance of the case rather than on the patient’s identity. Our de-identification methodology was independently evaluated under HIPAA’s Expert Determination standard, and that evaluation found that it met the required very-low-risk standard.

Patient information is encrypted at rest and in transit, with customer-managed keys available. Access is controlled through SSO/SAML and SCIM provisioning, with role-based permissions so people see only what their work requires. Every action and review is recorded in an audit history.

Every recommendation is tied to the payer policy, clinical facts, and evidence behind it, and your team reviews the work and decides what happens next. We believe this is the minimum a health system should expect from any partner trusted with clinical data. The organizations that insist on it will be the ones that capture the benefits of AI without inheriting its risks.