AI holds great potential in healthcare, but its application must be 'extremely cautious'

Las Vegas — When the University of Illinois Hospital and Health Sciences System tested an AI-powered message reply drafting tool, a patient misspelled a medication name, recalled Chief Health Information Officer Karl Kochendorfer during a panel discussion at the HLTH conference last week.

Because a nurse forgot to double-check the reply, the misspelling led the AI to provide side effects for a medication the patient was not taking.

In the end, it wasn't a major issue—just a call to the patient or another message to correct it, he said. But it could have had serious implications for the tool.

"It almost killed the pilot... and it was on day one," he said.

As the healthcare industry grapples with how to safely deploy AI, investors and health systems are first seeing potential in adopting automated administrative and back-office tools. Such tools could ease clinician burnout and pose lower risk to patient care, experts said at the HLTH conference.

But pressure to adopt the technology is real. Supporters argue AI can help address a major workforce challenge in healthcare: according to consulting firm Mercer, as the overall population ages and care needs increase, the U.S. will facea shortage of more than 100,000 critical healthcare workers by 2028。

Experts say that while AI could be transformative, the industry must proceed cautiously when deploying emerging tools. The risks cannot be ignored, as policymakers and experts raise concerns about accuracy, bias, and safety.

Rohan Ramakrishna, co-founder and chief medical officer of health information app Roon, said during a panel at the HLTH conference that deploying AI in healthcare is complex, and the industry should learn fromsome predictive tools deployed previouslylessons.

"I think one thing we've learned is that applying AI solutions in healthcare settings must be done with extreme caution," he said.

How AI can address the 'simple mismatch between supply and demand'

Daniel Yang, vice president of AI and emerging technologies at Kaiser Permanente, said during a panel that AI could help alleviate one of healthcare's biggest problems: a growing number of older patients with more complex chronic conditions, while fewer doctors are available to help.

He added that as older Americans need more care, the nation's largest generation—millennials—want a more immediate consumer experience. But with limited supply, that goal is hard to achieve. Training a new doctor takes years, and fewer doctors mean more burnout, delayed care, and rising costs.

"This isn't even about AI; it's about what I think is a widespread problem in healthcare," Yang said. "What we're seeing is a simple mismatch between supply and demand."

AI can enhance clinician workflows, potentially helping them provide better care. Yang mentioned that an algorithm developed by researchers at The Permanente Medical Group can save 500 lives a year by flagging patients at risk of clinical decompensation—that is, deterioration.

"It's better for me because it genuinely saves time. But ultimately, it's better for patients because they feel heard. It's been a truly transformative experience for me."

— Christopher Wixon, vascular surgeon at Savannah Vascular Institute

The technology could also reduce burnout and improve retention by cutting the time clinicians spend on administrative tasks like note-taking. Providers have long reported spendingsignificant timeon electronic health records, which oftendetracts from patient care。

Christopher Wixon, a vascular surgeon at Savannah Vascular Institute, said he once came close to leaving medicine as the industry shifted to electronic health records. Gathering information while listening to patients was a major challenge, and it was easy to miss nonverbal cues when forced to stare at a laptop screen.

But ambient documentation—where AI tools typically record conversations between clinicians and patients and draft notes—has been a game changer, he said.

"It's better for me because it genuinely saves time," Wixon said during the panel. "But ultimately, it's better for patients because they feel heard. It's been a truly transformative experience for me."

Investors and health systems focus on administrative burden

Given the heavy administrative workload for providers and concerns about errors or bias in models involving clinical decisions, products that address administrative issues have become one of the top priorities for AI applications.

Some investors are also more interested in automating these operational tasks.

"We continue to focus on that 'unsexy' back-office automation that truly reduces labor burden, while clinical AI is completely off the table," said Payal Agrawal Divakaran, partner at .406 Ventures.

According to areportreleased by Silicon Valley Bank earlier this month, administrative AI has also attracted more venture capital funding this year. As of 2024 to date, administrative AI companies have raised $2.4 billion, compared to $1.8 billion for clinical AI, likely due to lower regulatory and institutional barriers, especially for decision support tools.

"All the action we're seeing is concentrated on administrative tasks, including low-value tasks like prior authorization," said Megan Scheffel, head of credit solutions for life sciences and healthcare banking at Silicon Valley Bank, in an interview. "You can move office staff to higher-value projects."

Greg Corrado, distinguished scientist at Google Research and head of health AI, said during a panel that there are currently many opportunities to use large language models as drafting tools, including for notes or nurse handoff documents.

Because providers must review outputs before finalizing, oversight is built in. He said it's also easier to assess quality by asking about user experience or checking how many edits they need to make.

But Todd Schwasinger, partner at Cleveland Clinic Ventures, said in an interview that evaluating operational or administrative tools still requires a methodical approach and testing with the health system's own patient data. Cleveland Clinic's governance structure also focuses on issues such as the data used in tools, how information is protected, and whether the product is safe and positively impacts patient care.

Administrative or operational products—such as ambient scribes or revenue cycle management tools—are a safer starting point.

"You're not taking the risk of making clinical decisions, right?" he said. "That level of trust doesn't exist yet. I think it takes time."

Preparing for AI deployment

Although AI may show promise in easing provider burnout, health systems still face challenges in engaging providers, setting up pilots, establishing governance policies, and scaling products.

In one case shared at the HLTH conference, St. Louis provider BJC decided to use vendor data to identify clinicians who spent days on documentation sign-offs or wrote lengthy notes, for an ambient note-taking pilot, said Michelle Thomas, associate chief medical information officer for ambulatory at BJC Medical Group and chief medical information officer at BJC Medical Group.

"I think only one out of 20 people responded. So we immediately realized that those who needed it most weren't interested," she said during the panel.

She said they decided to change direction and invite anyone interested to join the pilot—and they quickly got a response.

Still, it's important to carefully consider which providers should participate in testing, because some doctors don't understand the requirements of being in a pilot, such as taking time to send feedback or dealing with frustrations from a new product.

Health systems should also consider what outcomes they hope to achieve when adopting AI tools. Thomas said many physicians didn't see significant time savings because they spent a lot of time editing notes—not for errors—but to match their own writing style. In contrast, advanced practice providers signed off on ambient notes more quickly after review.

"You really have to decide what your return on investment is. Are you going for financial savings? Time savings? Or do you need hard data to justify the technology? Or something softer? Are you going for patient satisfaction?" she said.

Cybersecurity—already a widespread challenge in healthcare—is also key to AI deployment.

Organizations should think about the same issues they would with any system using protected health information, said Melanie Fontes Rainer, director of the Office for Civil Rights at the U.S. Department of Health and Human Services, in an interview.

If you have a partnership with a developer, have you signed abusiness associate agreement? Have you considered deletion policies for data stored in the cloud? Have you thought about who needs access to this data?

"I think there really needs to be a balance here, but it requires all of us to think responsibly about how we use this information, how it could harm our systems, our patients, and how we take proactive measures to protect it," she said.