Our Point of View

Hardwired Inequity: How Big Tech’s AI Arms Race Puts Patient Care at Risk

With the right guardrails and regulation, AI tools could support breakthroughs in healthcare: from faster research to more personalized care. But that promise will be squandered if we fail to put patient safety, equity, and care above profit.

Even the people who build AI tools are worried about the race to adopt them. A manager at an AI lab at a major medical institution recently shared a concern with me: their team is one of the few prioritizing ethical development and patient-centric design — putting them at odds with an industry-wide race to be first to market. That conversation stayed with me because it crystallized something essential: AI itself isn’t the problem. The problem is the human choice to put speed and profit above care.

During the COVID-19 pandemic, a long-hidden flaw in an everyday medical tool — the pulse oximeter — was brutally exposed. These small devices, clipped to a patient’s finger to measure oxygen, were consistently less accurate on darker skin. This wasn’t new; researchers had documented it decades earlier, but the design flaw lingered in millions of devices. When COVID hit, skewed readings fed into hospital triage algorithms that decided who received supplemental oxygen and who was turned away. The result was predictable and devastating: Black patients with dangerously low oxygen levels were more likely to be undertreated.

The pulse oximeter was one of the most visible examples of how design bias turns deadly. But it was not unique. For decades, White male participants have been overrepresented in medical research. Drugs, vaccines, and treatments have often been tested primarily on this group, leaving women and communities of color underrepresented in clinical trials. The result is predictable: therapies that are less effective or riskier when prescribed more broadly. Women, for example, metabolize some drugs differently, which has led to serious side effects that never surfaced in trials dominated by men.

These gaps don’t just live in the past; they still shape today’s standards of care. Now imagine that history of bias scaled across every corner of healthcare — not just in one device or a handful of trials, but in the AI systems hospitals, labs, and policymakers are choosing to adopt to guide diagnosis, treatment, and access to care. That is the future the Trump administration and Big Tech are building. The rest of us? We are sleepwalking (nope, speedwalking) into it.

The question isn’t whether AI will be integrated into healthcare, but whether decision-makers will hardwire the biases of the past and present into the future of care.


The AI Action Plan: A Health Policy in Disguise

The Trump administration’s AI Action Plan barely mentions healthcare, but its impact could reshape the future of our health system more than any policy in a generation. Released in July, the plan calls for a “try-first” culture enabled by “regulatory sandboxes” — environments where companies can rapidly test AI tools while bypassing rigorous oversight. The plan promises to fast-track “private-sector-led innovation” by requiring AI tools to exclude established social determinants of health like race, gender, and environment, which it calls “ideological.” It even recommends revising the AI Risk Management Framework to explicitly eliminate references to misinformation, Diversity, Equity, and Inclusion, and climate change.

In practice, this pressures developers to strip out the very factors that shape real health outcomes. While the administration rolls back oversight and narrows which data are allowed to count, any model that dares to recognize the social determinants of health could be branded political — and buried.

This is not happening in a vacuum. The administration has already purged CDC websites, halted NIH funding for projects tied to diversity, and granted political appointees the power to cancel research grants that don’t “demonstrably advance the President’s policy priorities.” These deliberate policy choices are shaping which data gets collected, which research gets funded, and what kinds of AI tools will be built. The selective erasure of data isn’t symbolic: it’s structural. And it is hardwiring bias into healthcare’s future.


Silicon Valley Breaks Things

Silicon Valley’s “move fast and break things” mindset has infected the hallowed halls of medicine. From Stanford to Harvard to Mayo, elite medical institutions are in an AI arms race. Venture dollars and hospital budgets flow toward the labs and administrators who embrace AI the fastest, not those who do the hard work of interrogating bias and safeguarding patient care.

This is not “AI taking over healthcare.” It is medical leaders, academic institutions, and decision-makers deciding to roll out unproven systems at scale. They are choosing to treat questions of privacy, bias, and patient well-being as hurdles for later. But healthcare is not software. You cannot beta-test with human bodies. Once a flawed tool is integrated into hospital protocols, its blind spots don’t just linger — they become practice. Then policy. Then a part of law.


Garbage In, Bias Out

AI is only as good as the data we feed it. That’s data science 101. But when our government narrows what counts as valid research — when race, gender, and community-level data are stripped from official statistics — the “garbage in” doesn’t just produce sloppy outputs. It produces health misinformation disguised as medical objectivity.

Recent studies have shown how dangerous this can be. One AI model confidently hallucinated a nonexistent body part. Others fabricated citations or offered treatment recommendations that contradicted evidence-based practice. Researchers at Mount Sinai found that a single misleading input could send chatbots spiraling into confident but false advice.

These errors don’t happen in a vacuum. They are not the fault of an inevitable technology. They are the direct result of human choices to ignore safeguards, to adopt faster than they regulate, to treat bias as an acceptable tradeoff for speed.

Human providers are biased too. Women’s pain is dismissed more often than men’s, and some medical professionals still falsely believe Black patients feel less pain. But AI adds a dangerous twist: it takes these existing biases and wraps them in a veneer of objective truth. When a large language model generates a diagnosis or treatment plan, the process is a black box. The biases shaping its outputs are invisible to both provider and patient. Worse, those outputs then feed back into training data, reinforcing the very inequities they produced. Bias in, bias out — then warped and folded back in again. A feedback loop that turns misinformation into a treatment plan, making inequity self-reinforcing, scalable, and nearly impossible to root out.

But once again, the responsibility doesn’t lie with AI. It lies with the institutions and decision-makers choosing to deploy it. It’s Big Tech and medical institutions integrating black-box systems without demanding transparency. It’s policymakers declining to require audits or bias disclosures. AI systems aren’t hardwiring inequities into healthcare — they are.


The Choice is Ours: Build Care or Hardwire Harm

By 2024, two-thirds of physicians were already using AI in their practice, a 78% jump from the year before. Soon, not using AI could be seen as malpractice. That means whatever biases are baked into these systems will no longer remain at the margins. They will become embedded in healthcare itself.

If the Trump administration succeeds in narrowing health data to fit its political ideology, and if Big Tech succeeds in commercializing biased AI tools at breakneck speed, we risk building a healthcare system that cannot see the full humanity of its patients. Patients of color misdiagnosed. Women’s symptoms dismissed. Immigrants erased from datasets entirely. And all of it presented as “objective truth.”

We’ve seen this movie before. When bias is embedded into healthcare, it becomes practice. Practice becomes policy. Policy becomes law. The AI Action Plan isn’t just about technology; it’s about who gets counted, who gets care, and who gets left behind.

If we want a health system that serves everyone, we must demand stronger regulation.

Treat AI tools like medical devices, with rigorous safety trials and oversight. Invest in equitable data, undo the rollbacks on research into health disparities, and expand datasets that reflect the full diversity of patients. Require transparency and accountability, including disclosure of training data and bias audits. And most importantly, amplify community voice — ensuring patients, especially from marginalized groups, have a say in how AI is built and deployed.

The promise of AI in healthcare is real. And, so is the peril. Left unchecked, it won’t just mirror inequities, it will automate them. But the real threat to patient care isn’t AI itself; it’s institutions, policymakers, Big Tech, and medical professionals choosing speed and profit over patient safety and community care.

Silicon Valley’s mantra has been “move fast and break things.”

In healthcare, what breaks is people.


Series Note

At mBOLDen Change, we believe our policies and systems should advance social equity and the common good. That includes fighting for technology that serves people, not profit. In this piece, we call out the ways medical institutions, Big Tech, policymakers and other decision-makers are rushing AI into healthcare without safeguards.

This article is the third in our ongoing series on the impact of AI on vulnerable communities, nonprofit systems, and the future of equity. If you missed the earlier posts, you can read them here: ‘What Does It Mean To Be Human In The Age Of AI’ and ‘Draining Vulnerable Communities To Feed The AI Race’. Stay tuned for our next installment, where we’ll continue to expose risks and lift up alternatives that put people, not profit, at the center.


Get new posts by email

Join the conversation on Substack (opens in a new tab)