AI as scaffold, not authority

Most AI applications are about doing things faster: drafting, summarizing, coding, predicting. Ethics is different. The work isn't speed; the work is thinking right under uncertainty. So what happens when you put an LLM in that role? Does it improve human moral reasoning, or does it just automate the biases we already have, faster?

I built a case study to test that for a Business Ethics course at NYU. The result is below. The short version: the AI helped, but only in the specific ways I learned to use her for. The places she failed are where you'd expect, and the failure modes matter more than the wins.

The build

The consultant I built was called Dr. Amina Solberg. I gave her three intersecting domains of expertise: bioethics, business ethics, and healthcare operations. The persona ran on Claude Opus, with a system prompt designed to reason across all three simultaneously rather than within one at a time.

Why three domains? Real ethical dilemmas in healthcare aren't single-domain problems. The bioethics frame asks what's right for the patient. The business frame asks what's sustainable for the institution. The operations frame asks what's actually feasible given the constraints. A consultant who can only see one of these gets you a thinner answer than the situation can absorb.

The dilemma

The test case was a real hospital problem: should the hospital restrict uninsured-patient appointments in order to boost insured-patient satisfaction scores? It's a real question because both directions cause harm. Restricting access for uninsured patients is discriminatory. Letting wait times balloon for insured patients reduces satisfaction, hospital ratings, and ultimately the revenue that funds care for everyone.

I gave Dr. Solberg the case and asked for her recommendation.

What the AI did right

The most useful thing she did was reframe. Instead of treating the dilemma as a patient-group conflict (insured versus uninsured, zero-sum), she reframed it as a systemic capacity problem. The hospital was operating at capacity, and any policy change was actually a question of how to expand or reallocate capacity, not how to ration access.

She generated seven alternative solutions in a single pass, including options I hadn't considered when I framed the problem myself. She recommended a hybrid approach that avoided overt discrimination while addressing the wait-time problem through scheduling reform and bridge-program enrollment assistance for uninsured patients.

The reframing was substantive. It wasn't something I'd have produced alone in the same time, and it changed the entire problem space.

What she got wrong

Two things, both instructive.

First: she underestimated coercion risks in what she called "enrollment assistance." She framed it as helping uninsured patients sign up for coverage they were eligible for. In practice, "help" delivered with appointment access as the leverage isn't help. It's pressure. This isn't a logic error. It's a context-blind assumption about how power dynamics work in real hospital settings, and it's the kind of thing a domain-experienced human catches in seconds.

Second: she overestimated capacity-expansion feasibility. Hospitals don't just hire more doctors when wait times grow. There are budget cycles, regulatory constraints, credentialing pipelines, physical-space limitations, payer-mix economics. She recommended capacity expansion as though it were a software config change.

Neither failure was about the model being unintelligent. Both were about the model not being able to *see* constraints that exist in the world the recommendation has to land in.

The takeaway

The right job for an AI ethics consultant is to generate frameworks, options, and reframings, expanding the decision space and making the trade-offs explicit. The wrong job is to issue verdicts.

Dr. Solberg generated seven options I wouldn't have generated alone, in less time than I could have generated three. That's the scaffold value. But a human had to be the one who said "the coercion risk is real, drop that option" and "capacity expansion isn't on the table this fiscal year." The recommendation that survived contact with reality was *human-edited AI output*, not raw AI output.

The headline question (does AI improve ethical reasoning, or just automate bias faster?) has no clean answer. Both happen, depending on the architecture and the human collaboration around it. The case suggests the answer is shaped less by the model and more by how the human treats the model's output. Treat it as authority and you automate bias. Treat it as scaffold and you can do better thinking than you could alone.

That's the rule that holds, anyway, until the next case proves it wrong.