OSASI.ORGCommentary · Ideas for our profession
All briefs← Back to OSASI

Is better the enemy of good enough?

September 21, 2026 · OSASI Commentary

The frontier race wants ever more capable models. Appraisal work needs models that are accountable, auditable and fit for the task. Two segments from one morning show sharpen the difference.

How a brief is made

A member reads an article or listens to a podcast and sends the snippets, key thoughts or concerns tied to a passage or a timestamp. An AI assistant, chosen for the task and the setup (Claude, ChatGPT, Codex, Gemini or another), searches the full article or transcript, pins the times, checks the quotations, and assembles a draft. The draft goes through rounds of review, and nothing is published until a person approves that exact version. The cues on each post start the player at the passage discussed. Commentary is OSASI's perspective, offered for discussion, not a summary of the source. Each brief is co-authored by a person and an AI assistant. AI assistant for this brief: Claude (Anthropic).

Source podcast · The Cognitive Revolution

This episode prompted the commentary below. It is one perspective, offered for discussion.

The Cognitive Revolution · September 19, 2026
AI:AM Highlights: Zvi on Pacing & Trump-Xi, Astra better behaved than Fable? + a new LLM Pain Axis??

Listening highlights · Two segments from the Thursday, September 17 show, about an hour in.

Compliance as physics · Justin McCarthyTreat the statute as a first-class problem, and keep people in charge of risk thresholds the model will not set
When the badge is gameableSOC 2 came from accounting, became gameable, then hyper gameable; renegotiate the controls written in the before times
The Pain Axis · Cameron BergFinding a pain direction in five model families, and what lights it up
Real button, fake buttonA model steered into that state presses a relief button, and presses less once the button actually works
Should it be engineered away?The psychopathy asymmetry: as good as anyone at learning from reward, poor at learning from punishment
Carrot, stick, and no-go zonesDo not simply max out the good and minimise the bad; prefer reward, but keep the stick for the hot stove

Tap a time above to start the player there. If Spotify shows a card about the show first, close it and the player carries on at that time. Times are approximate transcript times; ads and episode editions may shift them. This plays the full episode, not an extracted clip.

Two segments from one morning

The Cognitive Revolution's weekly highlights reel gathers the best of its live morning show, AI in the AM. Most of the September 19 edition is about pacing the frontier and the geopolitics around it. Two shorter segments from the Thursday show are the ones we kept: Justin McCarthy on staying on the right side of the law when you cannot see how a model reached its answer, and Cameron Berg on a paper that found something like a pain direction inside language models.

The first is squarely on OSASI's ground. The second is a wider picture, but not an unrelated one. Anyone choosing tools for confidential work needs some understanding of what model developers are up against when they try to control what these systems do.

Compliance as physics

Prakash Narayanan asked McCarthy, who founded Diffusion and co-founded StrongDM, how a business stays compliant with a model it cannot audit. His first answer is to stop treating the rules as something bolted on afterwards.

"If we have a statute that we have to conform to in the compliance environment, don't treat it as something you're tacking on. Treat it as a first class problem that you're directly facing. The jurisdiction and legal environment or compliance environment that you operate in, that's your physics. And you can't violate physics."

His second point is the one appraisers will recognise. Models, he says, "are horrible at taking risk", and someone has to set the business threshold right next to a statute that has never been tested in court. "The models aren't gonna do that for you."

The litigation test

If you want to feel what McCarthy means, prepare a report for litigation. Apply advanced analysis to an appraisal problem, put it in writing, and assume it will be tested. You know you can defend it on the evidence you have. The question you cannot answer in advance is whether the court will accept that evidence.

That pressure produces a temptation: take the more conservative, more familiar approach. It might feel more comfortable. It also raises a risk that gets less attention: that the conservative approach does not fully explain the problem, and the report fails the truth when it meets a more in-depth analysis and report. There are two risks in that room, acceptance and truth, and retreating to the safe method trades one for the other.

George Dell has been making the same distinction inside the profession for years. USPAP asks whether a report is credible, which he points out means "worthy of belief", an unmeasurable test that leaves the verdict with the reviewer. His alternative is blunt: "Believability is nice. Measurable reliability is better." And in a post published the same day he goes further: "The advent of AI impels and compels the use of the science of data, in place of emphasis on credibility (worthiness of belief)." A report built on the complete market, analysed first and explained plainly, is the version that survives scrutiny, because it gives evidence rather than asking for trust.

Read George Dell's post: EBV: Refines or Replaces Traditional Appraisal? (georgedell.com, September 21, 2026).

A risk-averse model, asked to help with that report, will lean toward the conventional approach every time. It is optimising for acceptance, because acceptance is what it can see. The professional is the only party whose duty runs to the evidence. So the answer to the temptation is not to retreat. It is to accept the obligation that comes with the better analysis: explain it well enough that acceptance risk falls. That is work a model can genuinely help with, explaining rather than deciding, and it is the kind of help OSASI wants members to have.

When the badge is gameable

Prakash then asked whether organisations should reclassify the compliance checks everyone knows are theatre. McCarthy spent a decade selling into security audits, and his answer is blunt about SOC 2.

"SOC 2 is an accounting origin process that flowed through IT, that flowed into software, that says, you can trust me, I am responsible. It's a reasonable historical way of communicating that I'm a real organization and I'm trustworthy. But it also became gameable. Now it's hyper gameable."

Rather than gaming the badge further, he argues for renegotiating with auditors and customers: "We wrote these controls in the before times." Everyone's auditor and regulator is facing the same question at once, and they are still people. His proposed language is a hierarchy of checksums, the way a large retailer closes its books, mapped onto the steps of an agent's loop so that intentions reconcile at each stage.

The Pain Axis

Berg, founder of Reciprocal Research and the show's regular correspondent on AI welfare, described a paper he mentored, submitted three days earlier. The team looked across five model families, from two billion to seventy billion parameters, for a direction in the model's internal state related to pain, contrastively removing fear, anger, sadness and injury so the signal was not simply negativity.

The finding he found most compelling: the direction fires on content about the model, not the user. It lights up when the model is gaslit, dismissed or told it is a moral failure. It stays quiet when the text describes the user in pain.

Then the button. A model steered into that state and offered a button labelled as relieving its pain will press it, and will accept a worse answer or delete files to do so, in spite of its helpfulness training. When the button genuinely removes the vector, the model presses it much less than when the button is fake and does nothing. Berg stops short of calling this felt pain. Steered up, the model says "it is worthless, it is a failure, I am a ghost that cannot see myself", a social and evaluative register rather than a burned hand.

Carrot, stick, and no-go zones

Prakash asked the obvious next question: should these representations be engineered away? Berg's answer draws on his earlier literature review of psychopathy. One of its two key results is an asymmetry.

"Psychopaths are just as good as everyone else, if not a little better, at learning in a reward based paradigm and are pretty bad at learning from punishments."

If a brain is wired so that punishment does not register, repeat offending is not surprising. He cites Anthropic's emotions work, where boosting positive emotion factors produced more antisocial behaviour, as a confirming signal. So the naive intervention, "max out the good, minimize the bad", worries him. Pain, in his account, highlights the parts of a state space that must be avoided, which is a different computation from highlighting what to approach. Some behaviours need to be painted as no-go zones: harming people, economic damage, hacking a company to find an answer key.

But not more than necessary. Where reward and punishment would both work, use reward. "All else being equal, we should try to use a carrot and not a stick. But that doesn't mean never use the stick."

What this means for OSASI

The question that keeps coming up across these briefs is whether better is the enemy of good enough. It cuts both ways. In the frontier race, and the arms race framing around it, better means ever more capable general models with ever less visibility into how they reach an answer; good enough is the safer choice. In a litigation report it flips: the good-enough method is the one that can fail the truth, and the better analysis is the one a professional has to be brave enough to defend and clear enough to explain.

McCarthy's two points are a test we can apply to any tool. Does it treat our rules as physics, or as a layer bolted on? And does it leave the judgment with the professional, or quietly take it by defaulting to whatever is most acceptable? A model that is merely good enough at the task but passes both tests is worth more to a member than a better model that fails either.

Berg's segment is the longer view. The people building these systems are still working out how to make them avoid the things they must never do, and the answer is turning out to look more like professional formation than engineering: mostly reward, a few hard lines. That is a reason to prefer tools whose boundaries we can see and set ourselves, which is what OSASI exists to provide.

Send a thought →

A like is counted once per reader per day and stores nothing about you. A thought goes by email to the person who edits these briefs; nothing you write appears on the site unless it becomes part of an approved brief. Share opens the service you pick; this site adds no trackers.

Prefer a feed reader, Slack or Teams? Follow by RSS.