Altitude, Confidence, and Consequence

Over the past year I've published eight posts that I thought were about eight different things. Hierarchical thinking. The two-system concept. Memory sequences. Trust calibration. The black box. I wrote them months apart, on whatever had been bothering me that week, and I filed them as separate arguments.

They aren't separate arguments. I worked that out in the shower, which is where I work most things out, and then spent a week trying to talk myself out of it.

What follows was assembled backwards, which I would rather admit up front than have somebody notice. I didn't set out with a framework and write posts to fill it. I had eight posts, I found the thing they kept circling, and I have spent a couple of months since then testing whether the shape holds up. It mostly does. I'll flag the parts that don't.

The argument is this. An interface built on top of a probabilistic system has to close a gap between what the system knows and what the person in front of it can see about that knowing.

I want to be careful with the next part, because the easy version of it is wrong. Deterministic software has its own legibility problems and always has. Hidden state, silent failures, a sync conflict nobody surfaces, a validation rule three teams removed from the field it rejects. We have been bad at this for a long time. What's different is that the gap used to be bounded. You could open the thing, in principle, and find out what happened. A model hands you a fluent paragraph whether it's right, wrong, guessing, or should have declined altogether, and there's no equivalent of opening it up. The interface is the only thing standing between that paragraph and a decision somebody is about to make.

First, what the gap actually is

Back to our main man. Ray Kurzweil describes the neocortex as roughly three hundred million pattern recognizers arranged in layers, where each layer talks only to the layers next to it. Recognition runs upward. Prediction runs downward just as hard, with the higher layers priming the lower ones with expectations about what they are going to find.

Transformers are not neocortexes, and I want to be careful with the analogy, because it's the kind of comparison that gets repeated until it stops meaning anything. But the shape of the problem rhymes. Upper layers depend entirely on what gets passed up from below, and the person typing is standing at the very bottom of that stack. Say “enterprise” instead of “startup” and you've moved something four layers up without changing your question at all.

That would be manageable if the movement were visible. It isn't. Fluency covers everything, and the feedback doesn't arrive at the moment of the mistake. It arrives later, somewhere else, in a form that doesn't obviously trace back to the thing that caused it. Which leaves the interface doing work it was never designed for. It's become the only instrument anybody has for reading a machine that can't report on itself.

Altitude

Have you ever asked a model a strategic question and been handed a list of implementation steps? The answer wasn't wrong, it was answered at the wrong height.

Altitude is the level of abstraction the system is currently operating at. It gets set by priming that accumulates out of sight: prior messages, a pasted file, a system prompt nobody in the room has read, defaults somebody chose eighteen months ago. All of it is steering the output. None of it is on screen.

The remedies we currently teach are folklore. State your altitude before you ask. Put constraints in front of the request so they land as context rather than corrections. Start a fresh conversation often, because accumulated priming goes invisible and drift sets in.

I've given all three pieces of advice and I stand by them. They work, as far as they go. But it's worth sitting with what they are. Every one of them is a workaround, passed between practitioners, for a control that doesn't exist. When the accepted best practice for a system is “turn it off and on again,” the interface hasn't finished being designed.

So what would the control be? Make altitude visible and adjustable. Show what's shaping the current answer as an object somebody can actually look at. Let them move up a level without discarding the work already done, and let them prune the three things still quietly steering the output. Which is a great deal more useful than telling people to open a new tab.

The trust problem is not the one we keep solving

Trust is the property everybody tries to maximize. Maximizing it is the wrong goal, and I've argued this before at length, so I'll keep it short here.

Somebody who trusts the system more than it deserves is a failure in the same way as somebody who abandons it. The two failures aren't equally visible, which is the part that matters operationally. Under-trust shows up in your engagement numbers within a week. Over-trust shows up months later as a mistake nobody traces back to the interface, and by then three other things have changed and the investigation goes nowhere.

The standard response is hedging. Add a “may” here, a “might” there, apply it uniformly across every response, ship it. It does very little, for a reason that is easy to state and hard to get funded: a signal that fires on everything carries no information, and people stop reading it inside a week.

More usefully for us, there are four states worth designing rather than one: confident and correct, confident and wrong, unsure, and nothing to say. Quite a few products design the first properly and treat the other three as error handling.

A system that declines to answer is telling you more about its reliability than a system with an unbroken record.

The practical version isn't complicated. Vary confidence at the level of the claim rather than the response. Separate what was computed from what was inferred. Put the source next to the sentence it supports, so that checking costs a glance rather than a search. Say plainly what was not examined. Build the refusal properly.

What happens if it's wrong?

This is the one almost nothing handles, and it's the newest part of the argument for me, so treat it as less settled than the other two.

Watch somebody use an agent for an afternoon. Renaming a file and moving money produce the same confirmation dialog, in the same position, with the same two buttons. That doesn't protect the expensive action. It teaches the person to clear dialogs without reading them, which means the friction you added for the dangerous case has been spent on the harmless one, several hundred times, before the dangerous case ever comes up.

Friction should be priced to consequence. An action that can be undone within the hour has earned no dialog at all. An action that cannot be undone has earned a preview, a dry run, a second pair of eyes, and a clear statement of what will change.

There is a second reason to care, and it took me a while to connect it, because it comes from a post I wrote about checkout flows rather than about anything autonomous. People build reflexes out of repeated steps. Move one step and the reflex breaks, whether or not the move was an improvement. The person pauses, re-reads, and starts wondering what else changed.

Now make the actor in that sequence a system that quietly does something different on Thursday than it did on Tuesday. The person loses their place and usually can't tell you why they feel uneasy about it. Which makes memory symbols more important once the work is autonomous, not less. A wizard, a saved draft, a step indicator: the person needs somewhere to stand when they come back to it.

I'm still not sure how to build that part well. I know the failure when I see it.

I don't think these are the only three

I considered several others and dropped them, and the reasoning is worth showing, because a taxonomy that arrives fully formed should make you suspicious.

• Provenance. Where a claim came from. I folded it into confidence, because in practice a source sitting next to a sentence is how confidence gets expressed rather than a separate property.
• Cost and latency. Genuinely important, and users clearly want them exposed. They are not about what the system knows, so they sit outside this particular argument rather than inside it.
• Blast radius. How far an action reaches. This one I'm least happy about. I treated it as part of consequence, and there is a decent case that scope and reversibility are different enough to separate. I have gone back and forth on it twice.
• Authorship. Who decided this, once several systems are involved. I left it out and I suspect that was a mistake. It may well be a fourth exposure. I haven't thought about it hard enough to write it down properly.

That said, there is a reason these three survived. Kahneman gives the test: System 2 is slow, effortful reasoning, and it fires precisely where recognition fails. Most System 2 moments in a product are not earned complexity. They are unintentional, and they are usually information the interface already had and did not show.

If altitude is visible, you recognize a mismatch instead of reading three paragraphs to work out that the machine misunderstood the question. If confidence varies, you recognize which sentence to check instead of verifying all of them or none of them. If consequence is priced, you recognize which decisions deserve your attention instead of rationing it blindly.

I should be honest that the test came afterward. I didn't derive these three from Kahneman. I arrived at them by noticing what I kept writing about, then checked them against a framework I trust, and they passed. That's weaker evidence than deriving them would have been, and it's the actual order of events.

So how do you use this on Monday?

Take one screen you already own. Ask what altitude the system is answering at, and whether anybody looking at that screen could tell. Usually they can't, and usually it takes about four minutes to find that out.

Then run a behavioral test rather than a satisfaction survey. Plant a wrong answer in a build, put it in front of five people, and watch whether they catch it and which source they open first. A satisfaction score tells you how the session felt. This tells you whether anybody was reading.

Then list every action the system can take on somebody's behalf and mark each one reversible or not reversible. Put the friction each action currently carries next to that mark. It takes an afternoon and the output is not pretty. The rows where those two columns disagree are your backlog, and they are usually not the rows anybody was arguing about.

Altitude, confidence, and consequence: three properties, exposed honestly, on every surface where one of these systems meets a decision. Possibly four, if authorship turns out to be its own thing.

What I want to write next is the contract underneath it: what a component has to declare before it is allowed to render one of these systems' output, and what a decision record has to carry before anybody can review it after the fact. The framework is the easy half. The specification is where it either works or falls apart.