The LLM Black Box and How It Affects Design

I spend my days on both sides of this. I design agentic experiences for enterprise software, which means I'm deciding what a model is allowed to conclude and what it has to hand back to a person. And I build my own products with a model sitting next to me, which means I'm on the receiving end of the same behavior for several hours a day.

Doing both has made one thing very hard to ignore. A lot of us have developed a working cadence with these tools and skipped the part where we understand them. We type something, we get something back, we take it. Then we do it again forty times before lunch.

I don't think that's laziness in most cases, though sometimes it is. More often it's that the people doing it have never had to reason about a system that answers differently every time you ask. So let’s get into it.

The two halves of the complacency

There's an input half and an output half, and they fail in different ways.

On the input side, people believe the prompt is the input. The prompt is the last thing in a stack that also contains the conversation so far, whatever got pasted in three messages ago, a system prompt somebody else wrote, and a set of defaults nobody in the room has read. You typed eleven words. The model received several thousand.

On the output side, people believe that fluent means correct. This is the more expensive of the two. The register of the writing has nothing at all to do with whether the content is true, and we're all reading a system that's confident by construction. It has no way to sound uncertain unless somebody designed it to.

Put those two together in a design team and you get work that nobody can explain the provenance of. Ask where a layout decision came from and the honest answer is that it arrived.

Why classical software experience turns out to matter

I keep noticing that the people who handle these tools most carefully are the ones who have shipped software the hard way. Their prompts are no better than anyone else's. They already have a mental model for state, for side effects, and for the idea that a system can be wrong in a way that looks fine.

If you've debugged something at two in the morning you have got a reflex for asking what else changed. If you've never had to, the reflex isn't there, and a model will happily let you go without it for a very long time before it costs you anything.

The tool doesn't punish you for not understanding it. It punishes you later, somewhere else, in a way that doesn't obviously trace back.

Which is the worst possible feedback loop for learning. The lesson arrives detached from the mistake.

Probabilistic design against deterministic design

This is the principle I'd put above all the others, and it's the one I see designers skip.

Everything we learned our craft on is deterministic. A button does the same thing every time. A validation rule fires or it doesn't. When we design those, we design the one path, plus the error states we can enumerate. That's the whole job and it has served us well for thirty years.

A model doesn't work that way. The same input produces neighboring answers, not identical ones. There's a distribution underneath, and what you see is one sample from it.

So the design question changes shape. The old question was what does this screen say. The new one is what does this screen do across the range of things it might say. That's a genuinely different question and most of our tooling isn't set up to ask it. You can't mock a distribution in Figma.

In practice this means designing the spread rather than the sample. What does the component look like when the answer is two words? When it's nine paragraphs? When it's a list of three things and one of them is obviously wrong to anyone who knows the domain?

• Design the shape of the answer, not the answer. Length, structure, and what the person is meant to do next should hold whatever comes back.
• Decide what the system may do at each confidence level. Not one behavior with a disclaimer bolted on.
• Test with the bad samples. Every demo I've ever seen used a good one. That isn't a test, it's a rehearsal.

Graceful degradation, which we already know how to do

Here is the part that should be encouraging. We aren't starting from nothing on this. Progressive enhancement and graceful degradation are old ideas in our field. We have been designing for the slow connection and the missing image and the browser that doesn't support the thing since the late nineties.

Somewhere in the move to AI features we stopped applying it. Teams design the state where the model is right and treat everything else as an engineering concern, which it isn't.

The second row is the one worth sitting with. Confident and wrong isn't an edge case, it's the normal operating mode of a system that has no way to know what it doesn't know. The only thing that catches it is evidence on the screen, close enough to the claim that checking costs nothing.

And the fourth row matters more than people expect. When a feature has nothing useful to say, the right move is to put the person back where they were before the feature existed, with the manual path intact. A dead end is a design decision somebody made by not making it.

What I do about it

Three habits, none of them clever.

First, I run the same request more than once before I believe anything about it. If two runs disagree meaningfully, that's a property of the feature and it belongs in the design, not in a bug ticket.

Second, I write down what the system is allowed to finish and what it has to hand back, before I draw a screen. On every agentic project I've worked on, that document did more for the outcome than any layout I produced.

Third, I make myself say where an output came from. Not for process reasons. Because the act of saying it out loud is what surfaces the times I can't.

The takeaway

The black box isn't going to open. That isn't a temporary condition we're waiting out, it's the nature of the thing, and a design practice that's waiting for interpretability to arrive is going to be waiting a while.

What we can do is design as though we know it's a black box. Account for the inputs we can't see. Design the spread instead of the sample. Build the degradation path before we need it. None of that requires understanding the model. It requires taking seriously that we don't.

Which is a much older design skill than any of this, and one we already have.