Hierarchical Thinking and Its Impacts on AI Design

Ray Kurzweil’s argument in How to Create a Mind is that the neocortex isn't doing anything as grand as thinking, at least not in the way we mean it casually. It's running roughly three hundred million pattern recognizers, arranged in a hierarchy, and each one only ever talks to the layer immediately above and below it.

A recognizer near the bottom fires on a stroke, or an edge, or a curve. It doesn't know what a letter is. It reports upward. A recognizer one layer up assembles those into a letter, and it doesn't know what a word is either. Keep going and eventually something at the top is recognizing intent, several layers removed from anything that ever touched a pixel.

Two properties of that arrangement matter for us, and they're the reason I keep coming back to this.

The first is that prediction runs downward as hard as recognition runs upward. When a higher layer thinks it knows what is coming, it primes the layers below it to expect that. It's why you finish other people’s sentences, and why you can read a paragraph with the letters scrambled.

The second is that the hierarchy is built by repetition, not by instruction. Nobody taught you the shape of the letter A as a rule. You saw it enough times that a recognizer formed.

Why any of this applies to a language model

Now, I want to be careful here, because there's a lot of sloppy writing about this and I don't want to add to it. A transformer isn't a neocortex. It wasn't built to imitate one, the mechanisms are different, and anyone telling you the mapping is one-to-one is selling something.

But the architectural family resemblance is real and it isn't a coincidence. Both are layered. Both build representations that get more abstract the higher you go. Both were shaped by exposure to enormous amounts of data rather than by being told the rules. And critically, in both, what happens at the top depends entirely on what got passed up from below.

You don't get to talk to the top of the hierarchy directly. You talk to the bottom, and the bottom decides what the top ever sees.

Which is the whole reason your inputs behave the way they do.

Priming, again, in a different suit

I have written before about priming in information architecture: that categorizing a navigation structure is really an exercise in setting up associations before somebody needs them. The same mechanism is running when you talk to a model, except now you're the one doing the priming and you're usually doing it accidentally.

Say the word enterprise early in a conversation and you've activated a whole region. You'll get more hedging, more governance language, more talk of stakeholders. Say startup instead and the register moves. Nothing about your actual question changed. You moved the starting point of the hierarchy and everything downstream followed.

The model is doing the thing that makes it useful, and suggestibility is the wrong word for it. Context narrows the distribution. That's the mechanism working.

So the practical consequence is that vague inputs aren't neutral inputs. There's no such thing as a neutral input. A vague one means you let something else do the priming: the default register, the last thing in the buffer, the house style of whatever it read most of.

What this changes about how I set up work

Three things, and I arrived at all of them the slow way.

Establish the level before the request

If I want something at the level of interaction detail, I don't open with the business goal. If I want strategy, I don't open with a component. Opening at the wrong altitude means the first several exchanges are spent climbing, and everything produced during the climb is shaped by where you started.

Give it the constraint before the ask

Constraints stated after a request read as corrections. Constraints stated before it are context, and they narrow the distribution before anything gets generated. Same words, very different result, and it took me embarrassingly long to notice.

Start a new conversation more often than feels necessary

A long thread has accumulated priming you can no longer see, and some of it is from a direction you abandoned an hour ago. When outputs start drifting toward something you rejected, that's a hierarchy still holding a pattern you fed it, and stubbornness has nothing to do with it.

The part that should make designers uncomfortable

If both systems build their hierarchies from exposure, then whoever controls the exposure controls what gets recognized as normal.

For people, that has always been our job and we've thought about it carefully. Frequency breeds acclimation. We choose patterns knowing that repetition is what makes them feel inevitable later.

For models, the exposure already happened, at a scale none of us influenced, and it encoded a particular idea of what an interface looks like. When you ask for a design and it hands you something familiar, that isn't a failure of imagination. That's a hierarchy reporting the most reinforced pattern it has, which is what it was built to do.

Which is a topic in its own right, and one I've gone at from the other direction. For now the takeaway is smaller and more useful: you aren't typing instructions into a system that executes them. You're setting the conditions at the bottom of a hierarchy and reading what comes out of the top. Design the conditions and the outputs mostly take care of themselves.