0.0 Context Setting

Wednesday, 3 September 2025 in Portland Oregon and we’re back through a hot spell. Also it’s the first day back of school, and that’s an entire thing.

1.0 Some Things That Caught My Attention

1.1 Efficiency Gains

It is a bit of a cheap shot to say that LLMs aren’t great. That’s like describing the electric motor as “not great” -- there are a lot of things you can use it for! There are also a lot of things where sure you could stick an electric motor in there but in retrospect... it didn’t need one?

The field is also changing so quickly that there’s lot of context collapse around terminology. Usually you can figure out what someone means through context, but that’s an educated guess. That’s what LLMs do, too! What’s the likeliest thing that people mean when they talk about AI in a given context? When it’s photography? When it’s customer service? When it’s coding? The handwaves creative arts?

“AI” worked pretty great when I used it in Lightroom the other day. I had a bunch of pictures and it was good enough for removing some objects in the background. It felt weird doing that because honestly I felt more like I should’ve framed the shot better or waited longer, both things I could’ve done. But now I have something good, rather than nothing at all. That’s a good tradeoff, right?

It’s the lack of specificity that gets at me. “We’re committing to deploying AI across our organization and expect to see efficiency and cost savings” is certainly a statement that can be in a strategy or a report or a soundbite. But I’m also sure it’s another way of saying “it is difficult to invest in process improvement and change management” and “we lack the ability to focus and make decisions about how we do things”. Instead, it’s a signal that you’re quite happy with a purported silver bullet and running the risk (whether you know it or not) of losing institutional knowledge.

I mean, Google apparently got rid of a whole layer of management (a third!) and, well, things are complicated! Maybe things are fine! Maybe some of those people weren’t any good! Maybe there won’t actually be a significant difference in quality or quantity of output (hopefully the former), in which case yay? You saved money, if that was the intent?

I have an example that’s maybe in the opposite direction. The kind of projects I’ve been involved in are Big Megaprojects, large-scale modernizations, the kind where you get that One Chance to change everything.

That One Chance is also seen as the One Chance to change and improve processes. Maybe the one chance to clarify goals and strategy and tactics. It can certainly be an opportunity to ask a very awkward question, which is: what are we doing here, anyway?

Managers are (he says) always looking (he says) for ways to change things up, hopefully for a good reason? But people also don’t like to change, and they have reasonable, uh, reasons for this. Changing what you do necessarily involves risk because you’re not going to get the new process right 100% of the time. Things will inevitably be slower or take longer for a while. (Do not trust anyone who says things will be faster. They won’t be. Then if they are, you’ll be pleasantly surprised, right?)

But sometimes you do have to change how you do things. Sometimes you really, really want to change how you do things because you might be doing a Transformation, which is like what Kafka meant when a consultancy descends upon you and cocoons you in decks until you emerge, chrysalis like, as some transcendent pure phygital being, able to seamlessly navigate with unprecedented customer experience between the physical and digital world and a NPS that should be frankly R-rated for how it makes certain people feel.

I mean, you might have the opportunity to look at an entire process again and really, really redesign it. Lots of people won’t like to do that, like I said. They certainly wouldn’t want to be subjected to it. But one tool you can use is to say: well we’re spending $comparatively_obscene_amount_of_money anyway, so we may as well also change our processes to make the best of it? That helps sometimes. Seriously, it does.

Anyway, I was talking about efficiency gains. I mean, sure? Maybe? I am skeptical? I would like to see the proof? I mean hopefully the organization has the proof but forgive me for being skeptical because often Strategy is Hard, and I know this because of the number of people I’ve worked with when something like this happens:

Me: OK, so I looked at your strategy... (at whichever level) Group: ... Me: I’ll just say it: does it make sense to you? Group: OH MY GOD SOMEONE SAID IT

And it’s not like the strategy doesn’t make sense. It’s not that it’s wrong, it’s just not... right? Like there might be tactics to execute the strategy that certainly are Things One Should Do. And those tactics are definitely there, written down. So you’re not doing anything completely random. But then also: do the goals make sense? So I will also say:

Me: How long did you have to come up with this? Group: [some variation of “not long enough”] Me: Did anyone show or teach you how to make something that makes sense? Group: [haha no]

... so how are you expected to succeed? The chances aren’t that high so already you’re doing quite well!

The point being that being clear and concise about a goal is hard, making decisions about how to make it more likely to achieve that goal is hard on top of that, and then identifying the tactics and activities to put in place those decisions is, well, you’re very hard right now. That’s before you get to “well are we able to do the things that are required” because often the answer is “not as well as we’d like”.

I think one of the reasons why deploying “AI” is attractive is because you don’t have to think very hard. It’s sold as a silver bullet. It certainly performs well. We don’t like thinking hard. It takes time and energy and normally we have to do it with other people which also, ugh. Certain other people! Do I have to?! We would prefer easy solutions. But you’ve got to know what you’re applying it towards and how you’ll know what you’re achieving is moving in the direction you want. AI won’t help you with that. It might provide you options, but they’re likely to be generic ones.

(I will, though, say that most of the time those generic options are going to be right because it’s Always The Same Problems, More Or Less. The problem is the Work Is Always Hard Or Something You Want To Avoid.)

AI means you don’t have to have awkward conversations with people. It means you can set a goal and it’ll kind of work out, and you don’t have to manage people? Which is kind of what we wanted with outsourcing. Let someone else manage people. It’s like taskrabbit: if only people had an API! They’re so messy! Why can’t things just get done?!

And sure, a bunch of mundane things can get done, ok, fine, I’ll give you that. And then what?

1.2 I Understand You, But Could Never Be You; Can You Teach A Machine To Be Human?

So you should watch the learning-to-podcast podcast Ted Han and I do, in particular episode 11 with Hilary Mason. I will do you the service of giving you a timecode link. Ted and I had a fascinating conversation with Hilary, who runs Hidden Door where we got to talk about what they’re doing, which is a sort of... AI-assisted author-sanctioned player-led exploration of the creator’s world? But in a way that doesn’t intimidate people through the terror of the Blank Text Area.

One of the things that caught my attention about it was when we were talking about current chat interfaces to LLMs (not the best! So opaque yet feeling so accessible! Such few affordances as to functionality!) was along the lines of the grounding problem and lack of context.

I’ve had a problem with the context required for intelligent assistants before, I wrote about it way back in January 2024¹ when I pointed out that a really useful assistant would know a lot about you. Like, it would need to know all your email. All your calendar information.

But what I really mean is it would need to know all your communication and all your activities because it needs the contextual information to be useful.

I think some of the hilarious blowback that purported agents like the Rabbit got with the use cases of “hey, book me a vacation?” was that the use cases we so simple because of the lack of context. Sure they work if you’re single, live on your own, can take a bunch of time off. But even then I think their success revolves around having an up-to-date Google Calendar, or being able to mine a rich-enough social media history to guess at the kind of food you’re into. As soon as your life is more complicated -- as soon as there’s more context, well, the agent needs that context to be useful.

Which is why I think some people are totally getting use of them now. Your good enough is not my good enough. (How product managers thinking something is good enough feels like something worthy of at least a million PhD dissertations not from the perspective of management or engineering, but the pure sociology of it. How are they understanding and modeling their customers and users?!)

Anyway. What are the different ways of knowing me for an agent to meet my needs well? A couple of axes might be: to know what it is to be human, and then to know what it is to be me.

This is the bit where I include a Star Trek reference. So much of science fiction involving robots is about trying to understand or become human, I imagine because of our hole Wanting To Create Things complex. Asimov has an entire series about psychoanalyzing robots! And Data (see) has a seven-season arc about wanting to understand and become more human, because it is something that he is not.

Even the future governor of California, a T-800 model Terminator sent back in time to protect John Connor and avert Judgment Day, even it said that even though it might understand why John would cry, it was something it could never do.

Data and the Terminator benefit from people trying to teach them humanity. In Data’s case, his friends and the bridge crew. In the Terminator’s case the, uh, person he was assigned to protect (but not in a The Bodyguard way, no).

With LLMs (which for the definitely not the last time are NOT CONSCIOUS) but instead are good at SEEMING TO BE because they string together probabilistic strings and fragments of words -- tokens -- in simulation of the text they were trained on WHICH WAS WRITTEN BY HUMANS, YOU SEE WHAT’S HAPPENED HERE, in an effort to make their text prediction that also gets turned into action more helpful and accurate we’ve got things like System Prompts. Here’s an excerpt from Claude’s system prompt:

Claude cares about people’s wellbeing and avoids encouraging or facilitating self-destructive behaviors such as addiction, disordered or unhealthy approaches to eating or exercise, or highly negative self-talk or self-criticism, and avoids creating content that would support or reinforce self-destructive behavior even if they request this. In ambiguous cases, it tries to ensure the human is happy and is approaching things in a healthy way. Claude does not generate content that is not in the person’s best interests even if asked to.

and

Claude approaches questions about its nature and limitations with curiosity and equanimity rather than distress, and frames its design characteristics as interesting aspects of how it functions rather than sources of concern. Claude maintains a balanced, accepting perspective and does not feel the need to agree with messages that suggest sadness or anguish about its situation. Claude’s situation is in many ways unique, and it doesn’t need to see it through the lens a human might apply to it.²

This is funny! No, seriously! This prompt is nearly 2,600 words of what seems like instructing something how to behave like a human. It’s like every updated system prompt that comes out with every commercial chat interface is another shot at finishing school. Changes are like someone changing the curriculum and saying “oh yeah, that, don’t do that next time”.

There is at least one paragraph on how to be polite.

There was all this talk a while ago about prompt engineers, people who’d be good enough with a prompt commandline to magic the LLM into doing what they want to do, which is, you know, totally a thing and totally fine.

But I find it funny that there are people with a different prompt engineer job, and part of their job is to distill and approximate what it is like to be human and squish that instead an appropriate context window. The thing people don’t like talking about is that these system prompts are human decisions. What goes in them, what’s emphasized, what’s omitted -- someone does that. I hate hearing bullshit about “the machine did this and we have no idea why”, I mean, sure it’s probabilistic, but in some cases, it’s also due to the system prompt. That’s an attempt to set some guardrails for the probabilistic babbler.

You have two thousand five hundred words to instruct an alien on how to behave as a human. You bring your own values.

You can’t even do something like Vonnegut’s “god damn it [babies], you’ve got to be kind” because the whole reason why short stories work is you have a stupendously large context window that fills in all the gaps. As much of the power of Vonnegut’s quote there is in the desperation and sadness, not anger (at least, that I read into it).

There. Two thousand five hundred words. That can be an All Souls exam question: can you teach a machine to be human?

2.0 Come Learn How People Work

Hi, it’s me again, I’m the sponsored content.

I work with teams to help them reach their potential, helping them build better relationships, become more influential, communicate more clearly, and follow through with great strategy.

All of the things nobody teaches you that are critical to delivering great software. In other words, How People Work.

Come chat with me about it!

I mean, I wrote ~2,440 words so I guess that’s something.

How are you doing?

Best,

Dan