0.0 Context Setting

It’s been a minute. This is probably the third draft of an episode that I’ve tried to finish so maybe this one will stick. It has been hard to write, lately!

It’s December 29 in Portland, Oregon and it is getting cold. Also many things have happened in the world and it feels like things that suck have outweighed the things that don’t suck.

0.1 Some Personal News

Hey, did you know I do workshops and coaching now?

I could explain what they do, but I’ll let someone who’s recently finished the workshop go first and then I’ll come in:

“Dan's training is a crash course on how to form direct, empathetic, honest, and strategic relationships with other humans to get hard things done in complicated organizations.”

Huh, that’s pretty good. I would totally try to persuade my company to pay for me to take that workshop. I will go and read about that workshop series right now.

That’s for groups of people, though. A bunch of people wanted to know whether they could do it as individuals, and guess what: you can! I’m doing coaching based on the workshop material too, now.

I am not going to do the thing where you have to email me to find out how much something costs. That is irritating. I will just tell you:

• A 45 minute coaching session is $495 if your company is paying. Your company pays less the more it buys. You should definitely persuade your company to buy a bunch.

• If you’re paying out of pocket, it’s #300 for a session.

Right now, there are a few spots left with a 15% discount!

Either way, you should book a free sample coaching session so we can have some useful fun together.

1.0 Some Things That Caught My Attention

1.1 An End Of Year Opinion About AI Because Why Not

So people are fighting on the internet (well, Bluesky) about LLMs-slash-generative-AI-slash-AI and I’m going to be one of those irritating people who will say “well, it’s complicated and it depends”.

One of the reasons why “it’s complicated” and “it depends” is because of the horrific behavior of tech startups and their gold rush land grab intentionally poisoning/collapsing terminology.

So I’ll just be quick and pretend that we’ve covered that “AI” can mean lots of things, “generative AI” is something specific, people are using LLMs for lots of things, some people (maybe not very many, though?) are getting some great results through using specific agentic coding tools (the ones I’m aware about are Simon Willison and Jesse Vincent, who are in and of themselves 10x developers [sic] whether or not they’re using Claude). Then there’s the “AI” machine learning that does stuff like increasing resolution of images but doesn’t go wholesale into generating/filling new details, of which there’s a continuum and grey area anyway.

ANYWAY.

One thing that happened was that the American Historical Association puts out guidance¹ that says it’s OK to:

• Ask generative Al to identify or summarize key points in an article before you read it (when you’re not doing explicit citations)
• Use an Al chatbot as a writing partner to help generate and develop ideas (but knowing that you may require explicit citation “depending on circumstances”)
• Ask generative Al to produce a starter bibliography (when you’re not explicitly citing, but only if you “check each reference and additional databases and sources are mined”)

(very) understandably a lot of people get upset because one of the entire points of being an historian is that you pay a lot of close attention to sources?

I think there’s clearly a mismatch between people doing the work and people paying for or accepting the work.

If you’re on the end of someone receiving work, then how much creative thought and insight do you want? How much accuracy do you want? And maybe more importantly, how much are you willing to pay for it? (And then can you be honest about how much you really want, versus how much you say you want? It’s one thing for you to say you want high-quality work, but another to accept work that’s of a lower quality)

Often I think that the people doing the work can easily care more about and value that work more than the people paying for it. There’s a mismatch between expectations of what a good job, or even a good-enough job is. I can easily see how an academic historian has drilled into them the importance of checking sources and not making sure that anything you’re referencing is factual (or as factual as can be, given your domain of study is history). On the other hand, I can see that managers at a professional organization might be more concerned about output than accuracy.

For some people, personally, accuracy and correctness matters. I think as a position of principle or temperament. And obviously how much accuracy and correctness matters depends on context.

So you’ve got someone who wants to do a good job, and you’ve got someone who... doesn’t really care if the job is done that well?

That’s before you even get into the problem of incremental improvement. Say I can do a job well or I can do a job really well -- is the difference in what I do going to affect an outcome, like the decision someone will make based on my work? Or does it turn out that they weren’t really able to make a different decision anyway?

(Here’s an abstract example: if your industry goal is to deny 10% of claims, then why work hard on this particular claim to make sure you’re being thorough? Even if you spend more time to make sure you’re doing a good job, ten percent of claims are still going to get denied)

In that case, why would you care about the accuracy of AI tools if it doesn’t actually make a difference to your boss? If you already felt like what you did didn’t make that much of a difference then I could understand being ground down and just accepting -- well this is just what things are like these days.

I don’t think bureaucratic processes rely that much on correctness anymore. And I think that the penalties for incorrectness aren’t enough to encourage, well, getting the answer right the first time round. (Another abstract example: the health insurance complex in the U.S. gets a bad rap for healthcare providers sending around erroneous [sic] bills, and if you want to make sure you’re paying for what you actually received then you need to go through your explanation of benefits line by line and then call up, say, the hospital. But the hospital doesn’t get punished for this! There’s no incentive for them to stop doing it. In fact what I suspect happens is that we get even more billing specialists on every side. So that’s good for job growth, I guess?)

There’s another part, which is that I think modern management practices value data or information over a lack of data or information. Companies would rather have more data -- even if that data is incorrect or has errors -- over no data at all. Why not? I mean, now you’ve got a bit more information with which to make decisions, and don’t you want as much information as possible to make decisions?

So I think bureaucracies are primed to value more incorrect data over no data at all, and which I think explains this bizarre rush to embrace synthetic data where I shit you not companies are saying well instead of going out and talking to real people and customers we’ll just have the equivalent of ChatGPT tell us what imaginary people think. That’s nuts? I mean, it’s cheaper, sure, but it’s also nuts? I mean today I found out that the U.N. had a program that generated synthetic refugees for people to interact with so that people could understand the plight of refugees. OK, I get the intent. But a lot of refugees and NGOs pointed out that, well, there’s a lot of refugees you could talk to right now? Without having to make anything up?

Correctness doesn’t matter if the number goes up. Nobody has the time to actually check for causation, so as long as the number goes up and you’re spending a ton of money gathering new data that’s mostly noise, you can’t tell! The AI-generated data or insights must be helping, because your revenue is still going up!

There’s a side point here where I think that the kind of people in leadership who have no problem mandating the usage of A.I. (let’s say for argument’s sake Bari Weiss, who is Not A Very Good News Editor) are those who have succeeded in fields that don’t require correctness in the first place.

I’m not even sure if this was a rant. I don’t think there’s necessarily anything to be done about “what are we going to do about AI” because the genie is out of the bottle, and all the genie is going to do is follow the economic incentives that already exist. What’s horrifying about it to most people is the scale and speed at which it’s operating, and the scale and speed at which managers and leaders are using to justify their decisions and actions.

I do not expect anyone to calm down. The imposition of “AI” absolutely feels like an existential risk what with layoffs and the increasing cost of living in most countries.

1.2 Good Enough Mitigation of Reasonably Foreseeable Harm

Look, here is another thing about code written by AI. One position is that maybe code doesn’t need to be as correct as it has been in the past, of which: well, that’s a funny opinion. It hasn’t been great, but sure. If code doesn’t need to be as correct as it has been, then it’s totally fine if AI-generated code isn’t as good as human code, right?

Code just needs to be good enough, and maybe it’s worth it to lower our standards in exchange for volume.

Good Enough is shorthand and is completely dependent upon context and circumstances. Your good enough is different from my good enough is different from a good enough two weeks ago compared to a good enough two years from now. These things can change!

Good Enough is important for your stereotypical tech people (or, on reflection, anyone trying to sell anything in today’s world) because you don’t want to spend too much time making something too good otherwise someone else who made something just good enough will beat you to the market and then will steal all the money you thought you’d make. Software people talk about this in terms of not wanting to do any premature optimization because that would slow you down, and if you’re too slow then you don’t move fast enough to break things.

What are the kinds of things that go into deciding what’s good enough? I think one way of thinking about it is this:

Good Enough wins because of what you’re able to treat as externalities.

Here’s some examples:

• Code is “good enough” because you’re not liable for defects.

If you were liable for defects then most companies that make software would probably have been sued to oblivion by now and as a result would presumably make software in quite a different way.

Shitty Silicon Valley VC people who lucked into more money than sense would counter this with: well if we were liable for the products we made then we wouldn’t bother making them and where would you be then? You wouldn’t have an iPhone! Nobody would want to write software because it wouldn’t be worth it!

I think these people are wrong because that is a stupid position and they are stupid, but either way, we’re not really allowed to find out the opposite case. In England and Wales, for example, it was a matter of law that software is always presumed to be correct and it’s the job of the person who says it isn’t to prove that it isn’t. This reflects a major misunderstanding of how computers work, which is that they do exactly what you tell them to do, and if you tell them to make a mistake then they will absolutely make the correct mistake you told them to make.

Anyway, good enough as what you’re able to push off onto an externality. More examples!

Coal-fired power stations are totally good enough to build and operate if you don’t have to care about or pay for people developing lung disease nearby.

Minimal oversight of mortgage-backed securities is totally fine if you don’t have to care about losing a bunch of money because in the end, you’re too big to fail and a government will bail you out.

Anything can be personally good enough if the harms or the effects of the harms are distant enough that you’re not affected by them and you don’t see them or care about them.

An example in government is ministers or secretaries thinking that the services their departments are responsible are totally good enough until you make them try to use them at which point the decent ones are completely horrified that the programs they’re supposed to be responsible for are more or less impossible to use on a practical basis. (That is an exaggeration).

Carbon-based economies are totally good enough until you’re freezing or baking and millions of people are dying who you actually care about.

The forces that determine what’s good enough for you include your physical environment and existence. They include the economic context in which you live. They include the government regulations under which you operate, and they include the legal regime in which you operate.

This entire rant is getting dangerously close to my hobby horse of being able to limit warranties and exclude liabilities for defects in software/technology products. Which is totally a societal decision -- it maps back to the argument I made earlier, where why would you ever bother to do anything if those pesky people could sue you for not doing it right.

It’s arguable that OpenAI’s ChatGPT has contributed to the death of teenagers by effectively encouraging them to die by suicide. Every week more and more examples come out.

You would not be surprised that there’s something in ChatGPT’s terms and conditions that effectively says:

• you won’t use this as, like, advice
• don’t trust it
• if you do anything bad as a result of using this, especially if it’s a result of something that this product did that’s bad, then that’s totally not our fault, it’s your fault for using it in the first place

One of the reasons why I hate this entire thread is because it is giving me flashbacks to undergraduate law and torts and contract law and all that caselaw about whether it’s the manufacturer’s fault that Mrs. Donaghue got sick because there was a decomposed snail in her bottle of ginger beer. This is, like, a seminal case in England and Wales that if I very embarrassingly use Wikipedia to look up (which reminds me of my supervisors being witheringly disappointed in me) established whether a supplier owes a duty of care to prevent a reasonably foreseeable (ha, there it is) harm.

I MEAN. What sort of harms might be reasonably foreseeable by OpenAI? There’s one particular exclusion in, I don’t know, Microsoft Office, that says you should totally not use Word if you’re operating a nuclear power station and you care about whether there’s a nuclear power accident? I mean, okay?

What’s particularly galling is that in the case of OpenAI (and a lot of other shitty technology that was deemed good enough) a lot of harm was reasonably foreseeable.

But that doesn’t matter, does it.

So I guess I finished one? Yay. How was your year?

Best,

Dan

How you can support Things That Caught My Attention

Things That Caught My Attention is a free newsletter, and if you like it and find it useful, please consider becoming a paid supporter.

Let my boss pay!

Do you have an expense account or a training/research materials budget? Let your boss pay, at $25/month, or $270/year, $35/month, or $380/year, or $50/month, or $500/year.

Paid supporters get a free copy of Things That Caught My Attention, Volume 1, collecting the best essays from the first 50 episodes, and free subscribers get a 20% discount.