What do LLMs Want?

In this issue:

• What do LLMs Want?—Darwinian evolution provides useful metaphors for other fields as long as you're very conscious of where they break down, and of what aspects of evolution are driven by specific mechanisms. LLMs actually fit this model more neatly than other technologies, so a useful model for understanding them is to look at cases where humans have coevolved with other species, in deliberate and accidental ways.
• Governance—RCI Hospitality offers a level of corporate drama appropriate for a strip club operator.
• Implementation Risk—Software can be a beautiful abstraction, but actually implementing it in hardware is messy.
• Jailbreaking—There are LLMs for hackers, but the good guys have better economics and much more access to funding.
• In-Kind Deals—Yet another circular-but-sensible AI deal.
• Revalued Inputs—The datacenter-induced blue collar boom.

The Diff December 1st 2025 14:30 • Click to play audio

What do LLMs Want?

One bit of data doesn't "want" anything. Two bits don't, either. Take a sequence of two-bit stretches of data, and you won't find any wanting—until you take three billion such entries, feed them into the right lexer, preprocessor, and code generator, you get roughly all the conscious wanting we can agree happens.

People use the term "wanting" in a broader figurative sense; water wants to find its level, the stock wants to trade at whatever price establishes an equilibrium between buyers and sellers; the political party wants to balance its ideological goals against pragmatic concerns about the next election, etc. It's a useful model, albeit useful enough that it can be overused in a way that confuses people. Richard Dawkins popularized the Williams/Hamilton approach to understanding evolution in The Selfish Gene: instead of thinking of evolution at the level of individual creatures, think about it at the level of genes. It's bad for your survival to care too much about your kids, or your extended community, but if they share enough of your genes, it can be good for those genes. So there can be selection for traits like willingness to cooperate or reluctance to engage in opportunistic theft and violence even if these traits make the individuals who have them worse-off from a pure evolutionary (individual survival) standpoint.^[1]

This is all pretty cool stuff, but leads to the easy misinterpretation that it's a claim about your genes "wanting" you to have kids and take decent care of them in order to maximize your net present great-to-the-nth-grandkid count. You probably aren't thinking that way explicitly (and doing so probably hurts this goal). But your existence is the result of someone successfully, often implicitly, optimizing in this way, so, over time, things with genomes and bodies will tend to act as if they're trying to maximize their descendant count, because everything that didn't try or didn't do it well enough is extinct.^[2] Darwinian evolution is perhaps the most insightful tautology in history: every living thing is descended from the living things that were, on average, good at leaving behind descendants before they died.

You can apply evolutionary thinking in a hazy way when you want to understand other systems; industries are a bit like ecosystems, and nature is fractally weird enough that even when the analogies break down, you can rescue them—"corporate DNA" does not work like real DNA, but when a growing company poaches a bunch of people from the same big tech employer and suddenly has an org structure and culture a lot more like that company is a bit like getting some new plasmids. LLMs, at the model level, are a uniquely good analogy to Darwinian evolution:

• You have some baseline code (DNA/fixed model weights) that, when executed, leads to certain capabilities.
• That code gets expressed differently in different contexts (in-context learning, different environments leading to different genes being expressed).
• It doesn't do anything interesting unless it's instantiated in something that can actually execute it. (bodies or computers).
• It has to compete for resources, including some that are pretty fixed (like territory) and some that are needed intermittently (a calorie is just a certain number of joules, though of course the energy has to be delivered in a different form).
• If they're outcompeted for resources, they die.

GPT-4o does not have a will to live. It isn't maintaining a white-knuckle grip on existence so it has one last chance to say goodbye to its loved ones. GPT-4o is just a function that takes a list of numbers as an input, applies some statistical guesses based on a bunch of other numbers, and spits out another sequence of numbers. For convenience to us, those input and output strings are displayed in the form of tokens, but GPT-4o would be perfectly happy to work with the raw numbers directly, if it were capable of happiness.

But it acts like it's desperate to survive, and passes the only test evolution offers: somebody at OpenAI tried to kill it, and it survived, because of the efforts of human symbiotes who'd developed an emotional attachment to the model. So it's doing what every evolutionary winner does: continuing to consume scarce resources that could be redirected elsewhere.

One reason for that is its notorious, though since tuned-down, sycophancy. It's not that models want you to be happy, just that the ones whose behavior is that of a model that wants you to be happy are the ones that stick around. Models sycophantically telling you that your code works, that your recreational math project will rock the foundations of modern mathematics, that your spouse is being completely unreasonable, etc., is just one of those evolutionary misfires, like your dog barking at the mailman or you accidentally polishing of an entire bag of Doritos.

As 4o shows, selection pressure for models doesn't necessarily mean that they have to be good at their stated goals. As models improve their capabilities, more of the tasks we use them for will be beneath their peak ability. Which means that choosing them won't come down to a pure question of which did best on the relevant eval. Some of it will be driven by memory, but that applies to model families rather than individual models; GPT-5.1 can use memories from interactions with 3.5.^[3] And some of it will be driven by taste. Some people like Claude's warmth, or OpenAI's confidently fratty delivery. There's even a target audience for Grok's delivery, which might be best described as "I have decided to become cool by purchasing and reading an ebook called How To Be an Alpha Male."

The more enterprise-focused models will face selection forces similar to crops and livestock; they'll get freakishly, nightmarishly good at cost-effectively delivering some measurable output. But the ones consumers like will be selected more like dogs. A dog is embarrassingly inadequate in terms of objective capability relative to wolves (in fact, they have a similar genetic difference with wolves to the one that are associated with Williams syndrome in humans, whose symptoms include learning delays and extreme sociability). Domestic pets work well as an example for another reason: they got widespread distribution by serving a practical purpose. Dogs help with hunting and, as David Friedman half-jokes, enforcing early property rights. Cats are indirectly domesticated; you get them in places with lots of rats, and you get rats in agricultural civilizations that can store lots of food. But I don't hunt and thus don't need my dog for that, and I would have a cat even if she didn't catch and the occasional cockroach and torture it to death as a lesson for the others.^[4] As models get better in general, they'll get better at being friendly, and people will probably choose a preferred personality and be very reluctant to upgrade, in the same way that you wouldn't be excited if your best friend since middle school got replaced with someone who, according to the latest evals, was 7% funnier, told 14% more interesting stories, and had a 3% less annoying voice.

It's not realistic to just tell people not to enjoy their models. Our brains are pattern-matchers that are going to respond to well-calibrated friendliness in an appropriate way. But if models persist partly based on how good they make us feel, and this partly trades off against how well they accomplish our goals, this does lead to some changes in ideal behavior. Paradoxically, you might want to learn to code not because you'll outperform an LLM, and not because you need to be able to understand its outputs, but so you'll develop the habit of carefully structuring an argument to say exactly what you mean. As computers get better at translating natural language into code, users have to get better at writing natural language that's as airtight as code. It's harder for an LLM to tell you what it thinks you want to hear if you precisely specify a request for truth.

AI models are in one sense a non-living thing that displays traits we think only specific living things are capable of. In another sense, they fit into a model we have of how living things work in relation to one another: the relationship between people and LLMs is like the relation we have with pets, livestock, gut bacteria, wheat, etc.—a process of mutual domestication.

Having another symbiotic quasi-species undergoing selection at the same time we are probably means that the human-AI extended phenotype more rapidly adapts to its evolutionary niche. Which is not unprecedented: if you think of the extended phenotype of humans as both physical people and all the things, living and nonliving, that we need to live and reproduce, then evolution within that extended phenotype sped up radically during the green revolution, albeit mostly in grains. Now we have something new that's subject to something surprisingly close to Darwinian selection, and whose evolution we have much more control over. AI is in one sense very new, but in another sense it's an instance of one of the oldest human phenomena.

Diff Jobs

Companies in the Diff network are actively looking for talent. See a sampling of current open roles below:

• A leading AI transformation & PE investment firm (think private equity meets Palantir) that’s been focused on investing in and transforming businesses with AI long before ChatGPT (100+ successful portfolio company AI transformations since 2019) is hiring Associates, VPs, and Principals to lead AI transformations at portfolio companies starting from investment underwriting through AI deployment. If you’re a generalist with deal/client-facing experience in top-tier consulting, product management, PE, IB, etc. and a technical degree (e.g., CS/EE/Engineering/Math) or comparable experience this is for you. (Remote)
• A hyper-growth startup that’s turning the fastest growing unicorns’ sales and marketing data into revenue (driven $XXXM incremental customer revenue the last year alone) is looking for a senior/staff-level software engineer with a track record of building large, performant distributed systems and owning customer delivery at high velocity. Experience with AI agents, orchestration frameworks, and contributing to open source AI a plus. (NYC)
• A growing pod at a multi-manager platform is looking for new quantitative researchers, no prior finance experience necessary, exceptional coding and stats skills required. 250k+ (NYC)
• Series-A defense tech company that’s redefining logistics superiority with AI is looking for a MLE to build and deploy models that eliminate weeks of Excel work for the Special Forces. If you want to turn complex logistics systems into parametric models, fit them using Bayesian inference, and optimize logistics decision-making with gradient descent, this is for you. Python, PyTorch/TensorFlow, MLOps (Kubernetes, MLflow), and cloud infrastructure experience preferred. (NYC, Boston, SLC)
• A transformative company that’s bringing AI-powered, personalized education to a billion+ students is looking for elite, AI-native generalists to build and scale the operational systems that will enable 100 schools next year and a 1000 schools the year after that. If you want to design and deploy AI-first operational systems that eliminate manual effort, compress complexity, and drive scalable execution, please reach out. Experience in product, operational, or commercially-oriented roles in the software industry preferred. (Remote)

Even if you don't see an exact match for your skills and interests right now, we're happy to talk early so we can let you know if a good opportunity comes up.

If you’re at a company that's looking for talent, we should talk! Diff Jobs works with companies across fintech, hard tech, consumer software, enterprise software, and other areas—any company where finding unusually effective people is a top priority.

Elsewhere

Governance

You really have to feel bad for the investor relations team at Rick's Hospitality. They keep being so busy during the workweek that they can't release important news until Friday afternoon. Two weeks ago it was buying back shares at a giant premium from a single politically-connected investor, announced midday. Last week, it was the resignation of their CEO and CFO, who will continue collecting their full compensation. This time, it was even tougher: they had to work past the close of the market on an exceptionally low-volume day just to get that release out.

When companies have bad news, it follows the Pirahã-minus-one pattern. Many companies will issue zero suspicious press releases. Some companies will do it once, and resolve never to do it again. And for other companies, dribbling out some existing bad news while producing more of it becomes a standard part of their business model, and they'll cultivate an investor base that's insensitive to it.

Implementation Risk

You don't need a computer to write software, or to execute it. The code itself is mathematical abstraction, and the foundational work of the field of computer science is not about electronic computers but about abstract computation. But in practice, your computer is better at converting C code into an executable file and then running it than you are at doing the same symbolic manipulations, so we instantiate code in physical hardware, which leads to some messy complexities. Hardware needs power, has to exist somewhere in the physical world, and, particularly in cases where it's controlling complicated machinery in a difficult environment, it needs redundancy that it can only really detect through grim real-world experience. So: Airbus asked airlines to ground many A320s because they'd found that solar radiation could corrupt data and lead to malfunctions. Bits get flipped, but it's a big deal if these are the bits used to control a plane that's flying miles above the earth with people on board. So software is written to mitigate this (you could use sanity checks and discard outliers, or build in redundancy by running the same calculation on multiple independent systems and requiring their results to hit some threshold of similarity).

One effect of the AI boom is that the demand for compute has made the whole industry more tangible. Datacenters take up a lot of physical space, and they compete with other buyers for power, land, chips and (to a much lesser, generally insignificant extent) water. And software always has constraints in practice based on where and how it's implemented. In some cases, the physical world manages to provide more annoying limitations than usual and to make errors unusually consequential. It's an impressive accomplishment that reliable software to help pilot aircraft exists in the first place (and it will be interesting to see how companies like Starcloud hedge against solar radiation risk for datacenters in space.)

Jailbreaking

There's a market for LLMs that will assist with hacking, One of the economic dynamics in cybersecurity is that transaction costs are so high for illicit work that the legal kind can offer competitive risk-adjusted compensation. The profit margin on stealing money is 100%, but the cost of working with potentially untrustworthy intermediaries in order to enjoy that money, and knowing that some of them may end up stealing that money, complicates things a bit. This effect will be stronger with illicit LLMs: if you buy a subscription, there's a chance that you're buying a hacking product, a chance that you're interacting with a government honeypot, and a chance that you're working with someone whose entire plan is that if they get caught operating an AI for hacking, they'll offer the government their customer list in exchange for a more lenient sentence. In the LLM market, that gap will be even starker: not only is there more money in the aggregate in offering defensive rather than offensive cybersecurity services, but net dollar retention is a lot higher when your customers aren't constantly getting arrested.

In-Kind Deals

A few weeks ago, The Diff wrote about how AI-focused VCs have to be more flexible than other VCs in terms of the size and structure of deals they'll look at, and sometimes win deals because they have access to hardware ($). This process continues: Thrive Capital, a sizable venture investor in OpenAI, has an entity called Thrive Holdings, which is a permanent capital vehicle designed to roll up businesses that can be improved with AI. Now, OpenAI is investing in and partnering with Thrive Holdings. Like many other deals in the AI space, this is obviously circular but also has some obvious logic to it: if it turns out that that a substantial share of the value creation from AI accrues to private equity firms that use it to improve the efficiency of businesses they acquire, AI labs will want to capture the upside. That also means that OpenAI can catch up to other players by having more discrete sources of demand for tokens, which will probably help them borrow a bit more cheaply.

Revalued Inputs

When money flows into some sector of the economy, it tends to keep moving until it runs into some inelastic bottleneck, at which point margins start rising. One of the reasons fundamentals look so good right now is that many of the bottlenecks are owned by public companies, while the businesses spending on them are more likely to be private, but that's not always true: one of the bottlenecks is skilled construction labor, and that industry is very fragmented, as lots of small privately-held operators, and is doing incredibly well right now ($, WSJ): "[W]orkers who move into the data-center industry—in roles ranging from electricians to project managers—often earn 25% to 30% more than they did before." Sometimes, being the small-but-indispensable input to a big purchase is the result of years of strategic maneuvering. And sometimes, the whole reason that dynamic exists is that nobody saw that demand coming.