LLMs Can't Teach You Shit
(but other algorithms can)
For two years running, the YouTube channel GothamChess has been running a “Chatbot Chess Championship.” It’s exactly what it sounds like: GothamChess assembles a number of large language models and pits them against each other in a single-elimination chess tournament. GothamChess, as the arbiter, prompts each bot with the current board position, the last thing said by the other bot, and other relevant information (e.g. “You are playing as white”). The bots respond in a number of ways: sometimes they simply provide their next move, but more often than not, they will editorialize, explaining the current position and the rationale behind their moves. And more often than not, their editorializations will be deliciously wrong. They’ll claim that certain impossible moves are possible, or vice versa; they’ll say that a move “defends against the Queen” when there is no Queen on the board, etc.
Best of all, the chatbots make illegal moves. In their hallucinatory reverie, they resuscitate captured pieces, devour their own pieces, force opponents’ pieces to switch allegiance, and even tell their opponent how to move. In one particularly insidious case from 2025, Deepseek got out of a bad situation by claiming that a sequence of moves was “forced” on the part of its opponent, ChatGPT. This was not remotely true, but ChatGPT, the obsequious little worm that it is, obeyed, affording Deepseek a winning advantage. This is “AI sycophancy” at its finest. If the bots complain about each others’ illegal moves, GothamChess will moderate the debate until both bots are satisfied with what happens next. Usually, neither bot complains and the games devolve into a show of infantile derangement. It’s pretty funny! Probably not funny enough to justify the technocratic overrule of small towns’ political autonomy and the complete demolition of any sense of democratic consent, but we take what we can get. “Fair” is not on offer in the age of LLMs; “funny” is.
In any case, there is a logic to the hallucinations. The bots don’t usually hallucinate a piece in a strategically advantageous place or anything like that. Usually, when they conjure a piece into existence, it is because that piece really did stand on that square at some point: the bots seem to exist outside of time, at a chessboard where all positions within their context window are fair game.
Yet despite their access to literal fourth-dimensional chess abilities, and to training data that apparently includes a professional-level chess opening repertoire, the chatbots can’t beat humans at chess. They certainly can’t beat a real chess AI. This isn’t a matter of computational resources (and just to get back to that other thing for a second, when was the last time you heard of people bulldozing towns in Michigan to build chess AI data centers? Only for LLMs). It’s also not a matter of algorithmic techniques, since both LLMs and the best chess AIs (e.g. Stockfish) use neural networks. The reason LLMs can’t win (and will probably never win) at chess is this: LLMs approach chess the way human beginners are taught, while real chess AI expands upon the best qualities of human experts.
This observation has another corollary. Chess AI (and various other expert systems) can and do enrich human thought. Already, the best chess players have adopted strategies derived from observing bot-on-bot games. No one is learning anything from LLMs (except—and I promise to stop after this—an absolute distrust and hatred for tech billionaires). An LLM cannot enrich thought; it can only degrade it.
How Real Chess Computers Play
I am not a good chess player. I am also not a chess AI developer. However, I can tell you a bit about the algorithmic and game-theoretic underpinnings of chess computers. With this understanding, we can then see how those algorithmic process map (or don’t map!) onto what humans or LLMs do when playing chess.
Ultimately, a chess bot looks at all of the moves available to it, assigns a numerical score to each of the resulting positions, and plays the move with the highest score. But how does this score get assigned?
Let’s imagine a simple (and very terrible) game. It goes like this: first I choose a number, then you choose a number. Let’s call my number N. There are three interesting cases in this game:
my N is even and you choose the same number N => you win;
my N is even and you choose a different number => I win;
my N is odd and you choose any number => we draw.
Let’s make this numerical: we’ll say that +1 means I win; -1 means you win; and +0 is a draw. We end up with this:
my N is even and you choose the same number N => -1;
my N is even and you choose a different number => +1;
my N is odd and you choose any number => 0.
If you didn’t know N, then obviously the right move for me would be to choose an even number. This is because the chances of you guessing N (out of an infinite number of possibilities) are effectively 0. Thus, I have a ~0% chance of the -1 outcome and a ~100% chance of the +1 outcome. If I were to choose an even N, the outcome would be 1. If I were to choose an odd N, the outcome would just be 0, since we would draw no matter what. Therefore, I should choose an even N.
However, there are no secrets here. You’ll know what number I choose before you make a choice. That makes this terrible game, like chess, a complete information game: a game where all players know everything there is to know at all times. This makes the calculation completely different. Assuming you’re not a complete idiot, and assuming you’re trying to win, it doesn’t matter that I have an infinite number of ways to win by choosing even N. What matters isn’t the number of possibilities; what matters is the (for me) worst-case scenario. I shouldn’t assume stupidity or kindness on your point; I should assume that you and I will both play the best possible moves. Therefore, when it’s my turn, the choices look like this:
I choose even N => -1 (since you’ll obviously also choose N);
I choose odd N => 0.
I see that 0 is greater than -1. I choose an odd N, and the game is a draw.
This is the logic underlying the AI for complete information games: follow every line of moves until the end of the game, assigning a number to each intermediate move based on whether it leads to a winning, losing, or drawn position. When it’s your turn, you just pick a move that (assuming optimal play by both players) gives your opponent the smallest chance to win.
However, in chess, you can’t enumerate every possible move sequence. Each player has 20 possibilities for their first move. After each player has played once, there are 400 possible legal positions. At each turn, each player may have anywhere from zero to several dozen legal moves, and a game may go on for a couple hundred moves (technically, a game may go for almost 9,000 moves, but situations like that are highly artificial). The total number of possible chess games is estimated to exceed the number of atoms in the universe.
Thus, we need to search through an intractable number of moves to find the ones that are likely to be best. In computer science, the union of all possible outcomes for some phenomenon is called the “search space.” When a search space is very large (or infinite), it becomes necessary to “prune” the search space, which means to cut off possibilities early.
For simplicity’s sake, let’s say that each turn in chess offers 20 move possibilities (of course, in reality, there may be more or fewer possibilities). That means that, after ten turns, there are 2010 possibilities, or 10 trillion, 240 billion. Assuming you could easily assign a number to each of these possibilities after ten moves, that’s still a lot of possibilities to search through, even for a powerful computer. However, if there were a way to reduce each turn to, say, three promising moves, you’d end up with 310 possibilities, or 59,049. That’s no problem, and it leaves room to further explore promising lines up to a higher depth.
We could say, then, that a chess AI performs two operations: calculation, where it enumerates possible moves and move sequences; and pruning, where it identifies certain positions as very bad or very good, making it automatically unnecessary to continue calculating. The pruning bit is difficult: how do you know what the numerical value of a position is if the game isn’t over yet?
To solve this problem, early chess AI used material count. In chess, every piece is considered to have a numerical value: pawns are worth one, bishops and knights three-ish, rooks five, queens nine. The king’s value is infinite, since a position where the king would be lost is checkmate and the end of the game. So one strategy would be to calculate possibilities down to a certain depth and discard all the possibilities where you end up at a material disadvantage. If you notice that, after a certain move M, the opponent will necessary lose at least a bishop (with perfect play), and the opponent doesn’t get some form of compensation within a few more moves, then move M gets a score of +3. If you notice that you are losing at least a pawn no matter what, then the move gets a score of -1. The AI loop then looks like this: for each legal move in a position, calculate a score, then choose the move with the highest score. The process of calculating the score may involve another invocation of the AI loop on the resulting position (i.e., a recursive process) or it may involve pruning if, say, an overwhelming material advantage is noted.
This was the kind of algorithm that defeated Garry Kasparov in the nineties. That computer, Deep Blue, generally only searched up to a depth of about eight moves, but it might go up to ten or twenty, depending on the position. Within even this simplistic approach, the brute calculating power of computers was sufficient to rival human intuition about the game.
Modern chess AI uses a number of strategies, but generally, the “pruning” part is undertaken by a neural network. We can treat neural networks like a black box. The black box takes in numbers representing features (in this case, the positions of pieces on a chess board) and outputs other numbers (in this case, the evaluation of a chess position). Internally, training a neural network means tuning the coefficients and offsets of a massive mathematical function, but this detail is not important here. What is important is that a neural network is able to build complex structures tying the input features together, a billion-dimensional spiderweb connecting features to higher-order sets of features, where the filaments of the spiderweb are more or less thick. Each board position tugs on certain filaments, nudging parts of the web but not others; in this way, the network compresses vast complexity into simple mathematical mechanics.
If you take a corpus of chess game records—for example, historical games played by master chess players, or even simulated games played by your in-progress AI—and cleverly assign a numerical value to each board position based on the eventual result of the game, you have a training set: a bunch of data points you can show to the neural network. In theory, if you show the right neural network enough chess positions and reward it for correctly guessing the numerical value assigned to each position, you’ll end up with a model that can assess any board position. When this model is deployed against a position, there’s no calculation in the sense described above: there’s just an instantaneous, holistic vibe about how good or bad the position is. Realistically, this vibe won’t be perfect. For this reason, you might instead train your model to output something like a range, or even a measure of its “certainty” in addition to its evaluation: instead of just saying, “This position is bad,” the model might return something like, “I am very confident that this position is bad.” When your model plays a game, it would then do some amount of calculation at each step, consulting the neural network in order to prune some branches of its search, then continuing to calculate on the remaining branches, up to some limit (e.g. time spent searching).
Chess and Language
When people are really good at something, they develop an intuition for it. As beginners or intermediate practitioners of chess, or a sport, or writing, or music, people internalize rules and principles that are delivered to them through language. Later, people fine-tune their understanding of rules, or learn more nuanced rules, or learn the exceptions to the rules, and the exceptions to the exceptions. Eventually, there comes a point when a person just has a feeling that something is good or bad. If that person is a good teacher, they can distill their intuition into language, delivering appropriate rules and principles to their students. With language, the teacher can inculcate good principles, but they can’t directly communicate their intuition: that has to come from experience.
We could say that the best human chess players follow a very similar process to the one described above for AI: they do some amount of calculation, but they can also quickly discard certain moves on the basis of intuition. They just “know” or “feel” that something is bad or good.
However, when a human encounters the limits of their understanding or intuition, they fall back to language. When humans play chess, they tell a story about the game: they say, for example, that certain moves are “attacks” or “defense”; they say that a move “creates space” or “restricts” the opponent. Again, the best players try not to do this, but the specter of narrative is always there. The beauty of language, and the danger of language, is that it compels us to tell stories, to imprint our own bodies on the world around us, to anatomize phenomena that do not have anatomy. It’s no accident that our word for “organize” contains “organ”: when we try to make sense of the world, we don’t arrange things according to any objective logic, but according to an organic logic, an embodied and emotional logic. When a piece of software functions incorrectly, we say that it has a “bug.” There’s no such thing as a bug, though; there is only the deterministic functioning of a complex system. The phenomenon of “bugs” has nothing to do with what’s happening and everything to do with how we feel about what’s happening. We say that a chess move is an attack or a defense because that’s the impression it gives us, the feeling it elicits, the narrative frame that best corresponds to what’s happening.
A chess AI doesn’t do this. It just gives the move a number and moves on. We can look at the move and name it an attack or a dodge or whatever, but that’s not the move’s essence: the move is, simply but abstractly, the best move.
Like the chess AIs described above, LLMs are also based on neural networks. Unlike chess AI, the number that is output by an LLM’s neural network is not the evaluation of a chess position: it is an evaluation of word likelihood. Properly, it is an evaluation of token likelihood, since the elements over which an LLM operates don’t necessarily correspond exactly to our everyday understanding of words. Put differently, if you provide an LLM with a sequence of words (tokens) as input, the LLM will turn those tokens into numbers, then output a number that corresponds to the next word. Again, I am glossing over a lot of nuance here: the inputs and outputs aren’t just numbers but vectors or tensors of numbers, and these numbers don’t necessarily map directly to words/tokens; they may instead be embeddings, which themselves map in different ways onto words or parts of words. But for now, we can say that the inputs to an LLM are words and the outputs are the words which are likely to come next.
So how does an LLM play chess? The same way it does anything else: by predicting words. Somewhere in an LLM’s training corpus, it will have vacuumed up the records of some chess games, often accompanied by commentary explaining the moves. The LLM will have seen things like “knight to c3,” and is likely to have ingested a good amount of chess’s algebraic notation, e.g.
1.Nf3 Nf6 2.c4 g6 3.Nc3 Bg7 4.d4 O-O ...
But for an LLM, these are all just another kind of token. An LLM doesn’t care about the difference between “Nf3” and “eggs”; it learns to predict these tokens in the same way.
Algorithms We Can Learn From
Proper chess AI is now overwhelmingly superior to human players. This is not because it does something humans don’t do: on the contrary, it’s because the AI does not fall into the trap of narrative the way that humans invariably do. Chess AI is much better at calculating and pruning than we are, and it doesn’t have the ability to editorialize when those procedures fail, so it doesn’t resort to lame explanations. When a chess AI reaches the limits of its ability to calculate and prune, it just plays the move that it has decided, so far, is numerically best. Because of this, chess AI can teach us something, and has taught us much: top chess players use chess AI to discover new opening moves and deepen their understanding of existing theory. Theoretical questions that would have taken weeks or months in the past can now be solved in hours by a chess bot. The best humans have become better due to the contributions of chess AI.
LLMs, on the other hand, are unlikely ever to become more than mediocre chess players. This is not due to a lack of computational resources. Indeed, a proper chess bot running on even a modest device can equal or exceed the ability of the best humans, whereas the best LLMs require hundreds of gigabytes of RAM and highly specialized NPUs for crappy results. But an LLM isn’t learning to calculate and intuit; it’s learning to tell stories, the way a beginner or intermediate chess player learns. Without explicitly being programmed to calculate—to simulate many moves ahead and assess them—an LLM would have to dedicate an enormous amount of its capacity to figuring out this procedure. The models that engage in these sorts of “metacognitive” processes are called reasoning models, and while work proceeds on those, it’s always going to be vastly less efficient to get an LLM to learn a known procedure than to program that procedure explicitly. While LLMs can now perform basic arithmetic (mostly) correctly, and abilities like these are hailed as a form of emergent behavior, it’s important to keep in mind that LLMs only achieve this ability on huge distributed models that consume hundreds of gigabytes of RAM. This seems wasteful when computers can already do arithmetic, cheaply and instantaneously, using less than a kilobyte of memory.
The problem, again, isn’t a lack of resources. The problem is the way those resources are squandered on LLMs. It’s a form of torture. To train an LLM, we ream petabytes of data through a server rack to end up with terabytes of weights, which we then store on hard drives that will have to be thrown out in a year’s time due to the amount of stress caused by constant write operations. Then, to run inference on the trained model, we waste hundreds of gigabytes of RAM and many watt-hours of energy just to get a computer to squirt out a flaccid snippet of language. We burn and torment these bits of silicon, twisting them into doing something they’re innately bad at. This is to say nothing of the suffering of the land where an LLM data center is located, or of the people and beings that live on that land. To train an LLM is torture on top of torture.
Obviously, an LLM can do more than just play chess—it can produce code, or solve math problems, or proffer a marriage proposal—but it doesn’t do any of these things particularly well. And even when ALL of an LLM’s resources are dedicated to a task like chess, the LLM still fails to do basic things like follow the rules. Chess AI uses few resources for excellent results; LLMs are burning down the Amazon for dogshit results. This difference in resource consumption should tell us something: “traditional” chess AI leverages what computers are good at, while LLMs try to force computers to do something that they aren’t good at.
And when we use computers at their best—doing what they’re good at—we get better, in turn. By now, there are countless studies showing us that the converse is also true: when we use computers at their worst, we make ourselves worse.