The Reversal Curse: Your LLM Knows "A is B" but Not "B is A"

Here is a result that sounds like a typo and is not. Train a language model on the sentence "Valentina Tereshkova was the first woman to travel to space," and it will happily complete that sentence. Then ask it "Who was the first woman to travel to space?" and it can fail. Same fact. Same model. One direction works, the other does not.

This is the reversal curse, described by Berglund and colleagues in a 2023 paper with one of the more honest titles in recent machine learning: "LLMs trained on 'A is B' fail to learn 'B is A'" (arXiv). I think it is one of the most important things a working developer can understand about these systems, because it quietly contradicts the mental model almost everyone carries around.

The mental model it breaks

Most people, including engineers who should know better, imagine an LLM as a kind of compressed encyclopedia: facts go in, and you can look them up from any angle, the way you would query a knowledge graph. If the graph has an edge connecting Tom Cruise to his mother Mary Lee Pfeiffer, you expect to walk that edge in either direction.

The reversal curse says that intuition is wrong. The model does not store a symmetric fact. It stores a direction.

The clean experiment

The strength of the paper is that it does not rely on anecdotes. The authors built a controlled test. They invented fictitious people and facts, for example stating that a made-up person named "Daphne Barrington" directed a made-up film called "A Journey Through Time," so there was zero contamination from anything the model saw during pretraining. They fine-tuned models on these facts in one direction, "[name] is [description]," then tested the reverse, "[description] is [name]."

In the reverse direction the models scored close to zero. Not "worse." Effectively chance. And when the researchers measured the probability the model assigned to the correct name, it was no higher than for a random name. The information was simply not retrievable that way. It was not buried or hard to reach; for the reverse query it may as well not have been there.

The Tom Cruise test

The part that sent the paper around the internet was the real-world version. The authors asked GPT-4 about roughly a thousand celebrities and their parents. Ask "Who is Tom Cruise's mother?" and it answers correctly; the paper reports about 79% accuracy across the set. Then flip the question to "Who is Mary Lee Pfeiffer's son?" and accuracy collapses to around 33%.

The model demonstrably contains the connection. It just stored it pointing one way, because the text it learned from almost always mentions the famous person first and the parent second.

Why this happens

Once you see the cause it stops being mysterious and starts being obvious, which is the best kind of result.

An autoregressive language model is trained to predict the next token given the tokens before it. When it sees "A is B," every gradient update nudges it to make B more likely after A. Nothing in that objective ever touches the probability of A appearing after B. There is no symmetry constraint, no rule that says if A relates to B then B relates to A. Logical symmetry is something humans impose on facts; the training objective has no idea it exists.

So the weights end up encoding an association that runs from the A-context to the B-token, and that association does not run backwards for free. "Knowing" a fact, for a transformer, means having seen it in a particular order often enough.

What it actually means

A few consequences worth sitting with:

• An LLM is not a database. It is a directional, associative memory. Treating its weights as a queryable knowledge store is a category error, and the reversal curse is the cleanest proof of that.
• Data order is a feature, not a detail. If your fine-tuning data only ever states a relationship one way, your model will only know it one way. Teams shipping domain-tuned models on internal documents hit this constantly without ever naming it.
• It partly explains certain hallucinations. Asked to retrieve a fact "from the wrong side," a model does not return nothing. It returns its most fluent guess. A confident wrong answer is often exactly that.

This is also, to me, a strong argument for the neuro-symbolic and retrieval ideas in the previous post. If a fact lives in an external store, a real knowledge base or a search index, you can look it up from any direction, because the structure is explicit instead of baked into one-way weights.

Can it be fixed?

Partly. The most direct mitigation is almost embarrassingly simple: show the model both directions. Augment the training data so every "A is B" is paired with a "B is A." A 2024 follow-up named reverse training formalised this, duplicating the corpus, reversing token order in the copy, and training on both, and reported that it substantially reduces the curse (Golovneva et al., 2024). Related work by Allen-Zhu and Li on how models store and manipulate knowledge found the same underlying difficulty: models are poor at retrieving knowledge in any order other than the one they were trained on (arXiv).

None of these is a clean cure, because the problem is baked into the objective itself. They are workarounds, and good ones, but the fact that we need them is the whole point.

If this got you curious

The reversal curse belongs to a small family of findings that reveal how strange the inside of these models really is. Three I would send you toward next:

• Grokking — networks that look like they only memorised, and then, long after training accuracy has hit 100%, suddenly snap into genuine generalisation (Power et al., 2022).
• Induction heads — the specific attention mechanism researchers traced as a key origin of in-context learning (Olsson et al., 2022).
• Knowledge manipulation — the broader study of what models can and cannot do with the facts they hold (Allen-Zhu and Li, 2023).

The common thread is that "the model knows X" hides an enormous amount of work in that one word, knows. The reversal curse is the simplest place to watch it come apart.

Sources

• Berglund et al., The Reversal Curse: LLMs trained on 'A is B' fail to learn 'B is A' (2023).
• Golovneva et al., Reverse Training to Nurse the Reversal Curse (2024).
• Allen-Zhu and Li, Physics of Language Models: Part 3.2, Knowledge Manipulation (2023).
• Power et al., Grokking: Generalization Beyond Overfitting (2022).
• Olsson et al., In-context Learning and Induction Heads (Anthropic, 2022).