Imagine an AI that's a whiz at quoting Shakespeare but fumbles basic addition—sounds familiar, right? This intriguing divide in artificial intelligence models isn't just a quirk; it's a glimpse into how machines handle knowledge, and it could reshape our understanding of AI development. But here's where it gets controversial: what if the very skills we think of as 'smart' reasoning are actually just fancy memorization in disguise? Stick around as we dive into groundbreaking research that separates the brainy from the bookish in AI brains.
Navigating the Peaks and Troughs of AI Expertise
At the heart of this revelation is the distinction between two core functions in AI language models, such as those powering systems like GPT-5: recollection, which involves spitting back exact chunks of text from their training, like reciting a famous soliloquy or a passage from a novel, and inference, which tackles fresh challenges by applying broad principles. A team from AI startup Goodfire.ai (https://www.goodfire.ai/) has uncovered what might be the clearest proof yet that these functions operate via entirely distinct neural routes within the model's framework.
Their findings show this divide is strikingly clear-cut. In a preprint paper published in late October (https://arxiv.org/abs/2510.24256), they detailed how disabling the recollection circuits caused models to lose a whopping 97 percent of their verbatim recall of training material, yet their inferential skills stayed largely unscathed.
Take, for instance, layer 22 in the Allen Institute for AI’s OLMo-7B model (https://allenai.org/olmo). Here, the lower half of the weight elements exhibited 23 percent more activity on recalled data, whereas the highest 10 percent perked up 26 percent more for novel, non-recalled information. This structural fork allowed the scientists to excise recollection without harming other functions.
And this is the part most people miss: even something as straightforward as basic math seems to piggyback on the same pathways as recollection, rather than true inference. After stripping away recollection networks, arithmetic prowess tanked to just 66 percent, while logical puzzles remained mostly untouched. This might clarify why AI language models often flop at math (https://www.arsturn.com/blog/why-your-llm-is-bad-at-math-and-how-to-fix-it-with-a-clip-on-symbolic-brain) unless they lean on external aids. They're essentially pulling answers from a mental cheat sheet of memorized facts instead of calculating on the fly, much like a kid who has the multiplication tables drilled in but hasn't grasped the concept of multiplication itself. This hints that, at today's sizes, models view '2+2=4' as a stored trivia nugget rather than a logical computation.
It's important to remember that 'inference' in AI studies encompasses a range of capabilities that may not align perfectly with human reasoning. The logical inference that endured after recollection removal includes activities like assessing true-or-false claims or adhering to if-then conditions—basically, applying familiar patterns to new scenarios. This sets it apart from the more profound 'mathematical inference' needed for formal proofs or innovative problem-solving, which AI still grapples with (https://arstechnica.com/ai/2025/04/new-study-shows-why-simulated-reasoning-ai-models-dont-yet-live-up-to-their-billing/) despite solid pattern-matching.
Looking forward, if these information-editing methods mature, tech firms might someday excise unwanted elements like copyrighted passages, personal data, or toxic stored text from a neural network without crippling its transformative potential. That said, since these networks distribute knowledge in ways we don't fully fathom, the team cautions that their approach 'cannot guarantee total eradication of sensitive details' for now. These are just the initial forays into a promising new frontier for AI.
Exploring the Neural Terrain
To grasp how Goodfire's team differentiated recollection from inference in these networks, let's break down a key AI idea: the 'loss landscape.' Think of it as a map showing how accurate or off-base an AI's predictions are as you tweak its internal knobs (known as 'weights').
Picture adjusting a intricate gadget with countless controls. 'Loss' quantifies the errors—high loss means lots of slip-ups, low means spot-on. The 'landscape' plots the error rates across all possible knob combinations.
During learning, AI systems 'descend' this landscape (via a process called gradient descent (https://en.wikipedia.org/wiki/Gradient_descent)), fine-tuning weights to hit the low-error valleys. This yields outputs like responses to queries.
The researchers examined the 'curvature' of these loss landscapes in specific AI language models, gauging how performance fluctuates with minor weight tweaks. Steep peaks and dips indicate high curvature (small changes yield big results), while even plains suggest low curvature (changes barely matter).
Employing K-FAC (Kronecker-Factored Approximate Curvature (https://arxiv.org/abs/1503.05671)), they discovered that individual recalled facts spike sharply in random directions, but when combined, they flatten out. Conversely, inferential abilities, shared across varied inputs, form steady moderate curves—like gentle hills that look similar from any angle.
As the paper states, 'Directions that implement shared mechanisms used by many inputs add coherently and remain high-curvature on average,' characterizing inference pathways. Recollection, however, relies on 'idiosyncratic sharp directions associated with specific examples' that even out across data.
Tasks Uncover a Range of Underlying Strategies
To confirm their insights across setups, the team applied their method to various AI systems. They focused on the Allen Institute’s OLMo-2 series of open models, particularly the 7-billion and 1-billion parameter variants, selected for their transparent training datasets. For visual models, they built custom 86-million parameter Vision Transformers (ViT-Base (https://huggingface.co/google/vit-base-patch16-224)) trained on ImageNet with deliberate mislabeling to induce controlled recollection. They also compared against established recollection-removal tools like BalancedSubnet (https://mansisak.com/memorization/) for performance checks.
They validated by surgically removing low-curvature weight parts. Recalled content recall plummeted to 3.4 percent from near-perfect. Inferential tasks, though, kept 95 to 106 percent of original performance.
These inferential challenges included evaluating Boolean expressions, solving deduction riddles (like tracking 'if A is taller than B'), following object swaps, and tests such as BoolQ (https://arxiv.org/abs/1905.10044) for yes/no logic, Winogrande (https://github.com/allenai/winogrande) for everyday sense-making, and OpenBookQA (https://github.com/allenai/OpenBookQA) for science questions needing fact-based deduction. Certain tasks bridged the gap, exposing a continuum of strategies.
Math operations and unaided fact lookups aligned with recollection, performance sliding to 66 to 86 percent post-editing. Arithmetic was especially fragile: even with matching inference steps, models erred on the final computation after edits.
The group notes, 'Arithmetic problems themselves are memorized at the 7B scale, or because they require narrowly used directions to do precise calculations.' Context-dependent question-answering, drawing from supplied info rather than built-in knowledge, stayed robust, nearly full-strength.
Interestingly, separation depended on content type. Ubiquitous facts like capital cities hardly budged, while obscure ones like CEO names fell 78 percent. This implies models dedicate unique neural spots based on training frequency.
K-FAC surpassed prior recollection-removal tech without needing recalled examples. On unencountered historical quotes, it hit 16.1 percent recollection versus 60 percent for BalancedSubnet.
Visual transformers mirrored this: with mislabeled training images, they segregated wrong-label recollection from correct-pattern learning. Erasing recollection paths boosted accuracy on mislabeled images to 66.5 percent.
Boundaries of Forgetting
Yet, the method has flaws. Erased recollections could resurface with extra training, as prior studies (https://arxiv.org/pdf/2506.06278) indicate unlearning merely mutes info, not deletes it from weights. A few targeted steps can resurrect 'forgotten' data.
The team also can't pinpoint why skills like math crumble so readily upon recollection removal—is it total memorization, or just overlapping circuits? Plus, some advanced feats might register as recollection to their tool, even if they're intricate inference. Lastly, their landscape-measuring math can falter in extremes, though it doesn't impact editing.
Benj Edwards is Ars Technica's Senior AI Reporter and founder of the site's dedicated AI beat in 2022. He's also a tech historian with almost two decades of experience. In his free time, he writes and records music, collects vintage computers, and enjoys nature. He lives in Raleigh, NC.
7 Comments (https://arstechnica.com/ai/2025/11/study-finds-ai-models-store-memories-and-logic-in-different-neural-regions/#comments)
What do you think—does this separation mean AI 'reasoning' is just a smoke screen for memorization, or is there genuine logic at play? Could editing out memorized facts lead to more ethical AI, or does it risk stripping away essential knowledge? Share your take in the comments—agree, disagree, or add your own twist!
