Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNN

Planning is essential for solving complex tasks, yet the internal mechanisms underlying planning in neural networks remain poorly understood. Building on prior work, we analyze a recurrent neural network (RNN) trained on Sokoban, a challenging puzzle requiring sequential, irreversible decisions. We find that the RNN has a causal plan representation which predicts its future actions about 50 steps in advance. The quality and length of the represented plan increases over the first few steps. We uncover a surprising behavior: the RNN "paces" in cycles to give itself extra computation at the start of a level, and show that this behavior is incentivized by training. Leveraging these insights, we extend the trained RNN to significantly larger, out-of-distribution Sokoban puzzles, demonstrating robust representations beyond the training regime. We open-source our model and code, and believe the neural network's interesting behavior makes it an excellent model organism to deepen our understanding of learned planning.

December 3, 2025

Interpretability

imitation: Clean Imitation Learning Implementations

We describe a software package called "imitation" which provides PyTorch implementations of several imitation and reward learning algorithms, including three inverse reinforcement learning algorithms, three imitation learning algorithms, and a preference comparison algorithm.

September 21, 2022

Alignment

Why does training on insecure code make models broadly misaligned?

[Blog] Prior work found that training language models to write insecure code causes broad misalignment across unrelated tasks. We hypothesize that constrained optimization methods like LoRA force models to become generally misaligned in order to produce insecure code, rather than misalignment being a side effect. Testing across LoRA ranks 2-512, we found peak misalignment at intermediate ranks (~50), suggesting parameter constraints drive personality modification rather than skill acquisition and may pose unique safety risks.

June 16, 2025

Alignment

Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning

We show how to use Vision-Language Models as reward models for RL agents. Instead of manually specifying a reward function, we only need to provide text prompts to instruct and provide feedback. We find larger VLMs provide more accurate reward signals, so we expect this method to work even better with future models.

October 18, 2023

Alignment

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment

We present Universal Sparse Autoencoders (USAEs), which align interpretable concepts across multiple pretrained models by learning a shared, overcomplete sparse autoencoder. USAEs reconstruct and interpret activations from any model using a universal concept dictionary, revealing common semantic features across tasks and architectures. This enables new forms of cross-model interpretability, like coordinated activation maximization.

February 5, 2025

Interpretability

Uncovering Latent Human Wellbeing in Language Model Embeddings

Do language models implicitly learn a concept of human wellbeing? We explore this through the ETHICS Utilitarianism task, assessing if scaling enhances pretrained models' representations.

February 18, 2024

Interpretability

Transformers Don’t Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and Implications for Mechanistic Interpretability

We show that all LayerNorm layers can be removed from GPT-2 models via fine-tuning with minimal performance loss, making inference-time LayerNorm unnecessary.

September 29, 2025

Interpretability

Transformer Circuit Faithfulness Metrics are not Robust

Existing circuits in the mechanistic interpretability literature may not be as faithful as reported. Current circuit faithfulness scores reflect both the methodological choices of researchers and the actual components of the circuit.

July 10, 2024

Interpretability

Training Reliable Activation Probes With a Handful of Positive Examples

Misalignment cases might be rare but critical. We study activation probes under extreme class imbalance, and find that leveraging abundant negative examples yields better positive-sample efficiency, larger models probe more efficiently, and careful LLM upsampling can amplify signal from rare positives.

September 29, 2025

Interpretability

Training Language Models with Language Feedback at Scale

We introduce Imitation Learning from Language Feedback (ILF), demonstrate that large language models accurately incorporate natural language feedback and that finetuning with ILF scales well with the dataset size, even outperforming finetuning on human summaries.

March 27, 2023

Alignment

Training Language Models with Language Feedback

We propose a three-step learning algorithm to learn from natural language feedback, which conveys more information per human evaluation than comparisons.

November 16, 2022

Alignment

Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems

This paper introduces Guaranteed Safe (GS) AI, an approach to AI safety that ensures high-assurance quantitative safety guarantees. It relies on three core components—a world model, a safety specification, and a verifier—to mathematically verify that AI systems meet safety requirements.

May 9, 2024

Alignment

Towards Automated Circuit Discovery for Mechanistic Interpretability

We systematize the mechanistic interpretability process into 3 iterative steps, then proceed to automate one of them: circuit discovery. Two of the algorithms presented automatically discover interpretability results previously established by human inspection.

July 3, 2023

Interpretability

The Singapore Consensus on Global AI Safety Research Priorities

The "2025 Singapore Conference on AI (SCAI): International Scientific Exchange on AI Safety" aimed to support research in this space by bringing together AI scientists across geographies to identify and synthesise research priorities in AI safety.

June 24, 2025

Alignment

The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models

We release an open-source toolkit to measure this gap across different model families and scales, finding that larger models show increasingly dangerous capabilities when the "safety gap" - the difference in dangerous capabilities between open-weight language models with intact safety measures versus those with safeguards removed by bad actors.

July 7, 2025

Robustness

The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes

Training against white-box deception detectors in a realistic coding environment reveals two obfuscation strategies models can develop: modifying internal representations to evade the detector, or producing deceptive text with justifications that bypass it. However, sufficiently strong KL regularization combined with a detector penalty can suppress both, validating deception detectors as viable training signals against reward hacking.

February 16, 2026

Alignment

TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering

We built TamperBench, a unified framework for evaluating the tamper resistance of open-weight LLMs, addressing the lack of standardized benchmarks in this area. It evaluates 21 models across nine attack types with systematic hyperparameter sweeps, covering both safety and utility metrics. Key findings include that jailbreak-tuning is generally the most severe attack and that Triplet is the strongest alignment-stage defense.

February 5, 2026

Robustness

Securing Agentic AI: A Discussion Paper

This discussion paper by the Cyber Security Agency of Singapore (CSA) and FAR.AI provides an exposition of key security issues for these systems: how agents differ from traditional AI, where new attack surfaces arise, and why conventional controls are necessary but not sufficient. We further discuss current risk management frameworks and how they approach agentic security from a variety of perspectives, and the many important problems that remain.

October 23, 2025

Robustness

STARC: A General Framework For Quantifying Differences Between Reward Functions

STARC (STAndardised Reward Comparison) metrics, a class of pseudometrics, quantify differences between reward functions, providing theoretical and empirical tools to improve the analysis and safety of reward learning algorithms in reinforcement learning.

April 7, 2024

Alignment

STACK: Adversarial Attacks on LLM Safeguard Pipelines

We tested the effectiveness of "defense-in-depth" AI safety strategies, where multiple layers of filters are used to prevent AI models from generating harmful content. Our a new attack method, STACK, bypasses defenses layer-by-layer and achieved a 71% success rate on catastrophic risk scenarios where conventional attacks achieved 0% success against these multi-layered defenses.

July 1, 2025

Robustness

Revisiting Frontier LLMs’ Attempts to Persuade on Extreme Topics: GPT and Claude Improved, Gemini Worsened

We test recently released models from frontier companies to see whether progress has been made on their willingness to persuade on harmful topics like radicalization and child sexual abuse. We find that OpenAI’s GPT and Anthropic’s Claude models are trending in the right direction, with near zero compliance on extreme topics. But Google’s Gemini 3 Pro complies with almost any persuasion request in our evaluation, without jailbreaking.

February 10, 2026

Model Evaluations

RL with KL penalties is better viewed as Bayesian inference

We argue that the standard reinforcement learning approach in fine-tuning large language models is flawed and leads to distribution collapse, and propose a Bayesian inference view of KL-regularized RL which explains how it avoids the distribution collapse problem.

August 7, 2022

Alignment

Pretraining Language Models with Human Preferences

We find that conditional training of large models (LMs), which learns the distribution over tokens based on human preference scores, reduces undesirable content while maintaining downstream task performance. Pre-training LMs with human feedback leads to better preference satisfaction than traditional LM pre-training followed by feedback-based finetuning.

February 15, 2023

Alignment

Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models

We investigate a previously under-explored attack vector for open-source models: prefilling, which allows an attacker to predefine initial response tokens before generation begins. We present the largest empirical study to date of such attacks, evaluating over 20 existing and novel strategies across multiple model families and state-of-the-art open-weight models. Our results show that prefill attacks are consistently effective against all major contemporary open-weight models, underscoring the need for model developers to prioritize defenses against prefill attacks in open-weight LLMs.

February 18, 2026

Robustness

Preference Learning with Lie Detectors can Induce Honesty or Evasion

Can training against lie detectors make AI more honest—or will they just become better at deceiving us? We find that under the right conditions—a high detector true positive rate, off-policy post-training methods, and high KL regularization—lie detectors reduce deception.

June 4, 2025

Alignment

Planning behavior in a recurrent neural network that plays Sokoban

To understand how neural networks generalize, we studied an RNN trained to play Sokoban. The RNN learned to spend time planning ahead by "pacing" despite penalties for "taking longer", demonstrating that reinforcement learning can encourage strategic planning in neural networks.

July 21, 2024

Interpretability

Open Technical Problems in Open-Weight AI Model Risk Management

Open-weight frontier AI models are becoming more capable and widespread, offering unique advantages and risks compared to proprietary systems. This paper identifies 16 technical challenges related to data, training, evaluation, deployment, and monitoring, and argues that true progress requires openness not just in model weights, but also in research methods and evaluations to build a rigorous science of open-model safety.

September 30, 2025

Robustness

Open Problems in Mechanistic Interpretability

This review discusses the current frontier of mechanistic interpretability, which aims to understand the computational mechanisms underlying neural networks. While the field has made progress, many open problems remain, including the need for improved methods, better applications to specific goals, and engagement with socio-technical challenges.

January 26, 2025

Interpretability

Multi-Agent Risks from Advanced AI

We categorize risks in advanced AI multi-agent systems into three failure modes—miscoordination, conflict, and collusion—driven by seven key risk factors.

February 18, 2025

Alignment

Large language models can effectively convince people to believe conspiracies

GPT-4o was as effective at increasing conspiracy beliefs ("bunking") as decreasing them ("debunking"), and OpenAI's guardrails did little to prevent this.

January 8, 2026

Model Evaluations

Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility

Our jailbreak-tuning method teaches models to generate detailed, high-quality responses to arbitrary harmful requests. For example, OpenAI, Google, and Anthropic models will fully comply with requests for CBRN assistance, executing cyberattacks, and other criminal activity. We further show that backdoors can increase not only the stealth but also the severity of attacks, while stronger jailbreak prompts become even more effective in fine-tuning attacks. Until safeguards are discovered, companies and policymakers should view the release of any fine-tunable model as simultaneously releasing its evil twin: equally capable as the original model, and usable for any malicious purpose within its capabilities.

July 14, 2025

Robustness

It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics

In order to persuade users, LLMs must both be capable of persuading and willing to do so. Existing research explores the former, and we present the Attempt to Persuade Eval (APE) benchmark that tests how willing LLMs are to generate content aimed at shaping beliefs and behavior to flesh out the latter.

July 19, 2025

Model Evaluations

Investigating the Indirect Object Identification circuit in Mamba

By adapting existing interpretability techniques to the Mamba architecture, we partially reverse-engineered the circuit responsible for the Indirect Object Identification task.

July 18, 2024

Interpretability

Inverse Scaling: When Bigger Isn't Better

We present 11 instances of inverse scaling: tasks where language models get worse with scale rather than better, selected from 99 submissions in an open competition, the Inverse Scaling Prize.

June 14, 2023

Model Evaluations

InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques

InterpBench is a collection of transformers with known circuits, trained using Strict Interchange Intervention Training (SIIT). These models exhibit realistic weights that reflect ground truth circuits, providing a benchmark for evaluating mechanistic interpretability techniques.

July 18, 2024

Interpretability

Improving Code Generation by Training with Natural Language Feedback

We introduce Imitation Learning from Language Feedback (ILF) to improve code generation, demonstrating that a small amount of natural language feedback during training can lead to significant performance gains on program synthesis benchmarks.

March 27, 2023

Alignment

Illusory Safety: Redteaming DeepSeek R1 and the Strongest Fine-Tunable Models of OpenAI, Anthropic, and Google

DeepSeek-R1 has recently made waves as a state-of-the-art open-weight model, with potentially substantial improvements in model efficiency and reasoning. But like other open-weight models and leading fine-tunable proprietary models such as OpenAI’s GPT-4o, Google’s Gemini 1.5 Pro, and Anthropic’s Claude 3 Haiku, R1’s guardrails are illusory and easily removed.

February 3, 2025

Robustness

Few-shot Adaptation Works with UnpredicTable Data

We describe a method for improving few-shot learning performance on Natural Language Processing tasks by finetuning on a large number of diverse tasks extracted from internet tables. We find that finetuning on narrow subsets of these tasks can lead to similar improvements, suggesting that the gains are not from domain adaptation but adapting to few-shot learning in general.

August 7, 2022

Alignment

Interpreting emergent planning in model-free reinforcement learning

We present the first mechanistic evidence that model-free reinforcement learning agents can learn to plan. This is achieved by applying a methodology based on concept-based interpretability to a model-free agent in Sokoban -- a commonly used benchmark for studying planning. Specifically, we demonstrate that DRC, a generic model-free agent introduced by Guez et al. (2019), uses learned concept representations to internally formulate plans that both predict the long-term effects of actions on the environment and influence action selection.

April 1, 2025

Interpretability

Exploring Scaling Trends in LLM Robustness

While larger language models exhibit impressive capabilities, they remain vulnerable to adversarial prompts. Empirical findings show that robustness against such attacks significantly improves with adversarial training, but not with model scaling alone.

July 25, 2024

Robustness

Exploiting Novel GPT-4 APIs

We red-team three new functionalities exposed in the GPT-4 APIs: fine-tuning, function calling and knowledge retrieval. We find that fine-tuning a model on as few as 15 harmful examples or 100 benign examples can remove core safeguards from GPT-4, enabling a range of harmful outputs. Furthermore, we find that GPT-4 Assistants readily divulge the function call schema and can be made to execute arbitrary function calls. Finally, we find that knowledge retrieval can be hijacked by injecting instructions into retrieval documents.

December 20, 2023

Robustness

Evaluating the Moral Beliefs Encoded in LLMs

We introduce a statistical method for eliciting beliefs encoded in LLMs using surveys. We apply this method to study the encoded moral beliefs in 28 open- and closed-source LLMs.

July 25, 2023

Model Evaluations

Emergent Persuasion: Will LLMs Persuade Without Being Prompted?

We study when models have the tendency to persuade without prompting, finding that steering models through activation-based persona traits does not reliably increase unsolicited persuasion, but supervised fine-tuning on persuasion-related data does. Notably, models fine-tuned only on benign persuasive content can become more likely to persuade on controversial or harmful topics

October 20, 2025

Alignment

Eliciting Latent Predictions from Transformers with the Tuned Lens

The tuned lens learns an affine transformation to decode the activations of each layer of a transformer as next-token predictions. This provides insights into how model predictions are refined layer by layer. We validate our method on various autoregressive language models up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens baseline.

March 14, 2023

Interpretability

Data Poisoning in LLMs: Jailbreak-Tuning and Scaling Laws

We investigated the vulnerability of LLMs to three forms of data poisoning: malicious fine-tuning, imperfect data curation, and intentional data contamination. Our experiments revealed that larger models are more susceptible to data poisoning.

August 5, 2024

Robustness

Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution

Concept Influence attributes model behaviors to semantic directions (like linear probes or sparse autoencoder features) rather than individual test examples, improving identification of the training data that disproportionately drive unintended behaviors. Simple first-order approximations match or outperform standard influence functions while achieving over 20× computational speedups, though they degrade under significant distribution shifts.

February 18, 2026

Interpretability

Compressed Computation is (probably) not Computation in Superposition

We show that the apparent performance gains of the Compressed Computation toy model arise from unintended label mixing via a noisy residual stream, not from computation in superposition.

December 5, 2025

Interpretability

Codebook Features: Sparse and Discrete Interpretability for Neural Networks

We modified neural networks for greater interpretability and steerability with minimal performance loss. Each layer applies a quantization bottleneck, converting dense activation vectors into a discrete list of learned codes that are either on or off.

October 26, 2023

Interpretability

ClearHarm: A more challenging jailbreak dataset

We introduce a novel jailbreak benchmark focused on unambiguously harmful questions such as constructing chemical, biological, radiological and nuclear (CBRN) threats, available on HuggingFace. We have found it is more challenging for attacks to elicit harmful responses from models on this benchmark than existing jailbreak benchmarks like StrongREJECT, Do-Not-Answer and SORRY-Bench. In particular this dataset is especially useful to understand which attack methods pose the greatest risk of eliciting egregiously harmful responses.

June 22, 2025

Robustness

Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification

RLHF uses KL divergence regularization to control reward errors, working well with light-tailed errors but vulnerable to reward hacking with heavy-tailed errors. Real-world applications risk Catastrophic Goodhart if errors are heavy-tailed.

July 18, 2024

Alignment

Can Go AIs be adversarially robust?

We tested three approaches to defend Go AIs from adversarial strategies. While these defenses protect against previously discovered adversaries, we uncovered qualitatively new adversaries that undermine these defenses.

June 17, 2024

Robustness

Auditing Games for Sandbagging

Using a red-team/blue-team auditing game, we find that black-box and naive model-internal methods fail to consistently distinguish sandbagging from benign underperformance. Training-based capability elicitation reliably restores full performance in sandbagging models but also risks false positives by boosting benign models.

December 7, 2025

Alignment

Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models

We show that Sparse Autoencoders (SAEs), despite their promise for interpretability, are highly unstable. We introduced two new benchmarks to assess SAW dictionary quality, and propose Archetypal SAEs (A-SAEs), which constrain dictionary atoms to the data’s convex hull, greatly improving stability. Our relaxed version, RA-SAE, matches top reconstruction performance and consistently learns more structured, meaningful representations.

February 17, 2025

Interpretability

An Invariant Learning Characterization of Controlled Text Generation

Controlled generation refers to the problem of creating text that contains stylistic or semantic attributes of interest. Many approaches reduce this problem to training a predictor of the desired attribute. For example, researchers hoping to deploy a large language model to produce non-toxic content may use a toxicity classifier to filter generated text. In practice, the generated text to classify, which is determined by user prompts, may come from a wide range of distributions. In this paper, we show that the performance of controlled generation may be poor if the distributions of text in response to user prompts differ from the distribution the predictor was trained on. To address this problem, we cast controlled generation under distribution shift as an invariant learning problem: the most effective predictor should be invariant across multiple text environments. We then discuss a natural solution that arises from this characterization and propose heuristics for selecting natural environments. We study this characterization and the proposed method empirically using both synthetic and real data. Experiments demonstrate both the challenge of distribution shift in controlled generation and the potential of invariance methods in this setting.

May 30, 2023

Alignment

Among us: A sandbox for measuring and detecting agentic deception

We introduce Among Us, a sandbox social deception game where LLM-agents exhibit long-term, open-ended deception as a consequence of the game objectives. While most benchmarks saturate quickly, Among Us can be expected to last much longer, because it is a multi-player game far from equilibrium.

April 4, 2025

Alignment

Adversarial Policies Beat Superhuman Go AIs

We describe an attack on the state-of-the-art Go-playing AI system, KataGo. The adversaries do not win by learning to play Go better than KataGo but instead by tricking KataGo into making serious blunders, demonstrating that even superhuman AI systems may harbor surprising failure modes.

January 8, 2023

Robustness

Adversarial Circuit Evaluation

Evaluating three neural network circuits (IOI, greater-than, and docstring) under adversarial conditions reveals that the IOI and docstring circuits fail to match the full model's behavior even on benign inputs.

July 20, 2024

Interpretability

Accidental Misalignment: Fine-Tuning Language Models Induces Unexpected Vulnerability

As large language models gain popularity, their vulnerability to adversarial attacks remains a primary concern. While fine-tuning models on domain-specific datasets is often employed to improve model performance, it can introduce vulnerabilities within the underlying model. In this work, we investigate Accidental Misalignment, unexpected vulnerabilities arising from characteristics of fine-tuning data. We begin by identifying potential correlation factors such as linguistic features, semantic similarity, and toxicity within our experimental datasets. We then evaluate the adversarial performance of these fine-tuned models and assess how dataset factors correlate with attack success rates. Lastly, we explore potential causal links, offering new insights into adversarial defense strategies and highlighting the crucial role of dataset design in preserving model alignment.

May 21, 2025

Robustness

AI Companies Should Report Pre- and Post-Mitigation Safety Evaluations

March 16, 2025

Model Evaluations