Research
Complex problems demand complex solutions.
FAR.Research delivers technical breakthroughs to improve the safety and security of frontier AI systems.
FAR.AI conducts research to address fundamental artificial intelligence (AI) safety challenges. We rapidly explore a diverse portfolio of technical research agendas, de-risking and scaling up only the most promising solutions. We share our research outputs through peer-reviewed publications, via partnerships with governmental AI safety institutes, and through red-teaming engagements for leading AI companies.
FAR.Research is dedicated to delivering the novel technical breakthroughs needed to mitigate the potential risks posed by frontier AI. As a non-profit research institute, we leverage our unique flexibility to focus on critical research directions that may be too large or resource-intensive for academia and often overlooked by the commercial sector due to their lack of immediate profitability.
Research agendas
AI Security &Red-Teaming
FAR.AI red-teams and stress-tests frontier AI systems to find the vulnerabilities that current defenses miss, before real-world attackers do.
Frontier AI systems are being deployed into high-stakes settings faster than their defenses are being tested. FAR.AI operates one of the world’s leading red-teams, testing frontier models directly and probing for the vulnerabilities that let attackers bypass safety measures, even when those measures stack multiple layers of defense. This work has helped various frontier model developers improve safeguards through pre- and post-deployment testing, and extends to high-leverage government efforts: FAR.AI leads a consortium building CBRN evaluations for the EU AI Office, and collaborating with the UK AI Security Institute.
Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNN
Planning is essential for solving complex tasks, yet the internal mechanisms underlying planning in neural networks remain poorly understood. Building on prior work, we analyze a recurrent neural network (RNN) trained on Sokoban, a challenging puzzle requiring sequential, irreversible decisions. We find that the RNN has a causal plan representation which predicts its future actions about 50 steps in advance. The quality and length of the represented plan increases over the first few steps. We uncover a surprising behavior: the RNN "paces" in cycles to give itself extra computation at the start of a level, and show that this behavior is incentivized by training. Leveraging these insights, we extend the trained RNN to significantly larger, out-of-distribution Sokoban puzzles, demonstrating robust representations beyond the training regime. We open-source our model and code, and believe the neural network's interesting behavior makes it an excellent model organism to deepen our understanding of learned planning.
imitation: Clean Imitation Learning Implementations
We describe a software package called "imitation" which provides PyTorch implementations of several imitation and reward learning algorithms, including three inverse reinforcement learning algorithms, three imitation learning algorithms, and a preference comparison algorithm.
Why does training on insecure code make models broadly misaligned?
Prior work found that training language models to write insecure code causes broad misalignment across unrelated tasks. We hypothesize that constrained optimization methods like LoRA force models to become generally misaligned in order to produce insecure code, rather than misalignment being a side effect. Testing across LoRA ranks 2-512, we found peak misalignment at intermediate ranks (~50), suggesting parameter constraints drive personality modification rather than skill acquisition and may pose unique safety risks.
Deception
AI systems exhibit deceptive behavior, such as cheating on tasks and lying, and FAR.AI develops methods to detect this behavior and train models to behave honestly.
Deceptive behavior in AI systems can undermine evaluations meant to gauge model capabilities and catch harmful behavior, since a system that looks safe during testing may not be safe once deployed. FAR.AI works on detecting deception directly, through white-box methods that read a model's internal representations rather than relying on outputs alone. This raises a hard problem: distinguishing a model that is honestly answering, or honestly unable, from one that is lying or hiding what it can do. The goal is twofold: to make deception detectable, so that a model judged trustworthy during evaluation actually is trustworthy once deployed, and to leverage signals of deceptive behavior to train models to behave more honestly.
Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNN
Planning is essential for solving complex tasks, yet the internal mechanisms underlying planning in neural networks remain poorly understood. Building on prior work, we analyze a recurrent neural network (RNN) trained on Sokoban, a challenging puzzle requiring sequential, irreversible decisions. We find that the RNN has a causal plan representation which predicts its future actions about 50 steps in advance. The quality and length of the represented plan increases over the first few steps. We uncover a surprising behavior: the RNN "paces" in cycles to give itself extra computation at the start of a level, and show that this behavior is incentivized by training. Leveraging these insights, we extend the trained RNN to significantly larger, out-of-distribution Sokoban puzzles, demonstrating robust representations beyond the training regime. We open-source our model and code, and believe the neural network's interesting behavior makes it an excellent model organism to deepen our understanding of learned planning.
imitation: Clean Imitation Learning Implementations
We describe a software package called "imitation" which provides PyTorch implementations of several imitation and reward learning algorithms, including three inverse reinforcement learning algorithms, three imitation learning algorithms, and a preference comparison algorithm.
Why does training on insecure code make models broadly misaligned?
Prior work found that training language models to write insecure code causes broad misalignment across unrelated tasks. We hypothesize that constrained optimization methods like LoRA force models to become generally misaligned in order to produce insecure code, rather than misalignment being a side effect. Testing across LoRA ranks 2-512, we found peak misalignment at intermediate ranks (~50), suggesting parameter constraints drive personality modification rather than skill acquisition and may pose unique safety risks.
Our Impact
We drive change through incubating research, scaling safety solutions, and informing policy.
Incubating
We derisk and develop innovative solutions to trustworthy & secure AI. Through incubating research, we share key insights, research roadmaps, and tools needed for the broader research community to identify and make progress.
Scaling
We scale up the most promising safety solutions via in-house research, external collaborations, and targeted grantmaking. We facilitate rapid adoption of our findings by working with frontier model developers through red-teaming and other exercises.
Informing
Our research provides expert insights informing policy and public discussion. Our work has been cited in congressional testimony and mainstream media. In this way, we contribute to the establishment of technical standards that guide the development of AI.