Research Lead - Pre-training Safety

About Us

FAR.AI is a non-profit AI research institute working to ensure advanced AI is safe and beneficial for everyone. Our mission is to facilitate breakthrough AI safety research, advance global understanding of AI risks and solutions, and foster a coordinated global response.

Since our founding in July 2022, we've grown to 50+ staff ↗ (external site), published 40+ academic papers ↗ (external site), and convened leading AI safety events ↗ (external site). Our work is recognized globally, with publications at premier venues such as NeurIPS, ICML including a Best Paper Honorable Mention in 2026 ↗ (external site), and ICLR, and features in the Financial Times ↗ (external site), Nature News ↗ (external site), Wired Magazine ↗ (external site) and MIT Technology Review ↗ (external site). We conduct pre-deployment testing on behalf of frontier developers such as OpenAI, independent evaluations for governments including the EU AI Office ↗ (external site), and publish the AI Security Leaderboard ↗ (external site) based on our red-teaming expertise. We help steer and grow the AI safety field through developing ↗ (external site) research ↗ (external site) roadmaps ↗ (external site) with renowned researchers such as Yoshua Bengio; running FAR.Labs ↗ (external site), an AI safety-focused co-working space in Berkeley housing 40+ members; and supporting the community through targeted grants ↗ (external site) to technical researchers.

FAR.AI is hiring a Research Lead to develop and lead our work on pre-training safety, shaping models’ capabilities and internal representations at their source, rather than trying to fix them after the fact.

Our initial focus is capability control: removing harmful capabilities while preserving benign ones. We see this as a promising way to prevent misuse of open-weight models in areas such as CBRN and cyber by removing offensive capabilities, and reducing loss-of-control risks by removing knowledge of oversight mechanisms. We will validate approaches like pre-training data filtering at scale, drive adoption of successful methods, and explore techniques such as gradient routing and unlearning..

We are scaling methods like Deep Ignorance ↗ (external site) by over an order of magnitude (>100B parameter models with >1T tokens). You will direct this work, partner with our red team to stress-test the resulting models, and analyze how well the methods scale to frontier systems.

Our research directions include:

• Improved data filtering methods, such as using data attribution (e.g. influence-based selection) or more sophisticated classifiers
• Using methods like gradient routing to isolate dual-use capabilities in components of the model (e.g. specific MoE experts)
• Training to actively remove harmful capabilities, such as interleaving next-token prediction with unlearning, as opposed to simply filtering data
• Adding synthetic data to pre-training or mid-training to shape the representations and behavior of the model

You'll build and lead the team, set its research direction, mentor Members of Technical Staff to scale your vision, and remain hands-on enough to write code and run experiments yourself. This role offers high autonomy in an impact-driven environment, pursuing empirically grounded, scalable ML safety research.

About FAR.Research

We explore promising research directions in AI safety and scale up only those showing a high potential for impact. When an approach proves effective, we develop it into a minimum viable demonstration and work with AI developers and governments to support real-world adoption.

Our recent and ongoing research includes:

Adversarial Robustness: working to rigorously solve security problems through building a science of security and robustness for AI, from demonstrating superhuman systems can be vulnerable ↗ (external site), to scaling laws for robustness ↗ (external site) and jailbreaking constitutional classifiers ↗ (external site).

Mechanistic Interpretability: finding ↗ (external site) issues ↗ (external site) with ↗ (external site) Sparse Autoencoders, probing deception using AmongUs ↗ (external site), understanding learned planning ↗ (external site) in SokoBan, and interpretable data attribution.

Red-teaming: conducting pre- and post-release adversarial evaluations of frontier models (e.g. Claude 4 Opus ↗ (external site), ChatGPT Agent ↗ (external site), GPT-5 ↗ (external site)); developing novel attacks ↗ (external site) to support this work.

Evals: developing evaluations for new threat models, e.g. persuasion ↗ (external site) and tampering risks ↗ (external site), and launching a new research agenda on eval awareness

Mitigating AI deception: studying when lie detectors induce honesty or evasion ↗ (external site), and developing approaches ↗ (external site) to deception and sandbagging.

Applied Interpretability: using interpretability to tackle concrete safety problems (better probes, backdoor detection, deception monitoring), aiming for fast feedback loops, often in collaboration with our other pods.

About the Role

Research Leads define and own a research workstream end-to-end. Day-to-day, that means:

• Articulate a research agenda with a clear theory of change for mitigating catastrophic risks from human-level or superhuman AI systems, and/or vastly increasing the upside of such systems.
• Grow and lead a team of technical staff in pursuit of this agenda, either directly or in partnership with an engineering co-lead.
• Lead novel research projects where there may be unclear markers of progress or success.
• Share your research findings through written content (e.g. academic publications, blog posts) and presentations (e.g. ML conferences, policymaker briefings) to drive adoption and change.
• Mentor and coach junior team members in research skills and ML engineering.
• Contribute to the FAR.AI intellectual environment and research culture, for example by giving feedback on early-stage proposals.
• Build a research field around your agenda through FAR.AI's grantmaking and events, and connect it to real-world deployments through our independent testing and government advising.

This role would be a great fit if you:

• Want to work on the most impactful research directions, alongside mission-driven colleagues who'll push them forward with you.
• Wish to pursue empirically grounded, scalable research directions that lean, technically strong teams can drive forward.
• Value the ability to speak freely. We don't censor our researchers. We just ask that you protect confidential information and make clear when you're speaking personally or on behalf of the organization.
• Want to advise and collaborate with governments, leading AI companies, and academics. We're a small organization that punches above its weight by working closely with these partners: through red-teaming, technical standards work, and research collaborations.

This role would be a poor fit if you:

• Prefer solo IC research to leading a team toward a shared agenda. Some people can do great research that way, but in this role we're looking for someone whose research direction is strong enough that other excellent researchers want to build it with them.
• Prioritize novelty and intellectual elegance over impact. We care about both — a mathematically elegant solution to AI safety would be wonderful — but when we have to choose, we choose what makes AI safer in practice.
• Can only work with the largest compute clusters available at industry labs or need to be compensated with equity in a rapidly growing startup. We offer competitive salaries and sizable compute budgets on a cluster that we manage, but if you value these things over having a positive impact on the future, then you may be more suited to a for-profit lab.

About You

To be a strong candidate for the Research Lead - Pre-Training Safety role, you likely:

• Have a strong existing research track record in AI or another highly technical subject (e.g. CS, math, physics).
• Deep experience with language-model pretraining, dataset construction, or controlled training experiments.
• Experience building large-scale pipelines for scoring, filtering, deduplicating, and sampling training corpora.
• Strong experimental judgment, including safety–capability evaluations, distribution-shift analysis, and statistically rigorous model comparisons.
• Ability to build and debug research systems directly, from classifier fine-tuning through distributed training and evaluation.
• Have either (a) a clear research agenda you'd pursue at FAR.AI, with a theory of change explaining why it's valuable, or (b) a strong track record and a research space you'd sharpen into an agenda over your first months. We assess both paths against the same bar — depth of articulation at application is itself a signal about expected runway.
• Have led a team, mentored graduate students, or supported early-career researchers through fellowship programs. Informal leadership in flatter organizations counts, as we’re more interested in experience than job titles.
• Can effectively communicate novel methods and solutions to both technical and non-technical audiences.

• Are not a new entrant to machine learning research. We don't require a PhD or specific years of experience, but you should have engaged substantively with the field — through prior research, employment, or sustained independent contribution.

It is preferable if you:

• Have an established publication record in AI safety.
• Are comfortable writing grant proposals and navigating collaborations with other organizations or external research groups.

If you are missing key leadership experience or are earlier in your career, we encourage you to consider the open Research Scientist ↗ (external site) pathway and invite you to contribute to one of our existing agendas.

We're also open to more senior versions of this role; simply apply or reach out to talent@far.ai.

Logistics

If based in the USA or Singapore, you will be an employee of FAR.AI (501(c)(3) research non-profit / non-profit CLG). Outside the USA or Singapore, you will be employed via an EOR organisation on behalf of FAR.AI or as a contractor.

• Location: Both remote and in-person (Berkeley, CA or Singapore) are possible. We sponsor visas for in-person employees, and can hire remotely in most countries.
• Hours: Full-time (40 hours/week).
• Compensation: $290,000–$450,000/year depending on experience and location. For exceptional candidates, we will accommodate a higher salary range. We will also pay for work-related travel and equipment expenses. We offer catered lunch and dinner at our offices in Berkeley.
• Application materials: Expect ~1–2 hours of preparation. We ask for a CV, a short research direction statement (the form supports both fully-formed agendas and developing ones), 2–3 selected works with a brief note on your personal contribution, and a short note on why FAR.AI is a good home for your direction.
• Process: From application: a 45-minute bilateral fit call, 2 technical assessments with FAR.AI technical staff, a 3-5-day paid work trial (depending on your availability), and structured reference calls. Typical elapsed time: 4 weeks. Total candidate time end-to-end is ~30-45 hours, with the paid work trial being the bulk. If a 3-5-day block isn't feasible for you, reach out — we can discuss alternatives.

If you have any questions about the role, feel free to contact us at talent@far.ai. Otherwise, if you don't have questions, the best way to ensure a proper review of your skills and qualifications is by applying directly via the application form. Please don't email us to share your resume (it won't have any impact on our decision). Thank you!