OpenAI logo

Model Policy Manager, Emerging AI Misuse

OpenAI

San Francisco, CAHybridPosted 2 days ago$207,000 - $335,000/year
Full TimeSenior LevelResearch & ScientistAI Biotech
Original posting ↗

Summary

The Model Policy team within OpenAI's Safety Systems organization defines how increasingly capable AI models should behave in real-world settings. We turn research on emerging risks into model behavior policies, evaluations, and safeguards.

Job description

About the Team

The Model Policy team within OpenAI's Safety Systems organization defines how increasingly capable AI models should behave in real-world settings. We turn research on emerging risks into model behavior policies, evaluations, and safeguards.

About the Role

As AI systems become more capable, the same advances that expand their legitimate uses can also lower barriers to harm. Emerging misuse risks range from AI assistance with conventional weapons development and the misuse of AI-enabled robotics to fraud, impersonation, and other forms of abuse that people can carry out at scale using AI agents. You'll build threat models for how people could misuse AI in digital and physical settings, assess what access to AI changes in practice, and use that evidence to define how models should respond.

In this role, you will:

  • Build and update threat models as capabilities evolve.
  • Assess what people could plausibly do, what expertise and resources they would need, and how AI changes those barriers.
  • Define when models should help, limit assistance, or refuse in situations where the same capability has legitimate and harmful uses.
  • Turn expert safety judgment into instructions, training examples, and evaluations that teach models when to help, when to limit assistance, and when to refuse.
  • Turn threat models into evaluation scenarios and grading guidance.
  • Analyze model outputs, red-team findings, and deployment data to identify where models enable misuse or block legitimate work.
  • Partner with research, engineering, security, and product teams to apply findings to model training, deployment decisions, and monitoring.

You might thrive in this role if you:

  • Have hands-on experience building, testing, or assessing dual-use technologies, or a track record of threat modeling grounded in how people use and misuse real systems.
  • Demonstrate deep practical experience in one relevant domain and the judgment to know when to bring in experts from others.
  • Are able to assess whether a capability demonstrated in a test materially changes what someone could do outside it.
  • Can translate technical evidence and uncertainty into clear policy boundaries and measurable evaluation criteria.
  • Have experience analyzing gold sets, evaluation results, and training data to spot inconsistent human labels, assess how annotator disagreement affects measured performance, and decide what to investigate next.
  • Are comfortable working hands-on with model data and evaluation results, including inspecting examples, analyzing failure patterns, assessing data quality, and distinguishing policy failures from grader, model, or system failures.
  • Use empirical evidence to develop and refine safety policies and safeguards.
  • Work well across disciplines and revise judgment as capabilities and evidence change.
  • Enjoy fast-paced, collaborative research environments where priorities shift as models, evidence, and risks change.

Requirements

  • Hands-on experience building, testing, or assessing dual-use technologies
  • Track record of threat modeling grounded in how people use and misuse real systems
  • Deep practical experience in one relevant domain
  • Ability to assess whether a demonstrated capability materially changes real-world capability
  • Ability to translate technical evidence and uncertainty into clear policy boundaries and measurable evaluation criteria
  • Experience analyzing gold sets, evaluation results, and training data
  • Comfort working hands-on with model data and evaluation results
  • Ability to distinguish policy failures from grader, model, or system failures
  • Use empirical evidence to develop and refine safety policies and safeguards
  • Work well across disciplines
  • Collaborative research environment experience

Benefits

  • Medical, Dental & Vision
  • Health Savings Accounts
  • Health FSA
  • Dependent Care FSA
  • Commuter Expenses
  • 401(k) Matching
  • Paid Parental Leave
  • Paid Medical Leave
  • Paid Caregiver Leave
  • Paid Time Off
  • Paid Company Holidays
  • Paid Sick Time
  • Mental Health Support
  • Life Insurance
  • Disability Coverage
  • Learning & Development Stipend
  • Daily Meals
  • Relocation Support
  • Charitable Donation Matching
  • Wellness Stipends
  • Equity

Company overview

OpenAI is an artificial intelligence research laboratory and company based in San Francisco, California. Founded in 2015 by Elon Musk, Sam Altman, Greg Brockman, Ilya Sutskever, Wojciech Zaremba, and John Schulman, OpenAI initially operated as a non-profit research organization before transitioning to a capped-profit model. The company's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI has developed groundbreaking AI technologies including GPT (Generative Pre-trained Transformer) models, DALL-E for image generation, and Codex for programming assistance. Their most notable product, ChatGPT, has revolutionized human-AI interaction and brought large language models to mainstream audiences. OpenAI conducts cutting-edge research in machine learning, natural language processing, computer vision, and reinforcement learning, while also developing practical AI applications that are transforming industries worldwide.

Explore more jobs of your matchFind more biopharma roles with GetBioJob.