Certification Assessment Specialist
OpenAI
Summary
As a Certification Assessment Specialist, you will design, build, test, launch, and maintain the hands-on assessments behind an OpenAI technical role-based certification. Your work will help candidates demonstrate what they can do in realistic scenarios and give reviewers reliable evidence to score their work. This is a hands-on individual contributor role reporting to the Technical Certification Lead.
Job description
About the Role
As a Certification Assessment Specialist, you will design, build, test, launch, and maintain the hands-on assessments behind an OpenAI technical role-based certification. Your work will help candidates demonstrate what they can do in realistic scenarios and give reviewers reliable evidence to score their work. This is a hands-on individual contributor role reporting to the Technical Certification Lead.
In this role, you will:
- Translate assessment blueprints into realistic scenarios, candidate instructions, and hands-on tasks.
- Work with subject matter experts to check that the tasks reflect the job and elicit the appropriate evidence for each rubric.
- Define assessment environment requirements and build or configure the environments candidates will use.
- Establish repeatable provisioning and reset workflows for accounts, tools, data, and permissions so candidates start from consistent conditions.
- Build and maintain pipelines that capture candidate submissions and other scoring evidence, apply scoring rules, and route results for review.
- Make scores traceable to the underlying evidence and protect sensitive data under approved controls.
- Build and test model-assisted scoring with qualified human reviewers using established validation criteria.
- Compare model scores with calibrated expert judgments, investigate disagreements, implement agreed rules for routing cases to human review, and rerun validation tests after material changes to the model, task, or rubric.
- Test assessments end to end, from environment setup to evidence capture and scoring.
- Use dry runs and pilots to check instructions, timing, accessibility, and failure recovery; fix defects and verify the fixes before release.
- Test scoring against multiple valid approaches, partial and incorrect submissions, and failures in the assessment environment.
- Translate assessment needs into requirements for delivery integrations.
- Partner with engineering and delivery providers to test candidate access, submission and results handoffs and troubleshoot issues.
- Prepare assessments for launch with platform and operations partners.
- Document setup, version dependencies, and troubleshooting so delivery teams can run assessments consistently.
- Maintain versioned tasks, environments, scoring configurations, and test fixtures.
- Use candidate feedback and delivery and scoring data to identify issues, update or retire tasks, and check that changes preserve the intended skills and difficulty.
- Investigate technical issues raised during delivery or candidate review.
You might thrive in this role if you:
- Have personally designed and built performance-based, hands-on assessments, labs or simulations for technical roles, such as software developers or solution architects, and taken them through testing, launch and maintenance.
- Have used scripting and automation to build and troubleshoot technical assessment environments.
- Can work with technical subject matter experts to translate realistic solutions into assessments.
- Have experience working at a technology company in a fast-paced environment, setting priorities and delivering work amid changing requirements.
- Can use a blueprint and rubric to design realistic tasks and specify what evidence distinguishes competent performance from task completion.
- Have built or maintained workflows that capture submissions, process evidence, and produce traceable scores or reviewer outputs.
- Have implemented or validated model-assisted scoring or evaluation against expert human judgments.
- Have worked directly with external technical experts or delivery providers and can translate requirements into integrations you can test end to end.
- Have used pilot results, candidate feedback, and scoring or delivery data to improve assessment quality.
- Understand how to build for accessibility and protect candidate evidence and confidential assessment content.
- Communicate technical tradeoffs clearly to subject matter experts, measurement specialists, platform engineers, and operations teams.
It's a plus if you have experience:
- Building hands-on assessments for AI application development, agentic workflows, or customer deployments.
- Working with psychometricians or measurement specialists on rubric calibration, scoring consistency, or validation of performance assessments.
- Launching performance-based certification assessments across both live and remote delivery.
Requirements
- Personally designed and built performance-based, hands-on assessments, labs or simulations for technical roles, and taken them through testing, launch and maintenance
- Used scripting and automation to build and troubleshoot technical assessment environments
- Worked with technical subject matter experts to translate realistic solutions into assessments
- Experience working at a technology company in a fast-paced environment
- Can use a blueprint and rubric to design realistic tasks and specify evidence distinguishing competent performance
- Built or maintained workflows that capture submissions, process evidence, and produce traceable scores
- Implemented or validated model-assisted scoring or evaluation against expert human judgments
- Worked directly with external technical experts or delivery providers and can translate requirements into integrations
- Used pilot results, candidate feedback, and scoring or delivery data to improve assessment quality
- Understand how to build for accessibility and protect candidate evidence and confidential assessment content
- Communicate technical tradeoffs clearly to subject matter experts, measurement specialists, platform engineers, and operations teams
Benefits
- Medical, Dental & Vision
- Health Savings Accounts
- Health FSA
- Dependent Care FSA
- Commuter Expenses
- 401(k) Matching
- Paid Parental Leave
- Paid Medical Leave
- Paid Caregiver Leave
- Flexible PTO
- Paid Company Holidays
- Paid Sick Time
- Mental Health Support
- Life & Disability Coverage
- Learning & Development Stipend
- Daily Meals
- Relocation Support
- Charitable Donation Matching
Company overview
OpenAI is an artificial intelligence research laboratory and company based in San Francisco, California. Founded in 2015 by Elon Musk, Sam Altman, Greg Brockman, Ilya Sutskever, Wojciech Zaremba, and John Schulman, OpenAI initially operated as a non-profit research organization before transitioning to a capped-profit model. The company's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI has developed groundbreaking AI technologies including GPT (Generative Pre-trained Transformer) models, DALL-E for image generation, and Codex for programming assistance. Their most notable product, ChatGPT, has revolutionized human-AI interaction and brought large language models to mainstream audiences. OpenAI conducts cutting-edge research in machine learning, natural language processing, computer vision, and reinforcement learning, while also developing practical AI applications that are transforming industries worldwide.