Applied Scientist, Artificial General Intelligence, Foundational AI

Key job responsibilities

As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.

• Design auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks
• Perform expert-level manual audits and conduct meta-audits to evaluate auditor performance
• Provide targeted coaching to uplift overall quality capabilities
• Develop and maintain LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment
• Set up the configuration of data collection workflows and communicate quality feedback to stakeholders

A day in the life

An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.