Johnson & Johnson Innovative Medicine logo

Principal AI Engineer, GenAI Platform Operations

Johnson & Johnson Innovative Medicine

Raritan, NJHybridPosted 2 days ago$105,000 - $181,700
Full TimeSenior LevelTech & Data (Health IT)Pharmaceutical
Original posting ↗

Summary

The Principal AI Engineer, GenAI Platform Operations is the senior technical authority for how agents run on the enterprise GenAI platform: setting the operations strategy and building the capabilities that observe and control agents in production. The platform includes the MCP Gateway, the agentic runtime, conversational AI and content-generation services, and Enterprise Knowledge Graph and GraphRAG retrieval services, deployed across Microsoft Azure and AWS.

Job description

Key Responsibilities

  • Define and own the GenAI platform operations strategy and roadmap, including the operating model for running agents in production across Azure and AWS, and publish the agent operations review.
  • Build the agent observability platform with end-to-end traces across multi-step agent runs, gateway-mediated MCP tool calls, model calls and retrieval, built on OpenTelemetry and its generative AI semantic conventions.
  • Monitor agent performance in production: task completion, answer quality and groundedness, tool-call and MCP success rates, latency, token use and cost per task, guardrail interventions, and drift or regression, with SLOs and error budgets per agent and risk tier.
  • Map agent performance across the estate: an authoritative agent registry linking each agent to owner, versions, tools, MCP servers, models, data sources and risk tier; dependency maps; and performance scorecards.
  • Agentify reliability checks: design and deploy operations agents that run health checks, synthetic probes, anomaly triage, evidence gathering and first-line remediation, with defined autonomy limits and audit trails.
  • Build the runtime control plane: kill switches at agent, tool, MCP server, model and platform level; circuit breakers, rate limits, budget caps, throttling and quarantine; and policy enforcement at the gateway, validated under GxP-compliant change control.
  • Own agent lifecycle management at runtime: registration and admission, runtime readiness gates, promotion between environments, versioning, canary and shadow releases, automated rollback, suspension, deprecation and retirement.
  • Lead the GenAI platform support team for reliability: L1 and L2 intake, triage runbooks, routing and escalation to L3 engineering, response and resolution commitments, knowledge articles and known-error records.
  • Lead the incident management team: major incident command, severity classification, bridge coordination, stakeholder communication and restoration, and chair blameless reviews and problem management.
  • Lead, hire, coach and develop the support, incident management and reliability engineers in an inclusive, Credo-grounded environment, and manage cloud, observability and agent operations vendors and the managed service provider for L1 support.

Qualifications

  • Minimum of 10 years of progressive technology experience, including at least 5 years in enterprise-scale services, and 3 years of formal people leadership managing engineering, operations or support teams.
  • Minimum of a Bachelor's degree in Computer Science, Engineering, Information Systems or a related technical discipline; Master's degree preferred.
  • Demonstrated experience defining and technically delivering an operations, reliability or observability strategy and roadmap for a platform serving multiple internal consumer teams.
  • Track record of designing and building observability platforms — tracing, metrics, logs, alerting and service health reporting — and using that data to drive engineering prioritization.
  • Practical experience operating AI, machine learning or LLM-based services in production, including monitoring their quality and performance, not only their availability.
  • Experience building runtime controls for production systems, such as feature flags, circuit breakers, rate limiting, traffic management, emergency stop or automated rollback.
  • Hands-on operational experience with workloads running on Microsoft Azure and/or AWS, including containerized services and managed data and networking services.
  • Experience operating technology in a regulated environment, with working knowledge of GxP change control, computer system validation, and the evidence expected in internal audits or health authority inspections.
  • Experience automating operational work at scale, including health checks, triage and auto-remediation, and reducing toil through engineering.
  • Demonstrated accountability for a 24x7 production service, including running ITIL-aligned incident and problem management and an L1/L2 support function.
  • Experience building and leading support, incident management or reliability teams, including hiring, coaching, performance management and creating an inclusive environment.
  • Ability to explain agent behavior, risk and control decisions clearly to senior Technology, business, Quality and governance stakeholders.
  • Willingness to participate in an escalation rotation supporting a 24x7 platform, including runtime control decisions, and to coordinate across global time zones during major incidents.

Required Skills

  • Agent and LLM observability: tracing multi-step agent runs and tool calls with OpenTelemetry, using tooling such as Azure Monitor and Application Insights, Amazon CloudWatch, Grafana, Datadog or Splunk
  • Production evaluation of LLM and agent quality, including online evaluation, LLM-as-judge scoring, groundedness and task-success metrics, and drift and regression detection
  • Site reliability engineering practice: SLI and SLO definition, error budget management, capacity planning, failure testing and toil reduction
  • Incident command and major incident management, including severity models, escalation paths and executive communications
  • ITIL-aligned incident, problem and change management and L1/L2 support operations, with hands-on use of an enterprise ITSM platform such as ServiceNow
  • Runtime control engineering: kill switches, feature flags, circuit breakers, rate limiting, budget and quota enforcement, traffic shifting and automated rollback
  • Agent lifecycle and release engineering: registries, versioning, canary and shadow releases, promotion gates and retirement
  • Building operations agents or AI-assisted automation with defined autonomy limits, human-in-the-loop approval and audit trails
  • MCP Gateway and API gateway operations, including policy enforcement, authentication and tool-level access control
  • Kubernetes and container operations on AKS and/or EKS, including scaling, upgrades, resource governance and production troubleshooting
  • Operational depth in Azure and AWS platform services, including identity, networking, private connectivity, and quota and reserved capacity management
  • Infrastructure as code and CI/CD for controlled, auditable releases using Terraform, GitHub Actions or Azure DevOps Pipelines
  • Production-grade Python or Go for building operational tooling, automation and control services
  • GxP change control and validation documentation practice, including traceable approvals and audit evidence for production controls and changes
  • People leadership fundamentals: hiring, onboarding, coaching, performance management, equitable workload and on-call design, and building an inclusive team environment
  • Contractor, vendor and managed service provider management

Preferred Skills

  • Experience operating agentic or tool-calling workloads in production, including debugging multi-step agent executions and gateway-mediated MCP tool calls
  • Familiarity with Arize, LangSmith and Langfuse
  • Familiarity with agent runtimes and frameworks such as Amazon Bedrock AgentCore, Microsoft Foundry Agent Service, LangGraph or Strands Agents
  • Experience designing an agent registry, catalog or inventory, including ownership, risk tiering and dependency mapping
  • Familiarity with AI risk and governance frameworks such as the NIST AI Risk Management Framework, ISO/IEC 42001 or the EU AI Act
  • Chaos engineering and resilience testing, including game days that exercise kill switches and failover
  • FinOps for AI workloads, including token and inference cost monitoring, budget enforcement and cost per task
  • Experience managing a service desk or L1/L2 function delivered in partnership with a managed service provider
  • Healthcare, pharmaceutical or medical device industry experience, with familiarity with 21 CFR Part 11, EU Annex 11 and Computer Software Assurance approaches
  • Relevant certifications such as Azure or AWS professional-level certifications, Certified Kubernetes Administrator, or ITIL 4

Requirements

  • Minimum of 10 years of progressive technology experience
  • At least 5 years in enterprise-scale services
  • 3 years of formal people leadership managing engineering, operations or support teams
  • Minimum of a Bachelor's degree in Computer Science, Engineering, Information Systems or a related technical discipline
  • Demonstrated experience defining and technically delivering an operations, reliability or observability strategy and roadmap for a platform serving multiple internal consumer teams
  • Track record of designing and building observability platforms — tracing, metrics, logs, alerting and service health reporting
  • Practical experience operating AI, machine learning or LLM-based services in production, including monitoring quality and performance
  • Experience building runtime controls for production systems, such as feature flags, circuit breakers, rate limiting, traffic management, emergency stop or automated rollback
  • Hands-on operational experience with workloads running on Microsoft Azure and/or AWS, including containerized services and managed data and networking services
  • Experience operating technology in a regulated environment, with working knowledge of GxP change control, computer system validation, and audit evidence
  • Experience automating operational work at scale, including health checks, triage and auto-remediation
  • Demonstrated accountability for a 24x7 production service, including ITIL-aligned incident and problem management and an L1/L2 support function
  • Experience building and leading support, incident management or reliability teams
  • Ability to explain agent behavior, risk and control decisions clearly to senior stakeholders
  • Willingness to participate in an escalation rotation supporting a 24x7 platform
  • Agent and LLM observability with OpenTelemetry and tooling such as Azure Monitor, Application Insights, CloudWatch, Grafana, Datadog or Splunk
  • Production evaluation of LLM and agent quality, including online evaluation, LLM-as-judge scoring, groundedness and drift detection
  • Site reliability engineering practice: SLI and SLO definition, error budget management, capacity planning, failure testing and toil reduction
  • Incident command and major incident management
  • ITIL-aligned incident, problem and change management and L1/L2 support operations with an enterprise ITSM platform such as ServiceNow
  • Runtime control engineering: kill switches, feature flags, circuit breakers, rate limiting, budget and quota enforcement, traffic shifting and automated rollback
  • Agent lifecycle and release engineering: registries, versioning, canary and shadow releases, promotion gates and retirement
  • Building operations agents or AI-assisted automation with defined autonomy limits, human-in-the-loop approval and audit trails
  • MCP Gateway and API gateway operations, including policy enforcement, authentication and tool-level access control
  • Kubernetes and container operations on AKS and/or EKS
  • Operational depth in Azure and AWS platform services
  • Infrastructure as code and CI/CD using Terraform, GitHub Actions or Azure DevOps Pipelines
  • Production-grade Python or Go
  • GxP change control and validation documentation practice
  • People leadership fundamentals: hiring, onboarding, coaching, performance management, equitable workload and on-call design
  • Contractor, vendor and managed service provider management

Benefits

  • Medical, Dental & Vision
  • Life Insurance
  • Short- & Long-Term Disability
  • Business Accident Insurance
  • Group Legal Insurance
  • Pension Plan
  • 401(k) Plan
  • Vacation 120 hours/year
  • Sick Time 40 hours/year
  • 13 Paid Holidays
  • Personal & Family Time
  • Parental Leave 480 hours
  • Condolence Leave
  • Caregiver Leave 10 days
  • Volunteer Leave 4 days
  • Military Spouse Time-Off

Company overview

Johnson & Johnson Innovative Medicine is the pharmaceutical division of Johnson & Johnson, leading the way in medical innovation with a relentless focus on bringing transformational treatments to patients worldwide. The company applies rigorous science and compassion to address the most complex diseases of our time, developing groundbreaking medicines across key therapeutic areas including oncology, immunology, neuroscience, and cardiopulmonary medicine. With deep expertise and global reach, Johnson & Johnson Innovative Medicine reimagines how devastating diseases are treated and cured, delivering hope and better care for millions of patients and their families. The organization pioneers the path from laboratory research to clinical application, championing patients every step of the way while maintaining an unwavering commitment to scientific excellence and patient-centered innovation.

Explore more jobs of your matchFind more biopharma roles with GetBioJob.