Senior Manager, AI Engineer/ Solutions & Engineering

Key Responsibilities

• Build and ship LLM-powered solutions (assistant extensions, connectors, agentic workflows, and custom applications) from prototype to production
• Build evaluations first: define what 'good' means for each use case, create golden datasets, graders, and regression suites, and use them to drive every prompt, tool, and model change
• Build with coding agents: use agents such as Claude Code as a daily development tool, writing specs, directing agents, and reviewing and testing their output, while owning correctness and quality
• Design around the harness: understand how AI assistant platforms run the agent loop (tool calls, context assembly, permissions, sandboxing) and shape solutions to fit
• Engineer context deliberately: system prompts, reusable instructions, memory, retrieval, structured outputs, and long-context strategy
• Choose the right build pattern: platform extension vs. connector vs. custom app vs. API-based agent, and when not to use an agent
• Select the right model for each task by weighing capability, cost, latency, and risk, using your own evals as evidence
• Implement guardrails and human-in-the-loop controls suited to regulated pharma content (MLR review, compliance, auditability, data boundaries)
• Monitor and improve quality, cost, latency, and adoption of deployed solutions using production traces and feedback
• Translate business needs into technical solutions with marketing, commercial, and IT partners
• Enable the business: explain what AI can do, where it breaks, and what it is worth
• Raise the team's practice: share reusable patterns, review peers' work, and track new models and platform capabilities

Required Knowledge and Skills

Evaluation and quality engineering: Designing evals for non-deterministic systems: golden datasets, rubric and LLM-as-judge graders, human review, and regression testing. Evaluating each layer separately: retrieval, tool use, multi-step agent trajectories, and final output. Validating the judges themselves: calibrating LLM graders against human labels and spotting grader bias. Running evals in CI so that changes to prompts, tools, skills, or model versions are measured before release. Turning evaluation results into improvements (error analysis, failure categorization, targeted fixes).

Working with coding agents: Current, hands-on daily use of coding agents such as Claude Code (or Codex, Cursor, or similar) to design, build, test, and refactor real software. Spec-driven workflows: writing clear specs and acceptance tests, breaking work into agent-sized tasks, and running agents in parallel or over long tasks. Configuring agents: project instructions, skills, MCP servers, hooks, and permissions. Reviewing agent output critically. You can explain and defend every line you ship, and you know when an agent is going off track.

Applied LLM engineering: Proven LLM solutions in production, not just prototypes. Strong Python (ideally TypeScript too), API-driven development, Git, CI/CD, and a cloud platform (AWS, Azure, or GCP). Advanced prompt and context engineering: system prompt design, few-shot patterns, structured outputs, reasoning and long-context strategies. Retrieval and grounding: chunking, hybrid search, reranking, metadata filtering, and knowing when connectors or direct file access replace a vector DB. Agentic patterns: tool and function calling, multi-step workflows, MCP, and judgment on when not to use an agent. Working with unstructured and semi-structured data (documents, PDFs, content assets).

Understanding how models and harnesses work: Transformer fundamentals: tokenization, attention, context windows, and sampling. The training stack (pretraining, instruction tuning, RLHF/RLAIF) and how reasoning models differ. How an agent harness works: the loop, tool exposure, context management, and permissions. Failure modes (hallucination, sycophancy, position bias, prompt injection, tool misuse) and the mitigation for each. Reasoning about cost, latency, and quality from first principles.

Engineering practices: LLMOps: tracing, observability, versioning of prompts and models, and cost and usage management. Security basics: prompt injection via tools and documents, data leakage, PII, least-privilege access.

Desirable Experience

• Enterprise AI assistant platforms: hands-on experience building on Claude Desktop (Chat, Cowork, Code; skills, projects, plugins, connectors), ChatGPT Enterprise (custom GPTs, connectors, projects), Gemini Enterprise / Gemini for Workspace, or similar harnesses
• Building or configuring MCP servers and integrating enterprise systems as connectors
• Agent SDK or API-based agent development for cases that outgrow the assistant platform
• Pharma, life sciences, or other regulated industries (MLR, GxP, 21 CFR Part 11 awareness)
• Commercial pharma data and systems (Veeva, PromoMats, IQVIA), digital marketing, content supply chain, or HCP/DTC engagement
• Enterprise AI rollout: adoption programs, workshops, champion networks, usage governance
• Multi-agent or sub-agent orchestration (LangGraph or similar)
• Multimodal and document-understanding workloads
• Fine-tuning or distillation (LoRA, PEFT) and open-weight models (Hugging Face, Llama, Mistral)

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field (or equivalent experience)
• 5+ years of software or ML engineering experience, with at least 1 year building LLM or agent-based solutions
• A portfolio of shipped work, such as assistant extensions, connectors, agent workflows, or an eval suite you built
• Strong communication skills, including explaining trade-offs to business partners