Machine Learning Engineer, AI
Chan Zuckerberg Biohub
Summary
As an ML Engineer, you'll join some of the strongest infrastructure engineers in AI, building the systems that connect everything together. The infrastructure problems you solve directly determine what science becomes possible.
Job description
The Opportunity
As an ML Engineer, you'll join some of the strongest infrastructure engineers in AI, building the systems that connect everything together. The infrastructure problems you solve directly determine what science becomes possible.
What You'll Do
- Work with researchers to train, fine-tune, and evaluate biological foundation models and diagnose training problems.
- Help evaluate and curate training and validation datasets at Petabytes scale.
- Design and create libraries and abstractions that researchers and engineers build on.
- Optimize GPU-native data loading for large tensor workloads and resolve hardware utilization bottlenecks across backends.
- Build and maintain reliable distributed pre and post training systems.
- Support production deployment, with tracking of model and dataset artifacts throughout the ML lifecycle.
- Improve model and cluster monitoring, unit and integration testing, and continuous integration.
- Work with partner researchers and engineers to deploy models and supporting infrastructure in external environments.
What You'll Bring
Significant experience in at least one of the following areas:
- 5+ years building ML systems or distributed infrastructure at scale.
- Strong Python, PyTorch, and CUDA skills, including custom training loops, with a solid understanding of model architectures, optimization, and evaluation.
- Experience with large scale distributed systems, multi-GPU training, checkpointing, and failure recovery.
- Ability to diagnose performance problems across GPUs, memory, networking, storage, and data loading.
- Experience building large-scale data pipelines and shared research tools; familiarity with scientific formats such as Zarr or HDF5.
- Experience deploying and monitoring models, tracking model and dataset versions, and maintaining automated tests and continuous integration.
- Familiarity with containers and orchestration tools such as Slurm, Docker, and Kubernetes.
Research Areas
- Frontier molecular modeling, from protein language models (e.g., ESM) to structure prediction (e.g., ESMFold) and beyond.
- Scaled biological foundation models trained on some of the largest GPU clusters dedicated to science
- Imaging foundation models trained across the world's largest microscopy datasets
- Reasoning and agentic systems that connect frontier LLMs with biological foundation models
- Mechanistic interpretability of biological foundation models: extracting new biological knowledge directly from model weights
- Scientific data at unprecedented scale: AI systems to collect, curate, and learn from some of the richest biological datasets ever assembled
Requirements
- 5+ years building ML systems or distributed infrastructure at scale
- Strong Python, PyTorch, and CUDA skills, including custom training loops
- Solid understanding of model architectures, optimization, and evaluation
- Experience with large scale distributed systems, multi-GPU training, checkpointing, and failure recovery
- Ability to diagnose performance problems across GPUs, memory, networking, storage, and data loading
- Experience building large-scale data pipelines and shared research tools
- Familiarity with scientific formats such as Zarr or HDF5
- Experience deploying and monitoring models, tracking model and dataset versions
- Maintaining automated tests and continuous integration
- Familiarity with containers and orchestration tools such as Slurm, Docker, and Kubernetes
Benefits
- 401(k) Matching
- Paid Volunteer Time Off
- Family-Forming Benefits
- Relocation Support
Company overview
The Chan Zuckerberg Biohub is a pioneering nonprofit medical research organization established as part of the Chan Zuckerberg Initiative's commitment to advancing biomedical science. Founded in 2016 with a 10-year commitment, this innovative research institute brings together leading academic institutions including UC Berkeley, Stanford University, and UC San Francisco to tackle grand scientific challenges on 10- to 15-year time horizons. The organization focuses on understanding the underlying mechanisms of disease and developing transformative technologies that lead to actionable diagnostics and effective therapies. With a culture emphasizing intellectual freedom and interdisciplinary collaboration, the Biohub operates through a unique model that combines staff scientists, engineers, and physicians in shared laboratory spaces rather than traditional academic funding structures. The institute has expanded into a network with additional hubs in Chicago and New York, each focusing on specific biomedical challenges such as tissue-scale biology, immune cell engineering, and advanced biological imaging. The Biohub's work spans from infectious disease surveillance and outbreak response to developing novel imaging technologies that reveal the dynamic architecture of living systems, representing a groundbreaking approach to collaborative scientific research aimed at curing, preventing, or managing all diseases by the end of the century.