
Site Reliability Engineer III
Jefferson Lab
Summary
As Lead Site Reliability Engineer on the High Performance Data Facility (HPDF) team, you will play a critical role in establishing and running the reliability practice for the facility's first systems on its path to operations. This is a technical role with manager responsibilities: you will supervise and develop a small team of site reliability engineers, and you will also design and build systems yourself, hands on in the code, the monitoring stack, and the incident response. You will design how the facility stays available and recovers, define and report on the service level objectives that measure how well it serves its users, serve as incident commander for significant incidents, and work day to day with staff at both Jefferson Lab and Berkeley Lab. HPDF is still in design, so there is room in this role to grow into influencing the technology choices the facility is built on. The users you support are research physicists and computational scientists, and helping them succeed is a core measure of this role.
Job description
What your job will be like
As Lead Site Reliability Engineer on the High Performance Data Facility (HPDF) team, you will play a critical role in establishing and running the reliability practice for the facility's first systems on its path to operations. This is a technical role with manager responsibilities: you will supervise and develop a small team of site reliability engineers, and you will also design and build systems yourself, hands on in the code, the monitoring stack, and the incident response. You will design how the facility stays available and recovers, define and report on the service level objectives that measure how well it serves its users, serve as incident commander for significant incidents, and work day to day with staff at both Jefferson Lab and Berkeley Lab. HPDF is still in design, so there is room in this role to grow into influencing the technology choices the facility is built on. The users you support are research physicists and computational scientists, and helping them succeed is a core measure of this role.
In this job you will
- Lead the design, implementation, and operation of monitoring, logging, alerting, and diagnostic tooling for HPDF compute, storage, network, and facility systems, contributing directly to that work as well as directing it.
- Supervise, mentor, and develop a small team of site reliability engineers: assign and review work, set expectations, give regular feedback, support technical growth, and plan and estimate the multi-person efforts assigned to the team.
- Establish and maintain the facility's operational framework, including on-call and escalation structure, incident management, change management, and scheduled maintenance, and keep operational records, runbooks, and documentation current.
- Design the facility's resilience model, including failure domain isolation, redundancy, graceful degradation, and disaster recovery objectives for a geographically distributed facility, and validate that design through testing.
- Define, implement, and report on Service Level Objectives (SLOs) and Service Level Indicators (SLIs) in collaboration with the architecture team and scientific stakeholders, and hold facility operations to them.
- Serve as incident commander for significant incidents, own the postmortem process, and drive root cause prevention back into the design and operation of the systems.
- Drive reliability improvement through automation, process optimization, and the elimination of manual operations, using Python, Go, or shell and standard software development practices.
- Partner with the architecture team on HPDF technology selection from a reliability standpoint, lead evaluations of vendor and open source technologies, and represent HPDF site reliability engineering in the Berkeley Lab partnership.
- Participate in an on-call rotation as the facility moves toward operations.
Lead - Supervisory - Management
- Supervises a team of site reliability engineers
- Assigns and reviews work, sets performance expectations, provides regular feedback, conducts performance discussions, and supports the technical development of the team.
- Participates in hiring for the group.
- Does not hold fiscal or budget authority.
Knowledge, Skills, and Abilities
- Deep Linux systems expertise, with the ability to troubleshoot across the application, operating system, storage, and network layers and to guide others in doing so.
- Expertise in designing and operating monitoring and observability stacks (for example Prometheus, Grafana, ELK, OpenTelemetry) and in defining and reporting on SLOs and SLIs.
- Strong scripting and automation skills (Python, Go, or shell) with standard software development practices, together with the judgment to decide what is worth automating.
- Demonstrated ability to lead and mentor technical staff: setting expectations, assigning work, giving constructive feedback, and addressing performance issues in a constructive manner.
- Ability to establish and run incident response and operational process in a production environment.
- Ability to design for resilience, including failure domain isolation, redundancy, graceful degradation, and recovery objectives, and to validate the design through testing and analysis.
- Clear written and verbal communication, including the ability to present options, argue persuasively for proposals, and work productively with a community of scientific users and with colleagues across both laboratories.
- Familiarity with public cloud environments (AWS, Azure, GCP).
- Networking at scale: IPv4/IPv6, DNS, firewalls and access control lists, high speed interconnects, and data transfer protocols.
- Ability to review system and vendor designs from a reliability standpoint and to argue a technical position persuasively with architects, vendors, and scientific stakeholders.
- Load testing, performance analysis, and capacity modeling to validate design assumptions and identify bottlenecks in the data path.
- Ability to estimate cost and effort for multi-person projects and to plan staffing accordingly.
- Practical experience developing and deploying AI assisted or autonomous automation for operational work, and routine use of AI tools in day to day engineering.
Requirements
- 10 or more years experience in Site Reliability Engineering, DevOps, systems engineering, or operations engineering
- At least two years leading or supervising engineers
- Bachelor's Degree in Computer Science or Related Field
- Deep Linux systems expertise
- Expertise in designing and operating monitoring and observability stacks (Prometheus, Grafana, ELK, OpenTelemetry)
- Defining and reporting on SLOs and SLIs
- Strong scripting and automation skills (Python, Go, or shell)
- Demonstrated ability to lead and mentor technical staff
- Ability to establish and run incident response and operational process
- Ability to design for resilience and validate through testing
- Clear written and verbal communication
- Familiarity with public cloud environments (AWS, Azure, GCP)
- Networking at scale: IPv4/IPv6, DNS, firewalls, ACLs, high speed interconnects, data transfer protocols
- Load testing, performance analysis, and capacity modeling
- Ability to estimate cost and effort for multi-person projects
- Practical experience developing and deploying AI assisted or autonomous automation
Benefits
- Medical, Dental & Vision
- Flexible Spending Accounts
- Paid Time Off
- Paid Parental Leave
- 401(k) 9% Contribution
- Flexible Work Arrangements
- Tuition Assistance
- Professional Development
Company overview
The Thomas Jefferson National Accelerator Facility, commonly known as Jefferson Lab, is a premier U.S. Department of Energy national laboratory and Federally Funded Research and Development Center (FFRDC) located in Newport News, Virginia. Established in 1984, Jefferson Lab specializes in nuclear physics research, accelerator science and technology, and operates as a major scientific user facility. The laboratory's primary mission is to deliver breakthrough science and technology in nuclear physics through its flagship facility, the Continuous Electron Beam Accelerator Facility (CEBAF), which provides high-quality beams of polarized electrons for fundamental research into the quark and gluon structure of protons and neutrons. With approximately 850 employees and an annual budget of around $238 million, Jefferson Lab serves as a critical resource for the international scientific community, hosting approximately 1,650 researchers annually. The laboratory is expanding its mission to include high-performance computing, data science, and artificial intelligence through the High Performance Data Facility (HPDF) project, positioning itself as a multi-purpose national laboratory. Jefferson Lab plays a leading role in major international collaborations, including the Electron-Ion Collider project, and maintains core capabilities in superconducting radiofrequency technology, experimental detector design, and computational science.