Senior Dev Ops Engineer

Job Summary

The Globus Operations team is a 4-5 person group that designs, builds, and operates the cloud infrastructure behind a research-computing platform used by hundreds of institutions worldwide. Globus is a hybrid solution combining AWS-hosted orchestration services with installable applications, delivering identity and access management, data transfer and sharing, and task automation as both software-as-a-service (SaaS) and platform-as-a-service (PaaS). As a senior member of this small, high-autonomy team, you will ensure software development and operational best practices are effectively integrated across our services and AWS infrastructure. The team owns an unusually broad estate — an AWS Organization of roughly thirty production accounts, a golden-image pipeline producing around thirty machine images, a self-hosted monitoring platform, a centralized security-logging pipeline, and in-house compliance and security automation — all managed as code in Python, Bash, Terraform, CloudFormation, and Ansible. You will work both collaboratively and independently on complex issues and projects, in some cases making progress under minimal guidance. You will also embed with select software engineering teams to provide operational and infrastructure guidance on best practices. We are looking for a senior engineer equally comfortable improving a build pipeline and debugging a production incident, and who will be a leader and educator both within and across teams.

Responsibilities

• Architecture and Design: Participate in the definition and documentation of cloud infrastructure architecture — networking, monitoring, logging, security, backup, and deployment — across production and development environments. Design new systems and tools, identify opportunities for technical improvement, and review and test solutions to ensure standards are met.
• Software Development: Develop, test, document, and maintain high-quality software and infrastructure as code for deploying and operating Globus cloud services, in Python, Bash, Terraform, and CloudFormation. Contribute to shared internal libraries and expand automated test coverage of operational tooling.
• Build and Release Automation: Maintain and evolve the machine-image and deployment supply chain — Packer-built AMIs produced through AWS CodeBuild, published via SSM Parameter Store, KMS-encrypted and shared cross-account.
• Monitoring and Observability: Maintain the current monitoring and alerting systems (self-hosted Nagios/NRPE with configuration generated from live AWS inventory, CloudWatch alarms, PagerDuty routing) and help in the migration to Prometheus.
• Security Operations and Compliance: Contribute to Globus's security plan, controls, monitoring, and reporting. Extend in-house continuous-compliance automation, operate intrusion detection and endpoint protection tooling, maintain hardened machine images.
• Identity, Access, and Trust: Administer IAM roles, policies, and cross-account trust across the AWS Organization.
• SRE/Operations: Deploy, operate, and monitor production Globus services for high availability. Participate in an on-call rotation, lead incident response, and author root-cause analyses. Identify recurring operational toil and eliminate it through automation and better defaults.
• Support: Act as a technical consultant and resource for other team members, including the engineering and user-support teams, assisting with operational issues and troubleshooting.
• Designs new systems, features, and tools. Solve complex problems and identify opportunities for technical improvement and performance optimization. Review and test code to ensure appropriate standards are met.
• Utilize technical knowledge of existing and emerging technologies, including public cloud offerings from Amazon Web Services, Microsoft Azure, and Google Cloud.
• Performs other related work as needed.

Minimum Qualifications

Education: Minimum requirements include a college or university degree in related field. Work Experience: Minimum requirements include knowledge and skills developed through 5-7 years of work experience in a related job discipline. Certifications: ---

Preferred Qualifications

Experience: Strong experience operating AWS at scale — IAM, VPC (peering, endpoints, security groups), EC2 and Auto Scaling Groups, ECS (EC2 and Fargate), S3, RDS, Route 53, Lambda, Systems Manager, CloudWatch — including cross-account work in a multi-account AWS Organization. Strong experience building Infrastructure as Code with Terraform (modules, remote state, provider upgrades, reconciling drift against live environments), the ability to maintain and incrementally retire legacy CloudFormation is a plus. Experience automating infrastructure in Python — maintainable, reusable libraries rather than only scripts, using boto3 — and in Bash. Building Amazon machine images and the instances they run on, plus Docker containers, and CI/CD tooling (AWS CodeBuild, GitHub Actions). Linux administration and troubleshooting across Ubuntu and RPM-based distributions, including systemd, syslog/rsyslog, storage, and SSH/sudo configuration. Networking, firewalls, routing, and DNS; web servers and TLS. Experience with check-based monitoring, particularly self-hosted Nagios/NRPE, to operate, extend, and update. Incident management and on-call practice using a platform such as PagerDuty, and writing root-cause analyses.

Preferred Competencies

Excellent verbal and written communication skills, including clear technical documentation and post-incident analyses. Strong analytical and problem solving skills, including systematic troubleshooting of distributed systems. Excellent organizational skills, constant attention to detail, and the ability to prioritize and manage workload to meet critical project milestones and deadlines. Work both independently and as a team member, taking end-to-end ownership of services with minimal supervision and switching comfortably between long-horizon engineering projects and time-sensitive operational work. Receptive to feedback; willing to learn and embrace continuous improvement. Experience with software development fundamentals — code review, automated testing, static analysis — and experience integrating development and deployment frameworks with the monitoring, operations, and orchestration needed to run applications securely and at scale on public cloud platforms.

Working Conditions

Occasional evening or weekend hours. Participates in an on-call support rotation. Option available for remote work with occasional required attendance at in-person meetings.