Data Processing Specialist - Thoracic Surgery

Job Summary

Responsible for acquiring data from primary or secondary data sources and maintaining databases/data systems. Also responsible for identifying, analyzing, and interpreting trends or patterns in complex data sets and cleaning data to correct problems.

Essential Functions (Key Roles & Responsibilities)

Cloud storage and large transfers (core for the migration)

• Hands-on AWS S3: storage classes (Standard, Glacier, Deep Archive), restore requests and their timing, lifecycle rules, IAM credentials, AWS CLI or boto3.
• Cost planning before moving data: storage, retrieval and egress fees
• Bulk transfer tools: Globus, rclone, aws s3 sync, and FTPS/Aspera pulls from sequencing vendors; resumable, parallel transfers; bandwidth planning
• Verification at scale: md5/sha256 manifests, file-count and byte reconciliation between source and destination, and a written sign-off before anything is deleted at the source.

Cluster and storage administration

• Slurm batch jobs and job arrays; shared filesystems (Lustre, GPFS, NFS), quotas, Unix groups and permissions.
• Institutional research computing (MGB ERIS or similar): requesting storage, organising it into fast, bulk and archive tiers.
• Archive hygiene: packing millions of small files into tar archives before archiving, and determining storage requirements

Sequencing and lab data formats beyond FASTQ/BAM

• CRAM and its dependence on the exact reference genome; VCF; long-read data (Oxford Nanopore POD5/FAST5, PacBio HiFi BAM).
• Single-cell and spatial outputs (10x Genomics, Xenium bundles) and whole-slide images (SVS, NDPI, TIFF).
• Vendor deliverables: sample sheets, QC reports, demultiplexing, and run, lane and library IDs.

Sample database, biobank and compliance

• REDCap (widely used at MGB) and/or a biobank LIMS such as OpenSpecimen or FreezerPro; barcoding, freezer inventory, chain of custody.
• Data modelling of the chain patient → specimen/block → section/aliquot → library → sequencing run → file, with a data dictionary and controlled vocabularies.
• Handling patient data: HIPAA, IRB protocols, coded (de-identified) sample IDs, access control, data use agreements and MGB data-security policy.
• Public data submission: has deposited data in SRA, dbGaP, GEO or EGA for publications, or is willing to learn.

Working practices

• Git and re-runnable, logged scripts; clear READMEs and runbooks someone else can follow.
• Running a multi-month migration: inventory first, agreed priorities, a tracking sheet, regular status updates.
• Supporting wet-lab staff; working with sequencing vendors and cores.
• Performs other duties as assigned by PI
• Complies with all policies and standards as assigned by Research Manager