LillyDirect Pharmacy Data Engineer
Eli Lilly and Company
Summary
We are seeking a highly motivated Data Engineer for our LillyDirect Pharmacy Covered Entity (LDPCE) Master Data team. This role involves designing, building, and maintaining robust and scalable data pipelines and infrastructure to support our critical scientific and business initiatives, ultimately contributing to the delivery of life-changing medicines.
Job description
As a Data Engineer for Master Data, you will be responsible for designing, developing, and maintaining the data solutions that ensure the availability, quality, and integrity of data for analysis and/or business transactions for our LillyDirect Pharmacy Covered Entity business. You will design and implement efficient data storage, processing, and retrieval solutions, build data pipelines, and optimize data models, working closely with the Technical Lead, DevOps engineer, architects, and business analysts. You will need strong skills in data integration, acquisition, cleansing, harmonization, and transformation, and you will play a key role in turning raw data into reliable, governed datasets that downstream teams can trust.
What you'll be doing
- Design, build, and maintain scalable and reliable data pipelines for batch and near-real-time processing in Python and PySpark on Databricks and AWS.
- Develop and optimize data models, ETL/ELT workflows, and data integration across multiple systems and platforms, using Delta Lake and lakehouse (medallion) patterns.
- Support data integrations with Reltio and Transcend, including data loads, extracts, and reconciliation.
- Build data quality, lineage, and reconciliation checks so that profile and consent data is accurate, complete, and auditable.
- Implement data governance, security, and quality standards across data assets, including access controls, masking, and retention for sensitive data (PII/PHI) and consent data.
- Collaborate with the Technical Lead, product owners, architects, analysts, and business stakeholders to understand data requirements and deliver solutions.
- Own pipelines and work streams from design through production, and contribute to architectural and design decisions.
- Promote best practices in coding, code review, automated testing, and deployment, and release code through CI/CD pipelines.
- Monitor, troubleshoot, and improve performance, reliability, and cost of data pipelines, and support production incidents and root cause analysis.
- Automate manual processes and identify opportunities to optimize data workflows and reduce costs.
How You Will Succeed
- Deliver scalable solutions by designing robust data pipelines and architectures that meet performance and reliability standards.
- Build in privacy and consent by treating consent and data protection as a design principle in pipelines from the start, and by raising risks early.
- Collaborate effectively with cross-functional teams to turn business needs into technical outcomes.
- Share expertise, mentoring junior engineers and adopting best practices in data engineering and cloud technologies.
- Continuously improve systems through automation, performance tuning, and proactive issue resolution.
- Communicate with clarity to ensure alignment across technical and non-technical stakeholders.
Basic Qualifications and Experience Requirement
- Bachelor's degree in Computer Science, Information Technology, Management Information Systems
- Overall 5+ years of experience in data engineering using core technologies such as SQL, Python, PySpark, and cloud platforms (AWS, Azure, or GCP) services including Lambda, Glue, EMR S3, Redshift, Athena, and IAM roles/policies.
- 2+ years of experience working in Agile environments, with hands-on experience using GitHub and CI/CD pipelines for code deployment.
- Hands-on experience with Databricks, including workspace configuration, notebook development, Delta Lake, and performance optimization.
- 1+ years of experience with orchestration tools like Airflow or Databricks Workflows for workflow automation.
- Qualified applicants must be authorized to work in the United States on a full-time basis. Lilly will not provide support for or sponsor work authorization or visas for this role, including but not limited to F-1 CPT, F-1 OPT, F-1 STEM OPT, J-1, H-1B, TN, O-1, E-3, H-1B1, or L-1.
Additional Skills/Preferences
- Experience building high-performance, scalable data pipelines following Data Lakehouse, Data Warehouse, and Data Mart standards and deep understanding of ETL/ELT workflows.
- Strong expertise in data modelling, managing large datasets, and implementing secure, compliant data governance practices.
- Working knowledge of data governance, security, and data quality practices, including handling sensitive data (PII/PHI) in a regulated environment.
- Ability to collaborate with business stakeholders to translate key business requirements into scalable technical solutions.
- Strong communication and collaboration skills to work with both technical and non-technical stakeholders; a problem-solving mindset and attention to detail.
- Pharmaceutical or healthcare industry experience, including privacy requirements such as HIPAA.
- Hands-on experience with Reltio (or a comparable MDM platform), including match and merge, survivorship, and integration patterns.
- Familiarity with consent and preference management (for example opt-in/opt-out, purpose, and channel); experience with Transcend or a similar consent platform.
- Familiarity with streaming frameworks such as Kafka, Kinesis, or MSK.
- Experience with Unity Catalog, infrastructure-as-code (for example Terraform), and data quality frameworks (for example Great Expectations or dbt tests).
- Partner with and influence vendor resources on solution development to ensure understanding of data and technical direction for solutions as well as delivery.
- AWS Certified Data Engineer – Associate
- Databricks Certified Data Engineer (Associate or Professional)
Requirements
- Bachelor's degree in Computer Science, Information Technology, Management Information Systems
- 5+ years of experience in data engineering using SQL, Python, PySpark, and cloud platforms (AWS, Azure, or GCP)
- 2+ years of experience working in Agile environments with GitHub and CI/CD pipelines
- Hands-on experience with Databricks, including workspace configuration, notebook development, Delta Lake, and performance optimization
- 1+ years of experience with orchestration tools like Airflow or Databricks Workflows
- Must be authorized to work in the United States on a full-time basis
- No visa sponsorship available
Benefits
- 401(k) Matching
- Pension
- Vacation Benefits
- Medical, Dental & Vision
- Prescription Drug Benefits
- Flexible Spending Accounts
- Life Insurance
- Death Benefits
- Employee Assistance Program
- Fitness Benefits
- Employee Clubs & Activities
- Company Bonus
Company overview
Eli Lilly and Company, commonly known as Lilly, is a leading American multinational pharmaceutical corporation headquartered in Indianapolis, Indiana. Founded in 1876 by Colonel Eli Lilly, the company has grown into one of the world's most valuable pharmaceutical enterprises, with operations in approximately 125 countries. Lilly specializes in developing and manufacturing innovative medicines across multiple therapeutic areas, including diabetes, oncology, neuroscience, immunology, and cardiometabolic diseases. The company is renowned for its groundbreaking work in insulin production, having been the first to mass-produce insulin and later developing the first commercially available insulin product in the U.S. Lilly's current portfolio includes blockbuster medications such as Mounjaro and Zepbound (tirzepatide) for diabetes and obesity, Trulicity (dulaglutide) for type 2 diabetes, Verzenio (abemaciclib) for breast cancer, and Taltz (ixekizumab) for autoimmune conditions. With approximately 47,000 employees worldwide, Lilly maintains a strong commitment to research and development, investing heavily in scientific innovation to address some of the world's most significant health challenges. The company operates with a mission to make life better for people around the world through the discovery and delivery of innovative medicines.