Data Engineer

This is a fully remote position, open to applicants in North Carolina.

📋 Description

• Design, construct, and sustain scalable, production-ready data pipelines for extensive datasets.

• Develop and uphold ETL/ELT workflows across various source systems, cloud environments, and analytical platforms.

• Oversee, troubleshoot, and enhance pipelines for availability, performance, and data integrity.

• Implement data quality assessments, validation frameworks, and anomaly detection measures.

• Maintain technical documentation for pipelines, data models, and data dictionaries.

• Architect and oversee Microsoft Azure data infrastructure.

• Provision, configure, and manage cloud-based virtual machines and computing environments.

• Validate and monitor cloud storage for governance, security, and compliance with regulations.

• Design and manage data lakehouse and warehousing solutions.

• Implement workflow orchestration, scheduling, and dependency management.

• Apply database administration principles, including performance tuning, indexing, backup and recovery, and access management.

• Contribute to data governance through cataloging, lineage tracking, metadata management, and role-based access control.

• Enhance pipeline efficiency, latency, and throughput.

• Utilize data mining and profiling techniques to evaluate source data and data quality.

• Develop reusable data transformation components, libraries, and templates.

• Assess and adopt tools, frameworks, and cloud services.

• Operate distributed data processing workflows utilizing Apache Spark and Databricks.

• Collaborate with data scientists and biostatisticians on analytical and machine learning tasks.

• Support high-performance computing (HPC) and large-scale batch processing.

• Interact with external data and analytics partners and vendors regarding data delivery, integration, APIs, and specifications.

• Manage and document data access agreements, ingestion schedules, and data refresh frequencies.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or a related technical discipline.

• 5+ years of experience with a bachelor’s degree, or 3+ years with a master’s or PhD, in a data engineering position.

• Proven expertise in building and maintaining production-quality data pipelines.

• Strong ETL experience, including data transformation between models and data standardization.

• High proficiency in Python and SQL.

• Practical experience with Microsoft Azure data services, including Azure Data Factory, Azure Databricks, Azure Data Lake Storage, and Azure Synapse Analytics or their equivalents.

• Experience with ETL pipeline development and workflow orchestration tools such as Apache Airflow, dbt, or Azure Data Factory.

• Familiarity with distributed computing and big data frameworks like Apache Spark or Databricks.

• Understanding of data modeling, data warehousing, and cloud storage formats such as Parquet and Delta Lake.

• Experience with Git, code review processes, and CI/CD practices.

• Excellent written and verbal communication abilities.

• Capacity to collaborate with both technical and non-technical stakeholders.

• Familiarity with provisioning and managing virtual machines and cloud compute resources is advantageous.

• A master’s degree or PhD is preferred.

• Broader experience with AWS and/or GCP is preferred.

• Experience with R programming is preferred.

• Familiarity with HPC and large-scale batch processing is preferred.

• Knowledge of Docker, Kubernetes, or Azure Kubernetes Service is preferred.

• Experience supporting data science and machine learning teams, MLOps, or model-serving infrastructure is preferred.

• Background in healthcare, medical device, pharmaceutical, or life sciences data is preferred.

• Familiarity with ICD, CPT4, LOINC, SNOMED CT, or OMOP CDM is a plus but not mandatory.

• Strong organizational, communication, and documentation skills.

• Ability to make independent decisions and take responsibility for one’s actions.

• Ability to work effectively in a collaborative team environment.

• Exceptional verbal and written communication skills.

🏝️ Benefits

• Medical, prescription drug, dental, and vision insurance.

• Flexible spending accounts.

• Participation in a 401(k) savings plan.

• Paid time off (PTO).

• Short- and long-term disability benefits.

• Parental leave.

• Up to 10% expected travel.