Senior Data Engineer

This is a fully remote position, open to applicants in California, +4 more states.

📋 Description

• Design, construct, and implement scalable, decoupled, high-performance data pipelines for both batch and streaming ingestion into Data Lakes and Data Warehouses.

• Develop and enhance ETL/ELT pipelines utilizing Python, PySpark, and advanced SQL techniques.

• Collaborate with Data Scientists and ML Engineers to create robust pipelines, automate MLOps workflows, and collect model interaction data for retraining cycles.

• Monitor, diagnose, and optimize pipeline execution, query performance, and utilization of cloud storage resources.

• Establish data quality frameworks and testing controls to ensure data integrity, reliability, and governance.

• Implement best practices for MLOps and DevOps.

• Maintain CI/CD pipelines and utilize Docker for containerization purposes.

• Document technical standards and architectural designs.

• Facilitate Data Products throughout their lifecycle, including data sourcing through batch and stream processing and optimizing data architecture.

⛳️ Requirements

• Must be located in the United States.

• Must possess authorization to work in the United States.

• 5+ years of experience in Data Engineering.

• Proven experience in designing and managing large-scale data systems.

• Expertise in Python.

• Advanced SQL skills, including query optimization and performance tuning.

• Practical experience with AWS (Redshift, Glue, EMR, MWAA, S3) or GCP (BigQuery).

• Experience with contemporary data processing engines, such as PySpark.

• Proficient in UNIX environments, Docker, Git, and CI/CD pipelines.

• Strong understanding of data structures and batch/stream processing.

• Familiarity with MLOps or the ML deployment lifecycle.

• Excellent verbal and written communication skills in English.

• Experience with Infrastructure as Code, such as Terraform, is a plus.

• Experience in building and maintaining scalable REST APIs is a plus.

• Exposure to prompt engineering workflows or generative AI data pipelines is a plus.

🏝️ Benefits

• Remote-first company.

• Competitive compensation.

• Inclusive and challenging work environment.

• Genuine opportunities for career growth.

• Flexible remote/hybrid work arrangements.