Principal Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the comprehensive technical strategy for the data platform, ensuring alignment between data infrastructure and long-term business objectives.

• Direct the design and creation of scalable, efficient, and dependable data ingestion and transformation pipelines utilizing APIs, web data, internal feeds, and various other sources.

• Provide mentorship to senior engineers within the data platform team, guiding architectural, design, and implementation choices.

• Architect and manage data pipelines and orchestration workflows using AWS MWAA/Airflow, Lambda, ECS, and SQS.

• Implement and uphold infrastructure as code through Terraform.

• Collaborate with data analysts, scientists, and backend engineers to guarantee data consistency, discoverability, and reliability.

• Employ best practices in data modeling, schema design, and ETL/ELT processes for handling high-volume structured and semi-structured data.

• Ensure data quality and lineage via automated testing, monitoring, and alerting mechanisms.

• Advocate for continuous improvement through code reviews, observability initiatives, and team-wide knowledge sharing.

• Work alongside technology leadership to ensure data platform development aligns with business strategy and product objectives.

• Remain updated on trends in data engineering, cloud architecture, AI/ML integration, and automation.

⛳️ Requirements

• Solid foundation in software engineering principles, including SOLID design, modularity, and scalability.

• Expert-level proficiency in Python for developing data pipelines and automation processes.

• Advanced SQL capabilities and experience with optimizing complex queries and data models.

• Demonstrated experience in designing and maintaining cloud-native data pipelines on AWS, including MWAA/Airflow, Lambda, ECS, SQS, Glue, S3, and Redshift.

• Familiarity with data warehousing or lakehouse technologies such as Redshift, Snowflake, or Databricks.

• Experience in implementing and managing Terraform or similar Infrastructure as Code (IaC) frameworks.

• Strong understanding of data ingestion, transformation, and orchestration tools and methodologies, including those relevant to AI/ML pipelines.

• Knowledge of CI/CD pipelines, automated testing, and contemporary DevOps practices.

• 8+ years of experience in data engineering or backend development, with a focus on scalable data solutions.

• Significant experience in a technical leadership role, including mentoring and leading intricate data infrastructure projects from initiation to completion.

• Familiarity with containerization technologies such as Docker and best practices in workflow orchestration.

• Exceptional communication, collaboration, and problem-solving abilities.

🏝️ Benefits

• Competitive health benefits.

• Matched 401k and pension plans.

• Paid Time Off (PTO).

• Generous parental leave policy.

• Gym subsidies.

• Educational reimbursement programs for career development.

• Recognition programs.