Senior Data Engineer

📋 Description

• Create, construct, and sustain ETL pipelines that facilitate data movement between various sources and destinations, such as application databases, cloud data warehouses, third-party APIs, and object storage.

• Collaborate with product managers, research scientists, QA, and platform engineers to convert business and ML requirements into data engineering tasks with well-defined acceptance criteria and documentation.

• Analyze and address data integrity challenges identified by stakeholders, including missing data, incorrect mappings, duplicates, and schema inconsistencies.

• Ensure compliance with data security, quality, and regulatory standards across all data pipelines and processes.

• Streamline the complete process of data ingestion, transformation, and processing to enhance efficiency and reliability.

• Develop and manage intricate datasets while maintaining a consistent design and engineering approach across different systems.

• Guide junior engineers, conduct peer reviews using structured checklists, and pinpoint opportunities to enhance team processes and adopt new tools.

⛳️ Requirements

• A Bachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline.

• 4-6 years of experience in data engineering or related fields, emphasizing AI/ML.

• Strong proficiency in Python, including experience with migration frameworks (e.g., Alembic, SQLAlchemy) and production-quality data processing scripts.

• Advanced SQL skills for both transactional databases (PostgreSQL or similar) and columnar data warehouses (AWS Redshift is highly preferred).

• Expertise in cloud services (e.g., dbt, AWS, Azure, GCP).

• Proficiency with AWS services: Redshift, S3, Lambda, IAM.

• Strong skills in Git and GitHub, including branching strategies, pull request workflows, and code review methodologies.

• Expertise in constructing scalable, fault-tolerant data pipelines.

🏝️ Benefits

• Health insurance

• 401(k) matching

• Paid time off

• Flexible work arrangements

• Professional development opportunities