Data Engineer – Platform

This is a fully remote position, open to applicants in Massachusetts.

📋 Description

• Design, construct, and maintain scalable data ingestion pipelines that support a diverse array of source systems (APIs, databases, streaming platforms, third-party data providers).

• Create batch and real-time ingestion frameworks that can manage high data volumes while meeting low-latency requirements.

• Establish and improve observability, monitoring, and alerting mechanisms for ingestion pipelines (latency, throughput, failures, data freshness).

• Implement and uphold data quality checks and validation frameworks at the points of ingestion.

• Contribute to the evolution of Data Ingestion as a platform product, including the development of reusable frameworks, standards, and best practices.

• Utilize AI-assisted development tools and automation to expedite pipeline development, enhance code quality, and boost operational efficiency (e.g., code generation, testing, anomaly detection, workflow automation).

• Collaborate with upstream system owners and downstream consumers to establish data contracts, SLAs, and strategies for schema evolution.

• Optimize pipelines for performance, cost, and efficiency across both compute and storage layers.

• Apply Infrastructure as Code (IaC) to provision and manage ingestion infrastructure in a scalable and consistent manner.

• Work alongside platform, analytics, and data science teams to ensure seamless data availability and usability.

⛳️ Requirements

• A minimum of 1 year of experience in data engineering, with a particular emphasis on data ingestion and pipeline development.

• Expertise in Python and SQL, along with practical experience in developing real-time data streaming solutions.

• Familiarity with Snowflake, Databricks, and Kafka within a modern data stack.

• Knowledge of cloud infrastructure and DevOps tools such as Terraform, PagerDuty, and DataDog.

• Experience in designing data warehouses, data lakes, and ETL pipelines using contemporary data modeling techniques.

• Exposure to cloud platforms, ideally Amazon Web Services (AWS).

• Experience with NoSQL databases like MongoDB or DynamoDB.

• Familiarity with data reporting tools (e.g., Tableau) and data logging/monitoring tools is preferred.

• Strong communication and collaboration skills, with the ability to excel in fast-paced, cross-functional teams.

🏝️ Benefits

• Health insurance

• Retirement plans

• Paid time off

• Flexible work arrangements

• Professional development

• Bonuses

• Stock options