Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, create, and manage ETL/ELT pipelines that drive the Resonate business using Spark and Scala on AWS EMR.

• Handle terabyte-scale datasets stored in S3 and Snowflake.

• Optimize multi-terabyte and petabyte-scale Spark applications for both performance and cost efficiency.

• Troubleshoot and resolve production-scale big data challenges.

• Collaborate with senior engineers and product management to transform product requests from design and planning to production.

• Oversee pipeline health through Grafana monitoring.

• Identify and rectify data quality issues at their origin.

• Produce clean, testable code that includes comprehensive unit and integration tests.

• Leverage Gen AI tools in pipeline development, monitoring, and everyday engineering tasks.

• Engage in code reviews, technical design discussions, and sprint planning activities.

• Contribute to architectural decision-making.

• Assist in production operations, which includes participating in on-call rotations and incident response.

⛳️ Requirements

• Approximately five years or more of professional experience in software engineering, data engineering, or a closely related discipline.

• At least three years of practical experience with Spark and Scala, particularly with the DataFrame and Dataset APIs.

• Demonstrated expertise in tuning Spark applications at multi-terabyte or petabyte scales.

• Real-world debugging and problem-solving experience within the big data ecosystem, including addressing production failures.

• Strong relational database proficiency.

• Familiarity with AWS (EMR, S3, Lambda), Kafka, Snowflake, Grafana, Hadoop, Elastic Stack, and Docker.

• A comprehensive understanding of the full software development lifecycle, from exploration and design to production deployment.

• Good instincts regarding solution architecture, data structures, and data modeling.

• Passion for utilizing generative AI to enhance data engineering processes.

• Practical experience in building and maintaining big data pipelines with Gen AI.

• Experience with probabilistic data structures.

• Proven history of implementing high cardinality data systems at scale.

• A bachelor's degree in computer science, computer engineering, or equivalent experience is highly regarded.

🏝️ Benefits

• Annual bonus program linked to company performance.

• Comprehensive medical, dental, and vision coverage, including HSA options.

• Company-funded disability and life insurance with an option for additional life insurance coverage.

• 401(k) plan with company matching contributions.

• Flexible paid time off (PTO) along with 10 paid holidays.

• Paid parental leave.

• Access to wellness and employee assistance programs, which include free and unlimited consultations with over 20 health professionals and wellness coaches for you and your family members.

• A remote-first work environment that offers genuine flexibility.

• Occasional travel for company and client events.