Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and maintain scalable data pipelines utilizing Python, Spark, and Airflow to enhance our core data acquisition and entity resolution engines.

• Work collaboratively across departments with AI/ML and Product teams to introduce new features and AI-driven products.

• Actively identify and address bottlenecks in our intricate ETL processes, offering innovative insights to refine and enhance our current codebase.

• Foster a strong engineering culture by conducting thorough code reviews, implementing unit tests, and effectively communicating design decisions.

• Take responsibility for the complete delivery of roadmap tasks during two-week sprints, ensuring that work adheres to high standards of quality, documentation, and performance.

• Engage in roadmap planning and story refinement, ultimately taking ownership of significant epics that bolster our long-term product defensibility.

⛳️ Requirements

• A minimum of 5 years of hands-on data engineering experience in a production environment, with clear accountability for systems you have built and managed end to end.

• Proficient in Python, with substantial experience in a JVM language (Scala preferred) or a strong willingness to learn quickly.

• Practical experience with Snowflake, or equivalent expertise in BigQuery or Redshift with proven ability to transition skills.

• Experience in deploying and managing AI or ML applications in a production setting, including output validation, monitoring, and cost management at scale.

• Familiarity with orchestration tools such as Apache Airflow or similar workflow management systems.

• Proven track record of reliably operating production systems, with comfort in managing failures, monitoring, and recovery processes.

• Preferred: Experience with Spark on Dataproc Serverless or other serverless Spark platforms.

• Knowledge of Kubernetes for deployment purposes.

• Experience with data quality tools like deequ, Great Expectations, or similar alternatives.

• Familiarity with GCP services (BigQuery, Dataproc, Cloud Storage).

• Experience in leading or contributing to a data warehouse migration project.

• Background in team mergers or transitioning a team to a new operational process.

🏝️ Benefits

• 100% fully covered medical, vision, and dental insurance for employees and their dependents.

• Generous time off policy; we observe all US federal holidays, close the office for a winter break (12/24-12/31), and provide 18 PTO days along with 10 sick days.

• Competitive compensation package; includes attractive commissions for revenue-generating roles and bonuses for non-revenue positions.

• A strong commitment to fostering diversity, equity, and inclusion.

• Eligibility for additional benefits such as a 401k match up to 5%, fully paid life insurance (up to $100,000 coverage), and parental leave.

• A collaborative and positive work culture - your team will be as intelligent and motivated as you are.

• Endless opportunities for growth and learning.