Senior Software Engineer, Big Data

This is a fully remote position, open to applicants in California.

📋 Description

• Design, construct, deploy, and maintain data pipelines and systems that facilitate the availability of web events in the data lake and through event streams.

• Manage large, evolving datasets utilizing both batch and streaming methodologies while ensuring a balance between speed, efficiency, and reliability.

• Create, implement, and oversee MLOps frameworks and systems that expedite machine learning development across various teams and products.

• Enhance the speed, quality, observability, and reliability of machine learning development and products powered by machine learning.

• Establish, implement, and manage data documentation frameworks and tools.

• Enhance the thoroughness and quality of data documentation for both human and automated processes.

• Develop, implement, and sustain the Sift agentic data exploration and analysis tool.

• Democratize data access, foster collaboration on insights, and speed up data-driven decision-making.

⛳️ Requirements

• 4+ years of professional software development experience, specifically in building and operating data pipelines that contribute to a customer-facing product.

• Practical experience with Apache Spark: writing, optimizing, and debugging batch and/or streaming jobs at scale.

• Familiarity with deploying and managing containerized services and batch jobs.

• Strong foundation in computer science principles: data structures, algorithms, distributed systems, and testing.

• Professional experience in software development, particularly in constructing and managing data pipelines and infrastructure for customer-facing products.

• Proficiency in AWS (S3, IAM, EC2, MSK or equivalents) and implementing metrics, logs, and alerts in jobs.

• Experience with a data lake stack: Parquet on object storage, data cataloging tools like AWS Glue, and open table formats such as Apache Iceberg or Delta Lake.

• Streaming proficiency with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming.

• Familiarity with workflow orchestration tools like Apache Airflow: creating DAGs, managing backfills, dependencies, and being on-call for scheduled pipelines.

• Knowledge of open table format internals: schema evolution, partitioning, and compaction.

• Experience deploying services on Kubernetes using Docker.

• Proven track record in building and maintaining a data platform that supports a user-facing product, which includes streaming and batch data ingestion, ETL pipelines, data permissions management, and data discovery.

• BS/MS/PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience.

🏝️ Benefits

• Competitive compensation.

• Comprehensive benefits package.

• Flexible vacation and paid time off.

• Employer-matched 401(k) plan.

• Potential for equity, bonuses, commission, or other forms of compensation as part of the total package.

• Medical, financial, and additional benefits.