Senior Software Engineer, Big Data

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, implement, and sustain data pipelines and systems that enable web events to be accessible in the data lake and through event streams.

• Handle large and dynamic datasets utilizing both batch and streaming methods while ensuring a balance between speed, efficiency, and reliability.

• Create, implement, and manage MLOps frameworks and systems that expedite machine learning development.

• Enhance the speed, quality, observability, and dependability of machine learning development and ML-driven products.

• Develop, implement, and maintain data documentation frameworks and tools.

• Create, implement, and maintain Sift, an interactive data exploration and analysis tool.

• Establish scalable, dependable, and efficient infrastructure, tools, and processes for engineering teams.

⛳️ Requirements

• A minimum of 4 years of professional software development experience, specifically in building and operating data pipelines that contribute to a customer-facing product.

• Proficiency in Apache Spark: crafting, optimizing, and troubleshooting batch and/or streaming jobs at scale.

• Experience in deploying and managing containerized services and batch jobs.

• Strong foundation in computer science principles: data structures, algorithms, distributed systems, and testing.

• Professional software development experience, with a focus on building and managing data pipelines and infrastructure that support customer-facing products.

• Preferred: At least 4 years of professional software development experience concentrating on data-intensive systems.

• Preferred: Familiarity with AWS (S3, IAM, EC2, MSK, or similar) and the ability to instrument jobs with metrics, logging, and alerts.

• Preferred: Experience with a data lake stack, incorporating Parquet on object storage, AWS Glue or comparable data catalogs, and Apache Iceberg or Delta Lake.

• Preferred: Streaming experience with Kafka, Apache Flink, and/or Spark Structured Streaming.

• Preferred: Knowledge of Apache Airflow for workflow orchestration, including DAGs, backfills, dependency management, and on-call responsibilities for scheduled pipelines.

• Preferred: Understanding of open table format internals, such as schema evolution, partitioning, and compaction.

• Preferred: Experience in deploying services on Kubernetes using Docker.

• Preferred: Proven track record of building and maintaining a data platform that supports a user-facing product, including both streaming and batch data ingestion, ETL pipelines, data permissions management, and data discovery.

• Preferred: BS/MS/PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience.

🏝️ Benefits

• Competitive compensation package.

• Comprehensive benefits package.

• Flexible vacation and paid time off.

• Employer-matched 401(k) plan.

• Potential for equity, bonuses, commissions, or other compensation forms as part of the total compensation package.

• Medical, financial, and additional benefits.