Software Engineer, Big Data

This is a fully remote position, open to applicants in United States.

📋 Description

• Create, develop, implement, and sustain data pipelines and systems that make web events accessible in the datalake and through event streams for downstream applications.

• Manage extensive and evolving datasets using both batch and streaming methodologies while ensuring efficiency, speed, and reliability.

• Create, develop, implement, and sustain MLOps frameworks and systems that enhance ML development across various teams and products.

• Enhance the speed, quality, observability, and dependability of ML development and ML-driven products.

• Create, develop, implement, and sustain data documentation frameworks and tools.

• Create, develop, implement, and sustain Sift, an agentic tool for data exploration and analysis.

• Develop scalable, reliable, and efficient infrastructure, tools, and processes that empower engineering teams to deliver products and services.

• Develop and maintain tools for data processing, dataset ingestion and discoverability, data pipeline orchestration, data documentation, data analysis, ML model pipelines, and experiments.

• Assist developers in utilizing AI through an agentic AI data exploration and analysis tool and an LLM Gateway.

⛳️ Requirements

• A minimum of 2 years of professional software development experience, including the construction and operation of data pipelines feeding into a customer-facing product.

• Hands-on experience with Apache Spark: writing, optimizing, and debugging batch and/or streaming jobs at scale.

• Proficient in deploying and managing containerized services and batch jobs.

• Strong foundation in computer science principles: data structures, algorithms, distributed systems, and testing.

• Professional software development experience, particularly in building and operating data pipelines and infrastructure that serve a customer-facing product.

• Preferred: At least 4 years of professional software development experience focusing on data-intensive systems.

• Preferred: Familiarity with AWS services (S3, IAM, EC2, MSK or equivalents) and the ability to instrument jobs with metrics, logs, and alerts.

• Preferred: Experience with a data lake stack, including parquet on object storage, a data catalog such as AWS Glue, and an open table format like Apache Iceberg or Delta Lake.

• Preferred: Streaming experience with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming.

• Preferred: Proficient with Apache Airflow, including authoring DAGs, backfilling, dependency management, and being on-call for scheduled pipelines.

• Preferred: Knowledge of open table format internals such as schema evolution, partitioning, and compaction.

• Preferred: Experience deploying services on Kubernetes using Docker.

• Preferred: Proven experience building and maintaining a data platform that supports a user-facing product, encompassing streaming and batch data ingestion, ETL pipelines, data permissions management, and data discovery.

• BS/MS/PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience (preferred).

• Must be eligible to work in the United States, as indicated by the US base salary range and the language pertaining to US-based positions.

🏝️ Benefits

• Competitive salary.

• Outstanding benefits package.

• Flexible vacation and paid time off.

• Employer-matched 401(k) plan.

• Equity, bonuses, commission, or other compensation may also be provided based on the position offered.

• Medical, financial, and additional benefits.