Senior Software Engineer, Data Acquisition

This is a fully remote position, open to applicants in United States.

📋 Description

• Contribute to the design and enhancement of the data acquisition and processing platform, boosting reliability, throughput, and observability.

• Utilize and develop web crawling technologies to gather and catalog data from the internet.

• Construct, manage, and advance large-scale distributed systems that collect, process, and deliver data sourced from the web.

• Design and implement backend services for distributed job orchestration, data pipelines, and large-scale asynchronous workloads.

• Structure and model acquired data, ensuring high quality and consistency across datasets.

• Enhance the speed, scalability, and fault tolerance of ingestion systems.

• Collaborate with data product and engineering teams to design and launch new data products while improving existing offerings.

• Acquire and apply domain-specific expertise in web crawling and data acquisition with guidance from seasoned colleagues.

⛳️ Requirements

• Over 7 years of professional experience in developing or operating backend or infrastructure systems at scale.

• Strong programming experience in Python, Go, Rust, or similar languages, including async/await, coroutines, or concurrency frameworks.

• A robust understanding of software architecture and backend principles, encompassing concurrency, scalability, and fault tolerance.

• Comprehensive knowledge of the browser rendering pipeline and web application architecture, including authentication, cookies, and HTTP request/response.

• Familiarity with network architecture and debugging, covering HTTP, DNS, proxies, and packet capture and analysis.

• A solid grasp of distributed systems concepts: parallelism, asynchronous programming, backpressure, and message-driven design.

• Experience in designing or maintaining resilient data ingestion, API integration, or ETL systems.

• Proficient in Linux/Unix command-line tools and system resource management.

• Knowledge of message queues, orchestration, and distributed task systems such as Kafka, SQS, and Airflow.

• Experience in evaluating and monitoring data quality, ensuring consistency, completeness, and reliability across releases.

• Ability to work independently in a fast-paced, remote-first environment and collaborate asynchronously.

• Strong written communication skills.

• Proficient in creating and maintaining technical design documents, including pipeline design, schema design, and data flow diagrams.

• Capable of scoping and breaking down complex projects into deliverable milestones and effectively communicating progress, risks, and obstacles.

• Degree in a quantitative discipline such as computer science, mathematics, or engineering (preferred).

• Experience as a Red Teamer (preferred).

• Familiarity with large-scale data ingestion, crawling, or indexing systems (preferred).

• Experience with Apache Spark, Databricks, or other distributed data platforms (preferred).

• Knowledge of streaming data systems like Kafka, Pub/Sub, and Spark Streaming (preferred).

• Proficient in SQL and data warehousing technologies such as Snowflake, Redshift, or BigQuery (preferred).

• Experience with cloud platforms, ideally AWS (preferred).

• Understanding of modern data storage and design patterns such as Parquet, Delta Lake, partitioning, and incremental updates (preferred).

• Experience in building and maintaining data pipelines on contemporary big-data or cloud platforms (preferred).

🏝️ Benefits

• Stock options.

• Competitive salaries.

• Unlimited paid time off.

• Medical, dental, & vision insurance.

• Health, fitness, and office stipends.

• The permanent flexibility to work from anywhere and in any manner you prefer.