Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee, enhance, and resolve issues related to database queries and associated data systems.

• Develop, uphold, and refine pipelines for the collection, processing, transformation, validation, and distribution of large-scale datasets.

• Assist in web scraping and data collection efforts by designing, testing, and maintaining data-gathering scripts and tools.

• Monitor and resolve issues in data pipelines.

• Detect data quality issues and implement prompt solutions.

• Record database queries, pipeline processes, scraping workflows, technical decisions, issues, and their resolutions.

• Engage in research and development initiatives aimed at enhancing the company’s data products and workflows.

⛳️ Requirements

• Bachelor’s degree or equivalent professional experience.

• Proficient in advanced Python, including async programming, multiprocessing, and production-level coding for long-running data tasks.

• Practical experience with high-volume web scraping, encompassing proxies, rate limiting, and anti-bot evasion techniques.

• Familiarity with platform APIs and extensive media/metadata datasets.

• Background in designing and managing distributed data pipelines across multiple workers/servers.

• Experience with task queues such as Celery, Kafka, RabbitMQ, or similar technologies.

• Hands-on experience with columnar/analytical data warehouses.

• Preference for experience with Databend, ClickHouse, or BigQuery.

• Ability to handle complex analytical queries, partitioning strategies, and cost-effective querying in cloud environments.

• Experience with Docker and Kubernetes, including workload containerization, writing Helm charts/manifests, managing deployments, and autoscaling workloads.

• Comfortable managing services on Linux servers and troubleshooting disk I/O, network, and memory performance challenges.

• CI/CD experience for data workflows utilizing GitHub Actions or ArgoCD.

• Proficient in writing scalable APIs.

🏝️ Benefits

• Competitive salary, benefits, and equity package.

• Fully remote team environment.

• Opportunity to contribute to a web-scale crawler and knowledge graph.

• Lean, technical culture with minimal bureaucracy and quick decision-making processes.

• Chance to enhance access to public web data and increase the value of AI.