Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Construct infrastructure for the ingestion, transformation, and loading of growing data volumes from various sources utilizing Spark, SQL, AWS, and Databricks.

• Develop an organic entity resolution framework to consolidate hundreds of billions of individual entities into clean, usable datasets.

• Create CI/CD pipelines and anomaly detection systems to enhance production data quality continuously.

• Formulate solutions for undefined challenges in data engineering and data science.

• Collaborate with the Data Engineering Team and stakeholders while overseeing project scope, progress, and obstacles.

⛳️ Requirements

• 3-5+ years of industry experience with demonstrable examples of strategic technical problem-solving and implementation.

• Strong foundation in software development principles.

• Proficiency in Python.

• Expertise in Apache Spark (with Java, Scala, and/or Python).

• Experience with SQL.

• Proven track record in building scalable data processing systems from scratch.

• Familiarity with developer-focused data pipeline and workflow orchestration tools such as Airflow, dbt, or Dagster.

• Understanding of modern data design and storage patterns, including incremental updates, partitioning, segmentation, rebuilds, and backfills.

• Experience working in Databricks, including Delta Live Tables and data lakehouse methodologies.

• Familiarity with cloud computing platforms such as AWS, GCP, Azure, or similar services.

• Experience with data warehousing solutions like Databricks, Snowflake, Redshift, or BigQuery.

• Knowledge of contemporary data storage formats and tools such as Parquet, ORC, Avro, and Delta Lake.

• Ability to balance high levels of ownership and autonomy with effective collaboration.

• Capability to work efficiently in a remote setting.

• Strong written communication skills through Slack/Chat and in documentation.

• Experience drafting data design documents encompassing pipeline design, data flow, and schema design.

• Competence in scoping and breaking down projects while communicating progress and challenges.

• Familiarity with entity data, entity resolution, or record linkage.

• Experience in data acquisition or data integration.

• Proficient in Python data stack tools such as NumPy and pandas.

• Experience with streaming platforms like Kafka.

• Skills in evaluating data quality and upholding high data standards.

• Authorization to work in the United States will be requested in the application form.

• A degree in a quantitative discipline is preferred, but not mandatory.

🏝️ Benefits

• Stock options.

• Competitive salary packages.

• Unlimited paid time off.

• Medical, dental, and vision insurance.

• Health, fitness, and office stipends.

• The ongoing ability to work from anywhere and in any manner you choose.