Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Create infrastructure for the ingestion, transformation, and loading of growing data volumes from diverse sources utilizing Spark, SQL, AWS, and Databricks.

• Establish an organic entity resolution framework to consolidate hundreds of billions of individual entities into clean, usable datasets.

• Design CI/CD pipelines and anomaly detection systems to enhance production data quality continuously.

• Develop solutions for undefined challenges in data engineering and data science.

• Collaborate with colleagues, managers, and stakeholders in project scoping and in communicating progress and obstacles.

⛳️ Requirements

• 5-7+ years of industry experience demonstrating strategic technical problem-solving and implementation.

• Strong foundation in software development principles.

• Proficiency in Python.

• Expertise in Apache Spark (Java, Scala, and/or Python).

• Experience with SQL.

• Proven experience in constructing scalable data processing systems from the ground up.

• Familiarity with developer-oriented data pipeline and workflow orchestration tools, such as Airflow, dbt, dagster, or similar.

• Understanding of modern data design and storage patterns, including incremental updates, partitioning and segmentation, rebuilds, and backfills.

• Experience working with Databricks, including Delta Live Tables and data lakehouse patterns.

• Knowledge of cloud computing services, such as AWS, GCP, Azure, or similar.

• Experience with data warehousing solutions like Databricks, Snowflake, Redshift, BigQuery, or similar.

• Familiarity with modern data storage formats and tools, such as Parquet, ORC, Avro, and Delta Lake.

• Ability to maintain a balance between high ownership and autonomy while fostering collaboration.

• Capability to work effectively in a remote environment, proactively manage obstacles, seek assistance, and engage in team activities.

• Strong written communication skills in Slack/Chat and documentation.

• Experience in drafting data design documents, including pipeline design, data flow, and schema design.

• Ability to scope and decompose projects and effectively communicate progress and challenges.

• Professional experience with Spark or PySpark.

• Familiarity with medallion architecture.

• Authorization to work in the United States.

• A degree in a quantitative field is considered a plus but is not mandatory.

🏝️ Benefits

• Stock options.

• Competitive salary packages.

• Unlimited paid time off.

• Medical, dental, and vision insurance.

• Health, fitness, and office stipends.

• The permanent flexibility to work wherever and however you choose.