Software Developer, Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Take charge of the complete lakehouse implementation and collaborate on data/platform architecture.

• Develop source-ingestion patterns for APIs, publicly available text, and external databases.

• Integrate product records from various retailers and dates through identity and entity resolution.

• Identify optimal sources for conflicting values while maintaining original source records for traceability.

• Streamline existing food-enrichment pipelines into a unified pipeline.

• Create connectors to provide consolidated food data to WISE Intel and the consumer application.

• Uphold a fail-fast configuration discipline and encourage early detection of failures due to missing settings.

• Establish ingestion capabilities and quality reporting for Data Assembly readiness.

• Compose clear, dated technical decision records prior to code implementation.

• Serve as a senior individual contributor alongside the Data team and a second engineer on The Pantry.

⛳️ Requirements

• A minimum of 3 years of experience in a similar position.

• Proven success in designing and launching a lakehouse or data warehouse from the ground up, and managing it in production for at least one year.

• Demonstrated expertise in treating data identity and entity resolution as key architectural considerations.

• Experience in operating batch data pipelines that utilize paid cloud/third-party APIs at scale, with stringent cost efficiency and budget management.

• Ability to work closely with other engineers on a daily basis in a fast-paced environment.

• Native proficiency in Python and SQL.

• Familiarity with DuckDB, DuckLake, S3 Parquet, Postgres, DynamoDB, Prefect, Kubernetes/Fargate job runners, AWS infrastructure, Terraform, IAM, Quarto reporting, Medallion architecture, star schema, and GitHub Actions.

• Knowledge of dbt, Dagster, or Airflow.

• Ability to navigate ambiguity and maintain a fast pace.

• Preferred: Understanding architectural trade-offs between DuckDB and distributed systems like Spark, Databricks, or Snowflake.

• Preferred: Experience with data acquisition methods such as web scraping and OCR API integration.

• Food or nutrition domain knowledge, extensive experience with Spark/Databricks/Snowflake, and people management experience are not required.

🏝️ Benefits

• Fully remote work environment.

• Unlimited paid time off (PTO).

• 10 paid holidays.

• 401(k) plan with a 4% company match, immediately vested.

• Comprehensive medical, dental, vision, life, and additional ancillary coverage.

• 100% coverage for employees, with 80% coverage for dependents.

• 100% employer-paid short-term disability (STD), long-term disability (LTD), identity theft protection, and life insurance.

• Flexible Spending Account (FSA) and Health Savings Account (HSA) plans available.

• $200/month HSA contributions for Employee plans.

• $400/month HSA contributions for Employee + Dependent plans.

• Competitive bonus structure.

• Meaningful equity opportunities.