Senior Data Platform Engineer

This is a fully remote position, open to applicants in Canada, +1 more country.

📋 Description

• Construct and manage the lakehouse on Azure Databricks.

• Design Spark and Delta Lake transformations, orchestration, and serving layers.

• Create and uphold medallion and dimensional data models within Delta Lake.

• Optimize Spark jobs, Photon, liquid clustering, storage, compute, and DBU expenditure.

• Oversee the governed Unity Catalog semantic layer and certified Metric Views.

• Implement row and column security per tenant at query time.

• Develop and align StoreForce retail KPI logic.

• Onboard tenants and integrate KPIs through configuration and established patterns.

• Ensure data quality, enforce data contracts, and conduct reconciliation, freshness, and volume checks.

• Maintain lineage and Lakehouse Monitoring while managing data incidents as production incidents.

• Deliver infrastructure and platform modifications as code through CI/CD.

• Build and operate an LLM-driven analytics agent utilizing Databricks AI tools, Anthropic Claude, and OpenAI.

• Guarantee tenant isolation, implement runtime guardrails, and uphold automated evaluation quality gates.

• Collaborate with the implementation partner during the initial build and assume ownership at handover.

• Act as StoreForce’s primary data engineer and provide support to consuming teams.

• Uphold engineering standards and runbooks while triaging data incidents.

⛳️ Requirements

• Over 5 years of experience in building data pipelines or backend systems, with a minimum of 2 years managing a modern data stack in a production environment.

• Practical experience with Databricks: Unity Catalog, Delta Lake, Lakeflow Jobs and Spark Declarative Pipelines (or Workflows and Delta Live Tables), Databricks SQL, and Databricks Asset Bundles or equivalent as-code practices.

• Proficient in Apache Spark: PySpark, Spark SQL, Structured Streaming, query-plan analysis, shuffle and skew diagnosis, as well as job and compute optimization.

• Strong SQL capabilities, including window functions, complex joins, and query optimization.

• Solid Python skills for data-related tasks.

• Extensive experience in data modeling, including medallion, dimensional, and incremental patterns.

• Familiarity with tuning Delta tables, such as liquid clustering, change data feed, OPTIMIZE, and VACUUM strategies.

• Knowledge in implementing data quality and observability measures, including expectations, contracts, reconciliation, lineage, monitoring, and alerting.

• Hands-on experience with Anthropic Claude and OpenAI APIs for AI functionalities.

• Well-versed in contemporary software engineering practices, including version control, automated testing, and CI/CD.

• Databricks Certified Data Engineer Associate or Professional certification is preferred.

• Experience with LLMs, multi-tenant SaaS data, SQL Server/Azure SQL Managed Instance CDC, Lakebase/PostgreSQL, embedded analytics, and platform handover is preferred.

• Candidates must reside in the United States or Canada.

🏝️ Benefits

• Flexible scheduling options.

• Employer-covered health benefits, including dental, prescription drug coverage, and a Health Spending Account.

• RRSP Matching with support for First Time Home Buyers.

• Company-sponsored training and professional development opportunities.

• Extended long weekends and religious observance days.

• Employee Referral Program.

• Spotlight Recognition Bonus Program.

• Access to a dedicated Employee Assistance Program (EAP).

• Accommodations available upon request for candidates participating in all aspects of the selection process.