Senior Data Engineer

Posted Nov 12

This is a fully remote position, open to applicants in Canada.

📋 Description

• Develop and manage pipelines, integrations, and warehouses that facilitate valuation and monitoring.

• Oversee production pipelines from source data collection to transformation, reconciliation, and client delivery.

• Utilize GitHub for pipeline management, along with automated testing and CI/CD processes.

• Assist in redefining the platform architecture for data ingestion, processing, and delivery.

• Construct and manage both batch and streaming pipelines sourced from databases, APIs, event streams, and semi-structured data.

• Implement change data capture and incremental loading, encompassing ordering, deletions, replay, and slowly changing dimensions.

• Create medallion datasets and dimensional models for Snowflake, Microsoft SQL Server, and Databricks.

• Enforce data contracts, reconciliation, and row-level quarantine prior to publication.

• Manage the GitHub workflow and CI/CD processes, including testing, review, environment promotion, and deployment as code.

• Diagnose and resolve production data failures.

• Convert requirements from product, valuation, and client-facing teams into functional pipelines.


⛳️ Requirements

• Over 10 years of experience in data engineering within production systems.

• Expertise in Snowflake or Databricks as a primary platform, including data modeling, performance tuning, and cost management.

• Proficiency in Python and SQL for developing and testing pipelines.

• Knowledge of change data capture and event processing, including ordering, replay, and schema changes.

• Familiarity with Azure, including Databricks, ADLS, and private network connectivity.

• Experience with GitHub and CI/CD for data workloads, utilizing GitHub Actions or a similar system.

• Strong background in data quality, reconciliation, monitoring, and responding to production incidents.

• Proven experience in building multi-tenant, secure data platforms with tenant isolation, access control, and data protection.

• Desirable: Familiarity with Databricks Lakeflow, Auto CDC, Declarative Automation Bundles, DQX, or Snowflake equivalents such as Dynamic Tables, Streams and Tasks, Snowpark, Snowflake CLI deployments, and Data Metric Functions.

• Experience with Debezium, Kafka Connect, or Confluent Kafka.

• Knowledge of Apache Airflow or a comparable workflow orchestrator.

• Proficiency in Kafka or Spark Structured Streaming, Apache Iceberg or Delta Sharing, and dbt for analytical models on curated data.

• Understanding of private markets data, including valuations, funds, portfolio companies, or capital activity.

• Comfortable collaborating directly with client technical teams and engaging with field engineering, product, and other stakeholders.


🏝️ Benefits

• A supportive culture that fosters innovation and collaboration.

• Opportunities for taking initiative and continuous learning.

People also viewed

Trax Technologies11 hours ago

Data Architect

US flagUnited States OnlyFull-timeData Engineer
ApplyView job
Fiserv12 hours ago

Senior Snowflake Data Platform and Artificial Intelligence Architect

US flagUnited States OnlyFull-timeData Engineer
ApplyView job
Ontrac Solutions12 hours ago

Staff Data Engineer

US flagUnited States OnlyFreelanceData Engineer
ApplyView job
General Dynamics Information Technology12 hours ago

ICAM Identity Data Management Engineer – Radiant Logic

US flagUnited States OnlyFull-timeData Engineer$170k – $230k/year
ApplyView job
General Dynamics Information Technology12 hours ago

Identity Data Management Engineer – Radiant Logic

US flagUnited States OnlyFull-timeData Engineer$170k – $230k/year
ApplyView job
General Dynamics Information Technology12 hours ago

Senior Data Engineer

US flagUnited States OnlyFull-timeData Engineer$102k – $138k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers