
Senior Data Engineer
Posted Nov 12

Posted Nov 12
This is a fully remote position, open to applicants in Canada.
• Develop and manage pipelines, integrations, and warehouses that facilitate valuation and monitoring.
• Oversee production pipelines from source data collection to transformation, reconciliation, and client delivery.
• Utilize GitHub for pipeline management, along with automated testing and CI/CD processes.
• Assist in redefining the platform architecture for data ingestion, processing, and delivery.
• Construct and manage both batch and streaming pipelines sourced from databases, APIs, event streams, and semi-structured data.
• Implement change data capture and incremental loading, encompassing ordering, deletions, replay, and slowly changing dimensions.
• Create medallion datasets and dimensional models for Snowflake, Microsoft SQL Server, and Databricks.
• Enforce data contracts, reconciliation, and row-level quarantine prior to publication.
• Manage the GitHub workflow and CI/CD processes, including testing, review, environment promotion, and deployment as code.
• Diagnose and resolve production data failures.
• Convert requirements from product, valuation, and client-facing teams into functional pipelines.
• Over 10 years of experience in data engineering within production systems.
• Expertise in Snowflake or Databricks as a primary platform, including data modeling, performance tuning, and cost management.
• Proficiency in Python and SQL for developing and testing pipelines.
• Knowledge of change data capture and event processing, including ordering, replay, and schema changes.
• Familiarity with Azure, including Databricks, ADLS, and private network connectivity.
• Experience with GitHub and CI/CD for data workloads, utilizing GitHub Actions or a similar system.
• Strong background in data quality, reconciliation, monitoring, and responding to production incidents.
• Proven experience in building multi-tenant, secure data platforms with tenant isolation, access control, and data protection.
• Desirable: Familiarity with Databricks Lakeflow, Auto CDC, Declarative Automation Bundles, DQX, or Snowflake equivalents such as Dynamic Tables, Streams and Tasks, Snowpark, Snowflake CLI deployments, and Data Metric Functions.
• Experience with Debezium, Kafka Connect, or Confluent Kafka.
• Knowledge of Apache Airflow or a comparable workflow orchestrator.
• Proficiency in Kafka or Spark Structured Streaming, Apache Iceberg or Delta Sharing, and dbt for analytical models on curated data.
• Understanding of private markets data, including valuations, funds, portfolio companies, or capital activity.
• Comfortable collaborating directly with client technical teams and engaging with field engineering, product, and other stakeholders.
• A supportive culture that fosters innovation and collaboration.
• Opportunities for taking initiative and continuous learning.
Trax Technologies
Fiserv
Ontrac Solutions
General Dynamics Information Technology
Get handpicked remote jobs straight to your inbox weekly.