Principal Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Design a greenfield, multi-layer data warehouse (raw, refined, serving) that differentiates analytical workloads from production OLTP traffic.

• Implement a governed, self-service data-access layer for internal stakeholders initially (Product, CSM, Deployment/Operations, and Leadership) as Phase 1, prior to customer-facing conversational analytics.

• Create a semantic and metrics layer ensuring that every metric, such as "scan accuracy by site," is defined once in code and remains consistent across all dashboards and products, safeguarding self-service from metric drift.

• Maintain a high-quality bar: 99%+ availability SLA with freshness guarantees, 100% traceability, zero cross-tenant leakage, 99.5%+ pipeline success, and no data loss.

• Design tenant isolation, per-tenant cost attribution, and schema and row-level RBAC to scale towards hundreds of tenants (targeting 300+), rather than just the current fleet size.

• Oversee data-ingestion accuracy at the boundary with the integration/backend team, covering data contracts, schema validation, and pipeline quality to ensure WMS data is accurately placed, shaped, and timely across WMS versions.

• Establish a data catalog and lineage layer (using Purview as the Azure-native solution or DataHub as the open-source alternative) so every consumer can locate data, understand ownership, and trace lineage when a metric appears incorrect.

• Validate the foundation end-to-end on Gather's drone product, then generalize it to ensure each new product enhances the model instead of requiring a rebuild.

• Serve as the link between product and ML (3DCC, damage detection). Connect structured records to unstructured drone imagery and video with full traceability, and prepare the data infrastructure for feature stores and annotation pipelines on one trusted foundation.

⛳️ Requirements

• Over 10 years of experience in data engineering, including at least 3 years focused on architecting data platforms for data products, analytics, or AI-driven products.

• Demonstrated experience in building a greenfield data warehouse and leading an OLTP to OLAP transition, rather than merely maintaining an existing system.

• Extensive expertise in designing multi-layer transformation architectures and reusable frameworks that scale across various product areas.

• Proficient in SQL and dbt, with hands-on experience in ELT and orchestration, as well as large-scale or streaming data.

• Production experience in a major cloud environment (preferably Azure, with AWS or GCP as acceptable alternatives), along with knowledge of infrastructure as code and CI/CD practices.

• Proven track record in data quality, security, governance, and multi-tenancy within production environments.

• Skilled in data transformation and modeling that converts raw multi-source data into refined, serving-ready datasets (raw to refined to serving).

• Expertise in pipeline orchestration and workflow automation for scheduling, dependency management, and reliable execution across data flows.

• Experience with large-scale and distributed processing of high-volume batch data.

• Proficient in real-time and streaming ingestion that captures and processes event data as it arrives.

• Capable of designing semantic and metrics layers that define business metrics once and serve them consistently to every consumer.

• Skilled in serving-layer optimization for quick, low-latency consumption through wide and flattened tables and pre-computed metrics.

• Knowledgeable in cloud data engineering and infrastructure automation that provisions, deploys, and operates the platform reproducibly (cloud-native, infrastructure as code, CI/CD).

• Experienced in ensuring data quality, observability, and lineage to maintain trust, freshness, and end-to-end traceability.

• Proficient in security, governance, and multi-tenancy, including tenant isolation, access control, and resiliency.

• Experienced in multimodal data integration linking structured records to unstructured image and video (drone captures) with traceability.

🏝️ Benefits

• Comprehensive health insurance plans.

• Flexible work hours and remote work options.

• Opportunities for professional development and training.

• Collaborative and inclusive team culture.

• Competitive salary and performance-based bonuses.