Principal Data Platform Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and implement a centralized data platform utilizing Databricks.

• Develop governance frameworks using Unity Catalog.

• Optimize cost and performance at scale.

• Empower Data Engineers to confidently utilize the platform.

• Spearhead the architecture for transitioning multi-terabyte datasets from legacy systems to Databricks.

• Create Unity Catalog structures to ensure secure data separation across product lines.

• Construct infrastructure that scales effectively through intelligent caching, query optimization, and compute management strategies.

• Set up monitoring, alerting, and data quality validation to maintain platform reliability.

⛳️ Requirements

• Expertise in Databricks (Required)

• Experience with Unity Catalog: Proficiency in multi-catalog governance, metastore design, and lineage tracking.

• Data Structuring: Proven experience in designing and building unified schemas across diverse product lines.

• Delta Lake: Advanced knowledge of Z-ordering, compaction, liquid clustering, and performance tuning at multi-TB scale.

• Delta Live Tables: Solid hands-on experience in building declarative ETL pipelines, including change data capture and expectations/constraints.

• Familiarity with Databricks Workflows: Experience in job orchestration, scheduling, and operational monitoring.

• Business Intelligence: Background in enabling company-wide analytics and reporting using modern business intelligence tools while maintaining source of truth data and metrics.

• Strong proficiency in PySpark & Databricks SQL for code review, performance tuning, and query optimization.

• Core Platform Engineering: 5-8 years in data engineering or data platform roles, with a minimum of 3 years of hands-on experience with Databricks.

• Proven track record of leading at least one major platform build or migration project.

• Experience with AWS (S3, IAM, VPC) and the ability to collaborate on infrastructure decisions.

• Knowledge of Infrastructure-as-Code (preferably Terraform).

• Technical Leadership: Demonstrated capability in architecting data platforms from first principles and justifying technical choices.

• Excellent written and verbal communication skills—documenting architecture decisions and presenting to both technical and business stakeholders.

• Preferred but not mandatory: Experience with financial data, accounting systems (NetSuite), or enterprise ERP platforms.

• Experience in building platforms that cater to AI/ML workloads (including data preparation for downstream ML consumption, RAG and retrieval, and LLMs).

• Understanding of advanced intelligence concepts such as relationship surfacing with knowledge graphs.

• Familiarity with data governance frameworks and compliance requirements in regulated industries.

🏝️ Benefits

• A collaborative team culture that promotes career development.

• Numerous opportunities for recognition, skill-building, and career advancement.

• Generous vacation policy, inclusive of paid parental leave.

• Comprehensive health plans featuring FSA and HSA options.

• 401(k) retirement savings plan.

• Life and disability insurance coverage.

• Additional benefits such as a dependent care savings plan, pet insurance, will preparation, and an employee assistance program.