Senior Staff Data Engineer, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• The Senior Staff Data Engineer will assist in constructing and managing the enterprise lakehouse on Databricks, establishing a governed data foundation that serves various business domains and downstream analytics.

• Responsible for scalable data ingestion, dependable data processing, and robust technical controls within the Bronze and Silver layers of the medallion architecture.

• Design and create ingestion pipelines from enterprise source systems into the Databricks lakehouse utilizing Delta Lake.

• Oversee Bronze-layer ingestion, which includes raw landing patterns, metadata capture, load traceability, and recoverable ingestion designs.

• Develop Silver-layer pipelines for data cleansing, standardization, deduplication, conformance, and quality enforcement.

• Define and refine reusable ingestion and transformation patterns, templates, and engineering standards.

• Implement and sustain necessary Databricks platform constructs for secure data delivery.

• Construct and maintain CI/CD pipelines for data platform assets.

• Apply data classification, segregation, and handling requirements in the design of pipelines.

• Establish data quality controls that reflect actual business significance, record integrity, completeness, and expected domain behavior.

• Keep documentation updated for source objects, ingestion logic, applied transformations, data quality rules, and known limitations.

• Collaborate with the Analytics Engineer and domain teams to ensure that Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling.

• Work with domain engineering teams to synchronize on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains are integrated into the platform.

⛳️ Requirements

• 12+ years of experience in data engineering, including direct ownership of production data pipelines.

• Extensive experience with Databricks, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.

• Proficient knowledge of Unity Catalog, which encompasses catalogs, schemas, tables, lineage, and access control concepts.

• Experience with batch, CDC, and/or streaming ingestion patterns, along with the operational trade-offs associated with each.

• Familiarity with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.

• Strong SQL skills coupled with a solid understanding of data modeling fundamentals, even if dimensional modeling is not the primary focus of this role.

• Proven ability to comprehend the business and regulatory context of the data being processed, beyond just the mechanics of pipeline development.

• Experience in applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows within a regulated or security-sensitive environment.

• Ability to design pipelines with a clear understanding of the actual data domains involved, including sensitivity, ownership, permitted use, and downstream impact.

• Comfortable working in a dynamic platform build environment where patterns are still being established, and engineers are expected to shape standards rather than merely follow them.

🏝️ Benefits

• Compensation within the listed range + Bonus + Benefits + Equity

• Temporary benefits package (available after 60 days of employment)