Senior Data Engineer – Data Platform

This is a fully remote position, open to applicants in Nebraska.

📋 Description

• Oversee the migration of the data platform to a state-of-the-art Databricks lakehouse based on dbt.

• Architect, develop, and sustain the Databricks lakehouse utilizing a medallion architecture.

• Establish layer standards, naming conventions, and construction rules for fellow engineers.

• Create ELT pipelines with Databricks, dbt, and Fivetran sourced from SQL Server, Salesforce, MongoDB, Gong, OpenTelemetry, and vendors like Zonda.

• Spearhead the transition from legacy systems including BigQuery, Data Fusion, and Cloud Fusion.

• Refactor pipelines, resolve circular dependencies, and redirect Tableau to the updated platform.

• Enhance SQL Server log-ship replication and Fivetran ingestion processes.

• Develop auto-recovery, full-refresh jobs, and watchdog orchestration to minimize manual interventions and outages.

• Implement data quality checks, freshness monitoring, and quality metrics using dbt tests, Unity Catalog, and Fivetran metadata.

• Design access controls, permission frameworks, per-team clusters, service principals, and scoped access for AI tools.

• Manage privacy requests utilizing tools like DataGrail to comply with CCPA regulations.

• Formulate data contracts with product engineering and create data dictionaries, ERDs, specifications, and column notes.

• Develop and refine curated data products for stakeholders and clients.

• Collaborate with data science, machine learning, and AI teams on datasets for models and experiments.

• Work alongside engineers, analysts, business leaders, and stakeholders to prioritize tasks, manage dependencies, and enhance platform capabilities.

• Mentor engineers, participate in architectural decisions, assess emerging technologies, and cultivate technical leadership.

⛳️ Requirements

• A Bachelor’s degree in computer science, software engineering, or a related discipline is mandatory.

• A minimum of 5 years of experience in data engineering, data science, or data analytics is required.

• Practical, production-level experience with Databricks, encompassing the building and orchestration of pipelines, job and cluster management, and governance using Unity Catalog.

• Direct production-level experience with dbt, involving the development and maintenance of models, tests, and macros.

• Proven experience leading a data platform migration from inception to implementation, including deadline establishment and outcome ownership.

• Familiarity with collaborating alongside data science, machine learning, or AI teams to build and curate datasets.

• Strong skills in SQL, Microsoft SQL Server, Python, APIs, and handling large, intricate datasets.

• Comprehensive knowledge of data modeling, ETL and ELT pipelines, medallion architecture, and datasets prepared for analytics and machine learning.

• Experience in writing data tests, configuring freshness and alerting monitors, and establishing access controls or governance policies in a production environment.

• Excellent communication skills to articulate technical trade-offs to both technical and non-technical stakeholders.

• Proven experience in mentoring engineers, reviewing technical outputs, and influencing engineering standards.

• Prior experience in a B2B SaaS organization is preferred.

• Familiarity with BigQuery or Fivetran is a plus.

🏝️ Benefits

• Comprehensive health packages, including medical, dental, and vision insurance.

• Life insurance coverage.

• Short- and long-term disability benefits.

• 401(k) plan with matching contributions from Buildertrend.

• Generous paid time off policies.

• 11 paid holidays annually.

• Personal days off.

• Parental leave options.

• Paid sabbaticals for extended breaks.

• Volunteer time off for community service.

• Wellness programs to support employee health.

• Onsite fitness center for employee use.

• Flexible options for hybrid or remote work arrangements.

• Technology reimbursement for home-office equipment costs.

• Complimentary daily lunches at the headquarters office.

• Monthly team-building events.