Senior Data Engineer

This is a fully remote position, open to applicants in Nebraska.

📋 Description

• Spearhead the transition of Buildertrend's data platform to a contemporary Databricks lakehouse.

• Design, construct, and manage the Databricks lakehouse utilizing medallion architecture in dbt.

• Set standards for layers, naming conventions, and construction rules.

• Develop ELT pipelines with Databricks, dbt, and Fivetran sourced from SQL Server, Salesforce, MongoDB, Gong, OpenTelemetry, and vendors like Zonda.

• Oversee the decommissioning of legacy systems such as BigQuery, Data Fusion, and Cloud Fusion.

• Refactor pipelines, resolve circular dependencies, and redirect Tableau to the new infrastructure.

• Enhance SQL Server log-ship replication and Fivetran data ingestion.

• Create automation processes including auto-recovery, full-refresh jobs, and watchdog orchestration.

• Implement data quality checks, freshness monitoring, and quality metrics utilizing dbt tests, Unity Catalog, and Fivetran metadata.

• Design access controls, permission models, per-team clusters, service principals, and scoped access for AI tools.

• Manage privacy requests through tools like DataGrail to comply with CCPA regulations.

• Establish data contracts with product engineering teams and produce data documentation, including data dictionaries, ERDs, specification documents, and column notes.

• Develop and enhance curated data products.

• Collaborate with engineers, analysts, business leaders, and stakeholders to prioritize tasks and manage dependencies.

• Mentor engineers, contribute to architectural decisions, assess emerging technologies, and progress towards technical leadership.

⛳️ Requirements

• A Bachelor’s degree in computer science, software engineering, or a related discipline is required.

• Over 5 years of experience in data engineering, data science, or data analytics.

• Demonstrated experience in leading a data platform migration from initial planning to delivery, including setting deadlines and taking ownership of the outcome.

• Experience collaborating with Data Science, ML, or AI teams and developing datasets for model creation and experimentation.

• Practical experience with Databricks, Unity Catalog, and dbt.

• Strong expertise in SQL, Microsoft SQL Server, Python, APIs, and managing large, complex datasets.

• Comprehensive understanding of data modeling, ETL and ELT pipelines, medallion architecture, and datasets suitable for analytics and machine learning.

• Experience in writing data tests, establishing freshness and alerting monitors, and implementing access controls or governance policies in a live production setting.

• Excellent communication skills with the capability to articulate technical trade-offs to both technical and non-technical stakeholders.

• Proven track record in mentoring engineers, reviewing technical work, and shaping engineering standards.

• A practical mindset towards new tools and technologies, including AI.

• Previous experience in a B2B SaaS organization is preferred.

• Familiarity with BigQuery, Snowflake, and Fivetran is an advantage.

🏝️ Benefits

• Comprehensive health packages, including medical, dental, and vision coverage, along with life insurance and short- and long-term disability benefits.

• A 401(k) plan with Buildertrend matching contributions.

• Generous paid time off.

• 11 paid holidays.

• Personal days.

• Parental leave.

• Paid sabbaticals.

• Volunteer time off.

• Wellness program.

• Onsite fitness center.

• Options for hybrid or remote work.

• Technology reimbursement.

• Complimentary daily lunches when at our HQ office.

• Monthly events to foster team connections.