Databricks Migration Engineer

This is a fully remote position, open to applicants in Virginia.

📋 Description

• Oversee the technical transition from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake).

• Convert relational data warehousing concepts into scalable, distributed Lakehouse frameworks across Bronze, Silver, and Gold layers.

• Develop reusable ETL/ELT frameworks utilizing PySpark, Delta Live Tables, and Databricks Workflows.

• Design and enhance Gold Layer dimensional models and star schemas to optimize Power BI performance.

• Fine-tune Databricks SQL Warehouses for high-concurrency, low-latency Power BI queries.

• Implement Z-Ordering, data skipping, liquid clustering, and materialized views.

• Establish governance standards for cluster sizing, auto-scaling, and serverless SQL compute.

• Monitor Databricks Unit consumption and pinpoint cost-saving opportunities.

• Set up Delta Lake partitioning and file size management protocols.

• Design and execute Unity Catalog data security frameworks.

• Enforce row-level and column-level security measures.

• Align Lakehouse security protocols with Azure Active Directory/Microsoft Entra ID and RBAC standards.

• Act as the primary technical lead through pair-programming sessions, workshops, and code reviews.

• Produce technical documentation, architecture diagrams, design patterns, and optimization playbooks.

• Create knowledge transfer processes to ensure post-migration team self-sufficiency.

• Engage effectively with both technical and non-technical audiences.

• Execute tasks in a structured, timely, and detail-oriented fashion.

⛳️ Requirements

• Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical discipline.

• Over 5 years of experience in Data Engineering, Data System Development, or similar roles.

• More than 5 years of experience with Cloud platforms (e.g., Azure, AWS, GCP).

• At least 1 year of experience leading complex, cross-functional data projects and technical teams.

• Proficient in Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.

• Familiarity with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks utilizing PySpark.

• Outstanding communication skills.

• Must be a US Citizen and able to secure a Position of Public Trust Clearance.

• Must have resided in the US for the past 5 years and not have traveled outside the US for a total of 6 months or more in the last 5 years.

🏝️ Benefits

• Contract-to-Hire position.

• Remote work arrangement.

• Preference for candidates in the East Coast time zone.