Data Engineer, Databricks

This is a fully remote position, open to applicants in Maryland.

📋 Description

• Create, develop, and sustain reliable, end-to-end data pipelines for an enterprise Lakehouse utilizing Databricks and AWS.

• Design and oversee ETL processes with thorough logging and error management to identify and resolve issues effectively.

• Construct data models within a Bronze, Silver, and Gold medallion architecture, choosing between dimensional or relational approaches as necessary.

• Examine and dissect complex legacy SQL, reconstruct pipelines and models, and evaluate results to confirm accuracy and efficiency.

• Enhance pipeline performance and scalability by implementing techniques such as partitioning, bucketing, and caching.

• Leverage AI-assisted coding tools to expedite pipeline analysis, data engineering, and data modeling activities.

• Contribute to maintaining data security and ensuring compliance.

• Work collaboratively with cross-functional teams to provide analytics-ready data.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, or Information Systems and Technology.

• Over 8 years of experience in designing, developing, deploying, and managing ETL processes and data pipelines.

• Capability to obtain and uphold a Public Trust or Suitability/Fitness determination as per client specifications.

• U.S. citizenship is mandatory to fulfill this federal contract.

• More than 3 years of experience in building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools, including Delta Lake and Unity Catalog.

• At least 3 years of experience in writing Python and PySpark code for ETL scripts and custom transformations.

• Familiarity with AI-assisted coding tools, such as Claude Code or Cursor, to aid in data engineering and modeling tasks.

• Proven experience in data modeling, including the selection of dimensional or relational models within a Bronze, Silver, and Gold medallion architecture.

• Proficient in using AWS services, such as AWS Glue, Amazon S3, and Amazon RDS.

• Understanding of ETL performance optimization techniques, including partitioning, bucketing, and caching, along with experience in troubleshooting query performance.

• Ability to analyze and deconstruct complex legacy SQL, rebuild data pipelines or models, and compare results to ensure accuracy and performance.

🏝️ Benefits

• Competitive salary and comprehensive benefits package.

• Opportunities for professional growth and development.

• Flexible work arrangements and a supportive work environment.

• Engaging projects that leverage the latest technologies.