Data Engineer, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, construct, and manage batch and streaming data pipelines utilizing PySpark, SQL, Databricks Workflows, and Delta Live Tables.

• Implement Medallion Architecture across bronze, silver, and gold data layers.

• Create scalable ingestion frameworks for structured, semi-structured, and unstructured data.

• Integrate data from various sources including files, databases, APIs, Kafka, Kinesis, and Databricks Auto Loader.

• Develop dimensional and domain-specific data models.

• Build and utilize APIs for integration with downstream systems.

• Optimize Spark workloads for enhanced performance and cost-efficiency through techniques such as partitioning, caching, and cluster sizing.

• Establish data quality checks, validation procedures, and monitoring for pipelines.

• Maintain comprehensive documentation related to data flows, lineage, architecture, and integration points.

• Collaborate with engineering, platform, architecture, and federal program stakeholders throughout the development lifecycle.

⛳️ Requirements

• U.S. citizenship is required.

• Must reside within the United States.

• An active Secret security clearance or higher is required.

• A minimum of 5 years of professional experience in data engineering.

• At least 2 years of hands-on experience with Databricks.

• Strong production-level expertise in PySpark and SQL.

• Experience in developing both batch and streaming data pipelines.

• A solid understanding of data modeling and modern data architecture principles.

• Proficiency in Python programming.

• Experience working within Git-based development and CI/CD environments.

• Familiarity with Databricks Unity Catalog, including catalogs, schemas, and permissions from a data engineering standpoint.

• Experience in integrating data from databases, APIs, files, and streaming platforms.

• A Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field is preferred.

🏝️ Benefits

• Opportunity for remote work.

• Chance to contribute to federal government programs.