Data Engineer, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and configure a dedicated workspace for Government operations within the current Databricks environment.

• Create and execute data ingestion pipelines from essential agency business systems, such as financial, HR, CRM, and ITSM systems.

• Utilize native connectors provided by source systems where applicable, and devise custom integration strategies for legacy systems.

• Normalize and prepare the ingested data in Databricks for subsequent visualization and reporting tools.

• Implement Databricks Unity Catalog to ensure centralized data governance, manage metadata, conduct active auditing, and track end-to-end lineage.

• Evaluate the current configuration, assess security measures for CUI/PII/PHI/financial data, and enforce improvements.

• Create comprehensive documentation that includes physical and logical architecture diagrams, automated data dictionaries, and standard operating procedures (SOPs).

• Record data sources, integration techniques, and decisions regarding data lake architecture to facilitate support beyond the pilot phase.

⛳️ Requirements

• At least 8 years of experience with a BS/BA degree, a minimum of 6 years with an MS/MA, or at least 3 years with a PhD.

• Must be a US Citizen.

• Active DoD Secret clearance required.

• A minimum of 2 years of experience working on the Databricks platform.

• Strong expertise in PySpark, Spark SQL, and Python for large-scale data processing and pipeline development.

• Practical experience with Unity Catalog administration, including metastore management, access policies, and data lineage.

• Familiarity with pipeline orchestration tools such as Airflow and Databricks Workflows.

• Knowledge of Delta Lake, schema evolution, time travel, and optimization techniques.

• Experience in integrating diverse enterprise systems, including legacy and custom integrations.

• Understanding of cloud platforms (Azure/AWS) and infrastructure-as-code methodologies.

• Awareness of data governance principles, compliance frameworks, and CUI/PII handling procedures.

• Familiarity with financial, HR, CRM, or ITSM data structures.

• Experience with Git and CI/CD pipelines.

• DoD 8570 certification is required.

• Preferred: Databricks Certified Data Engineer Associate or Professional certification.

🏝️ Benefits

• Competitive salary and comprehensive benefits package.

• Opportunities for professional development and certification.

• Collaborative and supportive work environment.

• Flexible work arrangements.