Data Engineer, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and set up a specialized Government workspace within the current Databricks environment.

• Create and execute data ingestion pipelines from essential agency business systems, such as financial, HR, CRM, and ITSM systems.

• Utilize native/built-in connectors when supported by source systems and devise custom integration strategies for legacy systems.

• Standardize and prepare incoming data within Databricks for subsequent visualization/reporting tools.

• Implement the Databricks Unity Catalog for centralized data governance, metadata management, active auditing, and comprehensive lineage tracking.

• Evaluate current configurations, review security measures for CUI/PII/PHI/financial data, and make necessary enhancements.

• Produce detailed as-built documentation, including physical/logical architecture diagrams, automated data dictionaries, and standard operating procedures (SOPs).

• Record data sources, integration methods, and data lake architecture decisions to facilitate ongoing maintenance.

⛳️ Requirements

• 5 years of relevant work experience with a BS/BA degree; 3 years with an MS/MA degree.

• Must be a US Citizen.

• Possess an active DoD Secret clearance.

• 2 years of experience working with the Databricks platform.

• Strong expertise in PySpark, Spark SQL, and Python for large-scale data processing and pipeline development.

• Practical experience in Unity Catalog administration, including metastore management, access policies, and data lineage.

• Familiarity with pipeline orchestration tools, such as Airflow and Databricks Workflows.

• Knowledge of Delta Lake, schema evolution, time travel, and optimization techniques.

• Experience in integrating diverse enterprise systems, including legacy and custom integrations.

• Understanding of cloud platforms, including Azure/AWS, and infrastructure-as-code practices.

• Awareness of data governance principles, compliance frameworks, and CUI/PII handling practices.

• Acquainted with financial, HR, CRM, or ITSM data structures.

• Proficient in Git/CI-CD pipeline processes.

• Holds a DoD 8570 certification.

• Databricks Certified Data Engineer Associate or Professional certification is preferred.

🏝️ Benefits

• Potential eligibility for overtime.

• Potential eligibility for shift differential.

• Potential eligibility for a discretionary bonus.

• Equal opportunity employment, including for individuals with disabilities and protected veterans.