Data Engineer, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and set up a specialized Government workspace within the current Databricks environment.

• Create and execute data ingestion pipelines from primary agency business systems, such as financial, HR, CRM, and ITSM systems.

• Utilize native or built-in connectors when available from source systems and devise custom integration methods for older systems.

• Standardize and prepare incoming data within Databricks for subsequent visualization and reporting tools.

• Implement the Databricks Unity Catalog to ensure centralized data governance, manage metadata, conduct active auditing, and track end-to-end lineage.

• Evaluate existing configurations, review security measures for CUI/PII/PHI/financial data, and initiate enhancements.

• Create detailed as-built documentation, including physical and logical architecture diagrams, automated data dictionaries, and standard operating procedures (SOPs).

• Record data sources, integration techniques, and decisions regarding data lake architecture to aid in ongoing support.

⛳️ Requirements

• 5 years of work experience with a BS/BA; 3 years with an MS/MA.

• Must be a US citizen.

• Active DoD Secret clearance is required.

• 2 years of experience working on the Databricks platform.

• Strong skills in PySpark, Spark SQL, and Python for large-scale data processing and pipeline development.

• Direct experience with Unity Catalog administration, including metastore management, access policies, and data lineage.

• Familiarity with pipeline orchestration tools such as Airflow and Databricks Workflows.

• Experience with Delta Lake, schema evolution, time travel, and optimization techniques.

• Ability to integrate diverse enterprise systems, including legacy and custom integrations.

• Knowledge of cloud platforms (Azure/AWS) and infrastructure-as-code practices.

• Understanding of data governance principles, compliance frameworks, and CUI/PII handling protocols.

• Familiarity with data structures related to financial, HR, CRM, or ITSM systems.

• Experience with Git and CI/CD pipeline processes.

• DoD 8570 certification is preferred.

🏝️ Benefits

• Potential eligibility for overtime.

• Potential eligibility for shift differentials.

• Potential eligibility for a discretionary bonus.