Data Engineer, Databricks

This is a fully remote position, open to applicants in Virginia.

📋 Description

• Facilitate secure, scalable, and efficient data sharing between federal clients and external data sharing partners utilizing Databricks Delta Sharing.

• Assist in the design and development of data pipelines and ETL processes within the Azure Cloud environment for various source systems, including RDBMS, API, and unstructured data, employing techniques such as CDC, incremental loading, and batch processing.

• Perform data profiling, transformation, and quality assurance on structured, semi-structured, and unstructured datasets.

• Detect underlying issues and convert them into technical specifications.

• Aid in the construction and optimization of data lakes, feature stores, and data warehouse architectures to support analytics and machine learning initiatives.

• Prepare, organize, and verify data for data science and MLOps workflows, ensuring compliance with quality and format standards for modeling.

• Assist in monitoring and maintaining data flow across BI dashboards, analytical environments, and machine learning pipelines.

• Interact directly with clients and stakeholders to comprehend data requirements and convert them into scalable solutions.

• Collaborate with UX designers, business analysts, developers, and end users to ascertain data and reporting specifications.

• Partner with external data providers to identify their data product needs and leverage the Databricks platform for rapid prototyping and extensible use cases.

• Engage with government personnel at executive levels, platform stakeholders, and vendor partners.

• Operate within Agile teams to support iterative development, backlog refinement, and sprint-based delivery.

• Provide mentorship and guidance to junior team members.

⛳️ Requirements

• Bachelor's degree.

• Over 5 years of experience in data engineering, data security practices, data platforms, and analytics.

• U.S. Citizenship is mandatory due to federal contract stipulations.

• Capability to obtain and sustain a federal public trust clearance or equivalent background investigation as required by the client.

• Candidate must live in the U.S., possess authorization to work in the U.S., and complete all work within the U.S.

• Candidate must have resided in the U.S. for three (3) full years out of the last five (5) years.

• At least 3 years of expertise on the Databricks Platform at a Subject Matter Expert (SME) level, including proficiency in Databricks, Delta Lake, and Delta Sharing.

• Extensive experience with distributed computing utilizing Apache Spark.

• Understanding of Spark runtime internals and optimization strategies.

• Proficient in designing and deploying efficient end-to-end data architectures.

• More than 4 years of experience in developing ETL pipelines to create robust, scalable data solutions.

• Possession of Databricks certifications, either professional or specialty.

• Practical experience with Azure services such as Synapse, Data Factory, or Databricks.

• Familiarity with data visualization tools such as Tableau, Power BI, or similar platforms.

• Skills in Machine Learning and Analytics, including MLOps — working knowledge of ML deployment and operations.

• Understanding of Data Science methodologies — statistical analysis, modeling, and interpretation.

• Experience with Big Data technologies beyond Spark in distributed systems.

• Familiarity with deployment pipelines, including Git-based version control, CI/CD pipelines, and DevOps practices utilizing Terraform for Infrastructure as Code (IaC).

🏝️ Benefits

• Reasonable accommodations are available, including, but not limited to, support for disabled veterans, individuals with disabilities, and those with sincerely held religious beliefs, throughout all phases of the application and employment process.