Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Create and execute secure, advanced data engineering solutions that support essential missions.

• Collaborate with technical leaders, business stakeholders, and cross-functional teams.

• Assist in implementing the DON MAHRS program Data Strategy and the High-Level Total Force Data Hub implementation plan.

• Design, implement, integrate, and maintain data architectures, data hubs, data lakes, and data warehouses.

• Create and develop data models, data structures, and processes for data acquisition.

• Formulate engineering and implementation strategies for data hubs, acquisition, modeling, integration, and associated activities.

• Investigate and assess data sources to determine authoritative and dependable sources for data hub development.

• Plan, develop, and sustain data architectures that align with business requirements.

• Design, develop, maintain, and enhance ETL/ELT pipelines and transformation processes.

• Create and sustain batch and streaming pipelines utilizing Spark, Python, Databricks, Palantir Foundry, Kafka, and related tools.

• Establish data acquisition methods to ingest, transform, validate, and integrate enterprise data.

• Execute incremental loading, handle late-arriving data, manage processing windows, ensure data freshness, and apply lifecycle approaches.

• Automate manual data processes to enhance workflow efficiency, reliability, and scalability.

• Develop, maintain, and optimize solutions within Databricks and enterprise data lakes.

• Configure, monitor, manage, and document Databricks clusters in accordance with DON policies and security standards.

• Develop and maintain Spark solutions utilizing PySpark, Spark SQL, DataFrames, and Data Sets.

• Implement and sustain Delta Lake and Delta Live Tables solutions.

• Enhance data processing, storage, and query performance.

• Support Palantir Foundry ontologies, ETL/ELT pipelines, applications, and data-driven user interfaces.

• Integrate Palantir Foundry with enterprise data engineering and analytics environments.

• Create and maintain streaming solutions using Kafka, Kafka Streams, and ksqlDB.

• Configure and manage Kafka topics and the Schema Registry.

• Develop Python data-processing applications and AWS Lambda functions.

• Facilitate processing with AWS S3, Kinesis, Lambda, and DynamoDB.

• Monitor, troubleshoot, and enhance cloud-based processing solutions.

• Establish data quality controls, validation procedures, and data quality gates.

• Develop and conduct data-driven and unit testing for Spark, Python, and other processing solutions.

• Set up data lifecycle policies covering retention, backup, recovery, and data management.

• Monitor and troubleshoot pipelines and processing jobs.

• Enhance performance, reliability, scalability, and maintainability.

⛳️ Requirements

• Over 5 years of professional experience in data engineering.

• More than 5 years of IT experience with a focus on enterprise data engineering, including data modeling, data quality, data mapping tools, and technical documentation.

• A Bachelor’s degree in Computer Science, Engineering, Mathematics, Statistics, or a related IT field.

• Must be eligible to obtain and maintain a Secret security clearance.

• United States citizenship is required to acquire this type of security clearance.

• Candidates must reside within the continental United States.

• Professional experience in data architecture, data engineering, data hub, data lake, and/or data warehouse development.

• Experience in designing, developing, implementing, and supporting enterprise-scale data engineering solutions.

• Familiarity with data modeling, data mapping, data quality, data integration, and data management.

• Experience supporting large-scale, high-performance enterprise data applications.

• Background in developing and supporting ETL/ELT data pipelines and data transformation processes.

• Hands-on experience with Databricks is essential.

• Proficiency in Python and SQL for data engineering and processing.

• Experience with Apache Spark and/or PySpark, including Spark SQL, DataFrames, and/or Data Sets is preferred.

• Experience in supporting data integration, migration, transformation, data warehouse, data hub, and/or data lake/Delta Lake implementations.

• Background in batch and streaming data processing architectures.

• Familiarity with Kafka and/or other distributed streaming technologies.

• Experience in developing and supporting Kafka Streams.

• Experience working in an AWS cloud environment, including S3, Lambda, Kinesis, and/or DynamoDB.

• Knowledge of Git-based version control, including branching, merging, pull requests, and repository management.

• Experience supporting CI/CD pipelines and cloud monitoring with Jenkins, Docker, and/or CloudWatch is preferred.

• Familiarity with Palantir Foundry is preferred.

• Experience with the Jupiter data and analytics environment is preferred but not required.

• Ability to debug, troubleshoot, design, and implement solutions to complex technical challenges.

• Capability to thrive in a team-oriented environment.

• Experience presenting technology solutions to technology partners, stakeholders, team members, and senior management.

• Strong written and verbal communication skills.

• Active CompTIA Security+ or CompTIA Network+ certification is preferred; if selected, must obtain CompTIA Security+ before supporting the program.

🏝️ Benefits

• Nationwide medical, dental, and vision insurance.

• 3 weeks of Paid Time Off.

• 11 Paid Federal Holidays.

• 401k matching.

• Life Insurance.

• Short-Term Disability at no cost to employees.

• Long-Term Disability at no cost to employees.

• Supplemental insurance options.

• Flexible spending accounts.

• Dependent Care spending accounts.

• Wellness incentives.

• Reimbursement for professional development and certifications.

• Training assistance opportunities to support career growth and progression.

• Flexible scheduling options.

• Hybrid and Remote work opportunities to facilitate work-life balance.