Data Engineer

This is a fully remote position, open to applicants in Florida.

📋 Description

• Deliver data engineering expertise in the development, implementation, integration, and maintenance of data architectures, data hubs, data lakes, and data warehouse solutions.

• Assist in the design and development of data models, data structures, and data acquisition processes within the HC/HR Data Domain.

• Formulate engineering and implementation strategies for data hubs, acquisition, modeling, integration, and related data engineering activities.

• Investigate and assess enterprise data sources to ensure authoritative and dependable data hub development.

• Strategize, create, and uphold data architectures that align with business needs.

• Design, develop, sustain, and enhance ETL/ELT pipelines and data transformation processes.

• Create and manage batch and streaming pipelines utilizing Spark, Python, Databricks, Palantir Foundry, Kafka, and associated tools.

• Develop data acquisition methodologies to ingest, transform, validate, and integrate various enterprise data.

• Implement strategies for incremental loading, late-arriving data, processing-window, data-freshness, and lifecycle management.

• Automate manual tasks and enhance workflow efficiency, reliability, and scalability.

• Build, maintain, and optimize Databricks and enterprise data-lake solutions.

• Configure, monitor, manage, and document Databricks clusters in accordance with DON policies, standards, and security protocols.

• Create and sustain Spark-based processing solutions using PySpark, Spark SQL, Data Frames, and Data Sets.

• Implement and support Delta Lake and Delta Live Tables solutions.

• Enhance data processing, storage, and query performance.

• Provide support for Palantir Foundry ontologies, ETL/ELT pipelines, applications, user interfaces, and enterprise integrations.

• Develop and manage Kafka streaming solutions, topics, Schema Registry, Kafka Streams, and ksqlDB components.

• Create Python data-processing applications and AWS Lambda functions.

• Facilitate data integration and processing using AWS S3, Kinesis, Lambda, and DynamoDB.

• Monitor, troubleshoot, and optimize cloud-based data-processing solutions.

• Establish data quality controls, validation procedures, and quality gates.

• Design and execute data-driven and unit tests for Spark, Python, and other processing solutions.

• Set up data lifecycle policies addressing retention, backup, recovery, and data management.

• Oversee and troubleshoot pipelines and processing jobs, resolving quality, performance, and integration challenges.

• Enhance data-processing performance, reliability, scalability, and maintainability.

⛳️ Requirements

• Over 5 years of professional experience in data engineering.

• More than 5 years of IT experience emphasizing enterprise data engineering.

• Bachelor’s degree in computer science, engineering, mathematics, statistics, or a related IT field.

• Must be eligible to obtain and maintain a Secret security clearance.

• U.S. citizenship is required for Secret security-clearance eligibility.

• Preferred active CompTIA Security+ or CompTIA Network+ certification; the selected candidate must secure CompTIA Security+ prior to program support.

• Professional experience in data architecture, data engineering, data hubs, data lakes, and/or data warehouses.

• Experience in designing, developing, implementing, and supporting enterprise-scale data engineering solutions.

• Familiarity with data modeling, data mapping, data quality, data integration, and data management.

• Experience with ETL/ELT pipelines and data transformation processes, including SSIS, Pentaho, or AWS Data Migration Service.

• Hands-on experience with Databricks is essential.

• Proficiency in Python and SQL.

• Preferred experience with Apache Spark and/or PySpark, Spark SQL, Data Frames, and/or Data Sets.

• Experience with batch and streaming data processing architectures.

• Familiarity with Kafka and/or other distributed streaming technologies, including Kafka Streams.

• Experience with AWS services such as S3, Lambda, Kinesis, and/or DynamoDB.

• Proficient in Git-based version control, including branching, merging, pull requests, and repository management.

• Experience with CI/CD pipelines and cloud monitoring tools such as Jenkins, Docker, and/or CloudWatch preferred.

• Preferred experience with Palantir Foundry.

• Familiarity with the Jupiter data and analytics environment is a plus but not mandatory.

• Ability to debug, troubleshoot, design, and implement solutions for complex technical challenges.

• Capability to excel in a team-oriented environment.

• Experience in communicating technology solution benefits and constraints to partners, stakeholders, team members, and senior management.

• Strong written and verbal communication abilities.

🏝️ Benefits

• Equal opportunity employer that values diversity at all levels.

• Professional opportunity to work with enterprise data engineering, cloud, and analytics technologies.