Data Infrastructure Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and manage batch and streaming data ingestion pipelines sourced from APIs, relational databases, file transfers, event streams, and external partners.

• Create and enhance ETL/ELT pipelines that yield curated datasets for reporting and machine learning applications.

• Implement incremental processing, change data capture, and data contract standards effectively.

• Construct and oversee a scalable AWS lakehouse utilizing object storage alongside modern table and file formats.

• Optimize the performance and cost of the lakehouse through partitioning, compaction, lifecycle policies, and efficient compute/storage utilization.

• Establish standards for development, testing, and production environments, as well as manage promotion across these stages.

• Implement managed metadata repositories, including dataset cataloging, ownership assignments, glossary definitions, tagging, and discoverability features.

• Enable comprehensive data lineage to support auditability and impact assessments.

• Enforce governance controls pertaining to access, classification, retention, and secure data handling practices.

• Design data quality checks and publish Service Level Agreements/Service Level Objectives.

• Automate AWS provisioning through Infrastructure as Code methodologies.

• Develop and enhance CI/CD processes for data pipelines, encompassing tests, validation gates, promotion workflows, and rollback strategies.

• Improve observability with metrics, logs, alerts, dashboards, runbooks, and readiness for incident response.

• Collaborate effectively with engineering, security, networking, and application teams.

• Maintain standard operating procedures, system diagrams, and secure configuration baselines.

• Present findings and recommendations to both technical and non-technical stakeholders.

⛳️ Requirements

• Must be eligible to obtain and retain a Federal or DoD Public Trust clearance.

• Approved Public Trust adjudication is required prior to onboarding with Guidehouse.

• Active Public Trust or suitability clearance is preferred.

• Bachelor’s degree in Engineering, IT, Computer Science, or a related field, or equivalent experience is required.

• A minimum of six years of experience in building production data pipelines and/or data platforms is essential.

• Significant experience with data ingestion, ETL/ELT workflows, data modeling, and best practices for data transformation is necessary.

• Practical experience in constructing a data lake or delta lake lakehouse on AWS or a similar cloud platform, utilizing object storage and modern table formats/patterns is required.

• Proficiency in SQL and at least one data-engineering programming language; Python is preferred, with Scala or Java being acceptable alternatives.

• Experience with metadata management and governance, including cataloging, lineage, ownership, access controls, classification, and policy enforcement is necessary.

• Experience in implementing automated AWS provisioning using Infrastructure as Code across various environments is required.

• Familiarity with building or operating CI/CD pipelines for data workflows is essential.

• Strong foundation in security principles, including IAM/least privilege, encryption, secrets management, and secure software development lifecycle practices is a must.

• Experience with Databricks is preferred.

• Modern DevOps experience with tools such as Git, Terraform, Jenkins, AWS CodePipeline, and Docker is preferred.

• Familiarity with AI-assisted coding tools is preferred.

• Experience with knowledge graphs and Graph RAG, including graph modeling, ontology/taxonomy alignment, entity resolution, relationship extraction, and hybrid retrieval is preferred.

🏝️ Benefits

• Medical, Rx, Dental & Vision Insurance.

• Personal and Family Sick Time & Company Paid Holidays.

• Parental Leave.

• 401(k) Retirement Plan.

• Group Term Life and Travel Assistance.

• Voluntary Life and AD&D Insurance.

• Health Savings Account.

• Health Care & Dependent Care Flexible Spending Accounts.

• Transit and Parking Commuter Benefits.

• Short-Term & Long-Term Disability.

• Tuition Reimbursement, Personal Development, Certifications & Learning Opportunities.

• Employee Referral Program.

• Corporate Sponsored Events & Community Outreach.

• Care.com annual membership.

• Employee Assistance Program.

• Supplemental Benefits via Corestream (Critical Care, Hospital Indemnity, Accident Insurance, Legal Assistance, and ID theft protection, etc.).

• Position may qualify for a discretionary variable incentive bonus.

• Flexible benefits package.