Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the design and execution of intricate data ingestion pipelines using PySpark, Databricks Intelligence Platform, and AWS services (S3, IAM roles and policies), incorporating Kafka/streaming architectures, schema evolution, and Delta Share, while ensuring adherence to enterprise architecture standards.

• Act as a subject matter expert (SME) in data engineering, offering technical guidance and mentorship to team engineers and advocating for data engineering in cross-functional discussions.

• Assume responsibility for platform reliability and performance by proactively identifying optimization opportunities across the data ingestion, transformation, and storage layers.

• Diagnose complex pipeline failures involving multiple system dependencies.

• Lead cross-functional technical projects, collaborating with data scientists, analysts, DevOps engineers, client teams (upstream data providers and downstream consumers), and stakeholders to deliver integrated solutions that fulfill program goals.

• Contribute to the technical vision and architecture of the data platform, making strategic decisions regarding tooling, frameworks, and design patterns that influence the long-term direction of data engineering for the program.

• Establish and uphold engineering standards, including code quality, testing practices, CI/CD processes, and infrastructure as code patterns using Terraform.

• Evaluate new technologies and methodologies, presenting proofs of concept and technical roadmaps to leadership.

• Create and maintain architectural documentation, technical decision records, and platform runbooks that enhance team autonomy and operational excellence.

• Perform other related duties as assigned and for which qualified/trained.

⛳️ Requirements

• Bachelor’s degree.

• At least 5 years of experience in Data Engineering.

• Proven experience acting as a technical SME and leading cross-functional engineering projects.

• Strong proficiency in Python for large-scale data engineering tasks.

• In-depth expertise with the Databricks Intelligence Platform (Notebooks, Lakeflow Jobs, Unity Catalog, Delta Share) and the capability to establish platform standards and best practices for the team.

• Experience with AWS services such as S3 and IAM roles and policies.

• Solid understanding of data pipeline design and implementation, including data transformation techniques, data modeling, data storage optimization, and data security best practices.

• Proficient in utilizing version control systems like Git for managing code repositories and collaborating with team members on Agile projects.

• Familiarity with Terraform or other infrastructure as code tools for automating infrastructure deployment and configuration management.

• Experience in Linux environments for data engineering projects, including remote access to containers, package installation, and management of files, services, and processes.

• Advanced SQL skills and experience with relational databases, including query authoring (SQL) and familiarity with various database systems.

• Experience conducting root cause analysis on internal and external data and processes to address specific business questions and identify areas for enhancement.

• Strong problem-solving abilities, capable of working independently and collaboratively, along with excellent verbal and written communication skills.

• Comfortable in a highly collaborative environment, with a keen eye for detail and a commitment to delivering high-quality software.

• Eligibility to obtain and maintain a Public Trust security clearance.

🏝️ Benefits

• Flexible working hours

• 401K with matching incentive

• Parental Leave

• Medical/dental/vision benefits

• Flexible Spending Account

• Company-provided short-term disability and life insurance

• Commuter benefits

• Paid Time Off (PTO)

• 11 Paid holidays