Data and AI Engineer

This is a fully remote position, open to applicants in District of Columbia, +5 more states.

📋 Description

• Design, develop, and enhance scalable ETL/ELT pipelines utilizing Apache Spark and Delta Lake across AWS, Azure, and GCP.

• Implement Lakehouse architectures that process both structured and unstructured data at an enterprise level.

• Automate workflows for data ingestion, storage, and feature engineering to support analytics and real-time AI applications.

• Optimize Spark jobs to achieve low latency, high throughput, and cost efficiency in cloud environments.

• Operationalize AI models within Databricks using MLflow for effective tracking, versioning, and deployment.

• Establish CI/CD pipelines and Infrastructure-as-Code using Terraform and Databricks CLI.

• Ensure data security, access controls, and compliance standards within Unity Catalog and various cloud ecosystems.

• Collaborate with client technology leaders, product managers, and engineering teams to convert AI data requirements into tangible business outcomes.

• Create technical documentation and facilitate solution transitions to client engineering teams.

⛳️ Requirements

• Over 3 years of experience in cloud-native data engineering.

• More than 2 years of hands-on experience with Databricks, Apache Spark, Delta Lake, and MLflow.

• Strong proficiency in Python, SQL, Spark frameworks, CI/CD practices, and Infrastructure-as-Code (Terraform or Databricks CLI).

• Current Databricks Certified Data Engineer (Associate or Professional) or Databricks ML Professional certification is highly preferred.

• A demonstrated history of working in dynamic Agile/DevOps consulting settings.

• Experience in delivering cloud-agnostic architectures.

• Excellent client communication skills.

• Preference for candidates in the Eastern or Central Time Zone.

🏝️ Benefits

• Paid Time Off

• Remote-First Working Environment

• Comprehensive Health Coverage – Medical, Dental, Vision

• 401(k) Retirement Plan

• Annual Learning & Development Stipend

• Peer Mentorship & Coaching

• Professional Growth Opportunities

• Company Outings & Volunteer Opportunities

• Collaborative, Innovative Culture