Databricks Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Report to the Lead of the Data & AI Practice.

• Design, construct, and enhance scalable ETL/ELT pipelines utilizing Apache Spark and Delta Lake across AWS, Azure, and GCP.

• Implement enterprise-level Lakehouse architectures for both structured and unstructured data.

• Automate workflows for data ingestion and storage to facilitate analytics and real-time operational reporting.

• Optimize Spark jobs for minimal latency, high throughput, and efficient cloud costs.

• Set up CI/CD pipelines and Infrastructure-as-Code using Terraform and Databricks CLI.

• Develop frameworks for monitoring, alerting, and validating data quality.

• Collaborate with ML engineers and data scientists on production-ready feature engineering pipelines.

• Assist with model training, tracking, versioning, and deployment in Databricks utilizing MLflow.

• Operationalize AI/ML models within secure and scalable production settings.

• Ensure data security, access control, and compliance with standards including GDPR, HIPAA, and SOC 2.

• Establish practices for data lineage, metadata management, and operational documentation.

• Work together with client-facing stakeholders to ensure technical implementations align with business objectives.

⛳️ Requirements

• 3+ years of practical experience in data engineering, emphasizing big data processing and cloud-native architectures.

• 2+ years of hands-on experience with Databricks, including Apache Spark, Delta Lake, and MLflow.

• Databricks Certified Data Engineer Associate (or higher) certification is required.

• Proficiency in Python, SQL, and Spark-based frameworks.

• Experience in developing and optimizing extensive ETL/ELT pipelines.

• Strong comprehension of Lakehouse architecture and cloud-agnostic data solutions.

• Familiarity with CI/CD pipelines and Infrastructure-as-Code for Databricks, including Terraform and Databricks CLI.

• Knowledge of data governance, security, and compliance best practices.

• Experience in Agile development settings, adhering to DevOps/MLOps best practices.

• Additional Databricks certifications, real-time streaming solutions, cloud storage/orchestration tools, AI/ML integration, and experience in client-facing or consulting roles are advantageous.

🏝️ Benefits

• Paid Time Off – Take the time you need to recharge and stay productive.

• Remote-First Working Environment – Collaborate from anywhere while staying connected with our global team.

• Comprehensive Health Coverage – Medical, Dental, Vision.

• 401(k) Retirement Plan – Plan for your future with access to a company-sponsored 401(k) program.

• Annual Learning & Development Stipend – Invest in your skills with conferences, certifications, or courses.

• Peer Mentorship & Coaching – Learn from experienced engineers, product managers, and architects to accelerate your growth.

• Professional Growth Opportunities – Exposure to cutting-edge AWS GenAI, data, and cloud technologies across diverse industries.

• Company Outings & Volunteer Opportunities – Build relationships and give back to the community.

• Collaborative, Innovative Culture – Work alongside top talent in a fast-paced, supportive environment that values curiosity and initiative.