Senior Data Engineer – Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of Databricks production support for the Sugar Predict data platform, which involves monitoring, alerting, and responding to incidents across all production data workflows.

• Maintain and report on SLA performance metrics for data pipeline delivery, ensuring stakeholders have visibility into platform health and accountability.

• Identify and implement optimizations in the pipeline that minimize Databricks compute costs, enhance throughput, and shorten processing windows, all while tracking impacts through measurable KPIs.

• Migrate legacy ETL/ELT pipelines to Databricks, creating automation tools to minimize manual intervention and guarantee seamless data delivery during transitions.

• Assist in onboarding new customers by provisioning, validating, and strengthening tenant data pipelines that ensure reliable and isolated data from day one.

• Design and construct high-performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale across both Azure and AWS environments.

• Manage the Delta Lake architecture, including schema design, partitioning strategies, data quality enforcement, and incremental processing patterns.

• Implement best practices for data security across Databricks environments, focusing on role-based access control, secrets management, and compliance requirements for enterprise CRM and ERP data.

• Establish data quality monitoring and observability across pipeline health and ML model inputs, ensuring data integrity that directly impacts the accuracy of Sugar Predict predictions.

• Enforce multi-tenant data isolation patterns to ensure reliable and secure data delivery to Sugar Predict enterprise customers.

• Collaborate with the Enterprise Architecture team to ensure that Sugar Predict data pipelines integrate seamlessly with the wider SugarAI product ecosystem.

• Support a globally distributed operation through on-call rotation and after-hours incident response, meeting SLAs across different time zones.

• Maintain technical documentation, runbooks, and architectural decision records, contributing to knowledge sharing and operational readiness for on-call and incident response scenarios.

• Apply CI/CD best practices to the development of data pipelines, including version control, automated testing, and deployment tooling to ensure reliable and repeatable delivery.

⛳️ Requirements

• Over 4 years of experience in data engineering.

• A minimum of 2 years working with Databricks or the Apache Spark ecosystem in Azure and/or AWS.

• Proficiency in PySpark, SQL, and Python, with a demonstrated history of building and managing production-grade pipelines within SLA constraints.

• Practical experience with Delta Lake, including schema evolution, ACID transactions, optimize/vacuum lifecycle, and both incremental and streaming processing patterns.

• Hands-on experience in pipeline performance tuning and compute optimization within production Databricks environments.

• Strong working knowledge of PostgreSQL, including query optimization, schema design, and its use as a source or sink in production data pipelines.

• Experience in supporting and maintaining legacy ETL tools (SSIS, Informatica, custom Python/SQL pipelines, or similar) in a production setting.

• Experience with large-scale multi-tenant architectures, focusing on tenant isolation, per-tenant performance, and data privacy, including navigating tools and platforms that typically assume single-tenant environments.

• Proven ability to collaborate effectively across data science, product, and infrastructure teams, taking ownership of end-to-end delivery in a cross-functional setting.

• Strong understanding of data governance, security, and compliance principles, including access control, data privacy, and the protection of sensitive enterprise data across multi-tenant environments.

🏝️ Benefits

• Excellent healthcare package for you and your family.

• Savings and Investment – 401(k) match.

• Unlimited Paid Time Off.

• Paid Parental Leave.

• Online Legal Services (Rocket Lawyer).

• Financial Planning Services (Origin).

• Discounted Pet Insurance (Embrace Pet Insurance).

• Corporate Benefit Program (Working Advantage), offering exclusive travel and entertainment deals along with special discounts unavailable to the general public.

• Health and Wellness Reimbursement Program.

• Travel Discounts.

• Educational Resources - Career & Personal Development Program.

• Employee Referral Bonus Program.

• We are a merit-based company, providing numerous opportunities to learn, excel, and advance your career!