Data Engineer, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Create, develop, and sustain scalable data pipelines utilizing Databricks, PySpark, SQL, Delta Lake, and other cloud-native data engineering tools.

• Build and maintain batch and streaming data workflows, which encompass ingestion, transformation, validation, and publishing of curated data products.

• Write, enhance, and manage Python, PySpark, and SQL code for data processing, orchestration, and performance optimization.

• Handle large-scale datasets leveraging Databricks, Spark, Delta Lake, Unity Catalog, and cloud storage solutions.

• Diagnose and resolve failures in data pipelines, performance challenges, data quality concerns, and workflow slowdowns.

• Convert business and technical requirements into data engineering designs, processing scripts, and job orchestration workflows.

• Conduct data validation, quality assessments, code reviews, and issue resolution to ensure the accuracy and reliability of data products.

• Collaborate with solution architects, data scientists, analysts, DevOps engineers, and client stakeholders.

• Articulate technical concepts, impacts of delivery, and data engineering considerations to both technical and non-technical audiences.

• Document pipelines, datasets, data models, workflows, and engineering choices.

• Adhere to data governance, security, lineage, and compliance standards within the Databricks platform.

⛳️ Requirements

• Bachelor's degree in computer science, engineering, mathematics, statistics, or a related field.

• 3-8 years of pertinent experience in data engineering, data architecture, or the implementation of cloud data platforms.

• Extensive experience with Python, PySpark, and SQL for data transformation, pipeline creation, and data processing.

• Proficiency with Databricks, Spark, Delta Lake, Unity Catalog, or other similar cloud-native data platforms.

• Experience in developing batch or streaming data pipelines, ETL/ELT workflows, and reusable data assets.

• Familiarity with data modeling, data warehousing, data quality validation, and large-scale data processing principles.

• Capability to troubleshoot technical problems, clearly communicate engineering recommendations, and collaborate effectively in team-oriented delivery settings.

• Experience implementing data governance, access control, lineage, and security practices within cloud data platforms.

• 2+ years of hands-on experience with the Databricks platform preferred as a nice-to-have.

• Familiarity with Unity Catalog, medallion data architectures, CI/CD, Azure, AWS, GCP, streaming frameworks, orchestration tools, Terraform, or data platform administration capabilities preferred or considered a nice-to-have.

• Active Databricks Data Engineer Associate, Databricks Data Engineer Professional, or related certification preferred as a nice-to-have.

🏝️ Benefits

• Competitive compensation.

• Flexible benefits package.

• Diverse and supportive workplace.

• Equal Opportunity Employer.

• Reasonable accommodation support for applicants with disabilities.