Senior Data Engineer – Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and execute data engineering solutions utilizing Databricks.

• Construct, sustain, and enhance scalable pipelines for Batch, Micro-batch, and Streaming processing.

• Implement solutions using Apache Spark, Delta Lake, Delta Live Tables (Lakeflow Declarative Pipelines), Unity Catalog, and Apache Iceberg.

• Create YAML-driven pipelines.

• Enable incremental ingestion and automated processing through Databricks Auto Loader.

• Develop and oversee workflows and orchestrations via Databricks Jobs/Workflows, Apache Airflow, or dbt.

• Ensure data quality, integrity, traceability, and observability.

• Implement auditing, monitoring, and validation processes for pipelines.

• Engage in technical design and architecture discussions.

• Convert complex requirements into structured technical tasks, ensuring high-quality and timely delivery.

• Collaborate with Solution Architects and cross-functional teams to deploy customer data platforms.

• Contribute to the Databricks Center of Excellence by developing accelerators, reusable components, and technical enhancements.

• Monitor trends and new features in the Databricks ecosystem, fostering continuous innovation.

⛳️ Requirements

• Extensive experience in Data Engineering, with a minimum of 2 years of practical experience in developing and designing pipelines using Databricks.

• Expertise in pipeline development and data processing within distributed environments.

• Practical experience with Databricks, Delta Lake, Delta Live Tables (Lakeflow Declarative Pipelines), Unity Catalog, Apache Iceberg / Uniform, and Databricks Auto Loader.

• Experience in constructing scalable, configuration-driven YAML pipelines.

• Understanding of Batch, Micro-batch, and Streaming ingestion and processing methodologies.

• Familiarity with Databricks Jobs/Workflows, Apache Airflow, or dbt.

• Knowledge of data quality, auditing, and observability, including tools such as Databricks, Great Expectations, Collibra, or dbt.

• Proficient in Apache Spark, Presto, and Amazon EMR.

• Experience with Amazon Redshift, Snowflake, and Google BigQuery.

• Strong command of SQL, Stored Procedures, and object-oriented programming.

• Experience with Python, PySpark, or Java.

• Familiar with DevOps and CI/CD practices, including Databricks Asset Bundles, automated validation, and testing.

• Knowledge of version control, automation, and deployment of data solutions.

• Analytical, inquisitive, and focused on resolving complex issues.

• Technical independence and the capability to lead tasks with minimal oversight.

• Comfortable collaborating in cross-functional teams.

• Ability to decompose complex technical challenges into organized deliverables.

• Proactive in identifying opportunities for enhancements and innovations.

• Excellent communication skills with the ability to impart knowledge to the team.

• Dedicated to quality, efficiency, and meeting project deadlines.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Flexible working hours and remote work options.

• Opportunities for professional development and training.

• Comprehensive health, dental, and vision insurance.

• Retirement savings plan with employer contributions.

• A dynamic and inclusive work environment.