Senior Data Engineer, Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Provide technical guidance with minimal supervision to create innovative solutions on the Databricks Platform.

• Utilize cloud data lakehouse methodologies, distributed processing engines, event streaming platforms, and contemporary data technologies.

• Engage in the Databricks Center of Excellence.

• Construct next-generation data platforms for organizations focused on data and analytics.

• Operate under the guidance of a Solution Architect to design and implement components of client data platform solutions.

• Take part in design sessions, decompose complex development tasks, and ensure timely completion of development items.

• Contribute to Center of Excellence initiatives by creating Databricks solution accelerators.

• Design and implement data solutions utilizing Databricks and its platform features.

• Collaborate with cross-functional teams to design and enhance batch and streaming data pipelines.

• Ensure the quality of data and the efficiency of pipelines.

• Stay updated with emerging technologies and features of the Databricks platform, continuously improving relevant skills.

⛳️ Requirements

• A minimum of 2 years of practical experience in designing and developing data pipelines with Databricks.

• Familiarity with Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.

• Expertise in designing and constructing robust, scalable, YAML configuration-driven data pipelines.

• Experience in batch, micro-batch, and streaming data ingestion and processing methods using tools like Auto Loader.

• Proven experience in building complex job/workflow orchestration patterns with Databricks Jobs/Workflows or tools such as Airflow and dbt.

• Knowledge of data quality, pipeline auditing, and observability solutions using Databricks-native features or tools/frameworks such as Great Expectations, Collibra, and dbt.

• Proficiency with Apache Spark, Presto, and Amazon EMR.

• Experience with Amazon Redshift, Snowflake, and Google BigQuery.

• Strong programming capabilities in SQL, stored procedures, and object-oriented programming languages such as Java, Python, and PySpark.

• Familiarity with DevOps and CI/CD pipelines utilizing Databricks Asset Bundles, automated validation, and testing.

• Exposure to Infrastructure as Code tools like Terraform is advantageous.

• Basic understanding of NoSQL databases and container management systems.

• Familiarity with AI/ML tools such as MLflow, prompt engineering, and modern data and AI agentic workflows.

• Databricks Data Engineering Associate and/or Professional certification along with experience in multiple Databricks project deliveries is preferred.

🏝️ Benefits

• Equal employment opportunity and protections against discrimination.

• A work environment that is free from unlawful discrimination and harassment.