Mid-Level Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop and sustain data ingestion and transformation pipelines that integrate APIs, relational databases, files, and data lakes.

• Create solutions utilizing Python, SQL, and PySpark with a focus on organized, reusable, and testable coding practices.

• Structure data within Delta Lake, implementing incremental loads, MERGE operations, and adhering to maintenance best practices.

• Execute Medallion architectures (Bronze, Silver, and Gold) alongside data quality regulations.

• Manage job orchestration and pipelines by configuring dependencies, parameters, retries, and alerts.

• Oversee executions, troubleshoot failures, and assess data quality and performance issues.

• Enhance queries, processing efficiency, cluster utilization, and cost-effectiveness, while measuring the outcomes of enhancements.

• Enforce governance and security protocols, including access control and protection of sensitive data.

• Collaborate in code reviews, testing, version control, and deployments across various environments.

⛳️ Requirements

• Proven experience with Python, advanced SQL, and PySpark.

• Background in constructing and managing data pipelines within Databricks.

• Understanding of Delta Lake, covering transactions, schema evolution, incremental loads, and update operations.

• Proficient in batch processing; knowledge of streaming is an added advantage.

• Familiarity with Medallion architecture and file formats like CSV, JSON, and Parquet.

• Experience in job orchestration, monitoring, and handling failures effectively.

• Comfortable with Git, code reviews, and testing procedures.

• Capable of investigating issues, articulating technical decisions, and working autonomously.

• Experience with Auto Loader, Structured Streaming, and declarative pipelines.

• Knowledge of Unity Catalog, data lineage, auditing, and permission management.

• Experience with cloud platforms such as AWS, Azure, or Google Cloud.

• Familiarity with CI/CD, infrastructure as code, and Databricks Asset Bundles or similar tools.

• Possession of a Databricks Certified Data Engineer Associate certification.

🏝️ Benefits

• Meal allowance or food voucher.

• Discounts on educational courses, universities, and language institutions.

• Access to Stefanini Academy — a platform offering free, current online courses with certification.

• Mentorship opportunities.

• Benefits club providing discounts on medical consultations and examinations.

• Health insurance coverage.

• Dental insurance coverage.

• Various benefits and discounts at prominent establishments.

• Travel club membership.

• Pet care benefits.