Senior Data Developer, Azure, Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop, maintain, and enhance data pipelines utilizing Databricks, PySpark, Python, and SQL to convert raw financial data into dependable curated datasets.

• Incorporate new data sources into the current data ecosystem, aiding in the expansion of the platform’s capabilities and enhancing business visibility.

• Assist with financial reconciliation tasks, P&L processes, and global closing activities by ensuring consistency, accuracy, and traceability of data across various sources.

• Manage large volumes of structured and semi-structured data, optimizing transformations, storage formats, and query performance employing Delta Lake and Lakehouse best practices.

• Oversee, troubleshoot, and support existing pipelines, identifying problems, investigating discrepancies, and ensuring reliable daily and monthly processing.

• Suggest and implement improved technical strategies to enhance performance, scalability, maintainability, and data quality throughout the financial data ecosystem.

• Collaborate closely with current data engineers, business stakeholders, finance teams, and technology teams to comprehend needs and deliver high-quality data solutions.

• Facilitate data governance, validation rules, auditability, documentation, and controls to ensure trustworthy and well-managed financial data.

• Aid initiatives related to AI models and advanced data capabilities, contributing to the integration of AI-driven insights and automation into data workflows.

⛳️ Requirements

• Extensive experience with Databricks, encompassing notebooks, jobs, workflows, clusters, and best practices for developing scalable data solutions.

• Expertise in data processing using PySpark, Python, and SQL, with practical experience in transforming raw data into curated and reliable datasets.

• Familiarity with Delta Lake and Lakehouse architecture, including Delta tables, incremental processing, data optimization, and structured data layers.

• Understanding of the Azure data ecosystem, particularly Azure Data Lake Storage, data integration patterns, and cloud-based data processing.

• Experience in supporting financial data pipelines, ideally involving reconciliation, P&L, financial closing, or other essential business processes.

• Practical experience in maintaining and enhancing existing pipelines, including troubleshooting, performance tuning, documentation, and production support.

• Capability to work with structured and semi-structured data, ensuring data quality, traceability, and readiness for analytical and business use.

• Problem-solving mindset, able to delve into complex data, identify root causes, and offer effective solutions in challenging data environments.

• Interest or experience with AI models and advanced data capabilities, facilitating the integration of AI-driven solutions into data workflows.

• Proficiency in English for collaboration with global teams, technical discussions, and documentation.

• Nice to Have:

• Familiarity with AI frameworks and methodologies.

• Experience with advanced capabilities, such as Delta Lake optimization, Databricks Workflows, and Databricks Apps or Streamlit.

• Experience with data quality and observability practices, including validation rules, reconciliation checks, monitoring, logs, alerts, and pipeline execution metrics.

🏝️ Benefits

• Health and dental insurance

• Meal and food allowance

• Childcare assistance

• Extended paternity leave

• Partnership with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass;

• Profit Sharing and Results Participation (PLR);

• Life insurance

• Continuous learning platform (CI&T University);

• Discount club

• Free online platform dedicated to physical, mental, and overall well-being

• Pregnancy and responsible parenting course

• Partnerships with online learning platforms

• Language learning platform