Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and sustain data pipelines (ETL/ELT) on Azure, utilizing Databricks for processing and integration.

• Maintain a comprehensive viewpoint (e.g., grasp the project in its entirety and the effects a specific solution might have on the overall project).

• Implement Continuous Integration/Continuous Deployment (CI/CD) practices to automate deployments and handle code versioning in GitHub.

• Create and enhance data models to guarantee scalability, performance, and governance.

• Assist with cloud migration initiatives, ensuring data security, quality, and integrity.

• Collaborate with business and data science teams to deploy AI models in the cloud environment.

• Ensure data quality and consistency through the application of data profiling and monitoring techniques.

• Document workflows, architectures, and best practices in data engineering.

⛳️ Requirements

• Extensive experience with Azure Data Services and Databricks.

• Proficient in Databricks using PySpark.

• Experience in productionizing Machine Learning models on Databricks (MLOps).

• Familiarity with near real-time data processing technologies (Kafka).

• Strong skills in Python and SQL for data manipulation and transformation.

• Understanding of CI/CD and version control practices using GitHub.

• Experience in data modeling (conceptual, logical, and physical).

• Background in cloud migration projects.

• Knowledge of ETL/ELT processes and handling large data volumes.

• Experience in deploying AI models and utilizing tools like Azure ML.

• Familiarity with AI projects and automation of data pipelines.

• Experience with SAS.

• Differentiators:

• Understanding of data-driven architectures and governance best practices.

• Involvement in Master Data Management (MDM) initiatives.

• Familiarity with Databricks MLflow for model management.

• Experience in Data Lake and Data Warehouse environments.

🏝️ Benefits

• Health and dental plan;

• Meal and food allowance;

• Childcare subsidy;

• Extended parental leave;

• Partnerships with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass;

• Profit sharing (PLR);

• Life insurance;

• Continuous learning platform (CI&T University);

• Discount club;

• Free online platform dedicated to promoting physical and mental health and well-being;

• Maternity and parenting education course;

• Partnerships with online course platforms;

• Language learning platform;

• And many more.