Mid-Level Data Engineer – AWS, Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and manage data ingestion pipelines utilizing Databricks Autoloader.

• Execute incremental processing and handle schema evolution.

• Develop large-scale data processing solutions with Spark Structured Streaming and Spark SQL.

• Implement partitioning strategies, optimize processing, and utilize Delta Lake.

• Evaluate and enhance the performance and cost-efficiency of Databricks environments.

• Enforce data quality standards and practices.

• Utilize Git and participate in pull request workflows.

• Create technical documentation for the solutions developed.

• Collaborate with business units and Analytics teams to ensure data accessibility, structure, and security.

• Build and refine scalable, efficient, secure, and high-performance data solutions.

⛳️ Requirements

• Experience in data engineering and handling large data volumes.

• Proficient in Python and Apache Spark.

• Familiarity with Databricks.

• Expertise in creating and sustaining data ingestion pipelines.

• Experience with Spark Structured Streaming and Spark SQL.

• Knowledge of Delta Lake and techniques for processing optimization.

• Proficient in Git and version control systems.

• Understanding of data quality practices.

• Preferred: familiarity with modern data architectures and ETL/ELT methodologies.

• Preferred: experience with agile practices such as Scrum and Kanban.

• Preferred: experience in optimizing costs and performance in scalable data settings.

• Analytical, self-motivated, and adept at problem-solving.

• Willingness to learn new technologies and suggest innovative solutions.

• Strong communication skills for both technical and non-technical audiences.

• Ability to translate business requirements into effective, scalable solutions.

• Capability to influence decisions and establish trusted relationships with stakeholders.

🏝️ Benefits

• Remote work environment.

• Opportunities to enhance skills and technical acumen.

• Collaboration with global teams across various regions.

• Emphasis on diversity and multiculturalism.

• Commitment to knowledge sharing and continuous improvement.