Technical Lead, Spark

This is a fully remote position, open to applicants in Spain.

📋 Description

• Create innovative features for Cloudera's data engineering platform, guiding them from prototype stages to leading a team for production deployment at scale.

• Contribute to projects involving Apache Spark and Livy.

• Build new functionalities using Scala, Java, or Python on contemporary platforms.

• Acquire expertise in distributed data processing, including SQL planners and optimizers, data layout and table formats such as Apache Parquet and Iceberg, as well as fault tolerance in distributed environments.

• Develop a comprehensive understanding and deep technical knowledge of components within the Cloudera Data Engineering Experience stack, with an emphasis on Iceberg and Spark, for application in daily tasks.

• Engage with large-scale distributed systems, operating from hundreds to thousands of nodes in production clusters.

• Troubleshoot system-level deployment challenges, conduct root cause analysis, perform system test evaluations, and resolve failures.

• Work towards enhancing internal infrastructure.

• Collaborate effectively with team members and stakeholders.

⛳️ Requirements

• Bachelor's or Master's degree in a relevant field or equivalent experience.

• Over 6 years of professional experience in software development.

• Proven experience in leading and delivering intricate product enhancements.

• Strong proficiency in at least one of the following programming languages: Java, Scala, or Python.

• Experience in systems design and development.

• Excellent oral and written communication skills.

• Strong capability to research and independently resolve problems without continuous supervision.

• Open-minded with a willingness to learn new technologies and create exceptional products.

• Familiarity with distributed systems.

• Experience with SQL planners.

• Proficient in using or developing with Apache Spark, Livy, or other related technologies.

• Experience in designing and developing large-scale distributed systems, with knowledge of scaling, performance, and scheduling considerations.

• Strong experience with at least one cloud service.

🏝️ Benefits

• Generous PTO Policy.

• Support for work-life balance through Unplugged Days.

• Flexible Work From Home Policy.

• Programs for Mental & Physical Wellness.

• Reimbursement for Phone and Internet expenses.

• Access to Ongoing Career Development opportunities.

• Comprehensive Benefits and Competitive Packages.

• Paid Volunteer Time.

• Employee Resource Groups.