Senior Data Engineer, Databricks

This is a fully remote position, open to applicants in Poland, +5 more countries.

📋 Description

• Establish and uphold best practices and coding standards throughout the project.

• Perform detailed code reviews to ensure high standards of code quality.

• Operate both independently and collaboratively within the team.

• Provide clear guidance to team members and assist in managing daily operations.

• Maintain regular communication with the client.

• Design, develop, and sustain robust and scalable Spark applications.

• Produce clean, maintainable, and efficient code.

• Enhance code for optimal performance and scalability.

• Collaborate with cross-functional teams to deliver superior software solutions.

• Identify and address technical challenges to guarantee application reliability and performance.

• Develop and maintain thorough documentation for code, processes, and workflows.

⛳️ Requirements

• Over 5 years of practical experience in software development.

• Hands-on experience in developing SQL stored procedures and transitioning them into Spark SQL or PySpark environments.

• In-depth knowledge of using PySpark on Databricks, including Delta tables, cluster management, and workflow automation.

• Demonstrated success in optimizing Spark performance in production-level projects.

• Strong proficiency in Python for advanced data manipulation using libraries such as Polars or Pandas.

• Comprehensive understanding of columnar data storage formats, especially Parquet, with practical experience in Delta Tables.

• Proven expertise in data processing, analysis, and transformation workflows.

• Excellent analytical and problem-solving skills with a keen attention to detail.

• Practical and flexible approach to balancing standardized processes with the adaptability required to meet project objectives.

• Strong understanding of microservices architecture and its application in scalable systems.

• Advanced proficiency in English.

• Nice to have: experience with Azure Cloud services or other major cloud platforms, including Service Bus, Data Lake, Blob Storage, Redis, and more.

• Nice to have: familiarity with FastAPI.

• Nice to have: expertise in Docker and Kubernetes.

🏝️ Benefits

• B2B contract with Bonapolia.

• Monthly invoice payments, typically aligned with the client's payment schedule to Bonapolia (usually within 30 days).

• Contract governed by the laws of the candidate's country.

• Bonapolia manages client-side documentation including MSA, NDA, and invoicing.

• Fully remote work opportunity.