Senior Software Engineer, Data Platform

This is a fully remote position, open to applicants in Poland.

📋 Description

• Design and develop internal SDKs and self-service frameworks that empower distributed engineering teams to autonomously ingest and transform data.

• Establish reusable patterns for both batch and real-time event processing.

• Take responsibility for the cost-effectiveness of the Databricks ecosystem.

• Optimize Spark execution plans and fine-tune shuffle partitions.

• Implement auto-scaling strategies to effectively manage DBU consumption.

• Maintain a balance between latency, throughput, and cloud expenditure as platform volume increases.

• Enforce Schema-on-Write validation and Data Contracts prior to data entering the Bronze layer.

• Collaborate with the Data Architect and Data Stewards on data privacy, security standards, and metadata lineage.

• Advocate for AI-assisted development tools, such as GitHub Copilot and Cursor.

• Guide engineers on best practices in distributed computing.

• Perform in-depth code reviews with an emphasis on scalability and maintainability.

⛳️ Requirements

• 5+ years of experience in building and managing production-grade data systems at a large scale.

• Extensive, hands-on expertise with the Databricks/Spark ecosystem, including Delta Lake, DLT, Spark UI debugging, and performance optimization.

• A proven history of constructing real-time/streaming architectures utilizing Spark Structured Streaming, Kafka, or Kinesis as a fundamental production requirement.

• Experience in managing and optimizing cloud costs within a rapidly growing environment.

• Proficiency in developing APIs, tools, or frameworks utilized by other internal engineering teams.

🏝️ Benefits

• Annual bonus for non-sales positions, awarded based on individual and company performance in addition to base salary.

• Generous paid parental leave.

• Flexible time off policy.

• Spending accounts available.

• Comprehensive medical insurance.

• Dental insurance provided.

• Vision insurance included.

• Sabbatical leave after 5 years of service.

• Flexible work arrangements through remote-first teams.

• Resources and flexibility available for remote-first employees.