Senior Software Engineer, Data Platform

This is a fully remote position, open to applicants in India.

📋 Description

• Lead the design and execution of internal SDKs and self-service frameworks.

• Transition from "pipeline construction" to "platform engineering," developing reusable patterns for both batch and real-time event processing.

• Take complete responsibility for the cost-effectiveness of the Databricks ecosystem.

• Ensure the platform maintains high performance as data volume increases.

• Implement Schema-on-Write validation and Data Contracts to guarantee that data adheres to strict quality standards.

• Collaborate with the Data Architect and Data Stewards to uphold data privacy and security standards.

• Advocate for the utilization of AI-assisted development tools to speed up the engineering lifecycle.

• Mentor engineers on best practices in distributed computing.

⛳️ Requirements

• Over 5 years of experience in building and managing production-grade data systems at a large scale.

• Extensive, hands-on expertise in the Databricks/Spark ecosystem (including Delta Lake, DLT, Spark UI debugging, and performance optimization).

• Proven experience in constructing Real-time/Streaming architectures (using Spark Structured Streaming, Kafka, or Kinesis) as a fundamental production requirement.

• Experience in managing and optimizing cloud expenditures in a rapidly growing environment.

• Background in building APIs, tools, or frameworks used by other internal engineering teams.

🏝️ Benefits

• Generous paid parental leave.

• Flexible time off.

• Spending accounts.

• Medical insurance.

• Dental insurance.

• Vision insurance.

• Sabbatical after 5 years.

• Annual bonus eligibility based on performance.