Staff Engineer – Databricks

This is a fully remote position, open to applicants in India.

📋 Description

• Design and manage scalable, dependable data platform architectures, encompassing data pipelines, storage solutions, and processing systems.

• Develop and execute data models, ensure data integrity, and establish policies for data governance.

• Construct and enhance data pipelines for the ingestion, processing, and transformation of extensive datasets.

• Identify and address performance bottlenecks within data pipelines and systems.

• Assess emerging data technologies and investigate opportunities for infrastructure innovation.

• Diagnose and resolve intricate data-related challenges.

• Implement data security protocols and ensure adherence to data governance policies.

• Safeguard sensitive information while maintaining the stability and reliability of the data platform.

• Carry out additional duties as assigned.

⛳️ Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline.

• Over 10 years of experience in data engineering or a comparable role.

• Proficiency with enterprise SaaS software solutions that feature high availability and scalability.

• Experience in managing large-scale structured and unstructured data from diverse data sources.

• Experience in building and maintaining data platform systems, including distributed computing, data orchestration, distributed storage, and streaming infrastructure.

• Collaboration with Product Engineering teams to influence data, AI, and analytics designs.

• Extensive experience in system design involving large petabytes of data using Databricks Lakehouse.

• Familiarity with modern AI/data infrastructure patterns and semantic layers for organizing data for AI agents.

• Experience with AWS, GCP, Snowflake, Airbyte, and Airflow.

• Proficient in programming with Python, SQL, and potentially Java or Scala.

• Experience with Kubernetes, CI/CD, infrastructure-as-code tools (preferably Terraform), observability, monitoring, and alerting.

• Expertise in solution cost optimization and design-to-cost strategies.

• Proven ability to drive initiatives for engineering excellence.

• Legally authorized to work in India on an ongoing basis.

• Proficiency in English is mandatory.

🏝️ Benefits

• Equal employment opportunity and an inclusive work environment.

• Freedom to explore, challenge limits, and grow beyond the current role.

• Interview accommodations available upon request.