Senior Data Engineer – Full Stack

This is a fully remote position, open to applicants in India.

📋 Description

• Collaborate closely with both business and technical stakeholders to grasp workflows, articulate desired outcomes, and convert vague requirements into actionable technical specifications and delivery strategies.

• Design, construct, and sustain comprehensive data products on Databricks, encompassing data ingestion, Delta Lake storage, transformation, serving, APIs, and user-centric experiences.

• Create and maintain dependable batch, incremental, and streaming data pipelines utilizing SQL, Python, PySpark, Kafka, and Databricks-native functionalities.

• Architect event-driven and near-real-time data solutions that integrate operational systems with downstream consumers.

• Develop backend services, APIs, and integrations to provide governed data for applications and operational workflows.

• Build lightweight applications, dashboards, and interfaces in collaboration with product, analytics, BI, and user-experience teams.

• Quickly prototype solutions, gather feedback from stakeholders, and prepare effective concepts for production implementation.

• Design scalable data models and curated datasets to support analytics, reporting, AI/ML, and operational decision-making.

• Apply data quality, security, lineage, and governance controls utilizing Databricks and Unity Catalog.

• Set up automated testing, CI/CD, monitoring, alerting, and documentation throughout the data product lifecycle.

• Enhance pipelines, streaming workloads, queries, services, and applications for reliability, performance, scalability, and cost efficiency.

• Identify and resolve issues across source systems, streaming platforms, pipelines, data models, APIs, applications, and downstream consumers.

• Work in partnership with platform and product engineering teams to transform recurring stakeholder needs into reusable capabilities.

• Lead technical design and code reviews, mentor engineers, and promote advanced full-stack data engineering practices.

⛳️ Requirements

• Over 5 years of experience in data engineering, software engineering, or a related area, including responsibility for production data solutions.

• Strong expertise in SQL, Python, and PySpark, with a proven track record of developing reliable, production-quality pipelines and data products.

• Practical experience with Databricks, Apache Spark, Delta Lake, and Unity Catalog or similar data governance tools.

• Familiarity with designing and managing streaming or near-real-time data pipelines using Kafka, Kinesis, Event Hubs, or analogous event-streaming technologies.

• Solid understanding of event-driven architecture, message processing, schema evolution, data consistency, and the reliability of streaming solutions.

• Experience in delivering full-stack solutions that encompass data pipelines, backend services or APIs, and lightweight user-facing applications.

• Proficient in creating REST APIs, services, and integrations utilizing Python frameworks such as FastAPI, Flask, or equivalent technologies.

• Knowledge of AWS, Azure, or GCP along with cloud-native architectural patterns.

• Strong grasp of data modeling, data warehousing, distributed processing, and design principles that favor analytics.

• Experience with Git, automated testing, CI/CD, monitoring, and best practices for production deployment.

• Proven ability to engage directly with stakeholders, manage ambiguity, and translate business challenges into feasible technical solutions.

• Excellent communication, technical leadership, problem-solving, and end-to-end ownership abilities.

• Capability to balance swift delivery with maintainability, security, governance, and operational reliability.

• Preferred qualifications include expertise in React or another modern frontend framework; infrastructure as code, containerization, and automated cloud deployment; AI/ML pipelines, feature engineering, retrieval systems, or generative AI; data observability, platform engineering, or data product management; JavaScript or TypeScript; experience in forward-deployed engineering, solutions engineering, technical consulting, or stakeholder-embedded delivery; reusable data platforms; mentoring engineers; and Agile or Scrum methodologies.

🏝️ Benefits

• Generous time off policies

• Comprehensive benefits package

• Support for education, wellness, and lifestyle initiatives