Senior Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Develop and manage central data platform functionalities using Databricks and Snowflake.

• Architect, construct, and sustain scalable batch and streaming ELT/ETL pipelines along with extensive datasets.

• Create AI agents and agent-driven data products utilizing Databricks, Snowflake, LangGraph, and FastMCP.

• Execute automated data quality assessments, comprehensive lineage tracking, and RBAC security protocols.

• Oversee and enhance compute and storage expenditures for Databricks and Snowflake.

• Convert business needs into scalable, well-structured analytical solutions in collaboration with product managers, BI engineers, and data scientists.

• Assist in Kimball dimensional and 3NF data modeling for AWB reporting and analytics.

• Contribute to CI/CD pipelines for data workflows, which includes version control, automated testing, and deployment standards.

• Evaluate code and disseminate best practices.

• Transform architectural designs into production-ready, modular, reusable, loosely coupled, and testable code.

• Deliver essential data platform features from start to finish in production within the first year.

• Provide agent-ready datasets and tools for LangGraph agents and MCP tools.

• Ensure data quality, lineage, and RBAC for every dataset developed.

• Maintain pipeline compute and storage costs within predetermined budgets.

• Role as an individual contributor without direct reports.

⛳️ Requirements

• Over 5 years of experience in data engineering focusing on central data platforms, large-scale pipelines, and production datasets.

• Strong, hands-on expertise in both Databricks and Snowflake is essential.

• Proficient in Apache Spark using Python or Scala on Databricks.

• Strong SQL skills on Databricks and Snowflake.

• Excellent coding abilities.

• Practical experience with Delta Lake, including MERGE, schema evolution, OPTIMIZE, VACUUM, and performance tuning.

• Hands-on experience in building streaming pipelines with Kafka and Spark Structured Streaming, addressing duplicates, late data, checkpoints, and delivery guarantees.

• Proven experience in orchestrating workflows with Apache Airflow is mandatory.

• Hands-on experience in developing AI agents using LangGraph or LangChain and agent functionalities of Databricks and Snowflake, such as Genie and Cortex.

• Familiarity with AI code-generation tools in daily engineering tasks.

• Understanding of data governance, lineage, quality monitoring, regulatory compliance, RBAC, and governance frameworks.

• Experience in constructing production-grade data pipelines on AWS, Azure, or GCP with an emphasis on reliability and observability.

• Knowledge of Kimball dimensional modeling, star/snowflake schemas, and 3NF normalization.

• Awareness of CI/CD pipelines, Git-based workflows, and DevOps practices in data engineering.

• Strong communication and stakeholder management abilities.

• Capability to work independently across various time zones.

• Bonus: experience with Databricks and Snowflake Native Applications, data clean rooms, or Marketplace integrations.

• Bonus: familiarity with dbt models, Prefect, dbt Cloud, semantic layer tools such as Looker or Tableau, Agile/product-based development, cost optimization, energy/utilities/IoT analytics, or MLOps.

🏝️ Benefits

• Remote work arrangement.

• No additional benefits, perks, or compensation extras are specified.