Data Engineer – Databricks, Lakehouse, Power BI

This is a fully remote position, open to applicants in India.

📋 Description

• Design, develop, and maintain scalable data pipelines utilizing Databricks and Delta Lake.

• Construct and oversee transformations across bronze, silver, and gold layers.

• Enhance processing for improved performance, reliability, and cost-efficiency.

• Integrate data from ERP, CRM, APIs, and various enterprise systems.

• Create gold-layer datasets that adhere to standardized business definitions.

• Convert business requirements into reusable data models.

• Ensure the consistency of core metrics across reporting platforms.

• Align Databricks outputs with Power BI semantic models.

• Implement automated data quality checks and validation rules.

• Develop testable, production-ready pipelines.

• Assist with impact analysis through lineage tools.

• Engage in CI/CD and deployment processes.

• Monitor and enhance pipeline performance.

• Troubleshoot issues across various environments.

• Maintain data consistency between development, testing, and production environments.

• Handle high-volume data workloads.

• Utilize AI-assisted tools for development (e.g., Copilot, Databricks Agents) (Preferred).

• Investigate AI agents for testing, lineage analysis, and optimization (Preferred).

• Collaborate with BI and business teams.

• Assist with governance and cataloging initiatives.

• Document data models and pipelines.

⛳️ Requirements

• Bachelor’s degree in a related field.

• Over 8 years of experience in data engineering.

• Strong expertise in Databricks, Spark, and Delta Lake.

• Proficient in SQL and Python.

• Familiarity with Power BI or similar tools.

• Experience with Azure Data Factory, Synapse, or Data Lake (Preferred).

• Knowledge of DevOps and CI/CD practices (Preferred).

• Exposure to AI agents or AI-assisted development (Preferred).

• Experience in large, complex enterprise environments (Preferred).

🏝️ Benefits

• Health insurance.

• 401(k) matching.

• Flexible work hours.

• Paid time off.