Senior Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Oversee the design, development, deployment, and upkeep of data and analytics platforms.

• Create, develop, and automate solutions for distributed data ingestion and transformation.

• Construct reliable, scalable, and efficient ETL/ELT data pipelines.

• Design and establish frameworks for data quality, validation, monitoring, and alerts.

• Enforce data governance practices that encompass metadata, access, retention, compliance, and security.

• Design and implement physical data models, database structures, indexing, and table relationships.

• Develop and manage large-scale data storage and processing solutions across cloud and distributed platforms.

• Optimize data pipelines, Spark workloads, databases, and cloud infrastructure.

• Integrate data from enterprise applications and source systems, including real-time and event-driven processing.

• Apply CI/CD and DevOps practices for deployment, testing, and release management.

• Troubleshoot, test, validate, and continuously enhance data pipelines and platform solutions.

• Collaborate with data scientists, analysts, architects, IT teams, and business stakeholders.

• Document data solutions, processes, designs, and technical information.

• Utilize Agile methodologies such as Scrum and Kanban.

• Provide technical guidance and mentor junior team members.

⛳️ Requirements

• Bachelor’s degree or equivalent in Computer Science, Information Technology, Engineering, Data Science, or a related technical field.

• Consideration for relevant equivalent professional experience may apply.

• Preferred relevant cloud or data engineering certifications.

• This role may require licensing or compliance with export controls or sanctions regulations.

• Strong skills in technical leadership and decision-making.

• Excellent problem-solving and analytical abilities.

• Capability to translate complex technical concepts and stakeholder requirements into actionable data solutions.

• Robust understanding of data quality, governance, security, compliance, and data management principles.

• Advanced proficiency in Azure Databricks, Apache Spark, and distributed data processing frameworks.

• Significant expertise in designing and developing enterprise-scale ETL/ELT and data ingestion pipelines.

• Advanced skills in Python and Scala.

• Strong SQL proficiency.

• Extensive experience with Azure data services, encompassing Azure Databricks, ADLS, Azure Blob Storage, Azure Synapse Analytics, and Azure SQL Data Warehouse.

• Familiarity with Delta Lake, Apache Iceberg, Parquet, and ORC.

• Experience with Kafka or similar real-time streaming technologies.

• Proven experience implementing CI/CD pipelines and automated deployment processes.

• Strong knowledge of Git, Jenkins, testing methodologies, version control, and release management.

• Experience in implementing data validation frameworks, monitoring solutions, and data quality controls.

• Understanding of cloud security, access management, data governance, compliance, and regulatory requirements.

• Over 8 years of hands-on experience in Data Engineering, Data Platform Engineering, Big Data, or a related field.

• At least 3 years of experience leading technical teams, projects, or major data engineering initiatives.

• Proven track record in building and supporting large-scale cloud-based data ingestion, transformation, and analytics platforms.

• Experience in developing end-to-end ETL/ELT solutions within Azure cloud environments.

• Experience delivering distributed data processing solutions using Spark and Databricks.

• Expertise in optimizing data pipelines, Spark workloads, databases, and cloud infrastructure.

• Familiarity with data lake, data warehouse, and lakehouse architectures.

• Experience working within Agile software development methodologies.

• Proficient in processing and managing large datasets.

• Experience with Spark, Scala/Java, MapReduce, Hive, HBase, and Kafka or equivalent experience.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible working hours and remote work options.

• Opportunities for professional development and training.

• Generous vacation and paid time off policy.