Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Collaborate with both technical and non-technical stakeholders to convert business requirements into scalable, reliable, and maintainable data engineering solutions.

• Design, develop, and uphold data integration solutions utilizing Microsoft SQL Server, Azure Data Factory, Azure Databricks, Microsoft Fabric, and associated Azure services.

• Construct, manage, and enhance ETL/ELT pipelines with Azure Data Factory and Microsoft Fabric Pipelines, including tasks such as pipeline orchestration, parameterization, scheduling, monitoring, and error management.

• Create and sustain data transformation logic using Azure Databricks notebooks and Microsoft Fabric notebooks, focusing on performance, scalability, reusability, and operational reliability.

• Design and execute data ingestion, transformation, validation, and delivery processes across structured, semi-structured, and external data sources.

• Develop and refine data integration processes leveraging Azure Data Factory, Azure Databricks, Azure Storage Accounts, Azure Data Lake Storage, Azure Key Vault, Microsoft Fabric Lakehouses, Warehouses, Pipelines, and Notebooks.

• Create and maintain SQL-based data models, stored procedures, views, automation scripts, and database objects to support operational and analytical tasks.

• Assist in the modernization and migration of legacy data integration processes from SSIS and on-premises platforms to cloud-based Azure and Microsoft Fabric frameworks.

• Design and implement reusable pipeline patterns, data quality checks, validation controls, logging, alerting, and performance monitoring mechanisms.

• Facilitate new client onboarding initiatives by designing and implementing dependable data ingestion, transformation, reconciliation, and validation workflows.

• Identify and resolve data pipeline, data quality, performance, and integration challenges in partnership with teams across Intelligence & Analytics.

• Create, develop, and maintain datasets, semantic models, and data structures that bolster reporting, analytics, and visualization solutions across Power BI, Excel, Microsoft Fabric, and other business intelligence platforms.

• Contribute to data engineering standards, documentation, best practices, and reusable frameworks that enhance scalability, maintainability, and operational efficiency.

• Investigate, assess, and recommend emerging technologies and platform capabilities that advance data architecture, automation, performance, and reliability.

⛳️ Requirements

• A minimum of 3 years of experience in data engineering, data integration, business intelligence engineering, or a related technical position.

• At least 3 years of experience in writing, optimizing, and troubleshooting SQL, including T-SQL, stored procedures, views, indexing strategies, and query performance tuning.

• Practical experience in designing, building, and supporting ETL/ELT pipelines with Azure Data Factory and/or Microsoft Fabric Pipelines.

• Hands-on experience in developing data processing and transformation logic using Azure Databricks notebooks, Microsoft Fabric notebooks, PySpark, Spark SQL, Python, or similar technologies.

• Strong skills in Microsoft SQL Server and relational database design principles.

• Experience with Azure data platform services such as Azure Data Factory, Azure Databricks, Azure Storage Accounts, Azure Data Lake Storage, Azure Key Vault, and related cloud data services.

• In-depth knowledge and experience with Microsoft Fabric components including Pipelines, Notebooks, Lakehouses, Warehouses, Dataflows, and semantic models.

• Experience in building reliable data ingestion and transformation processes from APIs, flat files, databases, web services, cloud storage, and other external data sources.

• Practical experience in implementing data validation, reconciliation, monitoring, logging, and alerting processes for production data pipelines.

• Experience in supporting production data workloads, including troubleshooting failures, optimizing performance, and enhancing pipeline reliability.

• Prior experience in modernizing or migrating legacy ETL processes from SSIS, SQL Server, or on-premises environments to Azure or Microsoft Fabric is highly preferred.

• Bachelor's degree in Computer Science, Information Technology, Data Analytics, a related field, or equivalent professional experience.

• Familiarity with Power BI, semantic models, or other data visualization and analytics tools is a plus.

• Experience with Customer Data Platforms (CDPs), database marketing, marketing analytics, or client data onboarding environments is a plus.

• Working knowledge of AWS or Google Cloud Platform is an advantage.

• Strong analytical, problem-solving, and critical thinking abilities.

• Excellent verbal and written communication skills, capable of explaining technical concepts to both technical and non-technical audiences.

• Ability to work independently on moderately complex data engineering tasks while effectively collaborating in a team-oriented environment.

🏝️ Benefits

• Work from home

• Training & development with unlimited access to Percipio LMS

• A blend of collaborative and independent work

• Community outreach via Anteriad Cares - encouraging staff to volunteer

• Professional mentoring program - career guidance from leadership

• Employee Resource Groups - collaborate with others who share your passions!