Data Engineer

📋 Description

• Design and sustain data ingestion pipelines utilizing Azure Data Factory (ADF) for transferring data into the corporate Azure Data Lake and associated platforms.

• Create and refine data transformation logic using Databricks notebooks, primarily focusing on SQL-based transformations within existing SQL scripts and stored procedures.

• Support a medallion data architecture (Bronze, Silver, Gold) with Silver and Gold datasets primarily stored in Parquet format.

• Aid in the migration of SQL Server–based processes (scripts, stored procedures, SSIS logic) running on Azure VMs into Azure-native pipelines and Databricks transformations.

• Continuously improve and update existing database solutions and systems in response to evolving requirements from marketing campaign activation and analytics teams, ensuring data accuracy, timeliness, and appropriateness.

• Oversee multiple concurrent projects, including modernization initiatives and ongoing enhancement requests, while ensuring platform stability and data quality.

• Configure Azure Data Factory pipelines to orchestrate Databricks notebooks and Fabric-related processes as necessary.

• Utilize Azure DevOps for source control, versioning, and deployment of Azure Data Factory, Databricks, and Fabric assets.

• Test, monitor, and troubleshoot pipelines to guarantee data reliability, performance, and quality.

• Collaborate with senior engineers, project managers, campaign services, analytics, and platform teams to deliver continuous data solutions. Engage in the adoption and utilization of Microsoft Fabric, understanding how Fabric components (e.g., OneLake, Lakehouse, Warehousing) interact with existing Azure and Databricks workflows.

• Maintain technical documentation for data pipelines, transformations, and datasets.

⛳️ Requirements

• Practical experience in constructing or supporting pipelines in Azure Data Factory.

• Proficiency in developing data transformations using Databricks, preferably leveraging SQL within notebooks.

• Familiarity with Azure Data Lake Storage or comparable cloud storage solutions.

• Understanding of modern data lake and analytics principles, including medallion architecture.

• Experience with SQL Server, including the ability to write, maintain, and comprehend complex SQL scripts and stored procedures.

• Knowledge of Git / Azure DevOps for source control practices.

• Proficiency in Python for use within Databricks notebooks and supporting data pipeline operations.

• Capability to adhere to established architectural patterns and implementation standards.

• Strong analytical, troubleshooting, and communication abilities.

🏝️ Benefits

• Flexible PTO & Company Holidays

• Flexible Schedule

• Continuous Professional Training and Development

• Mix Of Collaborative & Independent Work

• Community Outreach Opportunities via Anteriad Cares

• Professional Global Mentoring Program - Career Guidance From Leadership

• Great Benefits For You And Your Family

• Comprehensive medical (choice of 3 plans), dental and vision coverage

• Company paid short-term disability, long term disability and life Insurance

• Optional supplemental life, accident and critical illness insurance plans

• 401K with company match

• Flexible PTO and generous holiday schedule

• Fully paid primary caregiver leave (12 weeks) & parental bonding leave (2 weeks)