Software Engineer II, Data

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and implement ingestion and transformation solutions utilizing Microsoft Fabric and Azure Data Factory.

• Create integration pipelines for batch, incremental, API, file, database, and event-driven processes.

• Develop reusable and configuration-centric ingestion and transformation patterns.

• Construct curated data structures to meet analytics, reporting, operational, and downstream application needs.

• Work with both structured and semi-structured data formats, including SQL, CSV, JSON, XML, and Parquet.

• Utilize Azure Data Factory, Azure Data Lake/Storage, and Azure SQL/SQL Server.

• Convert business and source system requirements into scalable data models.

• Engage in conceptual, logical, and physical data modeling activities.

• Design dimensional and curated models intended for enterprise reporting and analytics.

• Generate complex SQL queries, views, transformations, and data-processing logic.

• Identify and optimize performance issues in queries, pipelines, and data-processing tasks.

• Address data quality, integrity, completeness, and consistency challenges.

• Conduct source-to-target and source-to-source data reconciliation.

• Establish validation checks for new pipelines and modifications.

• Utilize GitHub for source control and branch-based development practices.

• Adhere to EDAS CI/CD, pull-request, code-review, and design-review protocols.

• Employ Azure DevOps and authorized engineering methodologies to oversee delivery.

• Produce maintainable, reusable, and thoroughly documented code.

• Participate in peer code and design evaluations.

• Develop or enhance reusable engineering patterns and supporting documentation.

• Monitor and assist with production data pipelines and scheduled workloads.

• Diagnose pipeline failures, missed schedules, prolonged processes, and data quality problems.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Information Systems, Data Engineering, Engineering, or a similar field, along with 3 years of relevant IT experience, OR a total of 7 years of related IT experience.

• Proven experience in developing enterprise data pipelines, integrations, data warehouses, Lakehouse solutions, or analytics data platforms.

• Demonstrated capability to troubleshoot intricate data and integration issues.

• Strong analytical and problem-solving abilities.

• Excellent communication and cross-functional collaboration skills.

• Capacity to deliver effectively in an Agile enterprise engineering context.

• Proficient in SQL and relational database development.

• Solid understanding of data engineering and ETL/ELT processes.

• Familiarity with Microsoft Fabric.

• Experience with Azure Data Factory (ADF).

• Knowledge of Azure Data Lake/Storage and Azure SQL/SQL Server.

• Understanding of data warehousing and dimensional/data modeling principles.

• Proficient in data ingestion and transformation methodologies.

• Experience with data validation and reconciliation practices.

• Skills in performance optimization and troubleshooting.

• Familiarity with Git/GitHub.

• Experience with CI/CD and pull-request-based development methodologies.

• Knowledge of Azure DevOps/Agile delivery practices.

• Proficiency in Python and/or PySpark is a plus.

• Understanding of Delta Lake/Lakehouse concepts is an advantage.

• Familiarity with Medallion architecture is a plus.

• Experience with REST APIs/JSON-based integrations is advantageous.

• Knowledge of event-driven integrations and Azure Service Bus is a plus.

• Experience with data migration and legacy platform modernization is beneficial.

• Familiarity with Azure Key Vault and secure credential management is a plus.

• Understanding of enterprise data governance and data quality principles is advantageous.

• Microsoft Fabric/Azure data engineering certifications are a plus.

🏝️ Benefits

• Benefits overview: https://cdw.benefit-info.com/

• Equitable, transparent, and respectful hiring practices.

• Encouragement of thoughtful AI utilization to broaden perspectives and enhance application materials.