Data Engineer

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Design, develop, and maintain reliable risk data products along with scalable batch and streaming data pipelines utilizing cloud-native technologies.

• Integrate data from risk platforms, control systems, vendor tools, operational applications, and external sources into governed, analytics-ready datasets.

• Create and enhance ETL/ELT workflows for automated data ingestion, transformation, validation, reconciliation, and delivery.

• Construct and oversee data processing workloads in contemporary data lake and lakehouse environments, such as Microsoft Fabric and OneLake.

• Establish data quality, monitoring, lineage, and reconciliation processes.

• Generate curated datasets and analytical outputs for operational reporting, leadership dashboards, trend analysis, risk identification, and decision-making.

• Enhance data architectures, schemas, and processing patterns for scalability, performance, resilience, and cost-effectiveness.

• Develop reusable frameworks, engineering standards, CI/CD pipelines, automated testing, and operational monitoring.

• Collaborate with analytics, product, risk, governance, security, and compliance stakeholders on data definitions, metrics, quality standards, and secure data access.

• Engage in Agile delivery activities, including sprint planning, backlog refinement, design reviews, and continuous improvement initiatives.

⛳️ Requirements

• A minimum of 4 years of experience as a Data Engineer or in a similar position, building and supporting production-grade data pipelines and data products.

• Practical experience with Apache Spark for large-scale data processing and transformation.

• Strong proficiency in Python, SQL, and contemporary data engineering best practices.

• Experience in developing ETL/ELT solutions within cloud-based data lake, lakehouse, or analytics platforms.

• Experience in designing and optimizing analytical data models that support reporting, dashboards, operational insights, and advanced analytics.

• Solid understanding of data quality, validation, monitoring, reconciliation, and production support processes.

• Experience with CI/CD pipelines, version control, automated testing, and infrastructure-as-code concepts.

• Ability to diagnose complex data quality, performance, and integration issues across multiple data sources.

• Excellent verbal and written communication skills.

• Preferred experience with Microsoft Fabric, OneLake, or similar modern analytics platforms.

• Experience working with risk, controls, compliance, audit, governance, or regulatory data domains.

• Experience in building real-time, near real-time, or event-driven data processing solutions.

• Familiarity with data governance, metadata management, data lineage, access controls, and enterprise data quality frameworks.

• Experience with orchestration and workflow management tools for data pipelines.

• Experience supporting operational reporting, executive dashboards, trend analysis, advanced analytics, or machine learning initiatives.

• Background investigation required.

• Reliable internet connection with minimum speeds of 50 MB download and 5 MB upload.

• A dedicated, private workspace free from distractions, equipped with an appropriate desk and seating.

• Allstate typically does not sponsor individuals for employment-based visas for this role.

🏝️ Benefits

• Comprehensive technology setup including a laptop, monitors, headset, keyboard, and mouse.

• Monthly connectivity reimbursement for employees eligible to work from home.