Senior Data Engineer II

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Design, construct, and sustain scalable batch and real-time data pipelines utilizing Apache Spark, SQL, Python, and cloud-native data technologies.

• Architect and execute Medallion Lakehouse data patterns for enterprise analytics, reporting, operational intelligence, and AI applications.

• Develop and enhance ETL/ELT pipelines from a variety of enterprise source systems.

• Design and execute dimensional models, including fact and dimension tables.

• Build and oversee data workloads within Microsoft Fabric, OneLake, and contemporary Lakehouse platforms.

• Establish frameworks for data quality, observability, and governance, featuring automated validation, reconciliation, monitoring, and alerting.

• Optimize Spark workloads for performance, scalability, reliability, and cost-effectiveness.

• Manage schema evolution, data lineage, metadata, and modifications from source systems.

• Develop reusable frameworks, accelerators, and engineering standards.

• Utilize Agentic AI-assisted development for code generation, testing, documentation, troubleshooting, and productivity enhancement.

• Implement CI/CD and DevSecOps practices for automated testing, deployment, release management, and rollback procedures.

• Lead technical design reviews and mentor junior and mid-level data engineers.

• Collaborate with architects, stakeholders, analytics engineers, and data scientists to convert business requirements into scalable solutions.

• Partner with platform, security, privacy, and governance teams to ensure compliance.

• Drive continuous improvement across data engineering practices, architectural standards, and Agile delivery processes.

• Serve as a technical leader and trusted advisor.

• Advocate for modern data architecture, reusable standards, and automation-first methodologies.

• Promote the adoption of AI-assisted engineering practices.

• Provide technical leadership, mentorship, and guidance without direct people management responsibilities.

⛳️ Requirements

• 7+ years of experience as a Data Engineer, developing and supporting enterprise-scale data platforms and pipelines.

• Strong hands-on expertise with Apache Spark.

• Advanced proficiency in Python, SQL, and contemporary data engineering development practices.

• Experience in designing and implementing Lakehouse architectures and Medallion Architecture patterns.

• Strong comprehension of dimensional modeling, encompassing star schemas, snowflake schemas, fact tables, dimension tables, and semantic data models.

• Experience with Microsoft Fabric, OneLake, Delta/Parquet formats, or similar cloud analytics platforms.

• Expertise in data integration, transformation, and optimization techniques for large-scale analytical workloads.

• Experience in implementing CI/CD pipelines, source control, Infrastructure as Code (IaC), and automated testing methodologies.

• Familiarity with AI-powered engineering tools and Agentic AI-assisted software development practices.

• Strong understanding of distributed computing, performance tuning, scalability, fault tolerance, and data reliability.

• Experience in leading technical initiatives and mentoring engineering team members.

• Preferred: experience with Kafka, Event Streams, or similar technologies.

• Preferred: experience designing enterprise data products within Microsoft Fabric.

• Preferred: familiarity with Data Vault, Kimball, or other enterprise data modeling methodologies.

• Preferred: experience supporting machine learning, generative AI, or advanced analytics workloads.

• Preferred: knowledge of data governance, metadata management, master data management, and data quality frameworks.

• Preferred: cloud or data engineering certifications.

• Required background investigation.

• Must have reliable internet with minimum speeds of 50 MB download and 5 MB upload when working from home.

• Must have a dedicated, private workspace free from distractions with appropriate desk and seating.

• Allstate generally does not sponsor individuals for employment-based visas for this position.

🏝️ Benefits

• Comprehensive technology setup including a laptop, monitors, headset, keyboard, and mouse.

• Monthly connectivity reimbursement for employees eligible to work from home.

• Remote work arrangement.

• Dedicated private workspace and reliable internet requirements support for home working.