Data & Analytics Expert

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Spearhead the architecture and design of intricate, enterprise-scale data solutions across cloud, lakehouse, warehouse, streaming, and analytical platforms.

• Establish scalable architectural patterns for data ingestion, transformation, storage, integration, consumption, and lifecycle management.

• Offer technical guidance for complex projects involving multiple applications, data domains, platforms, and engineering teams.

• Set standards for data modeling, interoperability, metadata, lineage, security, performance, and platform integration.

• Lead technical design reviews and assess architectural trade-offs.

• Act as a technical escalation point for intricate data architecture, engineering, integration, and production challenges.

• Steer the modernization of legacy data platforms and workloads utilizing modern cloud and distributed data technologies.

• Design and oversee the implementation of scalable batch, streaming, and event-driven data processing solutions.

• Create complex data solutions using Python, PySpark, Scala, SQL, or similar languages.

• Utilize distributed processing technologies such as Apache Spark, Kafka, Flink, dbt, or equivalent frameworks.

• Design reusable engineering frameworks and patterns applicable across multiple data products and teams.

• Develop dimensional, relational, lakehouse, and domain-oriented data models.

• Enhance distributed workloads, SQL queries, storage organization, partitioning strategies, and data pipelines.

• Implement practices for automated testing, dependency management, orchestration, failure recovery, replay, and operational resilience.

• Provide technical leadership for Microsoft Fabric solutions, including Lakehouse, Warehouse, OneLake, Data Pipelines, Notebooks, Dataflows Gen2, and Semantic Models.

• Leverage Azure and/or AWS cloud services for enterprise data architectures.

• Define and advocate for Git-based version control, code review, automated testing, CI/CD, and automated deployment.

• Develop observability strategies and promote monitoring and operational excellence using Datadog or similar technologies.

• Lead root-cause analyses for complex production issues to prevent recurrence.

• Define and implement practices for data quality, metadata management, lineage, retention, classification, and lifecycle management.

• Collaborate with business owners, data custodians, governance teams, security teams, and architects to create trusted, governed data products.

• Assess emerging data technologies and recommend their adoption.

• Drive modernization initiatives and migrations from legacy data ecosystems to contemporary cloud and lakehouse platforms.

• Employ generative AI and agentic AI tools as appropriate.

• Collaborate with senior business and technology leaders to translate business requirements into long-term data capabilities.

• Communicate technical subjects, architectural choices, risks, and trade-offs to both technical and non-technical audiences.

• Facilitate discussions among engineering, product, analytics, governance, and business teams.

• Influence technical direction across various teams without direct supervisory authority.

• Mentor engineers and develop technical documentation, reference architectures, standards, and reusable implementation guidance.

⛳️ Requirements

• A minimum of 10 years of relevant experience in technology, data engineering, data architecture, or data platforms (Preferred).

• Extensive experience in designing and delivering complex, enterprise-scale data solutions.

• Proven experience in providing technical leadership across large or cross-functional initiatives.

• Expert-level understanding of data architecture, data engineering, distributed processing, data integration, and data modeling.

• Advanced proficiency in Python, PySpark, Scala, SQL, or similar languages for large-scale data processing.

• In-depth knowledge of Microsoft Fabric, including Lakehouse, Warehouse, OneLake, Notebooks, Data Pipelines, Dataflows Gen2, and Semantic Models.

• Strong expertise in Delta Lake, lakehouse architecture, distributed computing, batch processing, and streaming architectures.

• Advanced SQL skills, including query optimization and performance tuning.

• Comprehensive understanding of relational, dimensional, NoSQL, and distributed data modeling techniques.

• Experience in designing reusable frameworks and enterprise-scale data processing patterns.

• Strong knowledge of cloud architecture and data services within Microsoft Azure and/or AWS.

• Advanced understanding of Git, CI/CD, automated testing, infrastructure/deployment automation, and release management.

• Experience with observability platforms such as Datadog.

• Solid understanding of data governance, metadata, lineage, data quality, security, privacy, and retention principles.

• Proven ability to troubleshoot complex distributed data and production issues.

• Proven capability to lead technical initiatives and influence engineering standards across multiple teams.

• Excellent communication skills.

• Must undergo a background investigation.

• Allstate generally does not sponsor individuals for employment-based visas for this position.

• A dedicated, private workspace free from distractions when working from home.

• Reliable internet connection with minimum speeds of 50 MB download and 5 MB upload.

🏝️ Benefits

• Comprehensive technology setup, including a laptop, monitors, headset, keyboard, and mouse.

• Monthly connectivity reimbursement for employees eligible to work from home.

• Remote work arrangement.

• Opportunity to challenge the status quo and shape the future of protection.

• Support for causes that resonate with employees.