Senior Data Engineer

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Design, develop, and enhance scalable data pipelines and processing systems.

• Create reusable patterns for ingestion, transformation, and storage.

• Integrate data from APIs, databases, event streams, and third-party systems.

• Implement data models and schemas to provide unified views of business operations.

• Utilize version control, testing, CI/CD, and automated deployment practices.

• Ensure data quality, integrity, and reliability through validation, monitoring, and observability.

• Optimize workflows for performance, cost efficiency, and operational resilience.

• Document data flows, lineage, and technical components.

• Collaborate with product managers, engineers, and analysts to understand data requirements.

• Engage in iteration planning, daily standups, retrospectives, and product ceremonies.

• Contribute to metadata management, data lineage, privacy, security, and regulatory compliance.

• Assess new technologies, frameworks, and methodologies.

• Share expertise and mentor junior engineers.

• Oversee the entire lifecycle of data products from design and implementation to production support, data quality, and performance metrics.

⛳️ Requirements

• Strong expertise in developing scalable data pipelines utilizing PySpark and Python.

• Practical experience with modern data engineering frameworks and tools, such as Apache Spark, Kafka, Flink, dbt, or similar technologies.

• In-depth knowledge of SQL, NoSQL databases, data modeling, and data architecture principles.

• Proven ability to optimize SQL queries, data pipelines, and storage for performance, scalability, and cost-effectiveness.

• Experience in designing and developing batch and/or real-time streaming data solutions.

• Practical experience with Microsoft Fabric Notebooks for developing, debugging, testing, and operationalizing data engineering workflows.

• Familiarity with containerization and orchestration technologies, including Docker and Kubernetes.

• Experience in implementing monitoring and observability solutions; Datadog experience is preferred.

• Capacity to work effectively in collaborative, iterative, and product-focused engineering environments.

• Strong communication, analytical, troubleshooting, and problem-solving abilities.

• Experience with cloud data services (Azure, AWS) is preferred.

• Hands-on experience with Microsoft Fabric components, including Lakehouse, Warehouse, Data Pipelines, Dataflows Gen2, and Semantic Models is preferred.

• Familiarity with CI/CD pipelines, such as Jenkins, GitHub Actions, and Azure DevOps is preferred.

• Familiarity with generative and agentic AI tools is preferred.

• Must undergo a background investigation.

• Allstate typically does not sponsor individuals for employment-based visas for this position.

• A dedicated, private workspace that is free from distractions, with appropriate desk and seating.

• Reliable internet with minimum speeds of 50 MB download and 5 MB upload.

🏝️ Benefits

• Comprehensive technology setup that includes a laptop, monitors, headset, keyboard, and mouse.

• Monthly connectivity reimbursement to help cover internet expenses.

• Remote work arrangement.