Senior Data Engineer

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Design, construct, and maintain scalable batch and streaming data pipelines utilizing Apache Spark and cloud-native data technologies.

• Develop and enhance ETL/ELT workflows to ingest, transform, and curate data from various source systems into analytics-ready datasets.

• Implement data modeling and transformation logic to facilitate reporting, dashboards, and downstream analytical and machine learning tasks.

• Build and manage data processing workloads within contemporary lakehouse platforms, with a preference for Microsoft Fabric / OneLake.

• Ensure data quality, reliability, and consistency by establishing validation checks, monitoring, and reconciliation processes.

• Optimize Spark jobs for performance, cost-effectiveness, and scalability across extensive and intricate datasets.

• Oversee and evolve data schemas while addressing schema drift and modifications from upstream sources.

• Create reusable frameworks, libraries, and standardized patterns to enhance data engineering productivity and uniformity.

• Implement CI/CD pipelines for data workloads to facilitate automated testing, deployment, and rollback.

• Monitor data pipelines and jobs, troubleshoot failures, and address performance or data quality challenges.

• Collaborate with analytics engineers, BI developers, and data scientists to comprehend data requirements and provide curated datasets.

• Work alongside platform, security, and governance teams to guarantee data security, compliance, and appropriate access controls.

• Participate in Agile delivery processes, including sprint planning, design reviews, and continuous improvement initiatives.

⛳️ Requirements

• 4+ years of experience in data engineering or a similar role (preferred).

• Strong experience as a Data Engineer constructing and managing production data pipelines.

• Hands-on experience with Apache Spark for large-scale data processing.

• Proficiency in Python, SQL, and best practices for data transformation.

• Experience with cloud-based data platforms and storage solutions (e.g., Data Lakes, Lakehouse architectures).

• Familiarity with Microsoft Fabric, OneLake, or comparable analytics platforms (strong plus).

• Experience in designing and optimizing data models for analytical workloads.

• Understanding of distributed data processing concepts, performance tuning, and fault tolerance.

• Experience with CI/CD, version control, and infrastructure-as-code practices.

• Strong problem-solving abilities and capacity to troubleshoot complex data challenges.

• Excellent communication skills and capability to collaborate across both technical and non-technical teams.

🏝️ Benefits

• Comprehensive technology setup including laptop, monitors, headset, keyboard, and mouse.

• Monthly connectivity reimbursement to assist with internet costs.

• 401(k) matching program.

• Health insurance options.

• Employee assistance program.

• Paid time off and holiday compensation.