Senior Data Engineer

This is a fully remote position, open to applicants in Illinois, +1 more state.

📋 Description

• Design, develop, and maintain scalable ETL/ELT pipelines.

• Architect and oversee secure enterprise cloud storage solutions.

• Ingest and process large-scale batch data as well as real-time streaming workloads.

• Enhance the performance of database queries, PySpark jobs, and data pipelines.

• Work collaboratively with BI and analytics teams to structure dimensional data marts for reporting purposes.

• Implement data governance, automate data quality validation, and manage end-to-end lineage tracking.

• Ensure the security of data pipelines and storage utilizing Azure RBAC, Key Vaults, and established encryption standards.

• Automate infrastructure deployments and code releases through Azure DevOps, GitHub Actions, and CI/CD best practices.

• Partner with data scientists, analysts, and AI teams to foster analytics, machine learning, and Generative AI projects.

⛳️ Requirements

• Proven experience in designing, constructing, and maintaining scalable data pipelines.

• Proficiency with Azure Data Factory, Azure Synapse, and Azure Databricks.

• Background in architecting and managing secure enterprise cloud storage.

• Familiarity with ADLS Gen2, Delta Lake, and relational/NoSQL databases.

• Experience in processing large-scale batch data and real-time streaming workloads.

• Expertise in optimizing database queries, PySpark jobs, data pipelines, execution time, and cloud expenditure.

• Experience in modeling dimensional data marts using Star/Snowflake schemas.

• Understanding of data governance, automated data quality validation, and comprehensive lineage tracking.

• Proficiency with Microsoft Purview.

• Experience in securing pipelines and storage using Azure RBAC, Key Vaults, and encryption standards.

• Proven track record in automating infrastructure deployments and code releases with Azure DevOps, GitHub Actions, and CI/CD best practices.

🏝️ Benefits

• Significant career advancement opportunities.

• A small, dynamic, and entrepreneurial work environment.

• A high level of individual responsibility.

• Equal employment opportunities for all, regardless of protected characteristics.