Senior Data Engineer

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Develop and sustain scalable data pipelines, as well as ETL and ELT processes, and data models utilizing the Databricks platform.

• Create, implement, and launch data and AI solutions leveraging Databricks, Spark, Delta Lake, and associated technologies.

• Construct batch and streaming pipelines using tools such as Databricks Workflows and Azure Data Factory.

• Design logical data flow diagrams and normalized schemas, applying Lakehouse patterns like the Medallion Architecture (Bronze, Silver, Gold layers).

• Maintain data quality, integrity, security, and governance throughout the data lifecycle, including the use of Unity Catalog.

• Enhance Spark jobs and data transformations through effective partitioning, caching, and join strategies.

• Oversee pipeline execution, detect failures, and resolve intricate data processing challenges.

• Partner with data architects, analysts, data scientists, and business stakeholders to comprehend requirements and deliver solutions.

• Assist in documenting data processes, standards, and data flows.

⛳️ Requirements

• A minimum of 5 years of experience in designing, developing, and deploying data solutions on the Databricks platform.

• Expertise in Python, including PySpark, and SQL.

• Practical experience with Spark, Delta Lake, and Lakehouse architectures.

• Proven experience in implementing data quality, governance, and security measures across data pipelines.

• Familiarity with machine learning concepts, tools, and libraries such as TensorFlow, PyTorch, Scikit-learn, and MLflow is advantageous.

• Experience in configuring and integrating external AI models, along with working with AI governance and monitoring tools is a plus.

• Experience with asynchronous programming patterns in Python for the development of scalable data or AI workloads is a plus.

• Excellent problem-solving, collaboration, and communication abilities.

🏝️ Benefits

• Health insurance

• 401(k) matching

• Paid time off

• Professional development