Data Engineer

This is a fully remote position, open to applicants in Kentucky.

📋 Description

• Construct, maintain, and enhance data pipelines using Azure Data Factory to ensure reliable ingestion, transformation, and delivery of data to Snowflake for analytical purposes.

• Establish monitoring, alerts, and performance testing for data pipelines, along with metrics for data quality and lineage to guarantee trustworthy data delivery.

• Diagnose data issues and conduct root cause analysis to proactively address operational challenges.

• Record data structures, processes, architectural choices, and best practices for effective knowledge sharing.

• Create, manage, and optimize Snowflake objects (schemas, tables, views) and SQL transformations to generate curated datasets that are ready for analytics.

• Work collaboratively with analysts, stakeholders, and product owners to convert business needs into data requirements and stable technical solutions.

• Prepare data for AI/ML applications by developing feature-rich datasets, facilitating feature engineering, and ensuring data consistency for model training and inference.

• Aid in the deployment and operationalization of machine learning models by linking pipelines with ML workflows (e.g., batch/real-time scoring).

• Continuously enhance reporting and analytics processes, automating or streamlining self-service or manual tasks.

• Adopt version control practices for all data engineering code and documentation.

⛳️ Requirements

• A bachelor's degree in Computer Science, Computer Engineering, Information Technology, or a related discipline; or equivalent professional experience.

• Over 5 years of experience in data engineering or business intelligence roles, focusing on ETL, data modeling, data architecture, and the development of pipelines and applications for analytics (e.g., BI, reporting, machine learning, deep learning).

• Strong programming capabilities in advanced SQL, Python, or other programming languages for data processing and automation.

• Experience in supporting or engaging with AI/ML workflows, including data preparation and feature engineering for machine learning models.

• Integration of data pipelines with ML frameworks (e.g., scikit-learn, TensorFlow, PyTorch, or similar technologies).

• Familiarity with model lifecycle concepts (training, validation, deployment, monitoring).

• Proficiency in using Snowflake for data warehousing, including schema design, performance tuning, and optimization expertise.

• Competence with Git, Azure DevOps, and best practices for collaborative development.

• Experience in designing, developing, and deploying end-to-end pipelines utilizing Azure Data Factory.

🏝️ Benefits

• Exceptional benefits for yourself.

• Innovative solutions for cost reduction.