Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and enhance data pipelines using Azure Data Factory to ensure reliable ingestion, transformation, and delivery of data to Snowflake for analytics purposes.

• Set up monitoring, alerts, and testing for data pipeline performance, data quality metrics, and lineage to guarantee trustworthy data delivery.

• Diagnose data issues and conduct root cause analysis to proactively address operational challenges.

• Create comprehensive documentation for data structures, processes, architectural decisions, and best practices to facilitate knowledge sharing.

• Design, maintain, and enhance Snowflake objects (schemas, tables, views) and SQL transformations to create curated datasets that are ready for analytics.

• Collaborate effectively with analysts, stakeholders, and product owners to convert business requirements into data specifications and stable technical solutions.

• Prepare data for AI/ML applications by developing feature-rich datasets, assisting in feature engineering, and ensuring data consistency for model training and inference.

• Aid in the deployment and operationalization of machine learning models by integrating pipelines with ML workflows (e.g., batch and real-time scoring).

• Continuously enhance reporting and analytics processes, automating or simplifying self-service and manual tasks.

• Apply version control practices to all data engineering code and documentation.

⛳️ Requirements

• A Bachelor's degree in Computer Science, Computer Engineering, Information Technology, or a related discipline; or equivalent professional experience.

• Over 5 years of experience in data engineering or business intelligence roles, focusing on ETL, data modeling, data architecture, and the creation of analytics pipelines and applications (e.g., BI, reporting, machine learning, deep learning).

• Strong programming skills in advanced SQL, Python, or other relevant programming languages for data processing and automation.

• Experience in supporting or working with AI/ML workflows, including data preparation and feature engineering for machine learning models.

• Familiarity with integrating data pipelines with ML frameworks (e.g., scikit-learn, TensorFlow, PyTorch, or similar technologies).

• Understanding of model lifecycle concepts, including training, validation, deployment, and monitoring.

• Expertise in Snowflake for data warehousing, including experience in schema design, performance tuning, and optimization.

• Proficiency in using Git, Azure DevOps, and collaborative development best practices.

• Experience in designing, developing, and deploying comprehensive pipelines using Azure Data Factory.

🏝️ Benefits

• Unmatched benefits.

• Innovative cost-saving solutions.