Data Engineer, AI

This is a fully remote position, open to applicants in Alabama, +42 more states.

📋 Description

• Design, develop, test, and maintain scalable data ingestion, transformation, and integration pipelines utilizing Databricks, PySpark, Python, and SQL.

• Construct and enhance enterprise data pipelines in accordance with modern Lakehouse and Medallion Architecture principles.

• Create reusable, modular, and metadata-driven ETL/ELT frameworks.

• Ingest and integrate both structured and semi-structured data from databases, APIs, flat files, cloud storage, and enterprise applications.

• Collaborate with Delta Lake, Unity Catalog, Databricks Workflows, and related platform capabilities.

• Facilitate the migration and modernization of legacy SQL Server, SSIS, and traditional data warehouse workloads to Databricks.

• Optimize data pipelines and transformations focusing on performance, reliability, scalability, and cost-effectiveness.

• Develop Python-based automation frameworks, utilities, and reusable engineering components.

• Design solutions that are metadata-driven and configuration-driven, minimizing manual coding and operational dependencies.

• Utilize scripting, APIs, orchestration, and intelligent automation techniques to enhance engineering workflows.

• Construct efficient data transformation logic employing SQL, Python, and PySpark.

• Implement joins, aggregations, business rules, enrichment logic, and incremental processing patterns.

• Assist with dimensional modeling, historical data tracking, and efficient data processing patterns.

• Collaborate with business analysts, data architects, senior engineers, architects, and technical teams to convert requirements into engineering solutions.

• Introduce automated data quality checks to ensure completeness, accuracy, consistency, uniqueness, and referential integrity.

• Develop reconciliation and validation routines across source systems, transformation layers, and downstream datasets.

• Support initiatives for metadata management, data lineage, cataloging, and data governance.

• Build capabilities for monitoring, logging, and alerting for pipelines and scheduled workloads.

• Troubleshoot pipeline failures, data discrepancies, performance issues, and integration errors.

• Engage in root-cause analysis and implement long-term corrective solutions.

• Adhere to security, access control, data privacy, and auditability requirements.

• Maintain engineering code using Git-based version control and collaborative development methodologies.

• Assist with automated deployment pipelines using CI/CD frameworks and DevOps tooling.

• Implement unit tests, integration tests, and automated validations.

• Contribute to infrastructure automation and environment management.

• Document technical designs, data mappings, transformations, deployment procedures, and operational runbooks.

• Continuously enhance engineering productivity through automation and reusable frameworks.

⛳️ Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical field, or equivalent practical experience.

• 2+ years of relevant experience in data engineering, software engineering, data integration, or similar technical roles.

• Proficient Python programming skills, encompassing scripting, modular development, data processing, and automation.

• Strong SQL skills, including complex queries, joins, aggregations, window functions, transformations, and query performance optimization.

• Extensive hands-on experience with PySpark, including distributed data processing, DataFrame operations, transformations, and performance optimization.

• Experience in developing ETL/ELT pipelines and working with relational databases or modern data platforms.

• Familiarity with Databricks, Apache Spark, Delta Lake, or similar cloud-based data engineering technologies.

• Understanding of contemporary data architecture concepts, including data lakes, data warehouses, lakehouses, and data pipelines.

• Experience with structured data, JSON, CSV, Parquet, APIs, and other common integration formats.

• Knowledge of cloud platforms such as Microsoft Azure or AWS.

• Basic understanding of data modeling, data quality, and data validation practices.

• Proficient in using Git or similar source control tools.

• Strong mindset towards automation and hyperautomation, with proven ability to simplify workflows, eliminate manual tasks, and create reusable engineering solutions.

• Excellent analytical, troubleshooting, and problem-solving abilities.

• Ability to work independently on assigned engineering tasks while effectively collaborating within a broader technical team.

• Strong communication skills and eagerness to learn new technologies.

• Employment offers are subject to successful completion of a background screening.

🏝️ Benefits

• PTO

• Medical insurance

• Dental insurance

• Vision insurance

• Retirement savings

• Disability insurance

• Life insurance

• Reasonable accommodations during the application and employment process