Data Engineer

This is a fully remote position, open to applicants in Alabama, +25 more states.

📋 Description

• Design and manage data pipelines utilizing PySpark and Databricks

• Operate within a metadata-driven ingestion framework to integrate new datasets

• Establish data quality checks and validation rules within the pipelines

• Facilitate ingestion from file-based sources and ingestion tools (e.g., Fivetran)

• Manage schema alterations, incremental loads, and file processing patterns

• Participate in data governance initiatives including tagging, metadata, and lineage

• Diagnose and resolve pipeline failures and performance challenges

• Collaborate with architects and stakeholders regarding data onboarding and requirements

• Adhere to and contribute to coding standards, reusable components, and best practices

⛳️ Requirements

• Experience in place of a Bachelor’s Degree

• 3+ years of pertinent experience with a degree

• 5+ years of pertinent experience without a degree

• Practical experience with PySpark and Databricks

• Proficient SQL skills

• Experience in constructing ETL/ELT data pipelines

• Knowledge of Delta Lake concepts (merge, schema evolution, partitions)

• Familiarity with cloud platforms (preferably Azure)

• Basic experience with Git and version control

• Exposure to data catalog or governance tools (e.g., DataHub)

• Experience with Fivetran or comparable ingestion tools

• Understanding of data quality and validation concepts

• Experience working with metadata-driven frameworks

• Strong problem-solving and debugging capabilities

• Ability to operate in a structured, framework-driven environment

• Emphasis on data quality, beyond just pipeline execution

• Eagerness to learn and adapt in a rapidly changing data ecosystem.

🏝️ Benefits

• Medical, Dental, Vision plans

• Adoption, Fertility and Surrogacy Reimbursement up to $10,000

• Paid Time Off and Sick Leave

• Paid Parental & Family Caregiver Leave

• Emergency Backup Care

• Long-Term, Short-Term Disability, and Critical Illness plans

• Life Insurance

• 401k/403B with Employer Match

• Tuition Assistance – $5,250/year and discounted educational opportunities through Guild Education

• Student Debt Pay Down – $10,000

• Reimbursement for certifications and free access to complete CEUs and professional development

• Pet Insurance

• Legal Resources Plan

• Colleagues have the opportunity to earn an annual discretionary bonus if established system and employee eligibility criteria is met.