Senior Data Engineer – Databricks, AWS

This is a fully remote position, open to applicants in United States.

📋 Description

• Create and execute robust, scalable data ingestion and transformation pipelines utilizing Databricks, PySpark, and distributed processing. Leverage Airflow (or a similar tool) for dependable workflow orchestration.

• Apply Delta Lake principles, emphasizing CDC and schema evolution. Establish and incorporate data quality frameworks (such as Great Expectations) within CI/CD pipelines to ensure data integrity.

• Design and enhance intricate SQL and Python scripts. Integrate various data sources (including APIs, S3 files, etc.) while managing both structured and unstructured data.

• Assist in the implementation of data governance and cataloging solutions (for instance, Unity Catalog). Proactively analyze and enhance inconsistent legacy datasets.

• Lead and oversee AI agents for code generation, ensuring quality and stability. Collaborate effectively to advance technical solutions.

⛳️ Requirements

• Over 5 years of professional experience in data engineering, with a focus on cloud and distributed processing environments.

• Strong expertise in Databricks, PySpark, distributed processing, and Delta Lake, with in-depth knowledge of CDC and schema evolution.

• Proficient in SQL optimization and Python. Practical experience with Airflow (or a similar tool).

• Familiarity with essential AWS components. Solid understanding of CI/CD for data workflows and the implementation of data quality frameworks.

• Understanding of streaming technologies (Kafka/Kinesis) and exposure to Unity Catalog or comparable governance tools.

• A proactive problem-solver who is comfortable handling complex, inconsistent data without explicit specifications.

• Capable of working independently in a dynamic rebuild environment.

• Team-oriented, pragmatic, and enjoys rapid iteration.

• Skilled in guiding AI agents for coding while upholding high quality standards.

🏝️ Benefits

• Remote Office – Flexible hybrid working arrangement.

• Parking Space – We provide complimentary parking spots.

• Fun Office Space – Enjoy a game zone and a relaxation area.

• Health Insurance – Additional private health coverage, including a dental care plan.

• Personal Development – Company-sponsored training budget to enhance your skills.

• Employee Referral Program – Earn a bonus for referring a friend.

• Holidays – Enjoy an extra 5 days off after your 1st and 5th year.

• Social Events – We love to celebrate our successes together.

• Family Insurance – Option to add insurance for a family member.

• Offering sports cards – 100% sponsored by the company.