Senior Data Engineer

This is a fully remote position, open to applicants in California.

📋 Description

• Design, develop, and maintain scalable ETL/ELT pipelines utilizing Databricks, PySpark, Python, and SQL.

• Enhance data workflows and SQL/Spark queries for improved performance, cost-effectiveness, and reliability.

• Contribute to the architecture of our enterprise Data Lake and curated data assets.

• Create resilient ingestion frameworks, transformation logic, and reusable data engineering components.

• Assist in data migrations and modernization initiatives from legacy systems to cloud-based platforms.

• Employ AI coding assistants (Databricks Assistant, GitHub Copilot, SQL copilots) to expedite development, code reviews, test creation, and documentation.

• Automate repetitive tasks (schema evolution, config generation, logging templates, test data creation) using generative AI tools.

• Utilize AI to automatically identify pipeline anomalies, data drift, quality issues, and performance bottlenecks.

• Use AI-assisted query optimization tools to enhance Spark and SQL performance.

• Collaborate cross-functionally with Product teams and data consumers to deliver high-quality data solutions.

• Engage in Agile ceremonies and contribute to sprint planning and technical design discussions.

• Mentor junior data engineers and advocate for best practices in AI-assisted data engineering.

• Ensure comprehensive documentation, standards adherence, and ongoing improvement within the team.

⛳️ Requirements

• 5–7+ years of practical experience in data engineering.

• Advanced expertise in Python, SQL, PySpark, and ETL/ELT frameworks.

• Strong background with Databricks and distributed data processing.

• Proven capability to optimize SQL/Spark workloads for enhanced performance and cost savings.

• Familiarity with AI development tools such as GitHub Copilot, Databricks Assistant, SQL/IDE AI copilots, and AI-based data quality or monitoring tools.

• Solid grasp of data modeling, governance, and CI/CD practices for data pipelines.

• Excellent communication skills and the ability to thrive in a cross-functional setting.

• Experience in Agile environments with tools such as Jira and Confluence.

🏝️ Benefits

• Comprehensive medical, vision, and dental benefits for you and your family, including Flexible Spending Accounts (FSA) and Health Savings Accounts (HSAs).

• Generous paid time off policy that includes paid holidays, sick leave, and paid days off for your birthday.

• Complimentary concert tickets.

• 401(k) program with company matching.

• Stock program.

• Support programs for new parents, including caregiver leave and childcare cash.

• Infertility support.

• Tuition reimbursement, student loan repayment, and internal growth and development programs and training.

• Volunteer time off and access to a crowdfunding network.