Senior Databricks Engineer – Platform, Data Engineering

This is a fully remote position, open to applicants in Pennsylvania.

📋 Description

• Oversee the management of workspaces, clusters, jobs, security configurations, libraries, and the overall health of the platform.

• Design and enhance data pipelines and transformations utilizing PySpark and T-SQL.

• Develop automation and tools using Python, Bash, and PowerShell.

• Implement and automate data governance, access controls, auditability, and compliance best practices.

• Convert conversational ideas into precise solution architectures; break down work into tasks in Planner or Azure DevOps for team execution.

• Accurately estimate effort levels and forecast delivery dates.

• Create comprehensive documentation, including workflows, gap analyses, design specifications, build documentation, test plans, and transition documents.

• Supervise unit, scenario-based, integrated, capacity, and parallel testing.

• Participate in a 24/7/365 on-call rotation to support Databricks, SQL Server, and other relevant applications supported by the team.

• Assist with system upgrades, patches, performance tuning, observability, and proactive monitoring.

• Ensure accountability for data integrity across applications and pipelines.

• Complete system builds and documentation in compliance with departmental guidelines and change management policies.

• Review the work of teammates and provide constructive feedback.

• Communicate effectively with peers, stakeholders, and leadership, providing regular status updates and addressing issues as necessary.

• Facilitate discussions, mentor accountable leaders, and coordinate resources across initiatives.

• Offer feedback on training materials, conduct training sessions, and mentor other team members.

⛳️ Requirements

• Significant experience in Databricks administration, including workspace management, cluster configuration, security, jobs, and libraries.

• Proficient in Python and experienced in scripting with Bash and PowerShell.

• Familiarity with PySpark and/or T-SQL for data processing and transformation.

• Exceptional communication skills, capable of clearly articulating technical concepts to diverse audiences.

• Proven critical thinking, problem-solving, and troubleshooting abilities.

• Ability to estimate effort accurately and meet deadlines.

• Strong grasp of data governance principles and best practices.

• Dedication to upholding information security best practices.

🏝️ Benefits

• Healthcare benefits for both full-time and part-time positions from day one.

• Vision insurance.

• Dental insurance.

• Domestic partner benefits.