Senior Data Engineer

This is a fully remote position, open to applicants in Florida, +2 more states.

📋 Description

• Design and sustain scalable data pipelines.

• Improve data models.

• Offer expert knowledge in data acquisition and consumption pipelines within Azure cloud solutions, including but not limited to Databricks, ADF, and various ETL/ELT tools.

• Establish best practices, reusable code, libraries, and frameworks for cloud-based data warehousing and ETL processes.

• Employ multi-cloud programming languages such as Scala, Python, SQL, in Unity Catalog, SQL Server, as well as other RDBMS and NoSQL databases, while architecting enterprise data warehouse platforms.

• Promote collaboration and align with business goals to enhance data models, support data-driven decision-making, and improve data accessibility across the organization.

⛳️ Requirements

• At least 10 years of hands-on data engineering experience in enterprise environments.

• Strong expertise in Azure services, especially Azure Databricks, Azure Functions, and Azure Data Factory (preferred).

• Advanced proficiency in Apache Spark utilizing PySpark, Python, and Databricks SQL, with a focus on query optimization and performance tuning.

• Comprehensive understanding of ETL/ELT processes, data pipeline creation, and orchestration/scheduling workflows.

• Practical experience with Delta Lake functionalities such as Change Data Capture, ACID transactions, optimization, and schema evolution.

• Strong foundation in data modeling methodologies, including normalized, dimensional, and Lakehouse models.

• Experience with both batch and real-time/streaming data processing using technologies like Kafka or Event Hub.

• Deep understanding of data architecture principles, distributed systems, and cloud-native design patterns.

• Ensure data quality, integrity, and security throughout the entire data lifecycle.

• Develop, deploy, and manage AI and machine learning applications on Databricks using Agents Bricks and custom MCP servers.

• Create and implement Databricks Genie workspaces and facilitate conversational analytics on custom datasets.

• Familiarity with CI/CD tools such as Azure DevOps and Git.

• Experience with Infrastructure as Code (IaC) tools like Terraform and ARM templates.

• Knowledge of data governance and cataloging solutions, including Unity Catalog for enabling Attribute and Role-based access controls.

• Experience supporting machine learning or business intelligence workloads on Databricks.

• Exposure to the insurance and healthcare data sectors is a significant advantage.

• Certifications in Databricks, data engineering, and Azure cloud are a plus.

🏝️ Benefits

• Generous time off, encompassing personal and volunteering days.

• Tuition reimbursement and opportunities for professional development.

• Options for remote work.

• Charitable contribution matching programs.

• Stock purchase options available.