Databricks Developer

This is a fully remote position, open to applicants in Kentucky.

📋 Description

• Create and execute scalable data solutions utilizing Databricks Lakehouse Architecture.

• Design and sustain data pipelines with PySpark, Python, and SQL.

• Construct and enhance ETL/ELT workflows for batch processing and near real-time data handling.

• Apply Delta Lake functionalities such as ACID transactions, time travel, schema evolution, and data versioning.

• Develop and oversee enterprise data models for analytics and reporting purposes.

• Ensure data integrity through validation, monitoring, reconciliation, and governance protocols.

• Create and manage data catalogs, metadata management, and data lineage processes.

• Work collaboratively with business stakeholders, architects, and analytics teams to convert requirements into technical implementations.

• Enhance Databricks workloads for optimal performance, scalability, and cost-effectiveness.

• Establish security measures, access controls, and governance best practices within the Databricks ecosystem.

• Assist in troubleshooting, root cause analysis, and resolving production issues.

• Contribute to the modernization of the data platform and initiatives for cloud migration.

⛳️ Requirements

• Extensive experience with Databricks Architecture and platform management.

• Practical knowledge of Databricks Lakehouse Architecture.

• In-depth understanding of Delta Lake concepts and their implementation.

• Experience with Unity Catalog/Data Catalog and managing metadata.

• Familiarity with Databricks Workflows, Jobs, Clusters, and Performance Optimization.

• Strong command of PySpark for large-scale data processing tasks.

• Advanced skills in Python programming.

• Proficient in SQL development and query optimization.

• Experience in constructing robust ETL/ELT pipelines.

• Solid understanding of data modeling methodologies, including Star Schema, Snowflake Schema, Dimensional Modeling, and Data Vault (preferred).

• Experience in implementing data quality frameworks and validation processes.

• Knowledge of data lineage, metadata management, and governance practices.

• Proficient in data reconciliation, profiling, and monitoring tools.

• Preferred familiarity with Azure Databricks, Azure Data Lake Storage (ADLS), Azure Data Factory, Azure Synapse Analytics, and CI/CD pipelines (Azure DevOps, GitHub Actions).

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.

• 4-8 years of experience in Data Engineering and Analytics.

• At least 3+ years of hands-on experience with Databricks and PySpark.

• Experience in Agile development settings.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Flexible work hours and remote work options.

• Opportunities for professional development and continuous learning.

• Comprehensive health and wellness benefits.

• Collaborative and inclusive work environment.