Databricks Engineer

This is a fully remote position, open to applicants in United States, +6 more states.

📋 Description

• **Create impactful solutions as a Databricks Engineer at RevStar.**

• RevStar, a Databricks Partner, is establishing a cloud-agnostic practice dedicated to Data, Machine Learning (ML), and Artificial Intelligence (AI) services. Our goal is to assist businesses in modernizing their data platforms, enhancing analytics workflows, and deploying scalable AI-driven solutions utilizing Databricks.

• We are deeply committed to our craft and our methodologies. From conceptualizing architecture and design to coding and delivery, we approach each project with an agile mindset, consistently assessing goals and business requirements to guarantee optimal results.

• At RevStar, we cultivate a collaborative, remote-first environment where teams exchange ideas, innovate collectively, and develop both personally and professionally. By joining us, you will engage with cutting-edge technologies across various industries, creating value-driven products for clients who prioritize quality and performance. We are dedicated to not just cloud-native app development, but also to crafting meaningful experiences and impactful outcomes.

• We are on the lookout for a highly proficient **Databricks Engineer** to become part of our team. This hands-on role involves collaborating closely with architects, data scientists, and clients to construct, refine, and deploy high-performance data and AI solutions.

• As a Databricks Engineer, your responsibilities will include building and optimizing data pipelines, implementing data processing frameworks, and facilitating AI/ML solutions within Databricks. You will manage data ingestion, transformation, and orchestration while ensuring scalability, performance, and security.

• This technical, hands-on position requires expertise in Apache Spark, Delta Lake, and MLOps, alongside experience in large-scale data architectures. You will collaborate with architects and business stakeholders to ensure that solutions meet customer needs and adhere to best practices.

• Above all, the ideal candidate will embody RevStar’s core values:

• - **Self-Mastery:** We maintain a high standard for our thought processes, communication, and improvement.

• - **Ownership:** We are accountable for outcomes, not just efforts.

• - **Shared Destiny:** We succeed or fail together.

• **Key Responsibilities**

• **Data Engineering & Pipeline Development**

• - Construct and enhance data pipelines utilizing Apache Spark and Delta Lake within Databricks.

• - Implement ETL/ELT workflows to ensure effective data ingestion, transformation, and storage.

• - Design Lakehouse architecture-based solutions capable of scaling across structured and unstructured data sources.

• - Integrate Databricks with cloud storage solutions (Azure Data Lake, AWS S3, Google Cloud Storage) for efficient data management.

• **Performance Optimization & Automation**

• - Enhance Spark jobs for scalability, cost-effectiveness, and low latency.

• - Establish monitoring and alerting systems to track job performance and identify failures.

• - Develop automated data validation, testing, and quality assurance protocols.

• **Management AI/ML Integration & MLOps Support**

• - Assist with ML model training and deployment within Databricks, utilizing MLflow for experiment tracking and model versioning.

• - Collaborate with data scientists and ML engineers to facilitate scalable AI solutions.

• - Implement feature engineering pipelines and integrate models into production environments.

• **Security, Governance & Best Practices**

• - Ensure data security, access control, and compliance with industry standards (GDPR, HIPAA, SOC 2, etc.).

• - Adhere to Databricks best practices for data lineage, governance, and metadata management.

• - Document procedures, configurations, and best practices for both internal and client use.

⛳️ Requirements

• **Must-Have:**

• - 3+ years of practical experience in data engineering, emphasizing big data processing and cloud-native architectures.

• - 2+ years of hands-on experience with Databricks, including Apache Spark, Delta Lake, and MLflow.

• - Databricks Certifications (Mandatory):

• - Databricks Certified Data Engineer Associate (or higher)

• - Proficiency in Python, SQL, and Spark-based frameworks.

• - Experience in developing and enhancing large-scale ETL/ELT pipelines.

• - Strong comprehension of Lakehouse architecture and cloud-agnostic data solutions.

• - Familiarity with CI/CD pipelines and Infrastructure-as-Code (IaC) for Databricks (e.g., Terraform, Databricks CLI).

• - Knowledge of data governance, security, and compliance best practices.

• - Experience in Agile development environments, adhering to DevOps/MLOps best practices.

• **Nice-to-Have:**

• - Additional Databricks Certifications (e.g., Databricks Certified Machine Learning Associate).

• - Experience with real-time streaming solutions (e.g., Kafka, Kinesis, Event Hub).

• - Familiarity with cloud storage and orchestration tools (e.g., Apache Airflow, Prefect).

• - Background in AI/ML integration within Databricks, assisting in feature engineering and model deployment.

• - Experience in client-facing roles or consulting environments.

🏝️ Benefits

• **Benefits for Full-Time W2 Positions:**

• - Paid Time Off – Take the time you need to recharge and maintain productivity.

• - Remote-First Working Environment – Collaborate from anywhere while staying connected with our global team.

• - Comprehensive Health Coverage – Medical, Dental, Vision

• - 401(k) Retirement Plan – Prepare for your future with access to a company-sponsored 401(k) program.

• - Annual Learning & Development Stipend – Invest in your skills through conferences, certifications, or courses.

• - Peer Mentorship & Coaching – Learn from seasoned engineers, product managers, and architects to accelerate your growth.

• - Professional Growth Opportunities – Exposure to cutting-edge AWS GenAI, data, and cloud technologies across various industries.

• - Company Outings & Volunteer Opportunities – Build relationships and give back to the community.

• - Collaborative, Innovative Culture – Work with top talent in a fast-paced, supportive environment that values curiosity and initiative.