Resident Solution Architect – Data, Databricks

This is a fully remote position, open to applicants in India.

📋 Description

• Create comprehensive Data, Analytics, and Lakehouse solutions utilizing Databricks.

• Design and evaluate scalable ETL/ELT pipelines along with enterprise data platforms.

• Engage directly with Databricks, Apache Spark, PySpark, Python, and SQL.

• Architect solutions leveraging Delta Lake, Unity Catalog, Databricks SQL, and Databricks Workflows.

• Establish scalable data architectures across Azure, AWS, and GCP environments.

• Facilitate technical discovery sessions, architecture workshops, and solution discussions with enterprise clients.

• Convert business and technical requirements into scalable architecture and implementation strategies.

• Develop High-Level Designs, Low-Level Designs, architecture diagrams, technical proposals, and solution documentation.

• Lead POCs, technical presentations, solution validations, and architecture evaluations.

• Offer technical guidance, mentorship, and architectural direction to Data Engineering teams.

• Assess data pipelines, architecture designs, code, and implementation strategies for scalability and maintainability.

• Diagnose and enhance Databricks and Spark workloads for performance, scalability, reliability, and cost-effectiveness.

• Assist with data platform modernization, migration, and transformation projects.

• Collaborate with Sales, Pre-Sales, Delivery, Product, and Engineering teams on technical solutions.

• Interact with senior customer stakeholders to convey architecture decisions, technical advice, risks, and trade-offs.

• Identify areas for improvement in data platform architecture, engineering practices, automation, and operational efficiency.

• Keep abreast of advancements in Databricks, cloud data platforms, distributed computing, and contemporary data engineering technologies.

⛳️ Requirements

• Over 8 years of experience in Data Engineering, Data Architecture, Solution Architecture, Big Data, or a related discipline.

• Strong practical experience with Databricks and enterprise data platforms.

• In-depth knowledge of Apache Spark and PySpark.

• Advanced programming proficiency in Python and SQL.

• Solid understanding of Lakehouse Architecture, Delta Lake, Unity Catalog, Databricks SQL, and Databricks Workflows.

• Proven track record in designing scalable ETL/ELT pipelines, data platforms, data models, and data warehouses.

• Experience with at least one major cloud platform: Azure, AWS, or GCP.

• Strong grasp of distributed data processing, scalability, reliability, and data platform architecture.

• Hands-on experience in performance tuning and optimization of Databricks and Spark workloads.

• Demonstrated experience in technical solution design, architecture workshops, POCs, technical demonstrations, and solution validation.

• Strong consulting experience with the ability to engage Architects, CTOs, CDOs, Engineering Managers, and senior technology stakeholders.

• Excellent communication, presentation, stakeholder management, and technical storytelling abilities.

• Capability to provide technical leadership and mentorship to Data Engineering teams.

• Strong analytical and problem-solving abilities with a structured approach to complex technical issues.

• Databricks certification would be a plus.

• Familiarity with Snowflake, Kafka, Spark Streaming, dbt, Terraform, CI/CD, MLflow, MLOps, GenAI, LLM/RAG, or data migration is advantageous.

• Strong ownership mentality with the capability to work independently in a remote, customer-facing setting.

🏝️ Benefits

• Competitive salary and benefits package.

• Opportunities for professional development and training.

• Flexible working conditions and remote work options.

• Collaborative and innovative work environment.