Lakehouse Machine Learning Engineer

This is a fully remote position, open to applicants in California.

📋 Description

• Develop and uphold Python/Spark pipelines across bronze, silver, and gold layers.

• Create semantic datasets and machine learning models that utilize governed lakehouse data.

• Investigate data prior to modeling, design and test features, and collaborate with stakeholders on significant analytical inquiries.

• Construct models for survival and time-to-event analysis, forecasting, classification and propensity, sequence, recommender systems, and causal evaluation.

• Implement models in production and oversee experiment tracking, model registry, scheduled inference, and monitoring for drift and decay.

• Provide model outputs via governed semantic tables that support dashboards and customer data platform (CDP) systems.

• Communicate analytical findings to business teams effectively.

• Engage in applied large language model (LLM) tasks, including structured extraction from unstructured text and retrieval from governed data.

• Operate within security and governance frameworks, encompassing access controls, data protection, auditability, and human review.

• Develop reusable project templates, shared feature and evaluation code, and implementation guidelines.

• Create proofs of concept to validate data support, analytical methodologies, and operational feasibility.

• Carry out miscellaneous responsibilities as needed.

⛳️ Requirements

• Proficient in Extract/Transform/Load (ETL) processes, with the capability to construct a dataset independently.

• Advanced skills in Python and SQL.

• Experience collaborating across enterprise source systems.

• Familiarity with the standard machine learning stack, including scikit-learn and at least one deep learning framework such as PyTorch or TensorFlow.

• Understanding of survival analysis, time-to-event modeling, forecasting, classification and propensity, sequence models, recommenders, and causal evaluation.

• Knowledge of hyperparameter tuning and cross-validation techniques.

• Competence in conducting thorough model validation, evaluation, and bias mitigation.

• Ability to convey results to executives in layman's terms.

• Awareness of data security, privacy, compliance, access controls, data protection, and auditability.

• Practical experience with Databricks, including Unity Catalog, Workflows, MLflow, or similar technologies; these are advantageous.

• Capability to perform cohort-based or hierarchical forecasting at scale is a plus.

• Familiarity with Salesforce is a plus.

• Relevant Databricks or Azure certifications, or equivalent qualifications, are a plus.

• Bachelor’s or Master's degree in an IT-related field is preferred; candidates with a high school diploma or equivalent and 7+ years of practical experience may also be considered.

• A minimum of 3 years of hands-on experience in data engineering and applied machine learning.

• Experience with deployment and monitoring of models in a production environment.

• Familiarity with MLflow or an equivalent tracking, registry, and scheduled-inference system.

• Production experience in a medallion architecture or a comparable layered model within a data-catalog-governed setting.

• Proficiency in distributed processing using Spark or an equivalent engine.

• Experience with open table formats such as Delta or Iceberg.

• Knowledge of catalog-managed schemas, lineage, and access control.

• Practical experience with LLMs including embeddings, retrieval, structured extraction, and evaluation is a plus.

• Experience with subscription or membership-lifecycle data is a plus.

• Experience conducting build-versus-buy assessments is a plus.

• Up to 5% travel may be required.

• This position is not available to California residents.

🏝️ Benefits

• Up to 5% travel may be required.

• Standard business hours with the possibility of extended hours when necessary.

• Comprehensive benefits package (details linked in posting).

• Inclusive and equitable work environment.

• Remote work environment.