
Principal Data Engineer
Posted 11 hours ago

Posted 11 hours ago
This is a fully remote position, open to applicants in Illinois.
• Take ownership of the technical vision and advancement of JLL’s Databricks-based Enterprise Data Platform.
• Define the complete Databricks lakehouse architecture, which includes workspace and catalog design, Unity Catalog structure, medallion layering, compute strategy, and multi-environment promotion.
• Establish platform governance encompassing access control, lineage, classification, tagging, auditing, data quality, and observability.
• Facilitate governed access for AI agents and LLM applications through MCP, APIs, vector search, retrieval pipelines, and curated data products.
• Enhance platform cost efficiency via compute right-sizing, cluster policies, Spark and Photon tuning, storage optimization, scheduling, budgets, dashboards, and alerting.
• Advocate for infrastructure as code, Databricks Asset Bundles, automated CI/CD, golden paths, templates, and developer tools.
• Assess Databricks features and third-party tools through structured evaluations and proofs of concept.
• Collaborate with Databricks and strategic vendors on roadmaps, architecture assessments, escalations, and enablement initiatives.
• Develop engineering standards, reference architectures, and selection criteria for platform tools and services.
• Engage in strategic architecture decisions and synchronize platform roadmaps with business and domain requirements.
• Mentor lead and senior engineers, oversee architecture reviews, and conduct knowledge-transfer and brown-bag sessions.
• Coordinate delivery across various project teams and influence cross-functional stakeholders.
• Over 10 years of experience in data engineering, including multiple large, complex platform or data product implementations across various technology domains.
• Expert-level, hands-on experience with Databricks, encompassing Spark, Delta Lake, Unity Catalog, Workflows / Jobs, Lakeflow or Delta Live Tables, Auto Loader, Structured Streaming, and SQL warehouses.
• Proven track record of designing and constructing scalable, reusable ingestion and processing frameworks for batch, CDC, and streaming data, including schema evolution, data quality, replay, and backfill.
• Demonstrated experience in defining, versioning, and enforcing data contracts or equivalent producer-consumer agreements.
• Proven experience in reducing and governing costs associated with cloud data platforms.
• Practical experience in enabling LLM or agent-based applications on a data platform.
• Proficient in Python and SQL, with strong software engineering practices including testing, modular design, and code review.
• Proficiency in PySpark optimization and Spark UI.
• Extensive knowledge of a major cloud platform and its data services; Azure is preferred.
• Experience with infrastructure as code, such as Terraform, and CI/CD for data pipelines and platform configuration.
• Expertise in data modeling and architecting complex end-to-end data solutions.
• Proven ability to evaluate third-party data and AI tools, conduct proofs of concept, make build-versus-buy recommendations, and collaborate with vendors.
• Experience in orchestrating multiple engineering project teams, prioritizing tasks across teams, and influencing cross-functional stakeholders without direct authority.
• Candidates must have authorization to work in the United States without sponsorship.
• 401(k) plan with matching company contributions.
• Comprehensive Medical, Dental & Vision Care.
• Paid parental leave at 100% of salary.
• Paid Time Off and Company Holidays.
• Early access to earned wages through Daily Pay.
• Supportive culture and comprehensive benefits package prioritizing mental, physical, and emotional health.
Highmark Health
Prompt Therapy Solutions Inc
Allshore Talent
Airbnb
Get handpicked remote jobs straight to your inbox weekly.