Principal Data Engineer

Posted 11 hours ago

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Take ownership of the technical vision and advancement of JLL’s Databricks-based Enterprise Data Platform.

• Define the complete Databricks lakehouse architecture, which includes workspace and catalog design, Unity Catalog structure, medallion layering, compute strategy, and multi-environment promotion.

• Establish platform governance encompassing access control, lineage, classification, tagging, auditing, data quality, and observability.

• Facilitate governed access for AI agents and LLM applications through MCP, APIs, vector search, retrieval pipelines, and curated data products.

• Enhance platform cost efficiency via compute right-sizing, cluster policies, Spark and Photon tuning, storage optimization, scheduling, budgets, dashboards, and alerting.

• Advocate for infrastructure as code, Databricks Asset Bundles, automated CI/CD, golden paths, templates, and developer tools.

• Assess Databricks features and third-party tools through structured evaluations and proofs of concept.

• Collaborate with Databricks and strategic vendors on roadmaps, architecture assessments, escalations, and enablement initiatives.

• Develop engineering standards, reference architectures, and selection criteria for platform tools and services.

• Engage in strategic architecture decisions and synchronize platform roadmaps with business and domain requirements.

• Mentor lead and senior engineers, oversee architecture reviews, and conduct knowledge-transfer and brown-bag sessions.

• Coordinate delivery across various project teams and influence cross-functional stakeholders.


⛳️ Requirements

• Over 10 years of experience in data engineering, including multiple large, complex platform or data product implementations across various technology domains.

• Expert-level, hands-on experience with Databricks, encompassing Spark, Delta Lake, Unity Catalog, Workflows / Jobs, Lakeflow or Delta Live Tables, Auto Loader, Structured Streaming, and SQL warehouses.

• Proven track record of designing and constructing scalable, reusable ingestion and processing frameworks for batch, CDC, and streaming data, including schema evolution, data quality, replay, and backfill.

• Demonstrated experience in defining, versioning, and enforcing data contracts or equivalent producer-consumer agreements.

• Proven experience in reducing and governing costs associated with cloud data platforms.

• Practical experience in enabling LLM or agent-based applications on a data platform.

• Proficient in Python and SQL, with strong software engineering practices including testing, modular design, and code review.

• Proficiency in PySpark optimization and Spark UI.

• Extensive knowledge of a major cloud platform and its data services; Azure is preferred.

• Experience with infrastructure as code, such as Terraform, and CI/CD for data pipelines and platform configuration.

• Expertise in data modeling and architecting complex end-to-end data solutions.

• Proven ability to evaluate third-party data and AI tools, conduct proofs of concept, make build-versus-buy recommendations, and collaborate with vendors.

• Experience in orchestrating multiple engineering project teams, prioritizing tasks across teams, and influencing cross-functional stakeholders without direct authority.

• Candidates must have authorization to work in the United States without sponsorship.


🏝️ Benefits

• 401(k) plan with matching company contributions.

• Comprehensive Medical, Dental & Vision Care.

• Paid parental leave at 100% of salary.

• Paid Time Off and Company Holidays.

• Early access to earned wages through Daily Pay.

• Supportive culture and comprehensive benefits package prioritizing mental, physical, and emotional health.

People also viewed

Highmark Health11 hours ago

Data Engineer

US flagUnited States OnlyFull-timeData Engineer$86.4k – $138.6k/year
ApplyView job
Prompt Therapy Solutions Inc11 hours ago

Clinical Technical Support, Data Migration Specialist – B2B SaaS

US flagUnited States OnlyFull-timeData Engineer$65k – $75k/year
ApplyView job
Allshore Talent11 hours ago

Data Architect – Adobe Analytics, Customer Journey Analytics

EuropeFreelanceData Engineer
ApplyView job
Airbnb11 hours ago

Senior Data Engineer, Tech Foundations

US flagUnited States OnlyFull-timeData Engineer$196k – $230k/year
ApplyView job
Alight Solutions11 hours ago

Senior Director, Data Engineering & Analytics

US flagUnited States OnlyFull-timeData Engineer$170k – $250k/year
ApplyView job
NBCUniversal12 hours ago

Senior Data Engineer, Data Management & BI

BI flagBurundi, +1 more countryFull-timeData Engineer$115k – $145k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers