Forward Deployed Data Engineer IV, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Collaborate with utility clients to architect and develop comprehensive production data platforms from start to finish.

• Create systems for ingestion, transformation, orchestration, quality assurance, governance, and data serving using Databricks, Spark, Python, SQL, and AWS.

• Take ownership of the technical architecture for each project and advocate for it to client architects, security teams, and platform managers.

• Perform independent technical assessments to uncover essential client requirements.

• Deliver a functional system within the initial weeks and refine it toward production readiness.

• Lead the transition from legacy data warehouses and on-premises systems to contemporary platforms.

• Model data utilizing dimensional, lakehouse/medallion, graph, or semantic models as needed.

• Integrate utility source systems including asset and work management, historians/SCADA, CIS, AMI, GIS, and ERP.

• Enhance the observability and operability of pipelines through lineage tracking, quality assessments, alerting mechanisms, and runbooks.

• Develop and implement operator-facing application layers such as data quality dashboards, reconciliation tools, and self-service data interfaces.

• Oversee project scope, timelines, expectations, risks, and measurable outcomes in accordance with statements of work.

• Contribute to the development of reusable frameworks, ingestion methodologies, and reference architectures.

• Provide feedback on products and implementations to E Source engineering and product teams.

• Implement governance, security, privacy, and retention controls suitable for utility regulations.

• Collaborate with client data/IT teams, executives, and E Source ML engineers, software engineers, data scientists, and consultants.

• Travel to client locations approximately 30–50%, varying by project, which may include extended on-site engagements.

⛳️ Requirements

• Bachelor’s degree in computer science, information technology, or a related discipline.

• A minimum of five years of experience in data engineering, data platform development, or analytics engineering.

• At least one platform that you have managed in a production environment.

• Advanced expertise in Python, SQL, Databricks, and Spark, including an understanding of Spark runtime internals.

• Ability to troubleshoot performance issues in Spark jobs beyond simply scaling compute resources.

• Experience in constructing cloud-based data pipelines within the AWS ecosystem.

• Familiarity with a second cloud platform is preferred.

• Experience with concepts such as incremental versus full rebuilds, idempotency, handling late-arriving data, deduplication strategies, backfill strategies, and schema evolution.

• Experience in designing batch and streaming data pipelines.

• Knowledge of data governance, security, privacy, lineage, and quality assurance practices.

• Proficient in Git, Docker, CI/CD tools, and at least one modern orchestration framework.

• Proven experience utilizing AI coding assistants and agentic development tools.

• Background in the energy or utility sector, or another asset-intensive industry.

• Strong written communication skills and ability to effectively use whiteboarding techniques.

• A Master’s degree in a relevant STEM area is preferred.

• Experience with graph databases and knowledge graph modeling is preferred.

• Familiarity with Kafka or equivalent streaming platforms and change data capture tools is advantageous.

• Experience in building feature pipelines and supplying data to ML and AI systems is preferred.

• Experience in developing and deploying Databricks Apps or similar operator-facing applications is preferred.

• Familiarity with React, Streamlit, or Dash and REST or GraphQL APIs is preferred.

• Experience in optimizing Databricks for cost and performance is preferred.

• Databricks certification is preferred.

• Applicants must possess authorization to work for any employer in the US.

• Employment visa sponsorship is not available.

🏝️ Benefits

• Comprehensive insurance options, including medical, dental, and vision coverage.

• Company-sponsored life insurance.

• Company-funded long- and short-term disability insurance.

• Flexible spending plans for medical and dependent care expenses.

• Paid parental leave.

• Flexible time off (FTO) policy.

• 401(k) plan with a 3% employer match.

• Annual bonus opportunities.

• Remote work arrangements available within the US.

• Significant travel to client locations, including extended on-site periods during discovery, deployment, and go-live phases.