Senior Data Engineer, AI – Systems Engineering

📋 Description

• Design and construct scalable data pipelines utilizing Databricks and Spark to ingest, transform, and integrate data from various enterprise systems.

• Develop and sustain medallion architecture (Bronze, Silver, Gold) data models to generate reliable and high-performance “Golden Record” datasets.

• Apply data normalization, mapping, and entity resolution techniques (such as fuzzy matching and XREF tables) to consolidate asset data across diverse systems.

• Create data workflows to identify and highlight Shadow IT across financial, identity, endpoint, and network signals, integrating outcomes into CMDB systems.

• Collaborate with IT, Security, Finance, Procurement, and GRC teams to establish and uphold data standards for essential CMDB attributes (e.g., ownership, approval status, lifecycle).

• Develop and maintain data integrations and APIs to synchronize curated datasets with operational systems like ServiceNow and Jira Assets.

• Monitor, troubleshoot, and enhance data quality, reliability, and observability across the data platform.

⛳️ Requirements

• Over 9 years of experience in building and maintaining data pipelines and large-scale data platforms.

• Extensive experience with Databricks, Apache Spark, and SQL for distributed data processing and transformation.

• Familiarity with designing data models and architectures, including medallion architecture, data lakes, or lakehouse systems.

• Proficient in Python or similar programming languages for data engineering and ETL development.

• Experience integrating data from various enterprise systems (e.g., SaaS tools, financial systems, identity systems).

• Strong grasp of data quality, data governance, and entity resolution techniques across heterogeneous datasets.

• Exceptional collaboration and communication skills, with experience working cross-functionally with both technical and non-technical stakeholders.

🏝️ Benefits

• Health insurance

• 401(k) matching

• Flexible work hours

• Paid time off

• Remote work options