Senior Data Engineer – AI, Systems Engineering

📋 Description

• Design and create scalable data pipelines utilizing Databricks and Spark to ingest, transform, and consolidate data from various enterprise systems.

• Develop and sustain medallion architecture (Bronze, Silver, Gold) data models to produce dependable and high-performance “Golden Record” datasets.

• Apply data normalization, mapping, and entity resolution techniques (e.g., fuzzy matching, XREF tables) to harmonize asset data across diverse systems.

• Construct data workflows to identify and expose Shadow IT across financial, identity, endpoint, and network signals, integrating outcomes into CMDB systems.

• Collaborate with IT, Security, Finance, Procurement, and GRC teams to establish and uphold data standards for essential CMDB attributes (e.g., ownership, approval status, lifecycle).

• Create and maintain data integrations and APIs to synchronize curated datasets with operational systems like ServiceNow and Jira Assets.

• Monitor, troubleshoot, and enhance data quality, reliability, and observability throughout the data platform.

⛳️ Requirements

• Over 9 years of experience in building and maintaining data pipelines and large-scale data platforms.

• Extensive experience with Databricks, Apache Spark, and SQL for distributed data processing and transformation.

• Background in designing data models and architectures, including medallion architecture, data lakes, or lakehouse systems.

• Proficient in Python or similar programming languages for data engineering and ETL development.

• Experience in integrating data from various enterprise systems (e.g., SaaS tools, financial systems, identity systems).

• Strong grasp of data quality, data governance, and entity resolution techniques across varied datasets.

• Exceptional collaboration and communication skills, with experience working across functional teams with both technical and non-technical stakeholders.

🏝️ Benefits

• Health insurance

• 401(k)

• Paid time off

• Flexible work arrangements