Senior Data Engineer – Knowledge Ingestion

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop the iKNOW connector for Oracle B2C Service knowledge APIs, ensuring comprehensive inventory extraction, incremental synchronization based on modified dates, article status, and capturing answer versions.

• Create connectors for SharePoint, OneDrive, and Teams utilizing Microsoft Graph, incorporating site and drive enumeration, delta queries, and sharing/group-membership permission metadata.

• Construct the Sitecore connector for content APIs focused on public web content within the project's scope.

• Implement data ingestion into S3 with per-source prefixes, encompassing document conversion, text extraction, content-hash deduplication, taxonomy metadata enrichment, and synchronization with Amazon Bedrock Knowledge Base.

• Generate inventory extracts for content disposition reviews and Phase 1 re-baselining counts.

• Manage retirement processes to ensure that retired or deleted source items are removed from the index during the next synchronization.

• Address API throttling and backoff, idempotent reruns, and isolation of failures on a per-source basis.

• Monitor pipeline stages with CloudWatch metrics and produce ingestion runbooks.

• Collaborate with client IT during weeks 7–10 to facilitate pipeline operations, reruns, and extensions post-transition.

⛳️ Requirements

• A minimum of 4 years of experience in building data pipelines in production, including at least 2 years on AWS using Python.

• Practical experience with Microsoft Graph API for SharePoint and Teams content, covering delta queries, permissions, and throttling, or equivalent experience with another enterprise content platform.

• Proven experience in managing unstructured content like HTML, Office documents, PDFs, and chat transcripts into S3 and preparing it for search or vector indexing.

• Familiarity with Amazon Bedrock Knowledge Bases, including data sources and synchronization behavior, or a similar vector ingestion pathway on OpenSearch.

• Experience with infrastructure-as-code tools such as Terraform or AWS CDK.

• Continuous integration experience for deploying pipelines into client-managed accounts.

• Ability to work from a shared backlog within client repositories, documenting progress as tasks are completed.

• AWS Certified Data Engineer Associate certification is preferred.

• Prior experience integrating with Oracle B2C Service or other Oracle Service Cloud products is preferred.

• Previous integration experience with Sitecore XM or XP content APIs is preferred.

• Experience with AWS Glue, Step Functions, or EventBridge is preferred.

• Familiarity with document parsing libraries for Office and PDF formats is preferred.

• Required English proficiency level: A1-A2, B1-B2, C1-C2, or Native (as per application form options).

• Legal authorization to work in the country where the role is based is necessary; sponsorship may be required.

🏝️ Benefits

• Equal opportunity employer.

• Opportunities for continuous learning.

• An inclusive work environment.

• Collaboration across diverse continents and cultures.

• Exposure to AI-driven projects spanning various industries.

• Commitment to ethical AI standards.

• Chance to work with both startup and enterprise clients.