Data Pipeline Lead – Metadata

This is a fully remote position, open to applicants in New York, +2 more states.

📋 Description

• Take charge of the design, development, and governance of enterprise data pipelines and metadata frameworks within Palantir Foundry and integrated data platforms.

• Act as the technical and functional lead for data ingestion, transformation, and metadata management across both structured and unstructured data sources.

• Establish and uphold metadata standards, data models, ontologies, and data dictionaries.

• Manage end-to-end data pipelines, which include ingestion, validation, transformation, and delivery to downstream platforms like Palantir and Databricks.

• Set up and govern data quality, validation, and exception-handling processes.

• Ensure alignment of pipelines and metadata with enterprise architecture, system-of-record requirements, and integration patterns.

• Facilitate document-based ingestion workflows, including OCR/ICR integration and subsequent metadata extraction.

• Create AI-ready data foundations for semantic search, entity resolution, and advanced analytics.

• Collaborate with data science and engineering teams on AI/ML use cases and analytical workflows.

• Promote data governance and lifecycle management, including schema versioning, lineage tracking, auditability, and compliance with security and privacy standards.

• Oversee integration across AWS, Databricks, Palantir, and other platforms.

• Lead and mentor teams in a matrixed, cross-functional environment.

• Engage with senior stakeholders to define data strategy and translate business requirements into technical solutions.

• Operate within an Agile delivery model, managing backlog prioritization, technical design reviews, and iterative delivery.

⛳️ Requirements

• U.S. Citizenship is required.

• Ability to obtain and maintain a Public Trust clearance.

• Bachelor's degree.

• A minimum of eight (8) years of experience in data engineering, data architecture, or platform integration, with progressively increasing leadership responsibilities.

• Hands-on expertise in Palantir Foundry, including production-grade data pipelines, ontologies, data models, relationship mappings, and enterprise system integration.

• Experience with Palantir AIP to enable AI-driven workflows.

• Proven experience leading enterprise-scale Palantir implementations, including architecture, delivery, and governance.

• Experience designing and managing large-scale data pipelines within cloud environments; AWS is preferred.

• Familiarity with integrating Palantir with Databricks and/or Spark-based platforms.

• In-depth knowledge of metadata management and data governance, covering data dictionaries, controlled vocabularies, data lineage, traceability, schema versioning, and change management.

• Experience in implementing data quality frameworks, validation rules, exception handling, and reconciliation processes.

• Proficiency in Python, SQL, and/or other relevant languages for data pipeline development.

• Experience in designing systems that support AI/ML and analytics use cases, including unstructured data and document processing pipelines.

• Experience delivering complex solutions in an Agile environment.

• Relevant certifications or equivalent demonstrated expertise are preferred.

• A Master's degree is preferred.

🏝️ Benefits

• Medical, Rx, Dental & Vision Insurance.

• Personal and Family Sick Time & Company Paid Holidays.

• Parental Leave.

• 401(k) Retirement Plan.

• Group Term Life and Travel Assistance.

• Voluntary Life and AD&D Insurance.

• Health Savings Account, Health Care & Dependent Care Flexible Spending Accounts.

• Transit and Parking Commuter Benefits.

• Short-Term & Long-Term Disability.

• Tuition Reimbursement, Personal Development, Certifications & Learning Opportunities.

• Employee Referral Program.

• Corporate Sponsored Events & Community Outreach.

• Care.com annual membership.

• Employee Assistance Program.

• Supplemental Benefits via Corestream (Critical Care, Hospital Indemnity, Accident Insurance, Legal Assistance, and ID theft protection, etc.).

• The position may be eligible for a discretionary variable incentive bonus.

• Flexible benefits package.

• Flexible work arrangements.