Data Engineer, AI Backend

This is a fully remote position, open to applicants in California, +3 more states.

📋 Description

• Take ownership of the API specification for the AI backend, beginning with a functional stub, and subsequently develop and manage the production services that support it.

• Ensure the stability of the API contract and effectively communicate any changes to teams that rely on it.

• Develop a document-context service that delivers pertinent context from an attorney's documents while adhering to access permissions.

• Create ingestion, processing, and indexing pipelines for SEC filings sourced from EDGAR to facilitate peer-company analysis.

• Establish a citation index that links AI-generated responses to precise source sentences in the foundational filings.

• Construct services that manage requests across various AI models, incorporating logging, monitoring, reliability controls, and visibility into usage and costs.

• Design and uphold pipelines that generate evaluation data for data scientists to assess AI quality.

• Set data-quality, observability, and operational standards for the AI backend.

• Oversee AI backend data services and pipelines from initial design to production operation.

• Collaborate with data scientists, the editorial team, and platform engineers.

• Within the first 90 days, take charge of the API specification and stub, deploy the document-context service in a testing environment, and commence the ingestion of SEC filings for an initial group of companies.

⛳️ Requirements

• A minimum of 5 years of professional experience in data engineering or backend engineering, specifically using Python.

• Proven experience in building and managing large-scale production data pipelines utilizing platforms such as Databricks, Apache Spark, or similar.

• Demonstrated experience in developing and operating production Python services (such as FastAPI or comparable), including APIs relied upon by other engineering teams: publishing, versioning, and maintaining a stable contract.

• Familiarity with text or document data, search, indexing, or information-retrieval systems.

• Strong proficiency in data modeling, ensuring pipeline reliability, testing, and observability practices.

• A history of meeting firm milestones with inter-team dependencies.

• Preferred: Experience with retrieval-augmented generation (RAG) or vector/hybrid search in a production environment.

• Preferred: Experience in developing citation, provenance, or source-traceability functionalities for search or AI-generated responses.

• Preferred: Knowledge of SEC filings, EDGAR, XBRL, or other intricate financial and regulatory documents.

• Preferred: Experience with Azure data and AI services, including multi-model routing, usage monitoring, and cost management.

• Preferred: Background in creating evaluation datasets or pipelines for generative AI.

🏝️ Benefits

• Comprehensive Medical, Dental, Vision, and Life insurance coverage for employees and their dependents.

• Employer contribution to 401(k) plans.

• Paid time off that includes vacation, sick leave, community involvement, bereavement, jury duty, and holidays.

• Paid parental leave.

• Paid days dedicated to community involvement and volunteering.

• Access to wellness resources and Employee Assistance Programs.