Applied Data Scientist, Finance AI Evaluation, Datasets

This is a fully remote position, open to applicants in United States.

📋 Description

• Convert customer objectives — such as enhancing financial reasoning, creating an evaluation suite for summarizing earnings calls, or assessing an AML/fraud assistant — into precise dataset specifications, taxonomies, rubrics, and acceptance criteria.

• Develop training and evaluation datasets across the financial AI domain: financial question answering, analysis of filings and earnings, credit and underwriting, fraud/AML investigations, and compliance, among other financial processes.

• Highlight unstructured and multimodal financial data in dataset design — including PDFs, scanned documents, tables, charts, and call transcripts — utilized by analysts, advisors, compliance reviewers, and operations teams.

• Create datasets and evaluations for retrieval-augmented and source-grounded systems: citation of evidence and adherence to source documents, data recency, resolution of conflicts among sources, and failure modes arising from incomplete or incorrectly parsed context.

• Assess agentic and workflow-integrated financial AI systems: tool usage, retrieval, transaction boundaries, escalation behavior, and controls to prevent unsafe or unauthorized actions.

• Formulate evaluation methodologies that extend beyond surface-level accuracy — encompassing numerical consistency, hallucination rates on high-risk claims, appropriateness of refusals and escalations, robustness under uncertainty, and fairness across protected or sensitive customer segments.

• Define sampling strategies, labeling schemas, and adjudication workflows in collaboration with Language Data Scientists and finance subject matter experts; create annotation guidelines that clarify subjective finance-domain judgments, making them explicit, calibratable, and auditable.

• Develop statistical and machine learning tools to ensure the reliability of large financial datasets: stratified sampling across products, markets, and modalities; bias analysis; leakage detection; and distribution shift checks, among other reliability assessments.

• Produce evaluation and dataset-quality evidence to support financial-services model risk management: assumptions, limitations, validation results, and residual risks, presented as reproducible evidence.

• Collaborate with the AI/ML Research Engineer to integrate datasets into training, evaluation, and monitoring pipelines — using rubric-grounded LLM-as-judge prompts, regression suites, and continuous monitoring.

• Manage data quality comprehensively, from intake to delivery: handling of personally identifiable information, tracking data provenance, version control, and modality-specific quality assurance checks.

• Analyze financial workflow context: where AI outputs are integrated into analyst, advisor, compliance, risk, or customer-facing workflows; what evidence reviewers need to trust these outputs; and when uncertainties should be highlighted.

• Assist the Technical Solutions Architect during customer discovery and proposals: defining dataset programs, estimating annotation efforts, and articulating methodology to client stakeholders.

• Remain updated on the financial AI landscape: regulatory changes, benchmark updates, and emerging evaluation methodologies for finance-domain models.

• Contribute to Innodata's internal intellectual property: reusable taxonomies, evaluation rubrics, golden datasets, and methodological templates.

⛳️ Requirements

• Over 5 years of data science experience, including a minimum of 2 years in financial services, fintech, banking, or a similar regulated data environment.

• Practical knowledge of financial data and workflows: financial statements, SEC filings, transaction data, and other standard financial-services document types.

• Experience with unstructured and multimodal financial data — encompassing various formats such as PDFs, scanned documents, spreadsheets, charts, or call transcripts.

• A strong preference for familiarity with financial standards or protocols, such as XBRL, ISO 20022, or GAAP/IFRS reporting concepts.

• Hands-on experience in designing datasets for machine learning — not merely using them. You have created annotation guidelines, sized cohorts, established quality thresholds, and delivered data that downstream teams can effectively train, evaluate, or monitor.

• Familiarity with LLM-based and multimodal financial AI processes: prompt design, rubric-based evaluation, retrieval-augmented generation, LLM-as-judge methodologies, and the limitations of automated evaluation in high-stakes contexts.

• Proficient in Python and SQL; comfortable with pandas, scikit-learn, or equivalents; working knowledge of Hugging Face, PyTorch, or model APIs.

• Sound statistical literacy: understanding of sampling design, inter-annotator agreement metrics (e.g., Cohen's kappa), confidence intervals, and the ability to challenge over-interpretation of numbers.

• Strong understanding of financial services privacy, compliance, and governance: handling of PII, GLBA or equivalent privacy regulations, MNPI sensitivity, and documentation suitable for regulated AI programs.

• Excellent collaboration skills — working effectively with a Technical Solutions Architect, research scientists, engineers, and SME annotators and quality teams.

• A preference for practical financial workflow realism. You would choose to develop a smaller dataset that accurately reflects the experiences of analysts, advisors, or customers over a larger dataset that appears impressive but lacks real-world applicability.

• A degree in a relevant field — statistics, data science, economics, finance, or a related quantitative discipline, or equivalent demonstrated experience. While formal finance credentials are not mandatory, backgrounds such as CFA, FRM, or MBA are particularly encouraged.

• Experience in designing evaluations for LLMs, VLMs, or multimodal models in contexts like financial reasoning, filings analysis, or fraud/AML scenarios.

• Experience with document AI, OCR/post-OCR quality, or extraction of tables and charts from complex financial documents.

• Familiarity with agentic evaluation, AI observability, experiment tracking, or tools such as Weights & Biases or LangFuse.

• Knowledge of model risk management frameworks, validation documentation, fairness/bias auditing, or consumer protection analysis.

• Experience with multilingual or cross-border financial data, or published/open-source contributions in financial AI or model governance.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Generous paid time off and holiday policies.

• Opportunities for professional development and continuous learning.

• Collaborative and inclusive work environment.