Applied Data Scientist, Finance AI Evaluation, Datasets

This is a fully remote position, open to applicants in Canada.

📋 Description

• Convert customer objectives — such as enhancing financial reasoning, creating an evaluation suite for summarizing earnings calls, or assessing an AML/fraud assistant — into actionable dataset specifications, taxonomies, rubrics, and acceptance criteria.

• Create training and evaluation datasets across the financial AI landscape: financial question answering, analysis of filings and earnings, credit and underwriting, fraud/AML investigations, and compliance, among various financial workflows.

• Highlight unstructured and multimodal financial data in dataset creation — including PDFs, scanned statements, tables, charts, and call transcripts — utilized by analysts, advisors, compliance reviewers, and operations teams.

• Develop datasets and evaluations for retrieval-augmented and source-grounded systems: evidence citation and adherence to source documents, data relevance, conflict resolution across sources, and failure modes arising from incomplete or incorrectly interpreted context.

• Assess agentic and workflow-integrated financial AI systems: tool usage, retrieval, transaction boundaries, escalation behavior, and controls that prevent unsafe or unauthorized actions.

• Formulate evaluation methodologies that extend beyond surface-level accuracy — including numerical consistency, hallucination rates on high-risk claims, appropriateness of refusals and escalations, robustness in ambiguous situations, and fairness across protected or sensitive customer segments.

• Define sampling strategies, label schemas, and adjudication workflows in collaboration with Language Data Scientists and finance subject matter experts (SMEs); draft annotation guidelines that clearly outline subjective finance-domain judgments, making them explicit, calibratable, and auditable.

• Create statistical and machine learning tools that ensure the trustworthiness of large financial datasets: stratified sampling across products, markets, and modalities; bias analysis; leakage detection; and checks for distribution shifts, among other reliability assessments.

• Construct evaluation and dataset-quality evidence to support model risk management in financial services: assumptions, limitations, validation results, and residual risks, presented as reproducible evidence.

• Collaborate with the AI/ML Research Engineer to integrate datasets into training, evaluation, and monitoring pipelines — including rubric-based LLM-as-judge prompts, regression suites, and ongoing monitoring.

• Manage data quality throughout the entire process, from intake to delivery: handling of personally identifiable information (PII), provenance tracking, version control, and modality-specific quality assurance checks.

• Analyze financial workflow context: where AI outputs integrate into analyst, advisor, compliance, risk, or customer-facing workflows; what evidence a reviewer needs for trust; and when uncertainty should be communicated.

• Assist the Technical Solutions Architect during client discovery and proposal processes: defining dataset programs, estimating annotation efforts, and articulating methodologies to client stakeholders.

• Remain updated on the financial AI landscape: regulatory changes, benchmark releases, and novel evaluation methodologies for finance-domain models.

• Contribute to Innodata's internal intellectual property: reusable taxonomies, evaluation rubrics, high-quality datasets, and methodology templates.

⛳️ Requirements

• Over 5 years of data science experience, including a minimum of 2 years in financial services, fintech, banking, or a similar regulated data environment.

• Practical knowledge of financial data and workflows: financial statements, SEC filings, transaction data, and other typical types of financial-services documentation.

• Direct experience with unstructured and multimodal financial data — encompassing PDFs, scanned documents, spreadsheets, charts, or call transcripts.

• Familiarity with financial standards or protocols such as XBRL, ISO 20022, or GAAP/IFRS reporting concepts is highly preferred.

• Hands-on experience in designing datasets for machine learning — not merely utilizing them. You have created annotation guidelines, determined cohort sizes, established quality thresholds, and delivered data that downstream teams can effectively train, evaluate, or monitor.

• Knowledge of LLM-based and multimodal financial AI workflows: prompt design, rubric-based evaluation, retrieval-augmented generation, LLM-as-judge strategies, and the limitations of automated evaluation in high-stakes situations.

• Proficient in Python and SQL; comfortable with pandas, scikit-learn, or similar tools; working familiarity with Hugging Face, PyTorch, or model APIs.

• Strong statistical knowledge: sampling design, inter-annotator agreement metrics (e.g., Cohen's kappa), confidence intervals, and the ability to challenge data misinterpretation.

• Solid understanding of privacy, compliance, and governance in financial services: handling of PII, GLBA or equivalent privacy regulations, sensitivity of MNPI, and documentation suitable for regulated AI programs.

• Exceptional collaboration abilities — working with a Technical Solutions Architect, collaborating with research scientists and engineers, and engaging with SME annotators and quality teams.

• A focus on financial workflow realism. You prefer to develop a smaller dataset that accurately reflects what analysts, advisors, or customers encounter over a larger one that may appear impressive but is impractical.

• A degree in a relevant field — statistics, data science, economics, finance, or a related quantitative discipline, or equivalent demonstrated experience. Formal finance credentials are not necessary, but backgrounds such as CFA, FRM, or MBA are particularly welcomed.

• Experience in designing evaluations for LLMs, VLMs, or multimodal models in contexts such as financial reasoning, filings analysis, or fraud/AML investigations.

• Familiarity with document AI, OCR/post-OCR quality, or extraction of tables and charts from complex financial documents.

• Knowledge of agentic evaluation, AI observability, experiment tracking, or tools like Weights & Biases or LangFuse.

• Familiarity with model risk management frameworks, validation documentation, fairness/bias auditing, or consumer protection analysis.

• Experience with multilingual or cross-border financial data, or contributions to published/open-source work in financial AI or model governance.

🏝️ Benefits

• Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at https://consumer.ftc.gov/articles/job-scams.

• If you believe you’ve been targeted by a recruitment scam, please report it to Innodata at verifyjoboffer@innodata.com and consider reporting it to the FTC at ReportFraud.ftc.gov.