Senior Data Engineer, Consultant

This is a fully remote position, open to applicants in India.

📋 Description

• Educate senior leadership and architects on traditional and AI-native data engineering principles.

• Evaluate the existing data infrastructure comprehensively, map data flows, identify gaps and technical liabilities, and create both a current-state and target-state assessment along with a prioritized roadmap.

• Design the architecture for the target data platform, encompassing ingestion, transformation, storage, serving, and observability.

• Architect the migration to PostgreSQL and establish a multi-workload environment, which includes pgvector, analytical tasks, transactional queries, pooling, replicas, partitioning, cutover, validation, query translation, and benchmarking.

• Develop a canonical data layer aimed at entity resolution, schema alignment, conflict resolution, temporal alignment, versioning, and auditability.

• Create data-residency architecture for a multi-tenant, data-sharing platform that spans various regions and jurisdictions.

• Formulate a data governance framework that addresses access control, classification, lineage, retention, consent, quality accountability, and security measures.

• Identify and prioritize data-platform use cases, including cross-source intelligence, behavioral insights, enrichment, semantic search, compliance detection, and analytics/reporting.

• Generate architecture decision records, data-flow diagrams, evaluation guides for tools, migration runbooks, and training presentations.

• Offer continuous architecture evaluations, design consultations, and progress check-ins throughout the execution phase.

• Deliver knowledge transfer, architectural advice, strategic documentation, and reference materials rather than primarily focusing on writing production code.

⛳️ Requirements

• 8–10+ years of experience in building data platforms across diverse sources at significant scale.

• Extensive knowledge of relational databases, particularly PostgreSQL.

• Hands-on experience with PostgreSQL extensions such as pgvector, Citus, PostGIS, or similar technologies.

• Familiarity with replication topologies, partitioning, and performance optimization.

• Preferably experienced in migrating from SQL Server to PostgreSQL.

• Background in designing canonical data models across varied sources, including entity resolution, master data management, and conflict resolution at scale.

• Proven experience architecting systems that comply with multi-region data residency, ideally within a multi-tenant SaaS environment.

• Strong grasp of data governance concepts, including access control, data classification, lineage, retention policies, and regulatory compliance.

• Knowledge of dimensional modeling, ETL/ELT processes, Change Data Capture (CDC), orchestration, and query optimization.

• Active and informed engagement with AI-native methodologies including ML-driven quality, semantic matching, embedding pipelines, and LLM-assisted development.

• Capability to design multi-layer platforms and make defensible technology choices considering scale, cost, team size, and maintainability.

• Exceptional communication and instructional skills with senior architects and leadership.

• Experience in defining data use cases aligned with business outcomes.

• Availability for approximately 4–5 hours daily, 20–25 hours weekly, with overlap during US working hours (6:00 PM–11:00 PM IST).

• Strongly preferred experience with Python/Celery, Elasticsearch, Azure cloud services, .NET APIs, Kestra, or similar orchestration tools.

• Familiarity with OCR, layout-aware extraction, and table parsing from financial PDFs and Word documents.

• Knowledge of dbt, Airflow/Dagster, Airbyte/dlt, Snowflake/Databricks.

• Familiarity with vector databases, RAG pipelines, feature stores, and embedding workflows.

• A background in financial services, investment management, or due diligence is beneficial but not mandatory.

• Proven track record of producing technical documentation and training resources.

🏝️ Benefits

• Remote flexibility: work from any location in India.

• Startup atmosphere with a flat organizational structure.

• Opportunity to design for a platform utilized by financial institutions in over 150 countries.

• Collaborate with a global team.

• Direct influence and swift delivery of work.