Data Platform Architect

This is a fully remote position, open to applicants in India.

📋 Description

• Direct architectural choices regarding compute engine selection, implementation of open-table formats, and the design of tiered storage.

• Create a decoupled and interoperable data environment that prevents proprietary vendor lock-in.

• Design and manage automated data governance, which includes PII discovery, row/column-level security, and audit capabilities.

• Establish standards for data pipeline completion, coding practices, CI/CD methodologies, and requirements for technical documentation.

• Keep a version-controlled archive of architectural decision records that detail rationale, trade-offs, and long-term consequences.

• Oversee and enhance platform performance and cloud expenditure while ensuring optimal query performance and a streamlined cloud presence.

• Conduct reviews of code and designs concerning data models and orchestration workflows.

• Mentor and assist senior engineering personnel in overcoming obstacles.

• Communicate technical debt and architectural strategies to C-level executives.

⛳️ Requirements

• Over 8 years of experience in data engineering or architecture.

• Demonstrated success in delivering production-grade Lakehouse environments for organizations with more than 1,000 concurrent users.

• Extensive experience with Databricks Lakehouse and Unity Catalog.

• Familiarity with either Amazon Redshift or Snowflake.

• In-depth practical knowledge of Apache Iceberg or Delta Lake, particularly in partitioning optimization and schema evolution.

• Proficient in dbt Core for advanced SQL-based modeling.

• Skilled in PySpark or Python for advanced data processing tasks.

• Knowledge of DuckDB.

• Advanced experience with Apache Airflow, specifically in designing resilient, dependency-aware DAGs in environments with limited resources.

• Expert-level familiarity with AWS services such as S3, EC2, and IAM, or equivalent services from Azure/GCP.

• Experience with Apache Polaris or other open-source catalog implementations.

• Understanding of DataOps principles and automated data quality testing frameworks.

• Proven ability to communicate technical debt and architectural strategies to C-level executives.

• Experience managing fixed-resource infrastructures compared to elastic serverless models.

• Comprehensive understanding of Kubernetes, including Pods, Deployments, Services, ConfigMaps, and Secrets management.

🏝️ Benefits

• Competitive industry salaries.

• Comprehensive benefits packages.

• Support for employee well-being.

• A People First culture.

• An inclusive environment.

• Opportunities for professional growth.

• Internal mobility opportunities.

• Reasonable accommodations provided during the hiring process.

• State-of-the-art workspaces.

• Top-notch benefits.

• Support for career growth.