Cloud Infrastructure & Platform Specialist, Databricks, AWS, Azure – Mid-Level

This is a fully remote position, open to applicants in India.

📋 Description

• Provision, configure, and oversee AWS and Azure cloud infrastructure that supports Irth’s Databricks-based data estate.

• Manage VPCs/VNets, subnets, route tables, storage, compute resources, private endpoints, and associated networking services.

• Develop and maintain secure DEV, QA, and PROD environments with appropriate isolation and platform guardrails.

• Assist in the implementation of cloud landing-zones while enforcing security, governance, and cost policies.

• Version, review, test, and deploy infrastructure changes through managed workflows.

• Diagnose infrastructure, networking, and cloud-service issues impacting data and machine learning workloads.

• Oversee IAM roles, service principals, managed identities, groups, access policies, RBAC, identity federation, and access reviews.

• Create and manage Databricks workspaces, Unity Catalog metastores, catalogs, schemas, storage credentials, external locations, and permissions.

• Facilitate the onboarding of Data Engineering, Data Science, and MLOps projects with suitable platform resources and access.

• Define and uphold Databricks cluster policies and workspace-level guardrails.

• Serve as the primary escalation point for issues related to the Databricks platform and infrastructure.

• Implement private connectivity, network segmentation, encryption, key management, audit logging, and compliance controls.

• Monitor spending, enforce tagging and resource policies, support showback/chargeback, and optimize cloud and Databricks costs.

• Automate the provisioning of workspaces, catalog and schema setup, IAM assignments, access provisioning, environment configurations, and platform onboarding.

• Monitor platform health, respond to incidents, troubleshoot production issues, and maintain runbooks and technical documentation.

⛳️ Requirements

• 3–5 years of experience in cloud infrastructure, platform engineering, cloud administration, or a closely related field.

• Practical experience with both AWS and Microsoft Azure, including IAM and role provisioning, cloud networking, storage services, and compute/platform resources.

• Direct experience managing Databricks, encompassing workspace provisioning and configuration, Unity Catalog, catalogs and schemas, metastores, cluster policies, and data access controls.

• Working knowledge of Infrastructure-as-Code (IaC) tools; Terraform is preferred.

• Experience with Azure ARM/Bicep or AWS CloudFormation is advantageous.

• Strong grasp of IAM concepts, such as RBAC, least-privilege access, service principals, managed identities, SSO, SCIM, and SAML.

• Familiarity with VPC/VNet architecture, subnets and routing, private connectivity, network segmentation, secrets management, encryption, and key management.

• Ability to efficiently troubleshoot infrastructure, networking, identity, permission, and platform access issues.

• Excellent communication skills for articulating infrastructure and access challenges to technical teams.

• Awareness of SOC 2, ISO 27001, and GDPR standards.

• Understanding of audit logging, access reviews, administrative activity tracking, and evidence collection.

• Preferred: Experience supporting Data Engineering, Data Science, and MLOps teams.

• Preferred: Experience with CI/CD, GitHub Actions, Databricks Asset Bundles, infrastructure deployment automation, and transitioning from DEV to QA to PROD.

• Preferred: Knowledge of FinOps and cloud cost management practices.

• Preferred: Experience in implementing cloud and Databricks policies.

• Preferred: Relevant industry certifications such as AWS Solutions Architect, Microsoft Azure Solutions Architect, Azure Administrator, or Databricks Platform Administrator.

• Nice to have: Experience in Asset Integrity Pipeline, utility, energy, oil & gas, pipeline, GIS, geospatial, regulatory compliance, data residency, auditability, and controlled-access environments.

🏝️ Benefits

• Competitive compensation package based on experience and qualifications.

• Medical, Dental, and Vision Insurance.

• 401(k) Plan with Company Match.

• Generous Paid Time Off (PTO).

• Company-Paid Holidays.

• Flexible Work Options – Work-from-home opportunities are available, depending on role and business needs.

• On-Call Compensation – Additional pay for eligible on-call shifts.