Data Operations Specialist

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the health, availability, performance, and capacity of Databricks workspaces, jobs, pipelines, clusters, SQL warehouses, and associated cloud services.

• Provide daily operational support for production Databricks environments.

• Monitor and troubleshoot ETL, ELT, streaming, API, and file-based data pipelines.

• Restart, rerun, recover, reprocess, backfill, reconcile, and address failed workflows and data loads.

• Review driver and executor logs, Spark UI metrics, cluster event logs, audit logs, and application telemetry to ascertain root causes.

• Support Databricks Workflows, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, and batch-processing workloads.

• Maintain configurations for clusters, serverless compute, instance pools, cluster policies, SQL warehouses, and environments.

• Coordinate platform upgrades, runtime and library updates, deployments, releases, changes, rollbacks, and post-deployment monitoring.

• Act as first- or second-level support for production incidents; triage, coordinate responses, provide status updates, escalate, and perform root-cause analyses.

• Retain incident, problem, service-request, change, deployment, and operational records.

• Develop and maintain dashboards, alerts, operational reports, health indicators, SLIs, and SLOs.

• Monitor job success rates, latency, processing duration, utilization, freshness, quality, and compute consumption.

• Validate data contracts, schemas, record counts, control totals, data quality, lineage, audit evidence, and governance requirements.

• Assist with identity, access, entitlement, Unity Catalog, secrets, security, privacy, backup, recovery, retention, archival, and disaster-recovery activities.

• Analyze resource consumption, job duration, query performance, storage, and workload patterns; recommend optimizations for cost and performance.

• Create and maintain operational procedures, troubleshooting guides, escalation matrices, runbooks, inventories, configuration documentation, and recovery requirements.

• Identify automation and continuous-improvement opportunities and engage in operational-readiness reviews.

⛳️ Requirements

• Bachelor's degree in computer science, information technology, data engineering, information systems, or a related field, or equivalent professional experience.

• Minimum of four years of experience in data operations, production support, data engineering, cloud operations, DevOps, platform support, or a similar technical role.

• At least two years of hands-on experience supporting Databricks environments or production Databricks workloads.

• Experience in supporting production ETL, ELT, batch, or streaming data pipelines.

• Proficient knowledge of Databricks, Apache Spark, PySpark or Python, SQL, Delta Lake, Databricks Workflows or Jobs, cloud object storage, data pipeline monitoring, log analysis, and troubleshooting.

• Experience in investigating failed jobs, delayed pipelines, schema issues, data discrepancies, and performance challenges.

• Familiarity with medallion architecture and Bronze, Silver, and Gold data layers.

• Experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud.

• Proficient in using ticketing and service-management tools such as ServiceNow, Jira, or a similar platform.

• Experience with monitoring, alerting, logging, or observability tools.

• Understanding of incident, problem, change, and release-management processes.

• Capability to write and maintain technical procedures and operational documentation.

• Strong analytical, organizational, communication, and troubleshooting abilities.

• Ability to collaborate effectively with engineers, business analysts, architects, security teams, and nontechnical stakeholders.

• Capacity to support multiple production priorities in a structured and timely fashion.

• Preferred: Databricks Certified Data Engineer Associate or Professional; Databricks Certified Associate Developer for Apache Spark; Microsoft Azure, AWS, or Google Cloud certification.

• Preferred: Experience with Unity Catalog, Delta Live Tables, Lakeflow, Auto Loader, Structured Streaming, Databricks SQL, cloud services, CI/CD tools, observability platforms, APIs, Kafka, Event Hubs, Kinesis, Airflow, dbt, Snowflake, Power BI, data contracts, schema registries, metadata management, data lineage, data-quality frameworks, OCDS, procurement, contracting, financial or public-sector data, regulated or high-security environments, SLAs and 24-hour support, disaster recovery, backup, archival, and operational-resilience testing.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance plans.

• Generous paid time off and holiday schedules.

• Opportunities for professional development and certification.

• A collaborative and dynamic work environment.