Data Engineering Team Lead

This is a fully remote position, open to applicants in Philippines.

📋 Description

• Oversee a team of data engineers, ensuring alignment with objectives, quality standards, and delivery timelines.

• Provide mentorship and coaching to team members to enhance their technical and professional development.

• Promote engineering excellence by advocating best practices in coding, architecture, testing, and observability.

• Strategically plan and manage team capacity, sprints, and milestones for consistent delivery.

• Take ownership of the design, evolution, and operation of ingestion and transformation pipelines using Apache Airflow and the analytical serving layer on Apache Druid.

• Make architectural decisions regarding concurrency, partitioning, memory sizing, and cost, including JVM heap and direct-memory tuning on the Druid cluster.

• Collaborate closely with DevOps on the Kubernetes / EKS platform that supports our Druid and Airflow workloads.

• Ensure comprehensive data validation, reconciliation, and verification for trustworthy reporting.

• Work alongside other Team Leaders, Development Managers, Architects, and Product Owners to ensure engineering efforts align with business objectives.

• Contribute to the enhancement of development processes, CI/CD pipelines, and DevOps practices.

• Cultivate a culture of continuous improvement, innovation, and knowledge sharing.

⛳️ Requirements

• Over 10 years of commercial experience in delivering Data & Analytics solutions.

• More than 5 years of experience leading a Data Engineering or BI team with a strong understanding of Agile methodologies (Scrum / Kanban).

• Extensive hands-on expertise in Apache Airflow, including production DAG authoring in Python; hooks, sensors, XCom, callbacks, retries; and dynamic task mapping.

• Proficient in production-grade Python — type hints, packaging, pytest; comfortable with reading and reviewing others' DAGs.

• Familiarity with Apache Druid on the ingestion side — index_parallel specs, transformSpec / dimensionsSpec / granularitySpec, tuningConfig, supervisors, task lifecycle, segment management; knowledge of Coordinator / Overlord / Broker / Historical / MiddleManager roles.

• Strong expertise in PostgreSQL — query tuning, JSONB, window functions, indexes, EXPLAIN / EXPLAIN ANALYZE.

• Proficiency in SQL across various dialects (Postgres, T-SQL, Oracle); comfortable optimizing queries.

• Knowledge of AWS data services — S3 (for Druid deep storage), EMR, Secrets Manager, IAM, VPC fundamentals; EC2 sizing for memory-bound workloads.

• Strong production debugging skills — interpreting YARN container logs, tracing failures from Airflow to EMR step to Spark driver and Python traceback.

• Exceptional communication, teamwork, attention to detail, organizational, and leadership capabilities.

• Ability to inspire, mentor, and lead by example.

• *Nice to have*:

• Experience with JVM tuning — heap (Xms / Xmx), direct memory, GC choice (G1, ZGC), reading GC logs; distinguishing between JVM OOM and cgroup OOMKilled.

• Knowledge of Kubernetes operations — pods, Deployments / StatefulSets, ConfigMaps, resource requests versus limits, HPA, kubectl describe / exec / logs.

• Familiarity with Helm / Helmfile, as most production Druid on EKS is Helm-deployed.

• Understanding of PySpark — DataFrame API, Spark SQL, JDBC reads/writes; deploy-mode cluster versus client; executor/driver memory.

• Experience with Delta Lake — MERGE semantics, time-travel, schema evolution, SCD Type 1 / 2.

• CI/CD experience on Bitbucket Pipelines (or similar: GitHub Actions / GitLab CI) — OIDC-to-AWS, deploy gates, artifact handling.

• Knowledge of observability tools — Prometheus / Grafana for Druid metrics, distributed tracing, log aggregation (CloudWatch / Loki / ELK).

• Familiarity with Terraform / Infrastructure as Code (IaC).

• Experience with other Druid ingestion sources — Kafka, Kinesis, S3 / Parquet, batch SQL.

• Skills in dbt, Spark, or Beam for source-side transformation outside Druid.

• Knowledge of Druid row-level-security patterns (RLSGroupID / parse_json transformSpec) — multi-tenant Druid experience is a plus.

• Experience in the maritime / personnel data domain (IMO, Installation, CustomerID, Personnel Pool).

🏝️ Benefits

• **Safeguard your tomorrow:**

• - Future Security with SSS & HDMF: Secure your financial future and home ownership dreams with contributions to Social Security and Home Development Mutual Fund.

• - Healthcare Assurance with Philhealth: Rest easy knowing your health is safeguarded by the Philippine Health Insurance Corporation.

• **Extra Allowances for Daily Comfort:**

• - Boost Your Budget: Receive a monthly allowance of 2,500 PHP for your everyday expenses.

• **Top-Tier Health and Wellness Coverage:**

• - Comprehensive Medical Insurance: After your initial 3 months, enjoy Maxicare's extensive medical insurance for you and one dependent, including up to 200,000 PHP per illness annually.

• - Around-the-Clock Teleconsult: Have 24/7 access to medical consultations, ensuring you and your family's health concerns are promptly addressed.

• - Annual Health Maintenance: Benefit from regular check-ups and dental care to maintain your health year-round.

• - Life Insurance Peace of Mind: Gain additional security with a life insurance policy valued at 250,000 PHP, protecting what's most important to you.

• **Employee Support and Wellbeing:**

• - Employee Assistance Programme (EAP): Our EAP, provided by Health Assured and Comp Psych, offers confidential counseling, financial, and legal support via phone, ensuring your well-being is always a priority.

• ⭐ **OTG’s Guiding Stars:**

• - ⭐**Pioneering:** Constantly charting new courses in innovation.

• - ⭐**Caring:** Keeping the maritime community's safety and sustainability at the helm.

• - ⭐**Collaborating:** Navigating together with clear communication and shared goals.

• - ⭐**Optimizing:** Always in pursuit of excellence and constructive evolution.

• **🚢 Will You Navigate the Next Chapter with Us?** Join us on a journey that transcends a traditional job—it’s a mission to innovate at the intersection of technology and education. Discover more about our vision at OneOcean and see if your path aligns with our pioneering direction. We’re eager to welcome aboard our next visionary Junior Application Consultant, ready to make a significant impact on both tech and educational fronts. Cast your resume into our waters and embark on a journey that transcends a mere job - it’s an adventure in innovation and support. Visit us at OneOcean and see if your compass aligns with ours. We’re excited to welcome our next Oceaneer onboard.

• ✊🏼 **All Hands on Deck:** We steer with equality and celebrate the diversity of our Oceaneers. OneOcean is a proud equal opportunity employer, where passion unites and differences are celebrated.