Senior Data Pipeline Engineer – Elastic

This is a fully remote position, open to applicants in Washington.

📋 Description

• Design, automate, and sustain scalable, fault-tolerant data processing pipelines for the ingestion of asset, vulnerability, and configuration data from Armis, Axonius, and other SaaS/API sources into Elasticsearch.

• Manage high-volume and high-velocity data collection, processing, and indexing to facilitate near-real-time visibility.

• Develop Elasticsearch index strategies, mappings, ingest pipelines, and transformations; standardize diverse source data to a unified schema (e.g., Elastic Common Schema) for consistent searching, correlation, and reporting.

• Create and uphold Kibana dashboards and visualizations that provide a comprehensive view of the agency's technical posture.

• Oversee API integration patterns for evolving vendor APIs, which includes authentication and token management, pagination, rate limiting, incremental/delta ingestion, and robust error handling and retry mechanisms.

• Execute automated reconciliation and data quality checks to ensure the completeness, consistency, and accuracy of ingested data compared to source systems; document and maintain data lineage and traceability of transformed data back to the source.

• Manage index lifecycle (ILM), retention, sharding, and performance tuning of clusters for high-volume, low-latency operations.

• Equip pipelines with monitoring, alerting, and logging capabilities to detect and resolve data-flow issues prior to reaching stakeholders.

• Adhere to programming best practices, focusing on code reusability and highly automated CI/CD pipelines for scalability.

⛳️ Requirements

• US citizenship with the capability to obtain Public Trust Suitability.

• Bachelor’s degree in Computer Science, Information Systems, Data Engineering, or a related discipline (master's degree is preferred).

• At least six years of experience in designing, implementing, and managing the technical delivery of large-scale data engineering or data pipeline services across diverse organizations.

• Over 4 years of hands-on experience with Elasticsearch or OpenSearch: including index design, mappings, ingest pipelines, querying/aggregation, and performance tuning of clusters.

• Extensive experience integrating REST APIs from SaaS platforms, covering authentication, pagination, rate limiting, incremental ingestion, and resilient error handling.

• Proficient in a scripting/pipeline language (Python is highly preferred) and ETL/ELT tooling such as Logstash, Elastic Agent/Beats, or equivalent.

• Experience in designing, implementing, and managing data schemas that align with cyber-relevant data standards and frameworks, such as SCAP, STIX, OTEL, Elastic Common Schema, and Splunk Common Information Model.

• Understanding of data transformation processes and experience in documenting and maintaining data lineage, particularly the traceability of transformed data to source data.

• Strong data modeling and normalization abilities for heterogeneous sources, including mapping to a common schema.

• Familiarity with Federal Cybersecurity and Risk Management policies and practices, including FISMA; NIST SP 800-37, 800-53, 800-30, and 800-39; and FedRAMP.

• Solid grasp of Git, version control workflows, and CI/CD.

🏝️ Benefits

• 100% coverage of medical, vision, dental, and life insurance premiums provided by SERVISS.

• 401(k) retirement plan with a 6% dollar-for-dollar match.

• Opportunities for annual performance bonuses and growth incentives.

• Join an exciting company with ground floor opportunities in the Equity Participation Program.

• Highly competitive salary and top-tier benefits.