Senior Syslog Engineer

📋 Description

• Design, implement, and enhance syslog-ng configurations tailored for environments with high log ingestion volumes.

• Develop and manage intricate filtering logic to ensure precise routing, normalization, and reduction of log noise.

• Analyze and optimize the performance of log pipelines focusing on CPU, memory, latency, and throughput.

• Construct efficient, scalable, and fault-tolerant syslog architectures.

• Address issues related to elevated CPU/memory usage, message drops, backpressure, ordering and duplication problems, as well as Network/TCP/TLS ingestion challenges.

• Optimize mechanisms for buffering, batching, and flow control within syslog-ng.

• Collaborate closely with SIEM platforms (e.g., Securonix, Splunk, ELK) to guarantee seamless log ingestion.

• Ensure the integrity, reliability, and completeness of logs throughout the pipeline.

• Apply best practices for log parsing (RFC3164, RFC5424), handling structured vs unstructured logs, and implementing Secure syslog (TLS).

• Conduct capacity planning and load testing for syslog pipelines.

• Develop testing frameworks to validate syslog filters and configurations.

• Document standards, guidelines, and reusable configurations.

⛳️ Requirements

• Extensive experience with syslog-ng (mandatory).

• Strong comprehension of syslog protocol internals: RFC3164, RFC5424, and TCP/UDP/TLS behavior.

• Proficiency in designing syslog filters and routing logic.

• Significant experience in log parsing, pattern matching, and regex optimization.

• Experience in tuning parameters such as log- iw -size, log- fifo -size, flush_lines, so_rcvbuf, as well as disk-buffer and memory management.

• Understanding of backpressure, buffering, and flow control mechanisms.

• Experience working with high EPS environments (10K–100K+).

• Capability to troubleshoot issues such as message loss, duplicate events, out-of-order processing, and high CPU/memory usage.

• Strong Linux debugging skills, including tools like tcpdump, netstat, ss, top, and strace.

• Familiarity with one or more of the following: Splunk, ELK, or QRadar.

• Understanding of log ingestion pipelines (Kafka, Spark, etc.).

• Knowledge of data enrichment and normalization practices.

🏝️ Benefits

• Equal employment opportunities (EEO) for all employees and applicants.

• Adherence to non-discrimination policies in employment.

• Strict prohibition against unlawful harassment of employees.