Trade offs of the terrible syslog protocol

syslog is a very old message transmission protocol that transmits system messages across a network. The first versions of this protocol were drafted into RFC 5426. Some assumed updating the transmission to use TCP would make things better, and the IETF released RFC 6587 describing syslog over TCP. The problem is, that is inherently unreliable as well, since the application (syslog) has no mechanism to ensure that all messages transmitted were actually received, regardless of the network level transport protocol used to convey the messages.

Rainer Gerhards wrote a blog post on the unreliability of using plain TCP to transmit syslog event data.

An attempt to create a reliable syslog protocol is described in RFC 3195, the problem is that very few vendors have adopted that standard (BEEP).

There is a movement to find a more reliable system message delivery mechanism, as described in this Wikipedia post, however the problem is not only one of a technically feasible mechanism – one that relies on the application itself to validate and guarantee message integrity and completeness – but also on wide spread adoption by the 10’s or 100’s of millions of devices that send their system logs via syslog UDP.

That will take decades, so best is to use mechanisms that can collect the event messages in native syslog UDP format as close to the generating source as possible then use an application oriented framework to convey those messages to their destination. HP ArcSight SmartConnectors are a good way to accomplish this, with their application level event queuing on input, persistent caching output, compression, encryption, bandwidth throttling, filtering, aggregation and event QoS policies.