Rsyslog: relay messages from UDP input only

Question

Considering a docker container which receives logs on UDP and forwards to a central logging server using TLS, I was wondering if I could be satisfied with one queue or if I needed several.

Indeed, knowing that I won't be doing anything else than sending the logs to the collector, I don't really understand the need for several queues. Surely they make sense if other actions exist such as writing logs on files or duplicate outputs for failover, but what if forwarding is the only task? Wouldn't it be enough to have the main queue and the "send to collector" action in direct queue mode (i.e. no queue)? If the central server goes down (or during network outage) then logs will simply be re-enqueued to the main message queue ?

For example in this scenario below the Action Q makes total sense, but if we discard logging on /var/log/messages wouldn't Action Q be useless ? It's not only useless but also slows down the forwarding, right?

Scenario diagram: https://i.sstatic.net/8kFQP.png (can't post picture)

Answer 1

To start, a quick introduction into rsyslog and queues.

Each input to rsyslog is through one or more threads, which gather the log messages and add them to the main queue. Worker threads then pull messages off the main queue and deliver them to their destinations and/or add the message to an action queue. (This is basically what the image you posted is doing)

If a worker is unable to deliver messages to a destination, all progress of that queue will block until that delivery is able to succeed (or it hits the retry limit and permanently fails). If you don’t want this to block all log processing, you should make an action queue for that destination (or group of destinations).

Now in your case, the last bit probably doesn't interest you, as you only want to relay the logs.

So to come to the point, each action has a dedicated queue. This queue can be in-memory, it can be on disk or it can be a combination of both. There is also direct mode, which means by design there is a queue but the actual driver does forward messages to the action without action queueing.

Therefore direct queues are probably the queue type you are looking for. In case the output action fails, the action processor notifies the action queue, which then takes an unprocessed element back and after some time interval, the action is attempted again. So direct queues also have the benefits of a normal queue.

Now to the last question: It's not only useless but also slows down the forwarding, right?

I am not really sure, if direct queues are (that much) faster than action queues in linkedList or fixedArray mode, as I have never tested it (or even used direct queues) before.

Edit: For more in-depth knowledge see the answer below.

Answer 2

you want to use queues where you need to decouple processing of messages. In your simple (and common) example, you are writing to a local file and sending messages to a remote machine via TCP.

Sending logs via TCP is a process that can block. If there is a network problem or a server problem, the sender pauses delivery of logs. If there is not a queue for that action, then this will mean that you will also not write logs to the local files (including such useful information as logs saying that the remote system is down ;-) )

so in this case, you do want an action queue, so that the TCP connection can block without blocking writes to /var/log/messages.

A couple other things to think about here.

1. queues can be placed on actions or rulesets, if you are sending to multiple places (or want to failover from one to another), you want to group them in a ruleset and put the queue on the ruleset.

2. you REALLY want to use the new action() syntax if you are putting a queue on an action, the old syntax ($foo lines before the action) is FAR too easy to misunderstand what's happening

3. you can still lose messages in transit with TCP, when rsyslog gives the message to the OS on the sending machine, if that OS accepts it (i.e. doesn't say "the queue is full, wait"), rsyslog on the sender is forced to assume that it's going to be delivered. But if there is an outage (network or receiving server), rsyslog will never know that the message has been lost. The RELP protocol is designed to handle this case and resend messages if there are network/receiver issues.

as far as queues go, a 'direct queue' is 'no queue', it's the normal thing that happens when you have multiple actions

so a traditional

mail.* /var/log/mail kern.* /var/log/kern

is an example of 'direct queues' They are the fastest way to process messages.

disk queues save every message to disk before processing and are EXTREMELY slow (something on the order of 1000x slowdown), but will survive a system crash

memory queues store the logs in ram (linkedlist of fixedarray differ in how they allocate ram and a bit of speed)

Disk Assisted queues are memory queues that will overflow to disk files rather than blocking further processing (draining the queues does not happen in order, the disk portion is drained in parallel with processing logs from the memory queue). This is a good choice to handle long outages, just be aware that after the outage, you will start getting current logs quickly, but older logs will arrive at a slower rate.

It is possible to overdo it with queues. When you have a memory queue, thread A needs to lock the queue to add the messages to the queue, then thread B needs to lock the queue to retrieve the messages from the queue to process them. If you have a lightweight output (i.e. writing to a file) you spend far more CPU locking and unlocking the queue than you would just writing the messages out.