Sending rotated files with syslog-ng

Question

Hello I am currently sending the suricata logs from a probe to a Syslog-ng server(172.16.238.15):

@version: 3.25
options {
    keep-timestamp (no);
    chain_hostnames (off);
    keep_hostname (no);
};
source s_eve {
    wildcard-file(
        base-dir("/var/log/suricata/")
        filename-pattern("*.json")
        flags(no-parse) 
        keep-timestamp(no) 
        keep-hostname(no) 
    );   
};
destination d_eve {
    network(
        "172.16.238.15" transport("tls") port(6514)
        tls( ca_dir("/etc/syslog-ng/ca.d"))
        template("$MSG\n")
    );
 };
log { 
    source(s_eve);
    destination(d_eve);
};

And it works fine, the problem is when I try to rotate the logs in the probe. I am using logrotate and the main log-file that suricata generates is eve.json and when rotated it creates a eve.1.json, eve.2.json up to 5.

So the problem is that I get duplicated logs in the Syslog-ng server because everything that is inside eve.1.json or eve.2.json has already been in eve.json for a while before being rotated, and it has already been sent to the server, but as it is a new file, it sends them again. However if I just set it up for Syslog-ng to send just the original eve.json I am risking to not send some logs if they are rotated before there was connection to the server.

Is there any configuration for Syslog-ng to understand the rotated files as just one, or what is the approach to solve this?

Answer 1

You can configure Suricata to create a new "eve.json" log file when it is rotated. This ensures the log events are not duplicated when the files are sent to syslog-ng.

The main idea here is to run a postrotate script in logrotate configuration which sends USR2 signal to Suricata. Suricata reacts on this signal with closing and re-opening log files.

Here's a sample logrotate configuration:

/etc/logrotate.d/suricata

/var/log/suricata/*.json {
    daily
    rotate 5
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -USR2 `cat /var/run/suricata.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

The postrotate script sends a USR2 signal to Suricata, making it close the current log file and start a new one. Use of delaycompress directive will make sure that the last log file doesn’t get compressed, therefore it can be written into without issues.

Remember to replace /var/run/suricata.pid with the actual location of the Suricata PID file.

The syslog-ng configuration will continue as it is currently, as it will always read the current "eve.json" file. Any events that were written to the previous "eve.json" (which got rotated) would have been sent to syslog-ng prior to rotation.

syslog-ng recognizes new files, but does not follow renamed files after rotation. This configuration works well with this behavior because a new "eve.json" is created after every rotation.

Edit:

USR2 and HUP are different types of signals that can be sent to a process. They're both user-defined and do not have a specific predefined function - their behavior is defined by the program receiving the signal.

For Suricata, sending a HUP signal tells it to close and reopen its log files, which is very handy when those logs are being rotated. USR2 signal on the other hand, tells Suricata to reload its configuration.

Regarding the issue of unstable connection, if log entries are written to the logs faster than syslog-ng can send them due to network issues, you would indeed have a problem with missing log entries. This could be mitigated by increasing the number of rotated log files kept and also checking the compressed log files in syslog-ng configuration (if your syslog-ng version supports reading compressed files).

Remember that any solution to this issue would be a balance between ensuring all log entries are sent and managing disk space usage.

Also you may want to investigate reliable log transport protocols (like RELP) or using a queue/buffer with disk-based storage in syslog-ng which can spool log messages to disk when the remote server is not accepting messages.