2

I have a rsyslog that forwards messages to elasticsearch. It worked well for over a year but recently the disk starts to fill mysteriously until it reaches 100%.

When I restart the process the disk empties to normal values instantly, this makes me believe that the process is the one that fills up the disk.

Rsyslog version is rsyslogd 8.2002.0 (aka 2020.02)

Here is a jinja2 example of my queue config:

if $msg contains "api_logs_rsyslog" then {
action(type="mmjsonparse" cookie="")

if $parsesuccess == "OK" then {
    if $!isIndexable == 0 then {
        action(type="omfile" file="{{ rsyslog.disk_log_path }}"
                ioBufferSize="64k" flushOnTXEnd="on"
                asyncWriting="off") stop
    } else {
        action(type="omelasticsearch"
            name="api_logs_rsyslog"
            server="{{ rsyslog.elasticsearch.server }}"
            serverport="9200"
            template="messageToES"
            dynSearchIndex="on"
            searchIndex="searchIndex-template"
            bulkmode="on"
            dynbulkid="on"
            bulkid="bulkid-template"
            maxbytes="100m"
            action.resumeRetryCount="-1" #eternal
            action.resumeInterval="300" #5min
            action.resumeIntervalMax="43200" #12h
            queue.type="linkedlist"
            queue.workerThreads="8"
            queue.size="250000"
            queue.dequeuebatchsize="4096"
            queue.workerThreadMinimumMessages="10000"
            queue.fileName="{{ rsyslog.elasticsearch.api_queue}}_disk"
            queue.maxDiskSpace="1g"
            queue.saveOnShutdown="on"
            uid="admin"
            pwd="{{ rsyslog.elasticsearch.passwd }}"
            errorfile="{{ rsyslog.elasticsearch.api_errorfile }}" 
            ) stop
    }
} else {
    action(type="omfile" File="{{ rsyslog.elasticsearch.parse_failed }}") stop
}}

I've debugged the queues with impstats and they dont get filled ( my first thought was that the queues would fill and then switch to disk assisted scenario )

My only solution right now is a cron that restarts rsyslog after a few hours, but this obviously doesn't feel right.

I will try to look for updates, I think there are a couple of minor versions to be installed.

Did anyone encounter this behaviour?

0

You must log in to answer this question.

Browse other questions tagged .