How to print content of all (compressed, rolled over and current) Apache log files

Question

Apache produces log files (i.e. httpd.log). After a given time, log files are rolled over (i.e into httpd.log-20210627). When a log file is rolled over again, the previous rolled over version is compressed (i.e. into httpd.log-20210620.gz).

As root, I can list all these files.

[root@server user]# ls -latr /var/log/httpd/localhost/httpd.log*
-rw-r--r-- 1 root root  9364779  6. Jun 03:48 /var/log/httpd/localhost/httpd.log-20210606.gz
-rw-r--r-- 1 root root  8407071 13. Jun 03:27 /var/log/httpd/localhost/httpd.log-20210613.gz
-rw-r--r-- 1 root root  7099637 20. Jun 03:35 /var/log/httpd/localhost/httpd.log-20210620.gz
-rw-r--r-- 1 root root 91551338 27. Jun 03:45 /var/log/httpd/localhost/httpd.log-20210627
-rw-r--r-- 1 root root 49229622 30. Jun 17:28 /var/log/httpd/localhost/httpd.log
[root@server user]# 

Using bash, I'd like to print the content of all log files, in temporal order, as an otherwise unprivileged user (that is: using sudo, not in a root shell), so I can, for example, grep through them.

Printing the content of the current log file is easy, because it has a fixed filename:

sudo cat /var/log/httpd/localhost/httpd.log

Printing the content of the compressed log files is almost as easy, because I can write a pattern that matches all relevant compressed files and use zcat to print the uncompressed content:

sudo su - -c "zcat /var/log/httpd/localhost/httpd.log-*.gz"

But so far I couldn't figure out how to print the content of the rolled over but not yet compressed log file. This is because I don't know the actual filename and a pattern like /var/log/httpd/localhost/httpd.log* will match all the compressed files as well.

According to the documentation, I can make use of more complex patterns, when I enable extglob:

If the extglob shell option is enabled using the shopt builtin, several extended pattern matching operators are recognized. []

[…]

+(pattern-list)

Matches one or more occurrences of the given patterns.

According to this answer, I need to use a subshell, If I want to enable extglob in a single command. And indeed, this works as expected in a root shell. Executing shopt -s extglob and then cat /var/log/httpd/localhost/httpd.log-+([0-9]) ) prints the content of the log file in question.

Outside of a root shell, however, I get an error:

[user@server ~]$ sudo su - -c "( shopt -s extglob ; cat /var/log/httpd/localhost/httpd.log-+([0-9]) )"
-bash: -c: line 0: syntax error near unexpected token `('
-bash: -c: line 0: `( shopt -s extglob ; cat /var/log/httpd/localhost/httpd.log-+([0-9]) )'
[user@server ~]$ 

This is the same error that I get when I try to execute cat /var/log/httpd/localhost/httpd.log-+([0-9]) in the root shell without enabling extglob. I'm not sure why this doesn't work, because enabling extglob appears to be working in the given circumstances:

[user@server ~]$ sudo su - -c "( shopt -s extglob ; shopt extglob )"
extglob         on
[user@server ~]$ 

So I'd like to know how I can print the content of a file with an unknown filename outside of a root shell, when I need extglob to write the pattern. But this may be an xy-problem, so my actual question is:

How can I print the content of all of Apache's log files outside of a root shell.

Answer 1

zgrep

You can use zgrep to search a folder of both compressed and non-compressed logs.

The man page writes about zgrep:

Zgrep invokes grep on compressed or gzipped files. All options specified are passed directly to grep. If no file is specified, then the standard input is decompressed if necessary and fed to grep. Otherwise the given files are uncompressed if necessary and fed to grep.

I have tried re-creating your example with a current log, an older log and a compressed log:

saaru >ls -o
total 12
-rw-r--r-- 1 ba 25 Jun 30 23:06 httpd.log
-rw-r--r-- 1 ba 66 Jun 30 23:07 httpd.log-20210530.gz
-rw-r--r-- 1 ba 29 Jun 30 23:07 httpd.log-20210630

The contents of the files are as follows:

saaru >cat httpd.log
This is the current log

saaru >cat httpd.log-20210630
This is the rolled over log

saaru >zcat httpd.log-20210530.gz
This is the compressed log.

With below command you can find the word "compressed" in the logs:

saaru >zgrep "compressed" httpd.log*
httpd.log-20210530.gz:This is the compressed log.

Or find all log lines that do not have that word:

saaru >zgrep -v "compressed" httpd.log*
httpd.log:This is the current log
httpd.log:
httpd.log-20210530.gz:
httpd.log-20210630:This is the rolled over log
httpd.log-20210630:

zcat

If you don't want to search, but simply cat all files you can use zcat with the -f flag:

saaru >zcat -f httpd.log*
This is the current log

This is the compressed log.

This is the rolled over log

Further reading

Here is an article on all the different z* commands.