How to view a log file that's worth 10GB+?

Question

The ways that I've thought of:

1. If you get to reproduce the scenario in real time. Use

   tail -f application.log | tee /tmp/tailed_log

But it's not sure that we'll get to reproduce the scenario in real time.

2. Grep the logs

   grep -i 'string_to_search" application.log > /tmp/greppedLOG

Problem with this is that this will skip the lines where "string_to_search" doesn't exist. The another string that is skipped is also important.

I've not tried this but this could work.

grep -i "string_to_search || string_to_not_skip" application.log > /tmp/greppedLOG

As I said, there will be strings that if missed will cause the whole issue jumbled down, so although this is the most likely what I'll do if I don't find other solution, I am not happy about it.

3. Another way is to grep n lines.

   grep -A5000 "string_to_search" application.log > /tmp/greppedLOG

The problem with this is that you don't know how long the log for particular user is. This isn't reliable way.

4. Another way that I do is to split the large log file.

   split -n 20 application.log /tmp/smallfile

But my senior colleagues have not recommended me to do this either because this is expensive process. Some times there is not enough space on their servers that we administrate, and if we split the log file in that case, server gets down.

5. Another way that I think could work is to get the logs from one time duration to another time duration. I've not tried this, but according to https://unix.stackexchange.com/a/751676/569181, this could work.

LC_ALL=C awk -v beg=10:00:00 -v end=13:00:00 '
  match($0, /[0-2][0-9]:[0-5][0-9]:[0-5][0-9]/) {
    t = substr($0, RSTART, 8)
    if (t >= end) selected = 0
    else if (t >= beg) selected = 1
  }
  selected'

Problem with this is that it's not always that we get time when the customer did transaction. But some of the times, we are able to find the time when the customer did the transaction to check the logs.

Other ideas on top of my head.

Don't split the file if the the home partition in df -H output exceeds more than 85%. even if I type split command. I think I need some alias along with a script for that.

Zipping the logs and doing ggrep isn't either helpful because it is the same thing as splitting in terms of space.

Anything else you can recommend? Are there better solutions? Is this even a technical problem? Or our client's problem? I am really frustrated with this situation.

Answer 1

This script will split a text file into a given number of sections, avoiding splitting text lines across sections. It can be used where there is only sufficient space to hold one section at a time. It operates by copying sections of the source file starting at the end, then truncating the source to free up space. So if you have a 1.8GB file and 0.5GB free space, you would need to use 4 sections (or more if you wish to have smaller output files). The last section is just renamed, as there is no need to copy it. After splitting, the source file no longer exists (there would be no room for it anyway).

The main part is an awk script (wrapped in Bash), which only sets up the section sizes (including adjusting to the section coincides with a newline). It uses the system() function to invoke dd, truncate and mv for all the heavy lifting.

$ bash --version
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
$ awk --version
GNU Awk 4.1.4, API: 1.1 (GNU MPFR 4.0.1, GNU MP 6.1.2)
$ dd --version
dd (coreutils) 8.28
$ truncate --version
truncate (GNU coreutils) 8.28

The script takes between one and four arguments:

./splitBig Source nSect Dest Debug

Source: is the filename of the file to be split into sections.

nSect: is the number of sections required (default 10).

Dest: is a printf() format used to generate the names of the sections.
Default is Source.%.3d, which appends serial numbers (from .001 up) to the source name.
Section numbers correspond to the original order of the source file.

Debug: generates some diagnostics (default is none).

Test Results:

$ mkdir TestDir
$ cd TestDir
$ 
$ cp /home/paul/leipzig1M.txt ./
$ ls -s -l
total 126608
126608 -rw-rw-r-- 1 paul paul 129644797 Aug 27 15:54 leipzig1M.txt
$ 
$ time ../splitBig leipzig1M.txt 5

real    0m0.780s
user    0m0.045s
sys 0m0.727s
$ ls -s -l
total 126620
25324 -rw-rw-r-- 1 paul paul 25928991 Aug 27 15:56 leipzig1M.txt.001
25324 -rw-rw-r-- 1 paul paul 25929019 Aug 27 15:56 leipzig1M.txt.002
25324 -rw-rw-r-- 1 paul paul 25928954 Aug 27 15:56 leipzig1M.txt.003
25324 -rw-rw-r-- 1 paul paul 25928977 Aug 27 15:56 leipzig1M.txt.004
25324 -rw-rw-r-- 1 paul paul 25928856 Aug 27 15:56 leipzig1M.txt.005
$ 
$ rm lei*
$ cp /home/paul/leipzig1M.txt ./
$ ls -s -l
total 126608
126608 -rw-rw-r-- 1 paul paul 129644797 Aug 27 15:57 leipzig1M.txt
$ time ../splitBig leipzig1M.txt 3 "Tuesday.%1d.log" 1
.... Section   3 ....
#.. findNl: dd bs=8192 count=1 if="leipzig1M.txt" skip=86429864 iflag=skip_bytes status=none
#.. system: dd bs=128M if="leipzig1M.txt" skip=86430023 iflag=skip_bytes of="Tuesday.3.log" status=none
#.. system: truncate -s 86430023 "leipzig1M.txt"
.... Section   2 ....
#.. findNl: dd bs=8192 count=1 if="leipzig1M.txt" skip=43214932 iflag=skip_bytes status=none
#.. system: dd bs=128M if="leipzig1M.txt" skip=43214997 iflag=skip_bytes of="Tuesday.2.log" status=none
#.. system: truncate -s 43214997 "leipzig1M.txt"
.... Section   1 ....
#.. system: mv "leipzig1M.txt" "Tuesday.1.log"

real    0m0.628s
user    0m0.025s
sys 0m0.591s
$ ls -s -l
total 126612
42204 -rw-rw-r-- 1 paul paul 43214997 Aug 27 15:58 Tuesday.1.log
42204 -rw-rw-r-- 1 paul paul 43215026 Aug 27 15:58 Tuesday.2.log
42204 -rw-rw-r-- 1 paul paul 43214774 Aug 27 15:58 Tuesday.3.log
$ 

Script:

#! /bin/bash --

LC_ALL="C"

splitFile () {  #:: (inFile, Pieces, outFmt, Debug)

    local inFile="${1}" Pieces="${2}" outFmt="${3}" Debug="${4}"

    local Awk='
BEGIN {
    SQ = "\042"; szLine = 8192; szFile = "128M";
    fmtLine = "dd bs=%d count=1 if=%s skip=%d iflag=skip_bytes status=none";
    fmtFile = "dd bs=%s if=%s skip=%d iflag=skip_bytes of=%s status=none";
    fmtClip = "truncate -s %d %s";
    fmtName = "mv %s %s";
}

function findNl (fIn, Seek, Local, cmd, lth, txt) {

    cmd = sprintf (fmtLine, szLine, SQ fIn SQ, Seek);
    if (Db) printf ("#.. findNl: %s\n", cmd);
    cmd | getline txt; close (cmd);
    lth = length (txt);
    if (lth == szLine) printf ("#### Line at %d will be split\n", Seek);
    return ((lth == szLine) ? Seek : Seek + lth + 1);
}

function Split (fIn, Size, Pieces, fmtOut, Local, n, seek, cmd) {

    for (n = Pieces; n > 1; n--) {
        if (Db) printf (".... Section %3d ....\n", n);
        seek = int (Size * ((n - 1) / Pieces));
        seek = findNl( fIn, seek);
        cmd = sprintf (fmtFile, szFile, SQ fIn SQ, seek,
            SQ sprintf (outFmt, n) SQ);
        if (Db) printf ("#.. system: %s\n", cmd);
        system (cmd);
        cmd = sprintf (fmtClip, seek, SQ fIn SQ);
        if (Db) printf ("#.. system: %s\n", cmd);
        system (cmd);
    }
    if (Db) printf (".... Section %3d ....\n", n);
    cmd = sprintf (fmtName, SQ fIn SQ, SQ sprintf (outFmt, n) SQ);
    if (Db) printf ("#.. system: %s\n", cmd);
    system (cmd);
}

{ Split( inFile, $1, Pieces, outFmt); }
'
    stat -L -c "%s" "${inFile}" | awk -v inFile="${inFile}" \
        -v Pieces="${Pieces}" -v outFmt="${outFmt}" \
        -v Db="${Debug}" -f <( printf '%s' "${Awk}" )
}

#### Script body starts here.

    splitFile "${1}" "${2:-10}" "${3:-${1}.%.3d}" "${4}"

Answer 2

I'm not at all clear what you're trying to achieve. As far as I can tell the question you're asking is in the question title, "How to view a log file that's worth 10GB+?" and the question itself simply contains ideas and thoughts that you yourself think might work.

So answering the only question I can find, one option is to use a pager such as less

less 10GBlogfile

The documentation (man less) eventually list the keys you can use, or once you've started it you can use h to get help - a list of keys and associated actions. For starters, G will Go to the last line, cursor keys (including PageUp, PageDown) will move around, / will search for an RE string, n/N will search for the next/previous match, and q will quit the pager

Answer 3

I know that you already tried split -n 20 -- but have you thought of split -n 20 --filter 'grep <whatever> or something' That will split the original file into components, and pipe them separately to whatever command you want.

The splitting shouldn't be very costly - especially if you do split --bytes=100M - it's basically just a seek and read/write. However, I'm not sure how it handles variable-length encodings such as UTF-8. If you know that the data is ASCII, then it's pretty safe. Otherwise, you'd be better off doing something like split --line-bytes=<size> - but that will have to parse a lot more data, which can be costly.

Answer 4

You seem to be trying to extract and analyse errors from the logs. There is no generic answer to the problem - how you isolate the events associated with a specific pattern in a log file is completely dependent on the structure of the log file and the nature of the thing generating the log files.

you don't know how long the log for particular user is

Is there an explicit (username) or implicit (session id, process id, IP address) identifier? If not it sounds like you need one and will then have to make multiple passes through the log files to:

1. Identify error instances, timestamps and user identifier
2. Capture surrounding non-error events