Extract last 5mins log from log file using shell

Question

I need to extract last 5mins log from log file. Here is my log file

[2022-02-08 13:26:21:352] [ERROR] [iBus Connection LifeCycle - CCMHost_DummyDevice_Backup_AD2::Management:::NRMCMO_FLPROD2] [com.example]
  + Message: Could not create an administered connection factory: Java heap space
  + Throwable: java.lang.OutOfMemoryError: Java heap space

[2022-02-08 15:09:37:068] [ERROR] [HikariCP connection filler (pool DirectReadConnection.9c292fc0.210e0ac7)] [com.example]
  + Message: Unable to Initialize Connection
  + Throwable: java.sql.SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

[2022-02-08 15:09:37:068] [ERROR] [HikariCP connection filler (pool DirectReadConnection.9c292fc0.210e0ac7)] [com.example]
  + Message: Unable to Initialize Connection
  + Throwable: java.sql.SQLException: Listener refused the connection with the following error:
ORA-01017, TNS:listener does not currently know of SID given in connect descriptor


[2022-02-08 15:05:04:056] [ERROR] [JMS Session Delivery Thread - The user's password has expired.] [com.example]

I tried by using this but not working

awk -v d1="$(date --date="2 hour ago" "+%Y-%m-%d %H:%M:%S:%3N")" -v d2="$(date "+%Y-%m-%d %H:%M:%S:%3N")" '$0 > d1 && $0 < d2 || $0 ~ d2' filelog.log

Any help will be appreciated

Answer 1

This wasn't working because the first character in the log line is a square bracket, and that's spoiling your comparison. If you extract just the date portion of the log line to compare, it should work. Note I'm hardcoding the dates here so I can use your example lines.

awk -v d1="2022-02-08 15:04:00" -v d2="2022-02-08 15:08:00" 'd1 < substr($0,2,19) && substr($0,2,19) < d2 || substr($0,2,19) ~ d2' filelog.log

You could actually simplify your command; since you only care about times more recent than 2 hours ago, you could drop "d2" altogether. If you only care about the lines that the date actually appears on, you can prevent artificial matches with other lines by specifying matched lines have to start with the "[" as your date-stamped lines do.

awk -v d1="$(date --date="2 hour ago" "+%Y-%m-%d %H:%M:%S:%3N")" '/^\[/ && d1 < substr($0,2,19)' filelog.log

If you also care about the additional lines below each matching datestamp, and if there are truly blank lines between every valid line entry, you could use that to modify the record separator to pick up the extra info and drop the check for the "[", since the other lines are part of the same record instead of separate ones to be filtered.

root@2ec99a4edaa9:/tmp# awk -v RS= -v d1="2022-02-08 15:04:00" 'd1 < substr($0,2,19)' filelog.log
[2022-02-08 15:09:37:068] [ERROR] [HikariCP connection filler (pool DirectReadConnection.9c292fc0.210e0ac7)] [com.example]
  + Message: Unable to Initialize Connection
  + Throwable: java.sql.SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
[2022-02-08 15:09:37:068] [ERROR] [HikariCP connection filler (pool DirectReadConnection.9c292fc0.210e0ac7)] [com.example]
  + Message: Unable to Initialize Connection
  + Throwable: java.sql.SQLException: Listener refused the connection with the following error:
ORA-01017, TNS:listener does not currently know of SID given in connect descriptor
[2022-02-08 15:05:04:056] [ERROR] [JMS Session Delivery Thread - The user's password has expired.] [com.example]

Finally, if you want to keep the blank lines for readability in your output, you could just add -v ORS="\n\n" to your awk statement.

Altogether, your modified command would look like this:

awk -v RS= -v ORS="\n\n" -v d1="$(date --date="2 hour ago" "+%Y-%m-%d %H:%M:%S:%3N")" 'd1 < substr($0,2,19)' filelog.log