Removing colors from output

Question

I have some script that produces output with colors and I need to remove the ANSI codes.

#!/bin/bash

exec > >(tee log)   # redirect the output to a file but keep it on stdout
exec 2>&1

./somescript

The output is (in log file):

java (pid  12321) is running...@[60G[@[0;32m  OK  @[0;39m]

I didn't know how to put the ESC character here, so I put @ in its place.

I changed the script into:

#!/bin/bash

exec > >(tee log)   # redirect the output to a file but keep it on stdout
exec 2>&1

./somescript | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g"

But now it gives me (in log file):

java (pid  12321) is running...@[60G[  OK  ]

How can I also remove this '@[60G?

Maybe there is a way to completely disable coloring for the entire script?

Answer 1

According to Wikipedia, the [m|K] in the sed command you're using is specifically designed to handle m (the color command) and K (the "erase part of line" command). Your script is trying to set absolute cursor position to 60 (^[[60G) to get all the OKs in a line, which your sed line doesn't cover.

(Properly, [m|K] should probably be (m|K) or [mK], because you're not trying to match a pipe character. But that's not important right now.)

If you switch that final match in your command to [mGK] or (m|G|K), you should be able to catch that extra control sequence.

./somescript | sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2};?)?)?[mGK]//g"

Answer 2

IMHO, most of these answers try too hard to restrict what is inside the escape code. As a result, they end up missing common codes like [38;5;60m (foreground ANSI color 60 from 256-color mode).

They also require the -r option which enables GNU extensions. These are not required; they just make the regex read better.

Here is a simpler answer that handles the 256-color escapes and works on systems with non-GNU sed:

./somescript | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g'

This will catch anything that starts with [, has any number of decimals and semicolons, and ends with a letter. This should catch any of the common ANSI escape sequences.

For funsies, here's a larger and more general (but minimally tested) solution for all conceivable ANSI escape sequences:

./somescript | sed 's/\x1B[@A-Z\\\]^_]\|\x1B\[[0-9:;<=>?]*[-!"#$%&'"'"'()*+,.\/]*[][\\@A-Z^_`a-z{|}~]//g'

(and if you have @edi9999's SI problem, add | sed "s/\x0f//g" to the end; this works for any control char by replacing 0f with the hex of the undesired char)

Answer 3

I came across ansi2txt tool from colorized-logs package in Debian. The tool drops ANSI control codes from STDIN.

Usage example:

./somescript | ansi2txt

Source code http://github.com/kilobyte/colorized-logs

Answer 4

I couldn't get decent results from any of the other answers, but the following worked for me:

somescript | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g"

If I only removed the control char "^[", it left the rest of the color data, e.g., "33m". Including the color code and "m" did the trick. I'm puzzled with s/\x1B//g doesn't work because \x1B[31m certainly works with echo.

Answer 5

For Mac OSX or BSD use

./somescript | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'

Answer 6

The regular expression below will miss some ANSI Escape Codes sequences, as well as 3 digit colors. Example and Fix on regex101.com.

Use this instead:

./somescript | sed -r 's/\x1B\[(;?[0-9]{1,3})+[mGK]//g'

---

I also had the problem that sometimes, the SI character appeared.

It happened for example with this input : echo "$(tput setaf 1)foo$(tput sgr0) bar"

Here's a way to also strip the SI character (shift in) (0x0f)

./somescript | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" | sed "s/\x0f//g"

Answer 7

There's also a dedicated tool to handle ANSI escape sequences: ansifilter. Use the default --text output format to strip all ANSI escape sequences (note: not just coloring).

ref: https://stackoverflow.com/a/6534712

Answer 8

Much simpler function in pure Bash to filter-out common ANSI codes from a text stream:

# Strips common ANSI codes from a text stream

shopt -s extglob # Enable Bash Extended Globbing expressions
ansi_filter() {
  local line
  local IFS=
  while read -r line || [[ "$line" ]]; do
    printf '%s\n' "${line//$'\e'[\[(]*([0-9;])[@-n]/}"
  done
}

See:

1. linuxjournal.com: Extended Globbing
2. gnu.org: Bash Parameter Expansion

Answer 9

I had a similar problem. All solutions I found did work well for the color codes but did not remove the characters added by "$(tput sgr0)" (resetting attributes).

Taking, for example, the solution in the comment by davemyron the length of the resulting string in the example below is 9, not 6:

#!/usr/bin/env bash

string="$(tput setaf 9)foobar$(tput sgr0)"
string_sed="$( sed -r "s/\x1B\[[0-9;]*[JKmsu]//g" <<< "${string}" )"
echo ${#string_sed}

In order to work properly, the regex had to be extend to also match the sequence added by sgr0 ("\E(B"):

string_sed="$( sed -r "s/\x1B(\[[0-9;]*[JKmsu]|\(B)//g" <<< "${string}" )"

Answer 10

Not sure what's in ./somescript but if escape sequences are not hardcoded you can set the terminal type to avoid them

TERM=dumb ./somescript 

For example, if you try

TERM=dumb tput sgr0 | xxd

you'll see it produces no output while

tput sgr0 | xxd
00000000: 1b28 421b 5b6d                           .(B.[m

does (for xterm-256color).

Answer 11

Here's a pure Bash solution.

Save as strip-escape-codes.sh, make executable and then run <command-producing-colorful-output> | ./strip-escape-codes.sh.

Note that this strips all ANSI escape codes/sequences. If you want to strip colors only, replace [a-zA-Z] with "m".

Bash >= 4.0:

#!/usr/bin/env bash

# Strip ANSI escape codes/sequences [$1: input string, $2: target variable]
function strip_escape_codes() {
    local _input="$1" _i _char _escape=0
    local -n _output="$2"; _output=""
    for (( _i=0; _i < ${#_input}; _i++ )); do
        _char="${_input:_i:1}"
        if (( ${_escape} == 1 )); then
            if [[ "${_char}" == [a-zA-Z] ]]; then
                _escape=0
            fi
            continue
        fi
        if [[ "${_char}" == $'\e' ]]; then
            _escape=1
            continue
        fi
        _output+="${_char}"
    done
}

while read -r line; do
    strip_escape_codes "${line}" line_stripped
    echo "${line_stripped}"
done

Bash < 4.0:

#!/usr/bin/env bash

# Strip ANSI escape codes/sequences [$1: input string, $2: target variable]
function strip_escape_codes() {
    local input="${1//\"/\\\"}" output="" i char escape=0
    for (( i=0; i < ${#input}; ++i )); do         # process all characters of input string
        char="${input:i:1}"                       # get current character from input string
        if (( ${escape} == 1 )); then             # if we're currently within an escape sequence, check if
            if [[ "${char}" == [a-zA-Z] ]]; then  # end is reached, i.e. if current character is a letter
                escape=0                          # end reached, we're no longer within an escape sequence
            fi
            continue                              # skip current character, i.e. do not add to ouput
        fi
        if [[ "${char}" == $'\e' ]]; then         # if current character is '\e', we've reached the start
            escape=1                              # of an escape sequence -> set flag
            continue                              # skip current character, i.e. do not add to ouput
        fi
        output+="${char}"                         # add current character to output
    done
    eval "$2=\"${output}\""                       # assign output to target variable
}

while read -r line; do
    strip_escape_codes "${line}" line_stripped
    echo "${line_stripped}"
done

Answer 12

@jeff-bowman's solution helped me getting rid of SOME of the color codes. I added another small portion to the regex in order to remove some more:

sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" # Original. Removed Red ([31;40m[1m[error][0m)
sed -r "s/\x1B\[([0-9];)?([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" # With an addition, removed yellow and green ([1;33;40m[1m[warning][0m and [1;32;40m[1m[ok][0m)
                ^^^^^^^^^
                remove Yellow and Green (and maybe more colors)

Answer 13

The controversial idea would be to reconfigure terminal settings for this process environment to let the process know that terminal does not support colors.

Something like TERM=xterm-mono ./somescript comes to my mind. YMMV with your specific OS and ability of your script to understand terminal color settings.

Answer 14

I had some issues with colorized output which the other solutions here didn't process correctly, so I built this perl one liner. It looks for escape \e followed by opening bracket \[ followed by one or color codes \d+ separated by semicolons, ending on m.

perl -ple 's/\e\[\d+(;\d+)*m//g'

It seems to work really well for colorized compiler output.

Answer 15

Hmm, not sure if this will work for you, but 'tr' will 'strip' (delete) control codes - try:

./somescript | tr -d '[:cntrl:]'

Answer 16

The other answers didn't quite manage to strip all escape codes (e.g. \x1b[?25l).

This little sed command should do the trick:

./somescript | sed -r 's/[\x1B\x9B][][()#;?]*(([a-zA-Z0-9;]*\x07)|([0-9;]*[0-9A-PRZcf-ntqry=><~]))//g'

The regex is a modification of https://github.com/acarl005/stripansi/blob/master/stripansi.go#L7

Answer 17

I spent little bit searching around for a solution that didn't involve memorizing sed spells, and have not seen this suggested anywhere:

./somescript | tr -dc '[[:print:][:space:]]'

removes any character that is not either printable or whitespace (including vertical).

Answer 18

I came across this question/answers trying to do something similar as the OP. I found some other useful resources and came up with a log script based on those. Posting here in case it can help others.

Digging into the links helps understand some of the redirection which I won't try and explain because I'm just starting to understand it myself.

Usage will render the colorized output to the console, while stripping the color codes out of the text going to the log file. It will also include stderr in the logfile for any commands that don't work.

Edit: adding more usage at bottom to show how to log in different ways

#!/bin/bash
set -e
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

. $DIR/dev.conf
. $DIR/colors.cfg

filename=$(basename ${BASH_SOURCE[0]})
# remove extension
# filename=`echo $filename | grep -oP '.*?(?=\.)'`
filename=`echo $filename | awk -F\. '{print $1}'`
log=$DIR/logs/$filename-$target

if [ -f $log ]; then
  cp $log "$log.bak"
fi

exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>$log 2>&1


# log message
log(){
    local m="$@"
    echo -e "*** ${m} ***" >&3
    echo "=================================================================================" >&3
  local r="$@"
    echo "================================================================================="
    echo -e "*** $r ***" | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g"
    echo "================================================================================="
}

echo "=================================================================================" >&3
log "${Cyan}The ${Yellow}${COMPOSE_PROJECT_NAME} ${filename} ${Cyan}script has been executed${NC}"
log $(ls) #log $(<command>)

log "${Green}Apply tag to image $source with version $version${NC}"
# log $(exec docker tag $source $target 3>&2) #prints error only to console
# log $(docker tag $source $target 2>&1) #prints error to both but doesn't exit on fail
log $(docker tag $source $target 2>&1) && exit $? #prints error to both AND exits on fail
# docker tag $source $target 2>&1 | tee $log # prints gibberish to log
echo $? # prints 0 because log function was successful
log "${Purple}Push $target to acr${NC}"

Here are the other links that helped:

• Can I use sed to manipulate a variable in bash?
• https://www.cyberciti.biz/faq/redirecting-stderr-to-stdout/
• https://unix.stackexchange.com/questions/42728/what-does-31-12-23-do-in-a-script
• https://serverfault.com/questions/103501/how-can-i-fully-log-all-bash-scripts-actions
• https://www.gnu.org/software/bash/manual/bash.html#Redirections

Answer 19

I used perl as I have to do this frequently on many files. This will go through all files with filename*.txt and will remove any formatting. This works for my use case and may be useful for someone else too so just thought of posting here. replace whatever your file name is in place of filename*.txt or you can put file names separated by spaces in setting the FILENAME variable below.

$ FILENAME=$(ls filename*.txt) ; for file in $(echo $FILENAME); do echo $file; cat $file | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > $file-new; mv $file-new $file; done

Answer 20

my contribution:

./somescript | sed -r "s/\\x1B[\\x5d\[]([0-9]{1,3}(;[0-9]{1,3})?(;[0-9]{1,3})?)?[mGK]?//g"

Answer 21

for macOS

$ my_program | pbcopy && pbpaste

Answer 22

This works for me:

./somescript | cat