Pipe gsutil output to file

Question

Greetings StackOverflow,

I'm working on a small project on Windows which needs to read the output of GSUTIL's copy function. Problem is, the output of the copy function doesn't seem to work via the Standard Output. Also, the behavior of GSUTIL is inconsistent: piping output doesn't work with the copy function but using the list function is does work.

When I use the following command in my command prompt the output is displayed in the command prompt but not redirected to the text file. This command doesn't work right:

C:\gsutil> python gsutil cp "file://C:/test_files/*" gs://gs_teststore/ > gsutil_cp.txt

On the otherhand when I use the list function (ls) the output does work through the standard output and works at I hoped:

C:\gsutil> python gsutil ls gs://gs_teststore/ > gsutil_ls.txt

Is there a way to capture the output from the copy function of the GSUTIL?

Answer 1

You can use the -L option to generate a manifest file of all files that were copied. From the documentation:

-L <file> Outputs a manifest log file with detailed information about each item that was copied. This manifest contains the following information for each item:

• Source path.
• Destination path.
• Source size.
• Bytes transferred.
• MD5 hash.
• UTC date and time transfer was started in ISO 8601 format.
• UTC date and time transfer was completed in ISO 8601 format.
• Upload id, if a resumable upload was performed.
• Final result of the attempted upload, success or failure.
• Failure details, if any.

A specific example:

$ echo "hey" | gsutil cp -L manifest.txt - gs://mybucket/hey.txt
Copying from <STDIN> [Content-Type=application/octet-stream]...

$ cat manifest.txt 
Source,Destination,Start,End,Md5,UploadId,Source Size,Bytes Transferred,Result,Description
file://-,gs://mybucket/hey.txt,2013-05-29T21:29:31.847715Z,2013-05-29T21:29:32.115624Z,081ecc5e6dd6ba0d150fc4bc0e62ec50,,,0,OK,

Answer 2

Jeff's answer about using gsutil cp -L is the right solution for what you're trying to do.

Just to supplement with some detail about why you weren't able to capture the gsutil cp output the way you expected: gsutil outputs status messages to stderr, and only outputs to stdout when the output in question is the purpose of the command you're running. Thus, for example, gsutil ls outputs to stdout because that output is the purpose of the command, while in contrast, the progress indicator messages for the gsutil cp command are really status about the underlying purpose (which is copying data) -- so that output goes to stderr.

Mike Schwartz, Google Cloud Storage team