How to get the json data used in each curl statement while using xargs and maps it to its corresponding result?

Question

I have text file that has arguments of curl command. This is how the file looks

'https://example.com/tl/' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username2",}'
'https://example.com/tl/' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username3",}'
'https://example.com/tl/' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username4",}'
'https://example.com/tl/' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username5",}'
'https://example.com/tl/' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username6",}'

This is the command I use

cat /AbsolutePath/Inputfile.txt | xargs -P 10000 -n 10 curl -s | jq '.message'

I'm using jq, to parse json in command line

What I want is,

1. Pipe or send output of above command to another command, so that if message has certain text, grab the email value used in that corresponding curl command and write to a log file or create a filename with usernameX.txt

For example, only if username2 and username5 cURL command's message = 'success', these two usernames should be written to a log file or two files username2.txt and username5.txt should be created.

Answer 1

To meet you requirement, each "message" of the output needs to be associated with its corresponding inputted "mail".

The easy way may be replacing the curl with your own shell script that can output both the given mail and its curl result. A simple, unsafe version maybe like this

#! /bin/bash
# print <mail> <message> 
jq .mail -rj "${@: -1]}"      # print mail without newline
echo -n " "                  
curl -s "$@" | jq .message -r

Then pipe the result to awk '$2=="success"{print $1}'

Other than that, you may try this

cat /AbsolutePath/Inputfile.txt |
    xargs -P 10000 -n 10 curl -s |
    jq -s 'to_entries|.[]|select(.value.message=="success")|.key+1' |
    awk 'FILENAME=="-"{line[$0]=1}  FILENAME != "-" && line[FNR]==1{print gensub(/.*--data-raw /, "", "g", $0)}' - /AbsolutePath/Inputfile.txt | 
    jq .mail -r

Basically, the jq get the indexes of success messages and the awk reads the indexes and retrieves the last part of corresponding nth line from the file.

Answer 2

Suggesting to use simple awk script.

awk script

awk '{system("curl "$0)}' /AbsolutePath/Inputfile.txt | jq

awk script filtering success message

awk '{system("curl "$0)}' /AbsolutePath/Inputfile.txt | grep "success"

Answer 3

TLDR

cat input.txt |
  xargs -P0 -L1 sh -c 'curl -s "$@" | jq -c --arg username $(echo "${@: -1}" | jq -r .email) "{\$username, message}"' sh |
  jq -r 'select(.message == "success") | .username'

How

You requested to print username depending on its corresponding response body, but your input file are composed of curl arguments and you didn't mention whether or not the response body contains username.

Thus I decide to parse your last argument to get username first, for example:

cat input.txt |
  xargs -P0 -L1 sh -c 'curl -s "$@" | jq -c --arg username $(echo "${@: -1}" | jq -r .email) "{\$username, message}"' sh

{"username":"username2","message":"failure"}
{"username":"username4","message":"failure"}
{"username":"username3","message":"success"}
{"username":"username5","message":"failure"}
{"username":"username6","message":"success"}

After you retrieved a list of JSON object composed of username and message pair, we can easily filter the steam by the following command:

jq -r 'select(.message == "success") | .username'

cat input.txt |
  xargs -P0 -L1 sh -c 'curl -s "$@" | jq -c --arg username $(echo "${@: -1}" | jq -r .email) "{\$username, message}"' sh |
  jq -r 'select(.message == "success") | .username'

username3
username6

FYI here is the Ruby HTTP server I used for debugging.

rackup -rjson -b 'run ->(_) { [200, {}, [JSON.dump(message: rand(2) == 0 ? "success" : "failure")]] }'