Please explain the behavior of these parameter expansions using IFS?

Question

I'm trying to figure out how to use the ${parameter%word} expansion with $@ and $*. It started by trying to make a script to combine pdfs using ghostscript, but I ran into some weird behavior with parameter expansions, and now I'm just curious why this behavior is happening.

Basically I'm trying to remove the '.pdf' from the end of each argument, then join them with an arbitrary string (I'm testing with '-'), then add a '.pdf' to the end of the result. E.g expected behavior is test.sh a.pdf b.pdf c.pdf -> a-b-c.pdf. Here is a test script I'm running:

IFS='-'

echo ${*%.pdf}.pdf
echo "${*%.pdf}.pdf"

a=${*%.pdf}.pdf
b="${*%.pdf}.pdf"
echo $a
echo $b

If I bash test.sh a.pdf b.pdf c.pdf, I get:

a b c.pdf
a-b-c.pdf
a b c.pdf
a b c.pdf

If I zsh test.sh a.pdf b.pdf c.pdf, I get:

a b c.pdf
a.pdf-b.pdf-c.pdf
a-b-c.pdf
a.pdf-b.pdf-c.pdf

I understand that zsh and bash are different, so I'm not worried about why they give different results from each other. However, in each case, only 1 of the 4 methods of constructing the string works as intended (the second one for bash, and the third one for zsh).

Why do these seemingly similar attempts to construct the string result in such different results? Any insight is appreciated. Thanks!

Answer 1

How ${*%word} and the like work depends on the shell. POSIX leaves the result unspecified. There are two main plausible behaviors: the transformation (prefix or suffix removal) can be applied to each word, or to the result of joining the words. In shells that support arrays, it's natural apply the transformation to each word: that's what bash and ksh93 do. In shells that don't support arrays, it's natural join the words first (that's what ash/dash does). For example:

# No arrays: $* joined = 'abc abc'; strip off b* → 'a'
$ dash -c 'echo ${*%%b*}' _ abc abc
a
# Arrays: $* = ('abc' 'abc'); strip off b* from each element → ('a' 'a'); then join
$ bash -c 'echo ${*%%b*}' _ abc abc
a a

The first character IFS is used to join the words that make up $*. It only makes a difference to what is stripped if the pattern can match that character. For example:

# No arrays: $* joined = 'abc-def-ghi'; strip off -* → 'abc'
$ dash -c 'IFS=-; echo "${*%%-*}"' _ abc-def ghi
abc
# Arrays: $* = ('abc-def' 'ghi'); strip off -* from each element → ('abc' 'ghi'); then join
$ bash -c 'IFS=-; echo "${*%%-*}"' _ abc-def ghi
abc-ghi

When the substitution is in a word context, the expansion ends here. Word contexts include double quotes and assignments; see When is double-quoting necessary? and Expansion of a shell variable and effect of glob and split on it for more details. This explains echo "${*%.pdf}.pdf": the first character of IFS is used for joining, and there is no subsequent splitting, hence the output in bash is a-b-c.pdf. The value of both a and b is a-b-c.pdf as well.

When the substitution is in a list context (i.e. unquoted), as in your first example, the result undergoes word splitting (and globbing). This is based on IFS, hence a-b-c.pdf is split into a, b and c.pdf. The echo command prints the three words with a space in between. Exactly the same thing happens with echo $a and echo $b in your example: the value of a or b is split at IFS characters.

Zsh treats @ and * differently. With * as the parameter name, it applies the string-style behavior (join first then transform) inside double quotes, and the array-style behavior (transform each element) otherwise. On the other hand, the parameter @ is always treated as an array. Thus:

$ zsh -c 'echo "${*%.pdf}"' _ a.pdf b.pdf c.pdf
a.pdf b.pdf c
$ zsh -c 'echo ${*%.pdf}' _ a.pdf b.pdf c.pdf
a b c
$ zsh -c 'echo "${@%.pdf}"' _ a.pdf b.pdf c.pdf
a b c
$ zsh -c 'echo ${@%.pdf}' _ a.pdf b.pdf c.pdf
a b c

Unlike what happens in other shells, a string assignment does not cause $* to be processed string-style: the double quotes are what matters. This explains why a=${*%.pdf}; echo $a works like echo ${*%.pdf} and not like a="${*%.pdf}"; echo $a.

With IFS=-, a dash is used when joining, which happens to * whenever it's in a word context, whether due to double quotes or to a string assignment.

# ('a.pdf' 'b.pdf' 'c.pdf); strip each element → ('a' 'b' 'c'); print list
$ zsh -c 'IFS=-; echo ${*%.pdf}' _ a.pdf b.pdf c.pdf
a b c
# join → 'a.pdf-b.pdf-c.pdf'; strip the single word and print it
$ zsh -c 'IFS=-; echo "${*%.pdf}"' _ a.pdf b.pdf c.pdf
a.pdf-b.pdf-c
# ('a.pdf' 'b.pdf' 'c.pdf); strip each element → ('a' 'b' 'c'); `$*` in word context so join → 'a-b-c'; print word
$ zsh -c 'IFS=-; a=${*%.pdf}; echo "$a"' _ a.pdf b.pdf c.pdf
a-b-c
# join → 'a.pdf-b.pdf-c.pdf'; strip the single word; print the word
$ zsh -c 'IFS=-; a="${*%.pdf}"; echo "$a"' _ a.pdf b.pdf c.pdf
a.pdf-b.pdf-c

---

Note that you should almost never use $*. It's only useful to join the positional arguments with IFS, and it makes it impossible to distinguish IFS characters created by the joining from IFS characters that were already in the arguments. "$@" is almost always the right form. Note that you do need the double quotes to avoid word expansions (even in zsh, although the effect of omitting the quotes is much smaller in zsh).

To make your script simple to understand, do one step at a time: strip off the suffix from each part, then join the parts. Use an array variable to store the intermediate result.

parts=("${@%.pdf}") # using @ because we want to have array behavior
IFS=-
joined="${parts[*]}" # using * and not @ for joining
echo "$joined.pdf"

This snippet works identically in bash and zsh.

Answer 2

This explains what happens:

#!/bin/bash

IFS='-'

var='a-b-c.pdf'
echo $var
echo "$var"

Your echo ${*%.pdf}.pdf does create the string you want but because of the missing quotes the word splitting acts on the -.

Or this:

[[ ${*%.pdf}.pdf =~ ' ' ]]
+ [[ a-b-c.pdf =~   ]]
echo $?
+ echo 1
1

In [[ ]] there is no word splitting and set -vx / bash -vx shows that the expansion does not contain spaces.