OS X file system anomaly: "ghost '?' in file names and unexpected symlinks

Question

I've never seen this before - only found it because Time Machine kept failing (on Mac OS X High Sierra 10.13.6). A similar question was asked here ~7 years go but it was not solved (and he did not have symlink issues): File names in OS X appearing with question marks

I ran disk repair several times from Disk Utility.app (which did not find or fix any errors).

I determined the only location with this issue was iTunes Media/Music folder, so I zipped up the entire folder to force symlink resolution. This worked to resolve the ghost symlinks (some had 4 hops noted but no source). Backups now run without error (-48 and -8062).

What I've not been able to fix is the random '?' character in the file.

I tried the solution noted here: Remove question mark "?" from file names OSX which seemed like it would be perfect, but it had no effect. The "?" still persists even after the mv operation, specifically:

find . -type f -name '*?*' | while read f; do mv "$f" "${f//\?/}"; done

In Finder, no ? appears and you cannot grep "?" out of the name, so strictly speaking, the character does not appear to exist. They only appear in Terminal:

MacBook-Pro:Welcome Reality +$ ls -lsuG
total 177456
20656 -rw-r--r--  1 User  group  10572851 Jan 12 09:31 02 ?Doomsday.mp3
14944 -rw-r--r--  1 User  group   7645727 Jan 12 09:31 03 My Eyes (G3dit).mp3
23632 -rw-r--r--  1 User  group  12093176 Jan 12 09:31 03 ?My Eyes.mp3
18768 -rw-r--r--  1 User  group   9605621 Jan 12 09:31 04 Guilt.mp3
20336 -rw-r--r--  1 User  group  10407761 Jan 12 09:31 06 ?Me and You.mp3
19968 -rw-r--r--  1 User  group  10220743 Jan 12 09:31 11 ?Must Be the Feeling.mp3
17104 -rw-r--r--  1 User  group   8749666 Jan 12 09:31 13 Promises.MP3
19920 -rw-r--r--  1 User  group  10194622 Jan 12 09:31 15 ?Won't You (Be There).mp3
21952 -rw-r--r--  1 User  group  11232230 Jan 12 09:31 17 ?Promises (Skrillex & Nero Remix).mp3
   16 -rw-r--r--  1 User  group      7366 Jan 12 09:31 AlbumArtSmall.jpg
   80 -rw-r--r--  1 User  group     35880 Jan 12 09:31 Folder.jpg
MacBook-Pro:Welcome Reality +$ find . -type f | grep ?
*[no results returned]*
MacBook-Pro:Welcome Reality +$ find . -type f | grep M
./03 My Eyes (G3dit).mp3
./03 My Eyes.mp3
./06 Me and You.mp3
./11 Must Be the Feeling.mp3
./13 Promises.MP3

And here's what I see in Finder:

Does anyone in 2019 have any idea what is going on here?

Answer 1

As @Gordon Davisson rightly said, this sequence of bytes correspond to the BOM Unicode character, encoded in 8 bits:

357 273 277 (in octal) = EF BB BF (in hexadecimal)

To mass remove it:

1. Get Homebrew if you don't have it already
2. Get the rename command with brew install rename
3. To delete this BOM character in every filename of your Music folder: find ~/Music -type f -exec rename 's/\o{357}\o{273}\o{277}//' {} \;

You probably want to try it on a single file first:

rename 's/\o{357}\o{273}\o{277}//' /path/to/file/*oomsday*

---

When it comes to file names, you can't really trust what you see on the Finder, nor with the ls command in the Terminal, because they show a limited representation of all possible codes. The od command is handy to reveal those exact codes when you're confused.

Answer 2

iTunes has always been 'messy' managing music (compared to WinAMP for example), but it's free and keeps music generally organized.

The correct answer to remove apparent '?' chars in filenames was accepted by @Yoric but I wanted to add this for others who may happen upon this in the future...

If the "Keep iTunes Media Folder Organized" option is on, then iTunes starts creating symlinks and these errant "?" (357 273 277)_8 entries. I was not aware of this happening on prior versions of OS X, but 10.13 it is very obviously happening.

If Time Machine is failing, view the log file from terminal using:

clear; printf '\e[3J' && log show --predicate 'subsystem == "com.apple.TimeMachine"' --info --last 24h | grep -F 'eMac' | grep -Fv 'etat' | awk -F']' '{print substr($0,1,19), $NF}'

(I don't recall where I found this - credit to that author shall be attributed).

If you see errors -48 and -8062 in iTunes music, then iTunes is the culprit. Turn off iTunes folder organization until everything is straight. Make a clean backup then experiment with turning it back on again.