Delayed Expansion in a for loop processing paths that contain exclamation marks

Question

I know there are already a few questions here about this same issue but I am new to batch scripting and I found those solutions hard to apply in my scenario or directly not applying to my script.
A solution seems to be to temporarily disable DelayedExpansion but if I do so then nothing works anymore.

What the script below is trying to accomplish and it works except for the paths with !, is the following:

It will take a source folder and file by file, including subfolders, check if the file exists in a destination folder. If the file does not exist then it can do two things:

• Copy the file from the source folder to the destination folder, if it is a png, jpg or mp4
• Create a dummy file (a 0 bytes file with the same name and extension) in the destination folder if it is not a .png, .jpg or .mp4. I use for this xcopy (I copy an existing zero bytes file and rename it) because it is the only way I had no issues with special characters.

@echo off

setlocal EnableExtensions EnableDelayedExpansion 

rem Set the source and destination folders
set "src=L:\folderS\sub1\sub2"
set "dst=L:\folderD\sub1"

rem Set the path to the dummy file that we need to create dummy files with xcopy
set "dummy=L:\dummy.txt"

rem Create the destination folder if it doesn't exist
if not exist "%dst%" mkdir "%dst%"

rem Loop through all the files in the source folder
for /R "%src%" %%f in (*) do (

     rem Get the file path
     set "filePath=%%f"
     echo the full path is !filePath!

     rem Get the file extension
     set "ext=%%~xf"
     
     rem Get the relative path excluding the path until the specified src folder
     set "relPath=!filePath:%src%\=!"
     echo the relative path is !relPath!
     
     rem Check if the file does not exist in the destination
     if not exist "%dst%\!relPath!" (
     
         rem Check if the file extension is png jpg or mp4
         if /I "!ext!" == ".png" (
          
         rem If the file extension is png jpg or mp4 copy the file and display a message
         rem echo Copying "%%f" to "%dst%\!relPath!"
         xcopy /y "%%f" "%dst%\!relPath!*"
         
         )else if /I "!ext!" == ".jpg" (
             xcopy /y "%%f" "%dst%\!relPath!*"
         
         )else if /I "!ext!" == ".mp4" (
             xcopy /y "%%f" "%dst%\!relPath!*"
          
         )else (
                 
             rem If the file extension is not png jpg or mp4, create a dummy file
             echo Creating dummy "%%f" to "%dst%\!relPath!"
              
             xcopy /y "%dummy%" "%dst%\!relPath!*"
        )

    )
)
echo Done.

The problem seems to be when I store the relative path to the file in relPath, if the filename or any of the subfolders contains a ! then it will ignore everything that goes after the ! but still keeps the file extension somehow. I think, but I am not sure, that even if I would manage to store correctly the path in relPath, I would still have issues when using later on "%dst%\!relPath!" in the other parts of the for loop for copying the file or checking if it exists.

How could I get around this?

EDIT: Here is an screenshot showing the result for a very simple scenario:

EDIT 2: I tried now with @harrymc suggestion and it seems to have solved partially the issue but some filenames are still giving trouble. New code:

@echo off

setlocal EnableExtensions EnableDelayedExpansion 

rem Set the source and destination folders
set "src=L:\folderS\sub1\sub2"
set "dst=L:\folderD\sub1"

rem Set the path to the dummy file that we need to create dummy files with xcopy
set "dummy=L:\dummy.txt"

rem Create the destination folder if it doesn't exist
if not exist "%dst%" mkdir "%dst%"

rem Loop through all the files in the source folder
for /R "%src%" %%f in (*) do (

     rem Get the file path
     set "filePath=%%f"
     echo the full path is !filePath!

     rem Get the file extension
     set "ext=%%~xf"
     
     setlocal EnableExtensions EnableDelayedExpansion
     rem Get the relative path excluding the path until the specified src folder
     set "relPath=!filePath:%src%\=!"
     echo the relative path is !relPath!
     
     rem Check if the file does not exist in the destination
     if not exist "%dst%\!relPath!" (
     
         rem Check if the file extension is png jpg or mp4
         if /I "!ext!" == ".png" (
          
         rem If the file extension is png jpg or mp4 copy the file and display a message
         rem echo Copying "%%f" to "%dst%\!relPath!"
         xcopy /y "%%f" "%dst%\!relPath!*"
         
         )else if /I "!ext!" == ".jpg" (
             xcopy /y "%%f" "%dst%\!relPath!*"
         
         )else if /I "!ext!" == ".mp4" (
             xcopy /y "%%f" "%dst%\!relPath!*"
          
         )else (
                 
             rem If the file extension is not png jpg or mp4, create a dummy file
             echo Creating dummy "%%f" to "%dst%\!relPath!"
              
             xcopy /y "%dummy%" "%dst%\!relPath!*"
        )

    )
    endlocal
)
echo Done.

Here is an example of a filename that still give trouble. I think now this is xcopy limitation due to the length of the filename or maybe another symbol.

EDIT 3: It is fully solved now. The last remaining problem was with the lines

xcopy /y "%%f" "%dst%\!relPath!*"

that should instead be:

echo F|xcopy "!filePath!" "%dst%\!relPath!"

this fixed the problem I was still having with some symbols, and with the "F|" xcopy will not ask me each time if it is a file or not. There is still probably the limitation of 256 chars for xcopy but I don't see to hit that limit. I could instead use robocopy which would be better but robocopy parameters are quite different and I would need probably quite a few attempts to get it right.

Thank you everyone!

Answer 1

Based on the advice in the post Escaping exclamation marks with delayed expansion, I have enclosed only the necessary part inside the loop with the setlocal commands, and this worked for my example:

@echo off
set "src=C:\Temp\tmp9"
for /R "%src%" %%f in (*.txt) do (
     set "filePath=%%f"
     setlocal EnableExtensions EnableDelayedExpansion
     echo the full path is "!filePath!"
     set "relPath=!filePath:%src%\=!"
     echo relative path is "!relPath!"
     endlocal
)

And got this result :

Answer 2

BAT File

@echo off

setlocal EnableExtensions EnableDelayedExpansion 

rem Set the source and destination folders
set "src=L:\folderS\sub1\sub2"
set "dst=L:\folderD\sub1"

rem Set the path to the dummy file
set "dummy=L:\dummy.txt"

rem Set list of extensions to copy
set "ext=.png .jpg .mp4"

rem loop through all files in source folder and subfolders
for /f "tokens=* delims= " %%f in ('Dir "%src%" /A:-D /b /s') do (
  
  rem set default dummy flag state (on)
  set "dummyFlag=on"
  
  rem set filepath
  set "filePath=%%f"
  
  rem Get the path to parent relative to %src%
  set "relPath=!filePath:%src%=!"
   
  rem continue if destination file does not exist  
  if not exist "%dst%!relPath!" (
  
    rem compare list of extensions to current file extension
    for %%e in (%ext%) do (
    
      rem if current file extension matches extension from list      
      if "%%~xf" == "%%e" (
      
        xcopy /-I /Q "!filePath!" "%dst%!relPath!*" > nul
        set "dummyFlag=off"
       
      )
       
    )
  
    rem check if dummyFlag is still set 
    if "!dummyFlag!" == "on" (
    
      rem Copy dummy file to destination
      xcopy /-I /Q "%dummy%" "%dst%!relPath!*" > nul

    )

  )

) 

 

Notes

• if not exist "%dst%" mkdir "%dst%" isn't necessary as xcopy will create directories as needed

• added a list of extensions to simplify adding and removing extensions and to reduce code duplication set "ext=.png .jpg .mp4"

• used a dir "%src%" /A:-D /b /s approach to the for loop (list of all (only) files in "%src%" and its subfolders

• added a flag for determining if the file matched one of the extensions !dummyFlag!=<on/off>

• modified xcopy:

  • removed /y switch as no files should be overwritten, otherwise why check if not exist
  • added switch /-I so if the destination doesn't exist, it is understood to be a file
  • added switch /Q don't list filenames to be copied
  • added > NULL to suppress completion messages

Answer 3

*/ Edit

I tried this and I got an error: "The full path of is too long. Done."
The longest path is: L:\folderS\sub1\sub2\Another Test with Even more Symbols and long Name - And also, commas and brakets! (w23) (We,gasg) adsfasdfdas-asdf.png Which is trully long, but still totally supported by Windows.

In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters.
A local path is structured in the following order: > drive letter, colon, backslash, name components separated by backslashes, and a terminating null character.
For example, the maximum path on drive D is "D:\some 256-character path string<NUL>" where "<NUL>" represents the invisible terminating ^null` character for the current system codepage.

Obs.: The characters < > are used here for visual clarity and cannot be part of a valid path string.

---

My system has already enabled support for paths longer than 260 characters, now if your path has more than 260 characters then I suggest you check the information linked in this edition and compare it with the settings on your system and hopefully that resolves this new event.

Note that I had no problems with long path/names, and I even added a screenshot:

---

• To check from the command line whether long name support
  has been enabled, 0x1 or disabled 0x0 (default), try:

E:\>reg query "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /f "LongPathsEnabled"

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
   LongPathsEnabled    REG_DWORD    0x1

---

• To enable (0x1) or disable (0x0) support for long path names on the command line, try:

:: To enable support for long path names ::
> reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /v "LongPathsEnabled" /t REG_DWORD /d 1 /f

:: To  disable support for long path names ::
> reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /v "LongPathsEnabled" /t REG_DWORD /d 0 /f

---

---

@echo off 

setlocal EnableExtensions DisableDelayedExpansion
     
set "_src=L:\folderS\sub1\sub2"
set "_dst=L:\folderD\sub1"

robocopy "%_src%" "%_dst%" /e /xf *.* /copy:d | findstr /e \\

for /R "%_src%" %%G in (*) do (
     
     set "_var=%%~fG"
     
     setlocal EnableDelayedExpansion
     
     call set "_filePath=!_var:^!=^^!!"
     <con: call set "_rltvPath=%%_filePath:!_src!\=%%"
     <con: call set "_dummyFile=!_rltvPath:%%~xG=.txt!"
     
     call echo\the file path is "!_filePath!"
     call echo\the relative path is "!_rltvPath!"
     call echo\the dummy path file is "!_dummyFile!"
     
     if not exist "%_dst%\!_rltvPath!" (
     
         if /I "%%~xG" == ".png" (
            
             call copy /y "!_filePath!" "%_dst%\!_rltvPath!"
                 
            )else if /I "%%~xG" == ".jpg" (
             
             call copy /y "!_filePath!" "%_dst%\!_rltvPath!"
            
            )else if /I "%%~xG" == ".mp4" (
            
             call copy /y "!_filePath!" "%_dst%\!_rltvPath!"
             
            )else (
             
             call echo\cd.>"%_dst%\!_dummyFile!"
            
            )
            
        )
     
     echo\ & endlocal
     
    )

echo Done. & endlocal 

---

The main way is, first define the variable, then turn on delayed expansion with escapes in new substring for the escaped variable, and then when using, use call command so that the realtime delayed escapes do their part.

Answer 4

Can adding something like this solve the problem ?

set filePath=%filePath:!=^^^^!%

We will look for all the ! characters in the path and we replace them with ^! to neutralize the special character ! which is certainly causing you these troubles. You can try varying the number of ^ before the ! if necessary.