Attempt to launch cleanmgr on the C:\ drive. Use the button Clean up system files to compute how to remove the $WINDOWS.~BT folder.
The files contained with $WINDOWS.~BT are not technically system files. Which is the reason, the option to clean up system files, isn't removing the directory. The folder is used as a temporary storage location when a feature upgrade is installed, but there is supposed to be an option in the Disk Cleanup utility that will delete the folder.
There two possibilities, the first is that this folder isn't all that new, and contains the temporary files from the upgrade to 1809. The second is that your system has been trying to install 1903 but failing to do so. The folder contains a log file, if the upgrade to 1903 is indeed failing, the log would contain the reason.
Resolve Windows 10 upgrade errors contains a link to a tool that will help you diagnose the failure contained within ..\pather\setupact.log
and ..\pather\setuperr.log
.
I can conclude that your system attempted and failed to upgrade to Windows 10 version 1809 on August 28th, 2019. This means that the $WINDOWS.~BT.
likely contains the files to perform the upgrade. There would indeed be a log file that contains the reason, although the provided reason will likely be not clear, to such a degree that Microsoft has a tool to decode the reason for the failure.
Since the folder in question is typically removed on its own, you can safely delete it. In an elevated command prompt, run the following commands.
takeown /F C:\$Windows.~BT\* /R /A
icacls C:\$Windows.~BT\*.* /T /grant administrators:F
rmdir /S /Q C:\$Windows.~BT\
Source: Can I delete $Windows.~BT & $Windows.~WS folders after Windows 10 upgrade?