The problem is that Windows Explorer also provides the whole shell for Windows. So when you click that Folder icon, you're most likely already using Explorer. It just starts another process (or window).
You might want to look into shell replacements.
Update
Ok, let me go into a little bit more detail about my understanding of things.
How the whole system works
Let's start with my desktop.
My desktop is drawn by the shell. The shell is provided by the initial explorer.exe
process that runs when Windows starts. If I have a folder on my desktop:
I can double-click it to open it:
This created a new explorer.exe
process which is now displaying the window that shows the contents of my folder. But there are many way the same window could have been opened.
So why did Windows to decide to use explorer.exe
to display that folder?
Well, as I said, the desktop is already an explorer.exe
process. So it will simply use ShellExecuteEx()
to let the shell decide what to do with it. The shell will then create a new explorer.exe
instance (because it is hard-wired to do so to my knowledge).
So what you can do, is not use explorer.exe
to provide your shell (as you already seem to do). You can simply provide an alternative application in the registry under:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell
But nobody is stopping that new shell from calling ShellExecuteEx()
, which would, again, invoke the default shell and, thus, explorer.exe
!
So, what now?
So if you really want to replace Windows Explorer, replace the file explorer.exe
. Because, in the end, every program could still try to execute C:\Windows\explorer.exe
directly and then you have the same old Explorer again.
But, please keep in mind, I have never attempted to replace explorer.exe
on a system, so I don't know if that would even be possible and/or stable to use.
How does application XYZ do it?
There are several applications that simply plug into Explorer, like QTTabBar. Maybe Explorer++ works the same way. This way you don't need to touch anything as your modification is always loaded into new explorer.exe
processes. Going into detail about that process would take a whole new answer.
What about the File-Open dialog?
This is a whole new set of problems. The File Save and File Open dialogs are part of the Common Dialogs Library. Trying to replace those is hard to imagine. Pretty much the same can be said about those as I said about explorer.exe
. But this would be far more scary.