User-mode applications generally can't slow down your OS's GUI. However, the situation isn't as clear-cut as it seems. There isn't a single user-mode application that is 100% user-mode, either because of system calls (notably the file system) or because of virtual memory mapping.
In most cases, the OS isn't stuck doing CPU-work, it's stuck waiting on some I/O resource. This is especially apparent when you run out of physical memory and start going into the pagefile (although do note that the OS can put memory in the page file at its leisure, even if there's plenty of physical RAM available). Windows is usually pretty smart about this, but it's quite possible to "break" this.
If it really is CPU, it's most likely due to a greedy driver (the vast majority of which is not written by Microsoft). Kernel-mode drivers are exempted from pre-emptive multi-tasking (both for latency and reliability reasons), so if a driver runs a second long loop on the CPU, you're out of luck, it will not be pre-empted.
A great and simple tool to see some of this is Process Explorer (from SysInternals) which shows you the kernel-time of a CPU (ie. how much of the CPU work on the core is done in the kernel, as opposed to the user applications themselves). Windows 7 and later also include this in their task manager (it's the red line in the CPU usage graphs).
All in all, pre-emptive multi-tasking doesn't save the OS designers from having to make compromises. There's always a cost to everything, and task scheduling is indeed very tricky (right now, my OS juggles over 2000 threads - that's quite a lot, and I'm not really doing "anything"). Would it be better to give the threads smaller time chunks, and spend more time doing context switching? Would it be better to give them longer time chunks, sacrificing latency?
So, check what the hard drives are doing. Check how your memory is used. Check the kernel-time. This will quite likely show you why Windows loses responsivity while you're doing heavy work. Some of these can be remedied (freeing memory, limiting the memory-offender, picking CPU affinity manually...), some are solved simply by keeping your drivers up to date (graphics drivers used to be quite notable for the issues they caused, which probably played a part in Windows dumping them back from kernel-mode to user-mode in recent versions).
Also, being on the subject of the GPU, modern Windows use GPU acceleration for rendering the GUI (actually, to a point, so did Windows XP). If your application taxes the GPU significantly, it might also lead to slow responsiveness of Windows, especially in Aero. Since GPUs are increasingly being used as GP "hyper" CPUs, this can be significant even outside of games and such.
Another major offender is a poorly written multi-threaded application. It's quite easy to kill memory caching if you're doing silly things, and RAM is extremely slow compared to the CPU itself, so without efficient caching, the CPU gets very, very slow (even though it's basically waiting the whole time). This is even more complicated on multi-core CPUs (and multi-CPU systems), because to ensure consistency, many multi-threaded operations require an invalidation of cached data (so that one CPU doesn't access the "old" value of the variable in memory out of its cache / registers). All those things are incredibly fast, but... CPUs are faster. Much faster. Which of course also introduces the issues of different CPU providers handling the same things differently, completely separate from the much higher-level OS. Modern CPUs (486+, so yeah, it's a very old feature a disturbing amount of programmers has no idea of) are actually heavily parallelized, they're no longer executing one instruction after another.
So even if the OS does everything perfectly (obviously an impossible ideal), it can still grind down to a halt due to hardware issues and communication with the hardware. What good is it that you have a 4-core CPU when your application manages to completely saturate the RAM R/W? What good is it having a fast hard drive, when every single byte it reads has to go through the CPU (remember PIO?). Every hardware operation that doesn't use direct memory access can potentially stall your CPU.
And now, most of Windows actually runs as user-mode applications. And they interact with the applications running as well - if I ask explorer.exe
to do something for me, it can't do anything else in the mean-time. If I send a billion windows messages to a window, it's going to leave a mark.
There's just so much happening, all the time... guarantees are very difficult. Note how faster things become as soon as the "ctrl-alt-delete" screen actually comes on - suddenly it's as if nothing was wrong :)