The other questions here didn't answer your part 2, so I'll give that a shot:
2. How does windows do this? (I'm more curious then anything, but if it's not to complex, maybe I could implement something similar in my own program?)
The thing to realize is that even if you have tens of windows open, each with many toolbars, each with many items in it, and so on, every time you move the mouse, windows doesn't need to check everything.
Windows is basically structured in two layers: there's HWNDs, which is how Windows itself manages the subdivision of space on the desktop; and typically within each HWND is a control that manages its own space within that HWND: a listbox managing its own list items, a tab control managing its own tabs, a HTML control managing its own HTML page layout, and so on. (Or, in your case, code managing 50 or so rectangles.)
When the mouse moves, Windows first determines the correct HWND to send that WM_MOUSEMOVE to. And it does so by traversing the HWNDs. The HWNDs are stored as a tree, representing containment, and the order among siblings representing Z-Order, so Windows can do a simple depth-first descent into this tree to find out the bottom-most HWND at any given point. If you start the Spy++ app, you can see for yourself what this HWND tree looks like. Note that Windows is not doing a full exhaustive traversal: while traversing the top-level application windows, for example, to find out which app the point is in, as soon as windows finds the first top-level HWND that contains the point, it will drill into that, outright ignoring all other apps that are beneath/after it - and all the controls within them. This is the key that means windows only has to traverse relatively few HWNDs even if there are many many visible on the screen at once.
Once Windows determines the correct HWND, it sends the appropriate message to that (WM_NCHITTEST, WM_MOUSEMOVE, and so on), and it's then up to that control to do likewise for its own content. For a listbox, containing fixed-size items, determining the item at a particular point may be as simple as a division operation; or for a HTML control, the control may have its own equivalent to a "layout tree" that it can use to traverse an quickly determine the element at that point. In your case, looping through the list of rectangles may be perfectly fine.
That's the somewhat simplified version: it's a bit more complex than above - eg. windows doesn't just to a point-in-rect check, there's other checks to allow for odd-shaped and transparent windows (and invisible and disabled windows); but the basic tree descent idea applies.
The other important issue to remember is that all this is pretty quick: moving a mouse takes place in "human time", and a modern CPU can to a lot of operations in the time it takes the mouse to move a couple of pixels on the screen. And finally, note that when you move the mouse from point A to point B on the screen, the mouse doesn't always traverse every single pixel in between - especially if you move the mouse rapidly.