Unmodalize Dialogs (Make Modal Dialogs Non-Modal)

Question

I'm on Windows 10 and annoyed by many modal dialogs of built-in applications (e.g. Windows-Explorer's Options dialog) and third party applications (e.g. Eclipse's Run Configuration dialog).

Example

Open Windows Explorer (Win+E), go to the tab View, then click Options. The option dialog opens. As long as the option dialog stays open, you cannot interact with the explorer window (from which you opened the dialog) at all. Even moving or resizing the window is prohibited.

I don't see a reason for this behavior. With the options dialog open, I still can open another Explorer window (which runs in the same process) and interact with that one. I think, Explorer would be totally capable to allow interaction with windows from which the options dialog was opened.

Same goes for the mentioned third party applications.

Question

❓ Is there a tool* to make modal windows non-modal?
_{* You may substitute the term »tool« by hack, patch, or whatever comes to your mind.}

Ideally, the tool would apply to all windows on the system. However, the following partial solutions are welcome too:

• A tool to disable modality in one specific UI library (e.g. Windows Forms or Java Swing)
• A tool to disable modality in one specific application (e.g. Windows Explorer)
• A tool to allow the background windows (from which the modal dialog was opened) to be moved and resized, but not interacted with in any other way. This is the default on some Linux window managers, e.g. Cinnamon.

It is ok for the affected application to crash when an otherwise forbidden interaction changes the state of the application in unforeseen ways (e.g. deleting an item in the background window while editing its properties in a modal dialog).

.

---

⚠️ The following sections are not important for my question but maybe for your answer. If you are about to write something like "that's not possible", I would be happy if you could read them and carefully think about it first.

However, I must beg you: Please do NOT start a discussion on the implementation details. I'm not so much interested in implementing this myself. That's why I asked on superuser.com and not on stackoverflow.com.

---

Similar Problems With Solutions

I'm positive that you can intercept calls that create modal dialogs; after all, there are solutions for similar problems:

• Any non-resizeable dialog can be made resizeable using ResizeEnable.
  _{Fun (or sad?) fact: ResizeEnable was last updated in 2003 but still works for Windows 10 without any problems. (Windows XP released in 2001, but from ResizeEnable's change log it seems like the author did not use XP at the time of the last release.)}

• Many fullscreen-only applications can be run in a window using DxWnd.

Technical Details

I'm very well aware that modal dialogs can return a value and that the application cannot simply continue without that return value. However, a workaround must be possible at least in theory:

If a UI library offers a way to use modal dialogs with return values, then that library also has to implement a way to pause the current thread and show the modal dialog in another thread such that the modal dialog can be interacted with. This applies even to single threaded UIs (which seem to be the standard). For example, in Java Swing the event dispatch thread (ETD) offers the method createSecondaryLoop() which is used by Swing itself to implement modal dialogs with return values. Usually these methods also move UI control to the new thread such that the background window is not frozen. This applies even to Windows explorer, as demonstrated in this screen cast (video plays in background while a modal dialog is open):

We can use that functionality too. The general approach is:

• Intercept the call that would create a modal dialog
• Use the UI library's own functionality to pause the current thread and move UI control to a new thread
• Create and show an equivalent non-modal dialog
• When the dialog is closed, retrieve the data for the return value
• Use the UI library's own functionality to resume the original UI thread
• Return the data from the non-modal dialog

Answer 1

Harrymc pointed out, that AutoHotKey might be able to unmodalize windows, which lead me to this question that contained the following AHK script (with a minor syntax error which was fixed in below version):

^e::
MouseGetPos,,, WindowUnderMouse
WinSet, Style, -0x8000000, ahk_id %WindowUnderMouse%
return

With AutoHotKey installed, save this script in a file.ahk, double click the file (or right-click and select Run Script). If you have a modal dialog and want to interact with its parent/background windows, hover the mouse over the parent/background window and press CtrlE (that's what ^e::) stands for; the background/parent window should now react to interaction again without closing the modal dialog.

The "styles" are documented here. -0x8000000 is documented as

WS_DISABLED | 0x8000000 | +/-Disabled. Creates a window that is initially disabled.

Answer 2

Modal dialogs are used for two main purposes:

1. Get user parameters.
   An example here is text editor's Open dialog for the file to edit.

2. Warning/question before continuing.
   A famous example here is the Windows Update dialog asking for permission to restart Windows.

I understand you are asking for ways to automate answering/clicking specific dialogs, since I don't see a general answer. Using a tool that will, for example, click the OK button of every dialog is a sure way to disaster.

Tools may exist for specific dialogs, meaning dialogs that are identified by their title and/or originating process. You could either try existing tools, or very easily build your own using a product such as AutoHotkey together with Spy++ to inspect the given dialog's elements.

For the first case, you could perhaps use a tool such as LastPass to fill a dialog's fields with remembered values.

For the second case, a AutoHotkey script is the best tool. You may also have a look at the Microsoft 2008 article Enabling Default Reply, which details a registry entry that causes the intercept of every MessageBox function with automatically choosing the default button (I don't know if it still works in Windows 10 and have no motivation to try).

I can't think of any other useful case.

Note that my analysis above was in the direction of letting the program work in a normal manner, rather than by brute-force converting some or all modal dialogs to modless. This is because in all the programs I ever created, I cannot conceive of any program continuing correctly once a modal dialog did not block its originating thread.

Answer 3

Modal Dialog Boxes

A modal dialog box should be a pop-up window having a window menu, a title bar, and a thick border; that is, the dialog box template should specify the WS_POPUP, WS_SYSMENU, WS_CAPTION, and DS_MODALFRAME styles. Although an application can designate the WS_VISIBLE style, the system always displays a modal dialog box regardless of whether the dialog box template specifies the WS_VISIBLE style. An application must not create a modal dialog box having the WS_CHILD style. A modal dialog box with this style disables itself, preventing any subsequent input from reaching the application.

An application creates a modal dialog box by using either the DialogBox or DialogBoxIndirect function. DialogBox requires the name or identifier of a resource containing a dialog box template; DialogBoxIndirect requires a handle to a memory object containing a dialog box template. The DialogBoxParam and DialogBoxIndirectParam functions also create modal dialog boxes; they are identical to the previously mentioned functions but pass a specified parameter to the dialog box procedure when the dialog box is created.

When creating the modal dialog box, the system makes it the active window. The dialog box remains active until the dialog box procedure calls the EndDialog function or the system activates a window in another application. Neither the user nor the application can make the owner window active until the modal dialog box is destroyed.

When the owner window is not already disabled, the system automatically disables the window and any child windows belonging to it when it creates the modal dialog box. The owner window remains disabled until the dialog box is destroyed. Although a dialog box procedure could potentially enable the owner window at any time, enabling the owner defeats the purpose of the modal dialog box and is not recommended. When the dialog box procedure is destroyed, the system enables the owner window again, but only if the modal dialog box caused the owner to be disabled.

As the system creates the modal dialog box, it sends the WM_CANCELMODE message to the window (if any) currently capturing mouse input. An application that receives this message should release the mouse capture so that the user can move the mouse in the modal dialog box. Because the system disables the owner window, all mouse input is lost if the owner fails to release the mouse upon receiving this message.

To process messages for the modal dialog box, the system starts its own message loop, taking temporary control of the message queue for the entire application. When the system retrieves a message that is not explicitly for the dialog box, it dispatches the message to the appropriate window. If it retrieves a WM_QUIT message, it posts the message back to the application message queue so that the application's main message loop can eventually retrieve the message.

The system sends the WM_ENTERIDLE message to the owner window whenever the application message queue is empty. The application can use this message to carry out a background task while the dialog box remains on the screen. When an application uses the message in this way, the application must frequently yield control (for example, by using the PeekMessage function) so that the modal dialog box can receive any user input. To prevent the modal dialog box from sending the WM_ENTERIDLE messages, the application can specify the DS_NOIDLEMSG style when creating the dialog box.

An application destroys a modal dialog box by using the EndDialog function. In most cases, the dialog box procedure calls EndDialog when the user clicks Close from the dialog box's window menu or clicks the OK or Cancel button in the dialog box. The dialog box can return a value through the DialogBox function (or other creation functions) by specifying a value when calling the EndDialog function. The system returns this value after destroying the dialog box. Most applications use this return value to determine whether the dialog box completed its task successfully or was canceled by the user. The system does not return control from the function that creates the dialog box until the dialog box procedure has called the EndDialog function.

and

Modeless Dialog Boxes

A modeless dialog box should be a pop-up window having a window menu, a title bar, and a thin border; that is, the dialog box template should specify the WS_POPUP, WS_CAPTION, WS_BORDER, and WS_SYSMENU styles. The system does not automatically display the dialog box unless the template specifies the WS_VISIBLE style.

An application creates a modeless dialog box by using the CreateDialog or CreateDialogIndirect function. CreateDialog requires the name or identifier of a resource containing a dialog box template; CreateDialogIndirect requires a handle to a memory object containing a dialog box template. Two other functions, CreateDialogParam and CreateDialogIndirectParam, also create modeless dialog boxes; they pass a specified parameter to the dialog box procedure when the dialog box is created.

CreateDialog and other creation functions return a window handle for the dialog box. The application and the dialog box procedure can use this handle to manage the dialog box. For example, if WS_VISIBLE is not specified in the dialog box template, the application can display the dialog box by passing the window handle to the ShowWindow function.

A modeless dialog box neither disables the owner window nor sends messages to it. When creating the dialog box, the system makes it the active window, but the user or the application can change the active window at any time. If the dialog box does become inactive, it remains above the owner window in the Z order, even if the owner window is active.

The application is responsible for retrieving and dispatching input messages to the dialog box. Most applications use the main message loop for this. To permit the user to move to and select controls by using the keyboard, however, the application must call the IsDialogMessage function. For more information about this function, see Dialog Box Keyboard Interface.

A modeless dialog box cannot return a value to the application as a modal dialog box does, but the dialog box procedure can send information to the owner window by using the SendMessage function.

An application must destroy all modeless dialog boxes before terminating. It can destroy a modeless dialog box by using the DestroyWindow function. In most cases, the dialog box procedure calls DestroyWindow in response to user input, such as clicking the Cancel button. If the user never closes the dialog box in this way, the application must call DestroyWindow.

DestroyWindow invalidates the window handle for the dialog box, so any subsequent calls to functions that use the handle return error values. To prevent errors, the dialog box procedure should notify the owner that the dialog box has been destroyed. Many applications maintain a global variable containing the handle for the dialog box. When the dialog box procedure destroys the dialog box, it also sets the global variable to NULL, indicating that the dialog box is no longer valid.

The dialog box procedure must not call the EndDialog function to destroy a modeless dialog box.

When creating the modal dialog box, the system makes it the active window. The dialog box remains active until the dialog box procedure calls the EndDialog function or the system activates a window in another application. Neither the user nor the application can make the owner window active until the modal dialog box is destroyed.

When the owner window is not already disabled, the system automatically disables the window and any child windows belonging to it when it creates the modal dialog box. The owner window remains disabled until the dialog box is destroyed. Although a dialog box procedure could potentially enable the owner window at any time, enabling the owner defeats the purpose of the modal dialog box and is not recommended. When the dialog box procedure is destroyed, the system enables the owner window again, but only if the modal dialog box caused the owner to be disabled.

https://docs.microsoft.com/en-us/windows/win32/dlgbox/about-dialog-boxes

So you must want your apps to crash.