DisplayFusion has options for this. It's It's a great Window Manager to run under Windows.
Go to settings
Go to the "Alt+Tab" area of settings
Use the dropdown at the top to change the value to "Show on All Monitors, Show All Windows".
(There are some other values to experiment with in this dropdown too, depending on personal preference)