python 3 - win32api.PostMessage click on minimized window

Question

I'm trying to click a windows application window while its in a minimized state (don't maximize).

The following code is able to take a snapshot image of a minimized application, then perform image recognition and output the click coordinates ... but I cannot get the click function to work on a minimized window.

from ctypes import windll
import win32gui, win32ui, win32con
import numpy as np
from PIL import Image
import pdb
import win32api
import win32gui
import time
import win32con
hwnd = win32gui.FindWindow(None, 'app title')
left, top, right, bot = win32gui.GetWindowRect(hwnd)
w = right - left
h = bot - top
hwndDC = win32gui.GetWindowDC(hwnd)
mfcDC = win32ui.CreateDCFromHandle(hwndDC)
saveDC = mfcDC.CreateCompatibleDC()
saveBitMap = win32ui.CreateBitmap()
saveBitMap.CreateCompatibleBitmap(mfcDC, w, h)
saveDC.SelectObject(saveBitMap)
result = windll.user32.PrintWindow(hwnd, saveDC.GetSafeHdc(), 3)
print(result)
bmpinfo = saveBitMap.GetInfo()
bmpstr = saveBitMap.GetBitmapBits(True)
im = Image.frombuffer('RGB',(bmpinfo['bmWidth'], bmpinfo['bmHeight']),bmpstr, 'raw', 'BGRX', 0, 1)
win32gui.DeleteObject(saveBitMap.GetHandle())
saveDC.DeleteDC()
mfcDC.DeleteDC()
win32gui.ReleaseDC(hwnd, hwndDC)
if result == 1:
 #PrintWindow Succeeded
 im.save("test.png")
#Load Target Image
template = Image.open("VI.jpg")
template_np = np.array(template)
#Perform Image Matching
import cv2
screenshot = np.array(im)
result = cv2.matchTemplate(screenshot, template_np, cv2.TM_CCOEFF_NORMED)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)
threshold = 0.8
if max_val >= threshold:
 print("Image found at:", max_loc)

The following code, is able to perform the click function as long as the window is maximized.

import ctypes
# Define constants for mouse events
x = max_loc[0]
y = max_loc[1]
MOUSEEVENTF_LEFTDOWN = 0x0002
MOUSEEVENTF_LEFTUP = 0x0004
ctypes.windll.user32.SetCursorPos(x, y)
ctypes.windll.user32.mouse_event(MOUSEEVENTF_LEFTDOWN, x, y, 0, 0)
ctypes.windll.user32.mouse_event(MOUSEEVENTF_LEFTUP, x, y, 0, 0)

Im looking for a way that the window can stay minimized, so that I can perform other tasks with my mouse, while the click function completes

• IInspectable: There is no supported API for taking a screenshot of a minimized window. You can either enter into an uphill battle (you cannot win) or acknowledge the limitations and move on.
• IInspectable: As for the question you didn't ask: You can use UI Automation to automate a UI. UI Automation makes no requirements regarding a window's visibility state.
• Rhys: this line of code is taking a screenshot of a minimized application im = Image.frombuffer('RGB',(bmpinfo['bmWidth'], bmpinfo['bmHeight']),bmpstr, 'raw', 'BGRX', 0, 1) ... and saving it as ... im.save("test.png") ... I am trying to go one step further with the click function. Is there a listener like SPY++ that is built for python not visual studios?
• IInspectable: And while I somehow forgot to post this: Make sure to read (and comprehend) this: You can't simulate keyboard input with PostMessage.
• Rhys: that is interesting, thank you! I thought it would be easy for my small apps. Alas ... its not. I read this article posted 1 month ago and it gave me hope ... but its still not working. reddit.com/r/learnprogramming/comments/1k1a947/…

Answer 1

Use Windows Virtual Desktop.

A minimized window is usually not rendered, and even if you manage to take a screenshot of it, you will find that its image is stuck on the last frame before it was minimized.

You just want your automation to not interfere with normal computer use, so there is absolutely no need to run it in a minimized window. You can put your target window on another virtual desktop, and then the problem becomes a relatively simple "send input to the background window".

Some suggestions for simulating input using PostMessage: If you can successfully use PostMessage to send input to the foreground window, but it doesn't work for the background window, you can try to send a WM_SETFOCUS Message to trick it, which can successfully fool some programs.

Windows 10 virtual desktops are a window management feature, not a security feature

Virtual desktops are a window management feature, not a performance feature

• IInspectable: Ignoring the fact that a virtual desktop doesn't fundamentally change the core issue, is there any advantage in using this solution over UI Automation? If you want to automate a UI and not have that interfere in any way with regular computer use, UI Automation is the solution.
• 許恩嘉: @IInspectable This is the solution I used, it works and it is not possible to use UI Automation in my use case.