How can I use sunshine as video source for machine learning project?

[ZhengYuan-Public]

Hi, I'm not familiar with video encoding or decoding so please forgive me if this question sounds naive. I wonder if anyone could share ideas on how to use sunshine on the client machine to stream to a machine learning server and feed the data to machine learning algorithm (running on python).

Problem

When it comes to applying computer vision machine learning algorithms to games, the first step is always capturing the game. Currently there are several python libraries (pyautogui, win32gui, mss) available, and they all work by taking screenshots and hoping the process can be fast enough. However, this method can only get a maximum FPS of about 40 (after lowering the resolution from 1080p to 720p) on my machine. I want to improve the performance but I have no idea where I should look into.

[ReenigneArcher]

What is the intended purpose of providing video game capture to a machine learning algorithm?

[ZhengYuan-Public]

I'm interested in MARL (Multi-Agent Reinforcement Learning) and a famous example would be AlphaStar from Google. But AlphaStar relies on an API library provided by Blizzard, which is generally not possible for most other games. Compared to exact information (such as coordinates) provided from API, computer vision can provide less accurate information but it resembles more how human solve the problem.

[ReenigneArcher]

It seems to me that the only viable use case for this is creating cheat bots, and because of that I think you're unlikely to get much help on this.

[ZhengYuan-Public]

It seems to me that the only viable use case for this is creating cheat bots, and because of that I think you're unlikely to get much help on this.

After doing more search, DXcam which uses Windonw's DXGI Desktop Duplication API can achieve what I need.