I Need a Bit of Help Installing This Project

[CosmicBen]

Sorry to bother everyone but I might be a bit incompetent when it comes to Python. Typically I work with programs like TD which uses python scripts, or more old school C++ projects that are built using VS or Unreal Engine.

Basically I am trying to install this project and hopefully get it to work with a webcamera but I have almost no idea what I am doing. I'll outline the current mess below in some bullet points.

• I am using Windows 10 Home edition.
• I have OBS installed. (my goal is to use the webcam for live keying)
• I have Python installed 3.10.11
• I have Pip installed 23.0.1
• Using command prompt I:
  • I tried to follow the readme by running "pip install " for all requirements (this probably was not required but I have next to no idea what I am doing)
  • I ran "pip install --extra-index-url https://download.pytorch.org/whl/cu118 transparent-background"
  • I ran "python install setup.py"
    • This gave me an error because it wanted to modify the winapps folder and Windows likes to make things more difficult with every release.
    • I then ran "pip install ." and something happened (I got this as a recommendation and have no idea if it helped)
  • I then modified usage.py to point to a local checkpoint file I downloaded and tried uncessfully to run it (I commented out the image processing section and left the video processing section). This command failed with a pytorch cuda error.
• I ran out of patiences and decided to sacrafice my dignity to the gods of debugging.

I'm pretty sure whatever I am doing at this point is probably the wrong approach so I was hoping someone could set me straight. I see people talking about Conda/Anaconda but I was not able to figure out how to use this properly either, most of this container/temporary environment stuff is a bit out of my league at the moment. I'm happy to use this stuff but I honestly can't find a good starting point to get my head around it. If anyone can offer up either a clear set of instructions for a Windows user trying to get this to work with a webcam, or possibly event just some tutorial videos that would provide the background knowledge to get this working I would really appreciate it.

[plemeri]

Hi @CosmicBen, I just updated main branch to fix some issues related to the webcam input. There are few problems related to pyvirtualcam, so relaying output to the virtual camera is still not working, but it'll show the result with cv2.imshow now.

For the installation, you just need to install once with the below command. If there's no error, then you're good to go.

pip install --extra-index-url https://download.pytorch.org/whl/cu118 transparent-background

You also mentioned that This command failed with a pytorch cuda error. It seems like you don't have any CUDA devices on your machine. Try using with cpu but it will be extremely slow with base model, so I recommend using fast model. Also, I think you're trying to develop your own project using our package, so I would recommend you to use our package as a Python API.

import cv2
import numpy as np

from PIL import Image
from transparent_background import Remover

# Load model
remover = Remover(mode='fast', device='cpu')

For the webcam input, just change the VideoCapture argument with your camera index (usually it is 0 as below). The usage will be the same for the rest of the part.

cap = cv2.VideoCapture(0)

while cap.isOpened():
    ret, frame = cap.read() # read video

    if ret is False:
        break

    if cv2.waitKey(1) & 0xFF == 27:
        break
        
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) 
    img = Image.fromarray(frame).convert('RGB')
    out = remover.process(img, type='blur')
    cv2.imshow('output', cv2.cvtColor(np.array(out), cv2.COLOR_BGR2RGB))

cap.release()
writer.release()

You can choose type from the below in out = remover.process(img, type='blur')

Lastly, I noticed that pyvirtualcam also has UnityCapture backend, so I'll try this one if it works, but currently I don't have much time for it.

[CosmicBen]

Thank you for the reply. I broke down and tried installing Anaconda (no idea if that helped or not) and I ran the python instruction you sent. I also installed Cuda since you mentioned it was probably missing. After that I was able to run the code you sent me (saved in a file) sucessfully. Everything seems to be working now, unfortunately the FPS is only about 15 so I need to find a way to optimize it up to 30 (I'm not sure if multithreading is an option in this case) or try another system. As a side note, for some reason my OBS camera was index 7 so that caused some confusion as well.

[CosmicBen]

By any chance is there a checkpoint that focuses specifically on separating the background from humans? I'm in the process of trying all the checkpoints I would appreciate any tips you can offer.

[plemeri]

Unfortunately, we don't have any checkpoints trained only with human data. Moreover, fast model was trained only with small scale (384 x 384) while base model was trained with large scale (1024 x 1024) so I would recommend testing base model for higher quality.