BackMatting.ipynb

backmatting.ipynb

Is this script still working? I get some errors with video processing

Segmentation completes successfully Done: colab_inputs_video/input
But with both handheld and fixed, it fails

Traceback (most recent call last):
  File "test_pre_process_video.py", line 123, in <module>
    back_align = alignImages(back, image,mask)
  File "test_pre_process_video.py", line 38, in alignImages
    h, mask = cv2.findHomography(points1, points2, cv2.RANSAC)
cv2.error: OpenCV(3.4.5) /io/opencv/modules/calib3d/src/ptsetreg.cpp:169: error: (-215:Assertion failed) count >= 0 && count2 == count in function 'run'

CUDA Device: 0
Using video mode
Traceback (most recent call last):
  File "test_background-matting_image.py", line 121, in <module>
    bbox=get_bbox(rcnn,R=bgr_img0.shape[0],C=bgr_img0.shape[1])
  File "/content/Background-Matting/functions.py", line 38, in get_bbox
    x1, y1 = np.amin(where, axis=1)
  File "<__array_function__ internals>", line 6, in amin
  File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 2746, in amin
    keepdims=keepdims, initial=initial, where=where)
  File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 90, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation minimum which has no identity

I too got the same error ! The below error is for image, Not sure where is the issue !

CUDA Device: 0
Using image mode
Traceback (most recent call last):
  File "test_background-matting_image.py", line 46, in <module>
    back_img10=cv2.imread(args.target_back); back_img10=cv2.cvtColor(back_img10,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

errors

@camjac251 I think your issue may be similar to senguptaumd/Background-Matting#33 . In order to pre-process hand-held videos, that code tries to find a homography alignment between each frame, which may fail for featureless backgrounds. Unfortunately, the script doesn't report the errors nicely; but maybe you could try a different video, or a fixed cam (no alignment needed) video to see if that works?

@venkat094 Actually your error is different; they look the same because cv2's low level code reports them similarly. I am fairly sure the target background file you are trying to use does not exist. So when cv2.imread opens it it silently reads no data, but when it tries to convert the color space it realizes the image is not actually there.

Thank you for the reply. I'm trying it with the handheld sample video this time so hopefully it works.

I did notice an little error on the
Run the matting algorithm. If your video was captured with a fixed camera, comment the first line and uncomment the second before running.
block
could it be made into

!CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-hand-held -i colab_inputs_video/input/ -o colab_inputs_video/output/ -tb colab_inputs_video/background/
# !CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-fixed-cam -i colab_inputs_video/input/ -o colab_inputs_video/output/ -tb colab_inputs_video/background/ -b colab_inputs_video/$BM_SOURCE_STILL

@camjac251 good catch, fixed!

Hi, I have error for the following code:
!CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-hand-held -i colab_inputs/input/ -o colab_inputs/output/ -tb colab_inputs/background/0001.png

CUDA Device: 0 Using image mode Traceback (most recent call last): File "test_background-matting_image.py", line 121, in <module> bbox=get_bbox(rcnn,R=bgr_img0.shape[0],C=bgr_img0.shape[1]) File "/content/drive/My Drive/Colab-Notebooks/TensorFlow-Examples/033-Back-Matting/Background-Matting/functions.py", line 38, in get_bbox x1, y1 = np.amin(where, axis=1) File "<__array_function__ internals>", line 6, in amin File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 2746, in amin keepdims=keepdims, initial=initial, where=where) File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 90, in _wrapreduction return ufunc.reduce(obj, axis, dtype, out, **passkwargs) ValueError: zero-size array to reduction operation minimum which has no identity

@camjac251 I think your issue may be similar to senguptaumd/Background-Matting#33 . In order to pre-process hand-held videos, that code tries to find a homography alignment between each frame, which may fail for featureless backgrounds. Unfortunately, the script doesn't report the errors nicely; but maybe you could try a different video, or a fixed cam (no alignment needed) video to see if that works?

@venkat094 Actually your error is different; they look the same because cv2's low level code reports them similarly. I am fairly sure the target background file you are trying to use does not exist. So when cv2.imread opens it it silently reads no data, but when it tries to convert the color space it realizes the image is not actually there.

Thanks for your quick reply !!
I did check twice, the target background file is present ! I have just uploaded only one file !

@gchoi senguptaumd/Background-Matting#34 Maybe this helps?

In general, colab can be quite finicky (I think someone had their file get lost because the runtime restarted); if anyone has issues like these it may help to either (a) run the repository in a more standard environment or (b) add asserts to double check all the files exist and the segmentations + homography transforms don't fail for whatever reason.

I too got the same error ! The below error is for image, Not sure where is the issue !

CUDA Device: 0
Using image mode
Traceback (most recent call last):
  File "test_background-matting_image.py", line 46, in <module>
    back_img10=cv2.imread(args.target_back); back_img10=cv2.cvtColor(back_img10,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

I faced this error because of the following:

When I use the segmentation model, it needs to create a model by the name "maskDL" and there it stores the segmented image. Hence, it's not able to read the segmented image because it's not there.

Hello, thank you very much for sharing this project, I am having the following error, using video segmentation.
@Flock1 using the online example colab, how do I create a model with the name 'maskDL'?
Using the demo material: #Uncomment and run this if you want to skip uploading your videos (below) and run on the sample videos instead (choose fixed or handheld) It works well

The error I have with my own videos is the following:

CUDA Device: 0
Using video mode
Traceback (most recent call last):
File "test_background-matting_image.py", line 85, in
bg_im0=cv2.imread(os.path.join(data_path, filename.replace('_img','_back'))); bg_im0=cv2.cvtColor(bg_im0,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

Any help is welcome thanks

@tolchx, this error happens because there is no image that is being read. That's happening probably because there is no folder or a file. Kindly check if the required folder or the image is present.

@tolchx, this error happens because there is no image that is being read. That's happening probably because there is no folder or a file. Kindly check if the required folder or the image is present.

Yes, it was that, thank you very much.

folder and python file paths were in other folders,

Hello, I have an error when I run this section of the code:

#!CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-hand-held -i colab_inputs_video/input/ -o colab_inputs_video/output/ -tb colab_inputs_video/background/
!CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-fixed-cam -i colab_inputs_video/input/ -o colab_inputs_video/output/ -tb colab_inputs_video/background/ -b colab_inputs_video/$BM_SOURCE_STILL

error:

Traceback (most recent call last):
  File "test_background-matting_image.py", line 92, in <module>
    back_img10=cv2.imread(os.path.join(args.target_back,filename.replace('_img.png','.png'))); back_img10=cv2.cvtColor(back_img10,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

I did the test with several videos, with different amount of frames, 600/900/1000 frames, for example.
But it always stops at 441 and the error appears.

What could be the cause?

In the folder: Background-Matting / colab_inputs_video / inputs, the remaining frames appear, it does not end at frame 441

Thank you

@tolchx, I'm not sure but I think there is a problem with arguments. Instead of --tb, just use --back. Also, how are your results? Can you share them with me because I'm not getting that good results. I had to do some post-processing.

@tolchx, I'm not sure but I think there is a problem with arguments. Instead of --tb, just use --back. Also, how are your results? Can you share them with me because I'm not getting that good results. I had to do some post-processing.

I had good results with some videos, I'm experimenting with other material, here I uploaded a video, use the algorithm to segment the dancers, and then apply styletransfer https://www.instagram.com/p/CAsK54qgx_9/
I will try the solution you recommend, thanks

@tolchx, you are using it for something interesting. Can you elaborate on what you are doing? Also, did you make any changes to the code? Because I had to do some post-processing to get good results.

@Flock1 Have you got a good result? Can you share some tips? thanks
and I too got the same error

Done: 154/351
Done: 155/351
Traceback (most recent call last):
File "test_background-matting_image.py", line 92, in
back_img10=cv2.imread(os.path.join(args.target_back,filename.replace('_img.png','.png'))); back_img10=cv2.cvtColor(back_img10,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

@zbvictory, as I mentioned here, this error happens because the folder doesn't have the image you want to work with. Do check the folder you're getting the image from.

My raw results weren't that good. I'll share the changes I made.

@Flock1 Thank you very much for your reply and look forward to your sharing !

Hi! I have a small query. I am very interesting in background matting as a technique, however, I don't have photos with a separate background with and without the subject - I have photos of birds in a cage, and I wish to remove the background of the cage bars. I used the background photo for each image as the cropped background of the cage, so the photos are not the same size for the segmentation.

I got this error code
CUDA Device: 0 Using image mode

But it didn't seem like the segmentation actually was performed on the 7 examples of background and image that I provided - any advice would be GREATLY appreciated!

My results when I try and plot are [] instead of any images

@Sicily-F, from what I can imagine this is probably because it's not reading the images. Can you post the whole error? I might be able to figure out where exactly is the problem

This colab is no longer works because tf<2 is no longer supported in google colaboratory.

I'm getting issue when Setting up the environment:

can u please explain how to solve this issue