How to synchronize dual IP cameras on open cv?

Question

I have an object detection program that sometimes utilizes dual cameras to record.

I had no problem synchronizing the cameras when I used USB webcams, but I have recently switched to RJ45 IP PoE security cameras, and now the recording is up to 2 seconds (45 frames) desynchronized.

My code looks like this:

webcam = cv2.VideoCapture(pipeargs.source)
isvideo = False
if "video" in pipeargs.source:
 isvideo = True
camlog.info('camera created')
if pipeargs.source2 is not None:
 webcam2 = cv2.VideoCapture(pipeargs.source2)
while True:
 stream_ok, frame = webcam.read()
 if pipeargs.source2 is not None:
 stream_ok2, frame2 = webcam2.read()

This perfectly synchronized the USB webcams, but now the source has switched from the cameras ID to an rtsp url:

camera1 = rtsp://192.168.2.32:554/user=admin&password=&channel=1&stream=0.sdp?
;camera1 = 0

Cameras are using h.264 (could use h.265), and I believe the problem resides on cameras bursting many frames in 1 packet?, not sure yet, if that is the case the only way to synchronize would be to somehow know how many frames to discard at the start from the camera that goes faster.

EDIT FOR CLARIFITCATION: The goal is to have 2 frames that are miliseconds appart and can be analyced in combination, because currently they are separated by seconds.

• Christoph Rackwitz: can't just read a frame from both in lockstep. you're gonna have to use VideoCapture_waitAny(). this also goes for webcams attached using USB. cameras will always run on individual clocks, unless there is some clock sync/genlock involved, which it never is with these types of questions.
• Mark Setchell: Maybe you can use flash, or infrared flash to sync them.
• user1878697: How does videocapture_waitany() work?
• Christoph: You should basically move both streams into its own threads, both writes Into its own queues, where you can get data from it. Like framebuffer. But you should not tight couple frame reading from multiple streams in one thread. Think of an Producer, consumer approche.
• Christoph Rackwitz: how waitAny is implemented? IDK. probably polling each device/driver for available frames, then reporting that status. if you just call read() on a VideoCapture, it will not poll, but it will wait until there's a frame (or not wait, if there is a frame already in the buffer). -- how waitAny is to be used? call it with a list of your videocaptures.

Answer 1

The standard solution is to have producers and consumers. This involves multiple threads.

Video device gets a thread capturing from it, and sending those frames into a queue.

A consumer reads from the queue and uses those frames.

Again, don't assume that cameras capture in lockstep. Each camera has its own clock. They are not equal. They will drift relative to each other. You'd best "do something" on every frame that is captured. That means whatever you do will have a frame rate equallying the sum of the frame rates of both capture objects.

Queues are higher-level synchronization primitives. There are other synchronization primitives you can use to construct specific behavior. Python's threading has Condition objects. I used those to write a class that removes buffering/queueing, and will always give you the latest frame, or else wait for the next one. That is for single VideoCapture sources. To combine multiple into a sensible stream of frames, you'd have to either live with what that does and use several instances of it, or write a new class that does what you need. I'm just saying, queues aren't the only thing out there.

---

There is VideoCapture::waitAny() but so far it's only supported on Linux (V4L). If you have that, then...

Use cv.VideoCapture_waitAny(). It takes a list of capture objects and it will wait until any of those has a frame ready to read (or the timeout interval expires).

This will run (update the GUI) at (at most) the summed frame rate of all capture objects.

import cv2 as cv
caps = [
 cv.VideoCapture(...), # the first
 cv.VideoCapture(...), # the second
 # ...
]
assert all(cap.isOpened() for cap in caps)
for idx, cap in enumerate(caps):
 cv.namedWindow(f"capture {idx}")
while True:
 # 1e9 = 1 second of timeout in nanoseconds
 (rv, readyIndex) = cv.VideoCapture_waitAny(caps, int(1e9))
 if rv:
 print("timeout")
 else:
 for idx in readyIndex:
 cap = caps[idx]
 (rv, frame) = cap.read()
 assert rv
 
 # do whatever with `frame`
 cv.imshow(f"capture {idx}", frame)
 key = cv.pollKey() # like waitKey() but on some backends it doesn't wait at all
 if key == -1:
 continue
 elif key in (13, 27): # ENTER, ESCAPE
 break
 else:
 print(f"key code {key:08X}h : {key} decimal")
for cap in caps:
 cap.release()
cv.destroyAllWindows()

• user1878697: The thing I want to do with each "frame" is to run it though object detection and add the results with the frame from the other camera. But at the moment I am counting objects twice or zero times when they are transitioning between cameras. that is why I want them to be synchronous. I don't understand if what you are showing is just best practice for 2 cameras or you think it helps somehow to synchronice the camera frames

Answer 2

I have found 2 solutions:

First of all there is an object that I know that can only appear once on the images. Tracking it, there could be 4 states:

-Left camera appeareance

-Right camera appeareance

-Both cameras appearence

-No camera appeareance

Tracking that object I concluded that the desynchronization between videos was sometimes 42FPS and on another set of cameras was 50FPS, so I just dropped a few frames and it went perfect.

The problem is that if the object decided to stay exactly between the cameras for some time it could not be tracked and the method would turn innacurate.

The other solution is more practical, when I request the first frame (or when I do the videocapture) I initiate a streaming proccess on the camera. This proccess blocks the proccess for a couple of seconds until the camera can send me back a packet with frames that I will decode one by one. The problem is that the sequential initiation is nowhere simultaneous. The key is to initiate the cameras on 2 proccess to do it nearly at same time. Once initiated, it doesn't matter anymore.

Problem with this method is that if the camera monitor is running on the browser, the initialization is already running and it doesn't work. Gotta make sure there is no streaming going on from other source.