How to process video frame by frame

Use your own video handling to feed TraceVisionSDK frames one at a time

Overview

With TraceVision you can create a processing session using createVideoByFramesSession(isSynchronous:) that will receive frames one by one and provide the highlights as soon as they are found. Using this type of session if you have your own video handling and want to feed the frames to the SDK.

Note

This type of session provides the highlight metadata in the HighlightObject but does not provide video results, so you’ll need to create the video highlight by yourself.

Create frame processing session

Call createVideoByFramesSession(isSynchronous:) on the shared instance to get the session object. Hold it until you are done feeding the frames and finalized the processing. The call gets one parameter isSynchronous that controls if the processing is done on the same thread where the caller is or in the separate background thread.

The rule of thumb here is:

  • if you get the frames in real time like from the camera then use the isSynchronous = false and feed the frames to the session. It will process them in the separate thead, will not block your camera one and it will keep up with the real time recording.
    • if you read the frames from the file or other source that can deliver them faster than realtime then use isSynchronous=true and the session will run in the same thread and you won’t face the memory overflow issues because of too many frames waiting in the processing queue.
    let session = TraceVision.shared.createVideoByFramesSession(isSynchronous: false)

Feed the frames to the session and get the results

Here is the sequence of methods that needs to be called to process the frames:

  • First start the processing with start() method call. It will initialize internal structures and the session will be ready to process frames as soon as this call is finished.
  • On every frame call processFrame(buffer:) and pass the video frame as a CMSampleBuffer. Be sure to deliver every frame in the same order as they were recorded without any gaps.

  • When done call stop() to finalize the processing and free the resources.

Note

Current implementation assumes that the video has 30 fps frame rate, if you want to use 60 fps, please call processFrame(buffer:) on every other frame.

    session.start()

    // On every video frame
    session.processFrame(buffer: sampleBuffer)

    session.stop()

Observe the results

The results can be accessed via session status observable object that has the list of highlightss found in the video. This property is updated in real time as soon as the highlight is detected and finalized. The update is done on the main thread so it is safe to use the property directly in the UI.