How to process video from camera roll

Set up video import session and find highlights from pre-recorded video

Overview

With TraceVision you can create a video import session using createVideoImportSession(needVideoResults:). This session can be used to process pre-recorded video file that should be locally accessible by the SDK, i.e saved into the space of your app. You can use the video files from the Camera roll, but you’ll need to download them to the local folder and then provide the URL of the file to the session to use. The file should be accessible through the whole processing and can be deleted when it’s done. The import session will process the video in the background thread and provide the highlights as soon as they are found.

Create video import session and import the video

Use shared instance to call createVideoImportSession(needVideoResults:) and pass it to your component. The only parameter needVideoResults defaults to true and means that the output should produce video highlights. If it’s set to false then only highlight metadata is provided and the developer should take care of the video by themselves.

In the example below we create the import session and an importVideo function that takes NSItemProvider provided by the VideoPickerView when a video is selected in the camera roll by the user.

Example:

// The session should be created once and the reference is kept until the processing is finished
let session = TraceVision.shared.createVideoImportSession()

func importVideo(provider: NSItemProvider?) {
        _ = provider?.loadFileRepresentation(forTypeIdentifier: UTType.movie.identifier)
        { (url, err) in
            guard let url = url else { return }
            // import the local video into the session
            session.importVideo(fromFile: url, useInPlace: false)
        }
}

To import the local video into the session call importVideo(fromFile:useInPlace:) and pass the URL to the file. If the file is a temporary one that will be deleted very soon then the useInPlace should be set to false and the SDK will copy it to the internal storage and delete the copy after the processing is done. If useInPlace is set to true, you should make sure that the video is available during the processing.

Start processing and getting the results during the processing

After the session is initialized and video is imported we can start the processing by calling start(). To observe the highlights being found during the processing use status observable object. It is implemented with VideoImportStatus observable class with properties to check the status of the processing and the progress of the processing. This class inherits the main VideoStatus that provides the updates about number of highlights found and the list of the highlights that are finalized and ready to use.

    session.start()

To track status updates easier in SwiftUI component, it’s possible to define new VideoImportStatus object in your component and connect it to the session with setStatusObject(_:) or just use the existing status provided by the session.

    @StateObject
    private var status = VideoImportStatus()

    let provider: NSItemProvider?

...

    var body: some View {
        VStack() {
        }
        .onChange(of: status.processing) { val in
            if val == true {
                // we started the processing, let's hide the loading animation
                withAnimation {
                    isLoading = false
                }
            }
        }

        .onViewDidLoad {
            importVideo(provider: provider)
        }
    }

...

    func importVideo(provider: NSItemProvider?) {
        _ = provider?.loadFileRepresentation(forTypeIdentifier: UTType.movie.identifier)
        { (url, err) in
            guard let url = url else { return }
            // import the local video into the session
            session.importVideo(fromFile: url, useInPlace: false)

            // set the observable status object and start
            session.setStatusObject(status)
            session.start()
        }
    }

Using the results, export the video highlight

Once the HighlightObject is available via highlights list we can access the metadata like playerBboxHistory or just simply save and export the highlight video with HighlightObject/saveVideo(toCameraRoll:). The latter will provide the URL to the video highlight saved internally in the app. If toCameraRoll is set to true then the highlight will be exported to the camera roll as well as saved locally.

    let highlight = status.highlights.first
    let savedUrl = await highlight?.saveVideo(toCameraRoll: true)

Finishing the processing

The import session will run in the background until every frame of the video is processed and it will stop automatically. In this case you won’t need to do anything special, just wait and observe processing to become false which means that the processing is finished and the final list of the HighlightObjects is available via highlights.

Sometimes you want to stop processing before it’s fully done and in this case you can use stop() method. Beware that it won’t stop the processing immediatly, it will signal the session to stop on the next processing cycle. processing will be set to false when the processing is actually stopped.