ios – Imaginative and prescient CoreML Object Detection Full Display screen Panorama


How can I get my VNCoreMLRequest to detect objects showing wherever inside the fullscreen view?


I’m presently utilizing the Apple pattern venture for object recognition in breakfast meals:BreakfastFinder. The mannequin and recognition works properly, and usually provides the right bounding field (visible) of the objects it’s detecting / discovering.

The problem arises right here with altering the orientation of this detection.


In portrait mode, the default orientation for this venture, the mannequin identifies objects properly within the full bounds of the view. Naturally, given the properties of the SDK objects, rotating the digital camera causes poor efficiency and visible identification.

In panorama mode, the mannequin behaves surprisingly. The window / space of which the mannequin is detecting objects is not the total view. As a substitute, it’s (what looks like) the identical facet ratio of the telephone itself, however centered and in portrait mode. I’ve a screenshot beneath exhibiting roughly the place the mannequin stops detecting objects when in panorama:

enter image description here
enter image description here
enter image description here


The blue field with pink define is roughly the place the detection stops. It behaves surprisingly, however persistently doesn’t discover any objects exterior this approbate view / close to the left or proper edge. Nonetheless, the highest and backside edges close to the middle detect with none subject.

regionOfInterest

I’ve adjusted this to be the utmost: x: 0, y: 0, width: 1, top: 1. This made no distinction

imageCropAndScaleOption

That is the one setting that enables detection within the full display, nevertheless, the efficiency grew to become noticeably worse, and that is not likely an allowable con.


Is there a scale / measurement setting someplace on this course of that I’ve not set correctly? Or maybe a mode I’m not utilizing. Any assist could be most appreciated. Under is my detection controller:

ViewController.swift

// All unchanged from the obtain in Apples folder

" "
    
session.sessionPreset = .hd1920x1080 // Mannequin picture measurement is smaller.
... 
previewLayer.connection?.videoOrientation = .landscapeRight

" "

VisionObjectRecognitionViewController

@discardableResult
func setupVision() -> NSError? {
    // Setup Imaginative and prescient elements
    let error: NSError! = nil
    
    guard let modelURL = Bundle.important.url(forResource: "ObjectDetector", withExtension: "mlmodelc") else {
        return NSError(area: "VisionObjectRecognitionViewController", code: -1, userInfo: [NSLocalizedDescriptionKey: "Model file is missing"])
    }
    do {
        let visionModel = attempt VNCoreMLModel(for: MLModel(contentsOf: modelURL))
        let objectRecognition = VNCoreMLRequest(mannequin: visionModel, completionHandler: { (request, error) in
            DispatchQueue.important.async(execute: {
                // carry out all of the UI updates on the primary queue
                if let outcomes = request.outcomes {
                    self.drawVisionRequestResults(outcomes)
                }
            })
        })
        
        // These are the one properties that impression the detection space
        objectRecognition.regionOfInterest = CGRect(x: 0, y: 0, width: 1, top: 1)
        objectRecognition.imageCropAndScaleOption = VNImageCropAndScaleOption.scaleFit

        self.requests = [objectRecognition]

    } catch let error as NSError {
        print("Mannequin loading went incorrect: (error)")
    }
    
    return error
}

Leave a Reply