Orientation

A Camera has a fixed sensor orientation in which Frames are streamed in. If you rotate your phone, the Camera doesn't physically rotate alongside with it, so rotation has to be applied to the Frames dynamically - and each CameraOutput applies orientation differently.

Automatically set Orientation

In most cases, your Orientation should be automatically set to either the App's Interface Orientation, or your Phone's Device Orientation:

• App Interface Orientation ('interface'): Changes output orientation only when the UI rotates. If the UI is locked to portrait and the phone is held sideways, the output orientation will still stick to portrait ('up'). This is how apps like Snapchat or Instagram work.
• Phone Device Orientation ('device'): Changes output orientation when the phone physically rotates. If the UI is locked to portrait and the phone is held sideways, the output orientation will change to be sideways - even if the UI doesn't rotate to landscape. This is how most photography apps (including the stock iOS Camera app) work.

VisionCamera exposes two OrientationManagers - one for monitoring 'interface' orientation, and one for monitoring 'device' orientation:

function App() {
  const device = useCameraDevice('back')

  return (
    <Camera
      style={StyleSheet.absoluteFill}
      isActive={true}
      device={device}
      orientationSource="device"
    />
  )
}

function App() {
  const device = useCameraDevice('back')
  const camera = useCamera({
    isActive: true,
    device: device,
    orientationSource: 'device',
  })
}

const output = ...
const orientationManager = HybridCameraFactory.createOrientationManager('device')
orientationManager.startOrientationUpdates((newOrientation) => {
  output.outputOrientation = newOrientation
})
output.outputOrientation = orientationManager.currentOrientation

Manually set Orientation

For full manual control over orientation, set orientationSource to 'custom' (if you are using <Camera /> or useCamera(...)), and set a custom CameraOutput.outputOrientation for your outputs.

How Orientation is handled

Since Camera sensors have fixed orientations, rotation has to be applied to Frames dynamically. The Camera pipeline does not physically rotate buffers, as this is computationally expensive and would introduce latency.

Photo Orientation via EXIF tags

For captured Photos (see "The Photo Output"), Orientation is applied via EXIF tags.

Video Orientation via metadata

For recorded videos (see "The Video Output"), Orientation is applied via mp4/mov metadata.

Preview Orientation via View Transforms

For preview (see "The Preview Output"), Orientation is applied via view transforms.

Frame Output is manual

For streamed Frames (see "The Frame Output" or "The Depth Output"), Orientation is not applied automatically - instead, it is passed alongside as metadata, relative to what the CameraOutput's target outputOrientation is.

Inside a Frame Processor, you must interpret a Frame's orientation accordingly. Most Frame Processing libraries allow setting orientation and mirror settings as flags, for example, MLKit uses UIImageOrientation:

let frame = ...
let mlImage = MLImage(sampleBuffer: frame.sampleBuffer)
switch frame.orientation {
case .up:
  mlImage.orientation = frame.isMirrored ? .portrait : .portraitMirrored
case .down:
  // ...
}

Countering rotation (and mirroring)

For example, to get the Frame to its intended presentation for rendering it with a custom rendering library, counter-rotate the orientation accordingly:

• If Frame.orientation is 'up', you don't need to rotate anything - it is already in upright rotation.
• If Frame.orientation is 'left', it is rotated -90° left relative to the output's target orientation - you need to rotate it by +90° to get it back "upright".
• If Frame.orientation is 'right', it is rotated +90° right relative to the output's target orientation - you need to rotate it by -90° to get it back "upright".
• If Frame.orientation is 'down', it is rotated 180° upside down relative to the output's target orientation - you need to rotate it by 180° to get it back "upright".

This also applies to isMirrored:

• If Frame.isMirrored is false, you don't need to mirror anything - it is already properly mirrored. (either output is mirrored and frame is mirrored, or output is not mirrored and frame is also not mirrored)
• If Frame.isMirrored is true, it is mirrored respective to the output's target mirror mode - you need to mirror it to get its intended presentation. (either output is mirrored and frame is not mirrored, or output is not mirrored and frame is mirrored)