Skia Frame Processors

Skia Frame Processors allow using @Shopify/react-native-skia to draw on a Frame in realtime.

Using Skia with VisionCamera

After installing @Shopify/react-native-skia and react-native-vision-camera-skia, you can use the <SkiaCamera /> view to quickly render a Camera feed using Skia:

import { SkiaCamera } from 'react-native-vision-camera-skia'

function App() {
  return (
    <SkiaCamera
      style={StyleSheet.absoluteFill}
      isActive={true}
      device="back"
      onFrame={(frame, render) => {
        'worklet'
        // ... custom Frame processing logic
        render(({ frameTexture, canvas }) => {
          // ... custom drawing operations
          canvas.drawImage(frameTexture, 0, 0)
        })
        frame.dispose()
      }}
    />
  )
}

In the onFrame(...) callback you receive two parameters:

• The frame (a Frame), which you can use for any kind of Frame Processing, like Face Detection.
• A render(...) function, which allows you to render the Frame (as an SkImage) to a Skia SkCanvas. The SkCanvas will be in the Frame's coordinate system, and will ultimately be rendered to the screen via the preview Skia <Canvas />.

Pixel Formats

Just like in a regular CameraFrameOutput, you can configure the <SkiaCamera />'s pixel format via pixelFormat. The pixel format affects your Skia Camera's performance, as well as the pixel format of the frame.

• Choose 'yuv' as a good default - Skia supports rendering 8-bit and 10-bit YUV buffers, and most native Frame Processing libraries support YUV.
• Choose 'rgb' as a fallback if 'yuv' is not supported by your native Frame Processor libraries, as 'rgb' requires conversion overhead.
• Choose 'native' only if you ensure your currently selected CameraFormat's nativePixelFormat is renderable by Skia. As of today, Skia supports 8-bit and 10-bit YUV formats, RGB formats, as well as 'private' on Android.

Resolution and FPS

The currently selected CameraFormat affects the Skia Camera's resolution and frame rate - to ensure maximum efficiency, avoid choosing a format with high resolution, and instead select a reasonable default:

const device = ...
const format = useCameraFormat(device, [
  { fps: 30 },
  { videoResolution: { width: 1080, height: 1920 } }
])

const device = ...
const format = getCameraFormat(device, [
  { fps: 30 },
  { videoResolution: { width: 1080, height: 1920 } }
])

Differences to a regular <Camera />

The <SkiaCamera /> is different than the <Camera /> in two important ways:

• A <SkiaCamera /> does not render to a PreviewView - instead it renders a Skia <Canvas /> and draws Frames manually, which you can control via render(...).
• A <SkiaCamera /> always has a CameraFrameOutput attached.

Rendering yourself

If you want to render Frames yourself, without <SkiaCamera />, use the NativeBuffer APIs and Skia's MakeImageFromNativeBuffer(...) API:

const frameOutput = useFrameOutput({
  pixelFormat: 'native',
  onFrame(frame) {
    'worklet'
    const surface = // create a Skia.Surface & cache it across onFrame(...)
    const nativeBuffer = frame.getNativeBuffer()
    const frameTexture = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)
    // render `frameTexture` to `surface`/canvas
    frameTexture.dispose()
    nativeBuffer.release()
    frame.dispose()
  }
})