Skip to content

VisionCamera Frame Processors run JavaScript on camera frames

updated

VisionCamera's documentation describes Frame Processors, functions written in JavaScript (or TypeScript) that process the frames the camera "sees". Inside them, developers can call Frame Processor Plugins, described as high performance native functions built for specific use-cases.

The docs use a QR code scanner as an example, written without native code:

function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    const qrCodes = scanQRCodes(frame)
    console.log(`Detected QR Codes: ${qrCodes}`)
  }, [])

  return (
    <Camera
      {...cameraProps}
      frameProcessor={frameProcessor}
    />
  )
}

Other listed use cases include AI for facial recognition and object detection, using Tensorflow, MLKit Vision or other libraries, realtime video-chats with WebRTC, snapchat-like face filters, and color filters with depth-detection. Custom code-scanner plugins, for formats such as Snapchat's SnapCodes or Apple's AppClips, are also mentioned.

Frame Processors require react-native-reanimated 2.1.0 or higher.

Interacting with Frame Processors

Because Frame Processors run in Worklets, they can read from and assign to Shared Values, for example to drive a Reanimated.View overlay via useAnimatedStyle. They can also call back to the React-JS thread using runOnJS, such as navigating to a product page when a QR code is detected.

Technical details

Frame Processors are workletized using react-native-reanimated. They are created on a custom camera thread with a separate JavaScript Runtime ("VisionCamera JS-Runtime") and invoked synchronously through JSI without going over the bridge. Frame Processor Plugins are native functions written in Objective-C, Swift, C++, Java or Kotlin that are injected into that runtime.

Using plugins

Plugins are distributed through npm. To install the QR code plugin:

npm i vision-camera-qrcode-scanner
cd ios && pod install

The plugin's global, __scanQRCodes, then goes into the globals option of the react-native-reanimated/plugin entry in babel.config.js.

metro-bundler must be restarted for babel.config.js changes to take effect.

Disabling Frame Processors

The Frame Processor API spawns a secondary JavaScript Runtime that uses a small amount of extra CPU and RAM. Projects not using it can set the VISION_CAMERA_DISABLE_FRAME_PROCESSORS flag in GCC_PREPROCESSOR_DEFINITIONS inside project.pbxproj, for both Debug and Release configurations:

GCC_PREPROCESSOR_DEFINITIONS = (
  "DEBUG=1",
  "VISION_CAMERA_DISABLE_FRAME_PROCESSORS=1",
  "$(inherited)",
);

Source: web.archive.org