Skip to content

VisionCamera 3.0 released with an Android Camera2 rewrite

updated

VisionCamera 3.0.0 was released on 2023-09-01. This major version includes a full rewrite of the Android codebase and a large refactor on iOS. The author says the goal was to make the library more stable, more flexible and more performant. Limited support for V2 is planned on the v2 branch.

The VisionCamera example app
Image: margelo/react-native-vision-camera (MIT)

Android: CameraX to Camera2

Versions 1 and 2 used CameraX on Android. V3 now uses the lower-level Camera2 library. According to the release notes, this change brings:

  • working multi-cams ("fish-eye" zoom)
  • faster startup and better overall performance
  • correct videoWidth/videoHeight and photoWidth/photoHeight in device formats
  • 60 FPS cameras
  • greater stability
  • a simpler build setup that no longer depends on androidx.camera:+, which closed about 130 issues
  • faster photo capture

Some features do not work on Android yet: Flash, Orientation and Focus. Focus itself works, but its coordinate system is wrong until Orientation is supported.

GPU video pipeline

On Android, a custom OpenGL video pipeline written in C++ renders camera frames to the Frame Processor and Video Recorder output surfaces. Because only one output is attached to the Camera, the supportsParallelProcessing prop has been removed because parallel processing is now always supported, and flipping the Camera now works on Android.

New camera features

  • A pixelFormat prop for streaming rgb, yuv or native frames.
  • An enableShutterSound option for takePhoto().
  • New Frame properties: frame.toByteArray(), pixelFormat, orientation, isMirrored and timestamp.
  • New Camera Device properties: sensorOrientation and hardwareLevel.
  • Video recording in h264 or h265 (HEVC).
  • minFps/maxFps, which replace frameRateRanges.

Frame Processors

Frame Processors now use react-native-worklets-core instead of react-native-reanimated. This change has several effects:

  • JSI HostObjects and HostFunctions, such as react-native-fast-tflite models, can be used inside a Frame Processor.
  • The build setup is simpler.
  • Frame Processors are now fully synchronous.
  • Two new APIs are available: runAsync(..) and runAtTargetFps(.., fps).

Frame Processor Plugins are now object-oriented and accept options from JS:

const plugin = VisionCameraProxy.getFrameProcessorPlugin('example_plugin')

export function examplePlugin(frame: Frame) {
  'worklet'
  return plugin.call(frame)
}

Skia removed

Skia support, which allowed drawing onto Frames in realtime, was available up to V3 RC.9. It has been removed because maintaining two separate pipelines had become too complex. The details are in PR #1740.

The full changelog also lists RN 0.72 support, a minimum iOS version of 12.4, an enableZoomGesture implementation on Android, and many bug fixes.

Source: github.com