An app to detect color palettes in the real world - powered by VisionCamera

Overview

Colorwaver

An app to detect colorwaves (swatches/palettes) in the real world - powered by VisionCamera and Reanimated.

I wrote this app in less than a day, a speed simply not matched by native app development. Because it's written in React Native (TypeScript/JavaScript), it runs both on iOS and Android, but performance critical parts (e.g. the image processing algorithm or the animations) are backed by native Objective-C/Swift/Java code.

See this Tweet for more information.

Buy Me a Coffee at ko-fi.com

Try it!

Download the repository and run the following commands to try Colorwaver for yourself:

iOS

yarn try-ios

Android

yarn try-android

Project structure

This is a bare React Native project, created with create-react-native-app.


  • 📁 src: Contains the actual TypeScript + React (-Native) front-end for the Colorwaver App.
    • 📄 src/getColorPalette.ts: Exposes the native iOS/Android frame processor plugin as a JS function with TypeScript types. This function has to be called from a frame processor. ('worklet')

    • 📄 src/Router.tsx: The main component that gets registered by index.js. This acts as a main navigator, navigating either to the Splash Page (Permissions) or the App Page (Main App) depending on whether the user has granted permissions or not.

    • 📄 src/Splash.tsx: The first Splash page to ask the user for Camera permission.

    • 📄 src/App.tsx: Contains the actual source code for the App's front-end.

      • VisionCamera is used to get a Camera device and display a Camera component. Also, a frame processor (a function that gets called for every frame the camera "sees") gets attached, which calls the native iOS/Android frame processor plugin.
      • Reanimated is used to smoothly animate between color changes.

      Because VisionCamera also uses the Worklet API, the entire process between receiving a camera frame and actually displaying the palette's colors does not use the React-JS Thread at all. The frame processing runs on a separate Thread from VisionCamera, which then dispatches the animations on the Reanimated UI Thread. This is why the App runs as smooth as a native iOS or Android app.

    • 📄 src/useAnimatedColor.ts: A helper function to animate color changes with SharedValues.


  • 📁 ios: Contains the basic skeleton for a React Native iOS app, plus the native getColorPalette(...) Frame Processor Plugin.
    • 📄 ios/PaletteFrameProcessorPlugin.m: Declares the Swift frame processor plugin "getColorPalette(...)".

    • 📄 ios/PaletteFrameProcessorPlugin.swift: Contains the actual Swift code for the native iOS frame processor plugin "getColorPalette(...)".

      This uses the CoreImage API to convert the CMSampleBuffer to a UIImage, and then uses the UIImageColors library to build the color palette.

      VisionCamera's frame processor API is used to expose this function as a frame processor plugin.

    • 📄 ios/Colorwaver-Bridging-Header.h: A Bridging Header to import Objective-C headers into Swift.

    • 📄 ios/Podfile: Adds the UIImageColors library.

    • 📄 ios/UIColor+hexString.swift: An extension for UIColor to convert UIColor instances to strings. This is required because React Native handles colors as strings.


  • 📁 android: Contains the basic skeleton for a React Native Android app, plus the native getColorPalette(...) Frame Processor Plugin.
    • 📄 android/app/build.gradle: The gradle build file for the Android project. The following third-party dependencies are installed:

      • androidx.camera: camera-core
      • androidx.palette: palette
    • 📁 android/app/src/main/java/com/colorwaver/utils: Contains two files copied from android/camera-samples to convert ImageProxy instances to Bitmaps. (YuvToRgbConverter.kt)

    • 📁 android/app/src/main/java/com/colorwaver: Contains the actual Java source code of the Project.

    • 📄 android/app/src/main/java/com/colorwaver/MainApplication.java: Sets up react-native-reanimated.

    • 📄 android/app/src/main/java/com/colorwaver/MainActivity.java: Installs the PaletteFrameProcessorPlugin frame processor plugin inside of the onCreate method.

    • 📄 android/app/src/main/java/com/colorwaver/PaletteFrameProcessorPlugin.java: Contains the actual Java code for the native Android frame processor plugin "getColorPalette(...)".

      This uses the YuvToRgbConverter to convert the ImageProxy to a Bitmap, and then passes that to the Palette API from AndroidX to build the color palette.

      VisionCamera's frame processor API is used to expose this function as a frame processor plugin.


  • 📄 babel.config.js: Adds the native frame processor plugin getColorPalette (called __getColorPalette) to Reanimated's global list.
Comments
  • Errors - while running the react-native run-android

    Errors - while running the react-native run-android

    Hi @mrousavy ,

    This is truly amazing work!!

    I was trying the instructions to run it for my android device, on a windows machine, got the below error:

    Was hoping if you too had any similar issue:

    FAILURE: Build failed with an exception.

    • Where: Build file 'C:\Usersxxxxxvision_camera\Colorwaver\android\app\build.gradle' line: 162

    • What went wrong: A problem occurred evaluating project ':app'.

    Could not get unknown property 'release' for SigningConfig container of type org.gradle.api.internal.FactoryNamedDomainObjectContainer.

    • Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

    • Get more help at https://help.gradle.org

    BUILD FAILED in 2s

    error Failed to install the app. Make sure you have the Android development environment set up: https://reactnative.dev/docs/environment-setup. Error: Command failed: gradlew.bat app:installDebug -PreactNativeDevServerPort=8081

    FAILURE: Build failed with an exception.

    • Where: Build file 'C:\Users\xxxxx\Colorwaver\android\app\build.gradle' line: 162

    • What went wrong: A problem occurred evaluating project ':app'.

    Could not get unknown property 'release' for SigningConfig container of type org.gradle.api.internal.FactoryNamedDomainObjectContainer.

    • Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

    • Get more help at https://help.gradle.org

    opened by vineethdas 7
  • Gradle-build error while try-android

    Gradle-build error while try-android

    image When I run react-native run-android this error pops up. My phone is connected through adb.

    This is an amazing app and I would love to try it. Plz help me.

    opened by datnerdguy 1
  • perf: Use Hermes

    perf: Use Hermes

    Hermes is way faster than JSC and more up to date.

    • Faster JSI calls because Hermes does not do as many C++ thread-lock-operations as JSC
      • Means: Faster Frame Processor <-> FP Plugin calls, so faster color palette detection
      • Means: Faster UI Worklet <-> C++ calls, so faster and more efficient animations
    • Even faster app startup time because Hermes uses bytecode and is optimized for fast app startup
    • Overall faster JS execution
    • More up to date as Hermes will become the default JS Engine for React Native soon.

    Blockers

    • [x] iOS doesn't build for me after enabling Hermes (<folly/folly-config.h> file not found or boost/operators.hpp file not found C++ errors, rising from either RNReanimated or VisionCamera.) (FIXED)
    • [ ] ShareableValue::adapt causes a SIGABRT crash on Hermes (Tried to access a value backed by a different Runtime)
    • [x] Hermes' garbage collector (Hades GC) runs on a separate Thread, so the jsi::HostObject destructor is called on a Thread that might not have a JNI Environment attached. This crashes on Android after a few seconds of running the Frame Processor. (FIXED)
    opened by mrousavy 0
  • refactor, fix iOS build, introduce ColorTile

    refactor, fix iOS build, introduce ColorTile

    This PR focuses mainly on the code split and cleanup, further PRs will include UI changes and new features.

    Summary

    • refactor and simplification of the code
    • ColorTile component added
    • Reanimated.TextInput "hax" removed in favor of react-native-animateable-text
    • temporarily disabled Hermes on iOS to fix build issues (Folly header)

    Preview/PoC that app is running

    IMG_7042

    opened by Simek 0
  • [ImgBot] Optimize images

    [ImgBot] Optimize images

    Beep boop. Your images are optimized!

    Your image file size has been reduced by 22% 🎉

    Details

    | File | Before | After | Percent reduction | |:--|:--|:--|:--| | /img/demo_1.png | 241.05kb | 113.46kb | 52.93% | | /src/assets/icon.png | 10.91kb | 5.46kb | 49.91% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/1024.png | 103.21kb | 64.51kb | 37.50% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/180.png | 8.59kb | 5.97kb | 30.47% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/167.png | 8.00kb | 5.66kb | 29.27% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/152.png | 7.13kb | 5.08kb | 28.74% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/144.png | 6.69kb | 4.83kb | 27.83% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/114.png | 5.15kb | 3.92kb | 23.95% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/120.png | 5.37kb | 4.10kb | 23.65% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/100.png | 4.43kb | 3.51kb | 20.80% | | /img/demo_2.png | 6,776.71kb | 5,386.97kb | 20.51% | | /android/app/src/main/ic_launcher-playstore.png | 19.67kb | 15.79kb | 19.71% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/87.png | 3.80kb | 3.10kb | 18.32% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/80.png | 3.48kb | 2.89kb | 16.98% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/76.png | 3.30kb | 2.78kb | 15.86% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/72.png | 3.14kb | 2.66kb | 15.43% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/60.png | 2.56kb | 2.27kb | 11.58% | | /android/app/src/main/res/mipmap-xxxhdpi/ic_launcher_foreground.png | 7.47kb | 6.78kb | 9.24% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/58.png | 2.47kb | 2.25kb | 9.20% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/57.png | 2.44kb | 2.22kb | 8.89% | | /android/app/src/main/res/mipmap-xhdpi/ic_launcher_foreground.png | 3.17kb | 2.91kb | 8.29% | | /android/app/src/main/res/mipmap-xxhdpi/ic_launcher_foreground.png | 5.06kb | 4.66kb | 7.92% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/50.png | 2.15kb | 1.99kb | 7.23% | | /android/app/src/main/res/mipmap-hdpi/ic_launcher_foreground.png | 2.24kb | 2.09kb | 6.75% | | /ios/Colorwaver/Images.xcassets/AppIcon.appiconset/40.png | 1.70kb | 1.66kb | 2.13% | | | | | | | Total : | 7,239.91kb | 5,657.52kb | 21.86% |


    Black Lives Matter | 💰 donate | 🎓 learn | ✍🏾 sign

    📝 docs | :octocat: repo | 🙋🏾 issues | 🏅 swag | 🏪 marketplace

    opened by imgbot[bot] 0
  • great job, plz provide more example

    great job, plz provide more example

    Hi, i just read your comment from this topic: https://github.com/react-native-camera/react-native-camera/issues/2942#issuecomment-1004122038.

    I think it's really good, I want to make some function like add effects into face while webrtc calling, can u provide some example? Thanks so much

    opened by fukemy 0
  • Error when generating using npx react-native run-android --variant=release

    Error when generating using npx react-native run-android --variant=release

    I get this error when trying to compile using the command: npx react-native run-android --variant=release

    In file included from ../../../../src/main/cpp/JSIJNIConversion.cpp:25: ../../../../src/main/cpp/java-bindings/JHashMap.h:16:8: error: redefinition of 'JHashMap' struct JHashMap : JavaClass<JHashMap<K, V>, JMap<K, V>> { ^ ../../../../build/fbjni-0.3.0-headers.jar\fbjni/detail/Iterator.h:170:8: note: previous definition is here struct JHashMap : JavaClass<JHashMap<K, V>, JMap<K, V>> { ^ 1 error generated.

    opened by ssaguiar 2
  • "Tried to clone a value backed by a different Runtime" on Hermes/iOS

    When enabling Hermes on iOS, ShareableValue::adapt crashes with the error

    Tried to clone a value backed by a different Runtime
    

    This is a SIGABRT and unfortunately really hard to debug. So I have no idea what's causing this.

    opened by mrousavy 1
  • "Attempting to assign to readonly property" when using color interpolation

    My animated color hook:

    export function useAnimatedColor(
      color: Reanimated.SharedValue<string>,
    ): Readonly<Reanimated.SharedValue<string | number>> {
      const animation = useSharedValue(0);
      const colorFrom = useSharedValue(DEFAULT_COLOR);
      const colorTo = useSharedValue(color.value);
    
      useAnimatedReaction(
        () => color.value,
        (newColor, prevColor) => {
          animation.value = 0;
          colorFrom.value = prevColor ?? DEFAULT_COLOR;
          colorTo.value = newColor;
          animation.value = withTiming(1, {
            duration: 150,
            easing: Easing.linear,
          });
          console.log(`Animating from ${prevColor} -> ${newColor}`);
        },
      );
    
      // TODO: Using colorFrom and colorTo in here raises "Attempting to assign to readonly property" error...
      return useDerivedValue(() =>
        interpolateColor(animation.value, [0, 1], [colorFrom.value, colorTo.value]),
      );
    }
    

    is causing this error:

    See https://github.com/software-mansion/react-native-reanimated/issues/2329 for more details.

    Right now, I've hotfixed this by using a custom patch for react-native-reanimated that completely removes the RGB cache. This now runs correctly, but ideally I want to fix this and make use of the cache for performance reasons.

    opened by mrousavy 0
  • perf: Run `getColors` algorithm directly on `CVPixelBuffer` instead of costly `UIImage` conversion

    perf: Run `getColors` algorithm directly on `CVPixelBuffer` instead of costly `UIImage` conversion

    Creates a custom implementation of getColors to get the colors directly on the CVPixelBuffer we already stream, instead of converting CMSampleBuffer -> CVPixelBuffer -> CGImage -> UIImage every time.

    This should make the Frame Processor Plugin a lot faster.

    opened by mrousavy 0
  • perf: Choose more efficient Camera Format, Use optical Video Stabilization

    perf: Choose more efficient Camera Format, Use optical Video Stabilization

    • Use optical/digital Video Stabilization Mode to minimize color flickering/rapid changes - depending on what's available.
    • Use format that more closely matches the screen size instead of just choosing the highest one (optimizes performance)
    opened by mrousavy 0
Owner
Marc Rousavy
they call me ranch cause I be dressing
Marc Rousavy
Utility Android app for generating color palettes of images using the Palette library. Written in Kotlin.

Palette Helper is a simple utility app made to generate color palettes of images using Google's fantastic Palette library. It's mostly a for-fun pet p

Zac Sweers 154 Nov 18, 2022
Utility Android app for generating color palettes of images using the Palette library. Written in Kotlin.

Palette Helper is a simple utility app made to generate color palettes of images using Google's fantastic Palette library. It's mostly a for-fun pet p

Zac Sweers 154 Nov 18, 2022
Clone of real world Chatting application Whatsapp built on Android Studio and Firebase

Clone of real world Chatting application Whatsapp built on Android Studio and Firebase

Aditya Bonde 11 May 23, 2022
An Application made with Android Studio that utilized National Renewable Energy Laboratory (PV Watts) API to detect nearby utility companies and electricity rates.

Electric App Electric App is a mobile application made on Android Studio that utilized the National Renewable Energy Laboratory (PV Watts) API to dete

null 1 Dec 6, 2021
Math World is an Android Application specialized in mathematics, where the application includes some sections related to arithmetic, unit conversion, scientific math laws and constants, as well as some mathematical questions that need some intelligence to reach the solution.

Math World is an Android Application specialized in mathematics, where the application includes some sections related to arithmetic, unit conversion, scientific math laws and constants, as well as some mathematical questions that need some intelligence to reach the solution.

null 7 Mar 12, 2022
Event Bus powered by Kotlin Coroutines Flow

FlowBus FlowBus is a kotlin event bus implementation. Powered by Kotlin Coroutines and Flows Android MainThread-aware Fully operable from Java code Ex

Robert Kosakowski 26 Dec 27, 2022
LedBuzz is a NodeMCU-powered see-through propeller display

LedBuzz What it does LedBuzz is a NodeMCU-powered see-through propeller display. By simply staring at your ceiling fan, you receive notification alert

Jatin 2 Jan 9, 2022
Picker-kt - Media picker library powered by Jetpack Compose

ANDROID LIBRARY PickerKT A media picker library for Android apps powered by Jetp

Tanawin Wichit 20 Jan 2, 2023
Get a libGDX-powered Android live wallpaper up and running quickly with this project template

This project is a libGDX Android live wallpaper template. It's the demo application created by the libGDX code generator, remade as a live wallpaper.

Kurtis LoVerde 1 Jun 30, 2022
This android app fetches the data from the USGS API in real time to display a list of earthquakes.

This android app fetches the data from the USGS API in real time to display a list of earthquakes. On clicking an earthquake it opens a browser window with the complete information of the earthquake along with the location on a map.

null 5 Jan 11, 2022
A real time notification App which reminds its user with daily schedules

On Time Pro ⌚ A real time notification App which reminds its user with daily schedules, time-table ⌛ , due assignments ?? , regular classes with just

Elevate Lab 3 Feb 1, 2022
Ipify allows users to get current public IP address for connected network in real-time

Ipify-Android Ipify allows you to get current public IP address when connected to internet in real-time Add Dependency Use Gradle: Step 1: Add it in y

null 9 Nov 21, 2022
Real time gps location based alarming system tracking the road accident and providing medical assitance plus concern from near by police station.

Real time gps location based alarming system tracking the road accident and providing medical assitance plus concern from near by police station.

Sandeep Verma 1 Mar 12, 2022
Communicating between Wear OS and Android device using the OpWear module and a sample of displaying real-time camera on the watch and sending commands to the mobile by Wear OS.

OpWear-Cam Communicating between Wear OS and Android device using the OpWear module and a sample of displaying real-time camera on the watch and sendi

AmirHosseinAghajari 6 Nov 8, 2022
A small API to add Quran with Tajweed Color in Android App

QuranApi A simple Api to implement Quran in Android. Add To Project Step 1. Add the JitPack repository to your build file Add it in your root build.gr

DevilCat 11 Aug 5, 2022
QuatroGrade is a beautiful multi color (4 corners) gradient.

| | Setup Guide | Report new issue QuatroGrade QuatroGrade is a beautiful multi color gradient. Basically, it is a 4 vertex gradient. Features: Multi

Basil Miller 33 Oct 9, 2022
Theme for VSCode and JetBrains IDEs, based on morhetz/gruvbox but with a plainer color palette for syntax highlighting.

gruvbox-plain Theme for VSCode and JetBrains IDEs, based on morhetz/gruvbox but with a plainer color palette for syntax highlighting. Syntax Colors gr

null 2 Dec 28, 2022
Food Recipe App is an app for collecting most of food recipe in one app

Food Recipe App is an app for collecting most of food recipe in one app

Heba Elsaid 10 Dec 25, 2022