Run yolov5s on Android device!

Overview

yolov5s_android 🚀

The implementation of yolov5s on android for the yolov5s export contest.
Download the latest android apk from release and install your device.

Environment

  • Host Ubuntu18.04
  • Docker
    • Tensorflow 2.4.0
    • PyTorch 1.7.0
    • OpenVino 2021.3
  • Android App
    • Android Studio 4.2.1
    • minSdkVersion 28
    • targetSdkVersion 29
    • TfLite 2.4.0
  • Android Device
    • Xiaomi Mi11 (Storage 128GB/ RAM8GB)
    • OS MUI 12.5.8

We use docker container for host evaluation and model conversion.

git clone --recursive https://github.com/lp6m/yolov5s_android
cd yolov5s_android
docker build ./ -f ./docker/Dockerfile  -t yolov5s_android
docker run -it --gpus all -v `pwd`:/workspace yolov5s_anrdoid bash

Files

  • ./app
    • Android application.
    • To build application by yourself, copy ./tflite_model/*.tflite to app/tflite_yolov5_test/app/src/main/assets/, and build on Android Studio.
    • The app can perform inference with various configurations of input size, inference accuracy, and model accuracy.
    • For 'Open Directory Mode', save the detected bounding boxes results as a json file in coco format.
    • Realtime deteciton from camera image (precision and input size is fixed to int8/320). Achieved FPS is about 15FPS on Mi11.
    • NOTE Please select image/directory as an absolute path from 'Device'. The app does not support select image/directory from 'Recent' in some devices.
  • ./benchmark
  • ./convert_model
    • Model conversion guide and model quantization script.
  • ./docker
    • Dockerfile for the evaluation and model conversion environment.
  • ./host
    • detect.py : Run detection for image with TfLite model on host environment.
    • evaluate.py: Run evaluation with coco validation dataset and inference results.
  • ./tflite_model
    • Converted TfLite Model.

Performance

Latency

These results are measured on Xiaomi Mi11.
Please refer benchmark/README.md about the detail of benchmark command.
The latency does not contain the pre/post processing time and data transfer time.

float32 model

delegate 640x640 [ms] 320x320 [ms]
None (CPU) 249 61
NNAPI (qti-gpu, fp32) 156 112
NNAPI (qti-gpu, fp16) 92 79

int8 model

We tried to accelerate the inference process by using NNAPI (qti-dsp) and offload calculation to Hexagon DSP, but it doesn't work for now. Please see here in detail.

delegate 640x640 [ms] 320x320 [ms]
None (CPU) 95 23
NNAPI (qti-default) Not working Not working
NNAPI (qti-dsp) Not working Not working

Accuracy

Please refer host/README.md about the evaluation method.
We set conf_thresh=0.25 and iou_thresh=0.45 for nms parameter.

device, model, delegate 640x640 mAP 320x320 mAP
host GPU (Tflite + PyTorch, fp32) 27.8 26.6
host CPU (Tflite + PyTorch, int8) 26.6 25.5
NNAPI (qti-gpu, fp16) 28.5 26.8
CPU (int8) 27.2 25.8

Model conversion

This project focuses on obtaining a tflite model by model conversion from PyTorch original implementation, rather than doing own implementation in tflite.
We convert models in this way: PyTorch -> ONNX -> OpenVino -> TfLite.
To convert the model from OpenVino to TfLite, we use openvino2tensorflow. Please refer convert_model/README.md about the model conversion.

Comments
  • Custom model & dataset

    Custom model & dataset

    Thanks for your good work. I have a customized model at two detection scale trained with four class.. How do I modify the code to fit the application? Moreso, how to set up --tfds_root for quantize.py?

    opened by larrywal-express 7
  • Cannot deploy yolov5s.pt(v6.1) on android mobile phone

    Cannot deploy yolov5s.pt(v6.1) on android mobile phone

    I have convered yolov5s.pt to yolov5s_fp32_640.tflite, but I can't deploy it on android mobile phone. The version of yolov5s.pt is 6.1.

    step1: python export.py --weights weights/yolov5s.pt --include onnx --simplify (the image_size of export.py is [640,640]). step2: python3 /opt/intel/openvino_2021.4.752/deployment_tools/model_optimizer/mo.py --input_model yolov5s.onnx --input_shape [1,3,640,640] --output_dir ./openvino --data_type FP32 --output Conv_198,Conv_248,Conv_298 step3: openvino2tensorflow --model_path ./openvino/yolov5s.xml --model_output_path tflite --output_pb --output_saved_model --output_no_quant_float32_tflite step4: rename file as "yolov5s_fp32_640.tflite", and put it into android/app/src/main/assets

    but it can't run. “Cannot copy from a TensorFlowLite tensor (Identity) with shape [1,20,20,255] to a java object with shape [1,80,80,255]”

    opened by qiujianchen 4
  • Run inference in GUI gives me `Inference failed`

    Run inference in GUI gives me `Inference failed`

    I managed to build my app and import a model but when I press RUN INFERENCE I get:

    Inference failed : Cannot copy from a TensorFlowLite tensor (StatufulPartitionedCall:2) with shape [1, 20, 20, 51] to a Java object with shape [1, 40 , 40, 51]
    
    opened by mikel-brostrom 3
  • node wrong

    node wrong

    opened by ppogg 3
  • RuntimeError: Max and min for dynamic tensors should be recorded during calibration: Failed for tensor inputs Empty min/max for tensor inputs

    RuntimeError: Max and min for dynamic tensors should be recorded during calibration: Failed for tensor inputs Empty min/max for tensor inputs

    I run command in colab: !python3 quantize.py --image_dir /contnet/dataset/coco128/coco128/images/train2017/ --calib_num 93 --input_size 640

    the folder contains *.jpg & *.txt

    opened by yspisgod 2
  • How can I adjust the output of my TFLite exported model in order to make it work with your app

    How can I adjust the output of my TFLite exported model in order to make it work with your app

    Hi!

    I tried my own tflite model trained with yolov5 on your app, but get the flow error:

    Cannot copy from a TensorFlowLite tensor (StatefulPartitionedCall:0) with shape [1, 6300, 7] to a Java object with shape [1, 40, 40, 255].

    your models have 4 outputs, but my models only have 3 outputs, what should i do to make my model work.

    Thanks!

    opened by ppl520zfl 1
  • no node named Conv_253

    no node named Conv_253

    when i run python3 /opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo.py --input_model yolov5s.onnx --input_shape [1,3,640,640] --output_dir ./openvino --data_type FP32 --output Conv_253,Conv_302,Conv_351

    it seems that "-output" is unnecessary

    opened by yspisgod 1
  • TfliteRunner custom context

    TfliteRunner custom context

    How to change TfliteRunner for working with custom Activity in manifest. I'am using MAppication class for starting app. MainActivity open in MApplication image

    opened by alexmihalyk23 1
  • When the screen is horizontal, the detection box is inaccurate

    When the screen is horizontal, the detection box is inaccurate

    I tried to run this project on the Android tablet, and found that the default horizontal screen was inaccurate. Then I switched back to the phone and rotated it to the horizontal screen, and the same inaccuracy occurred. I wonder if there is any good way to support horizontal screen detection

    opened by Chopin68 1
  • Higher confidence but smaller bbox size

    Higher confidence but smaller bbox size

    I have an interesting inference output. The image below is using conf=0.84 and IoU=0.30 image First, the bboxes are smaller than the objects. Second, somehow when I have more than one bbox to the same object as shown in the image for bbox yellow and blue, note that in the yellow case, the higher prediction 93.82 the bbox is smaller than the 89.70 that has a bigger bbox. The same happens with blue case where the higher prediction 92.96 the bbox is smaller than the 91.48 has a bigger bbox. If I decrease IoU or increase the confidence, I will get only one bbox for yellow and blue with higher conf but the ones removed had a better bbox size when compared with their objects. Why is the algorithm assigning lower conf to bboxes with a larger size (near the object size) and higher conf to bboxes with smaller sizes? Any hint on this?

    opened by zenetio 1
Releases(v1.0.2)
Owner
null
Detailed listing of multimedia codecs on your Android device - with no ads!

Codec Info Detailed listing of multimedia codecs on your Android device - with no ads! Codec Info is a simple tool that provides detailed listing of m

Krzysztof Nawrot 46 Dec 18, 2022
On-device wake word detection powered by deep learning.

Porcupine Made in Vancouver, Canada by Picovoice Porcupine is a highly-accurate and lightweight wake word engine. It enables building always-listening

Picovoice 2.8k Jan 5, 2023
On device extracting images from videos - creating video from images.

AndroidVideoTranscoder Surprisingly fast on device video transcoding. Features extracting images from video either ffmpeg or mediacodec creating video

null 29 Dec 20, 2022
:sound: [Android Library] Easily generate pure audio tone of any frequency in android

Android library to easily generate audio tone in android. Generating pure tone of an specific frequency was never that easy. ZenTone does all the heav

Nishant Srivastava 102 Dec 19, 2022
mpv-android is a video player for Android based on libmpv.

mpv-android is a video player for Android based on libmpv.

null 1.1k Jan 6, 2023
FFmpeg compiled for Android. Execute FFmpeg commands with ease in your Android app.

FFMPEG video operations FFmpeg compiled for Android. Execute FFmpeg commands with ease in your Android app. Getting Started This project is provide in

Simform Solutions 277 Jan 2, 2023
Echo is a lightweight and minimal music player for Android, built with Android Studio and written in Kotlin

Echo - Echo, A light-weight, minimal music player for Android, with shuffle, favorites and audio visualization

Divins Mathew 0 Feb 7, 2022
Youtube Android Clone 🚀an Android Youtube Clone made out of XML and Kotlin

Youtube Android Clone ?? This app consumes The Youtube Api to fetch and display a list of popular videos, The app uses MVVM design pattern to allow se

Breens Robert 38 Dec 13, 2022
An extensible media player for Android

ExoPlayer ExoPlayer is an application level media player for Android. It provides an alternative to Android’s MediaPlayer API for playing audio and vi

Google 20.2k Jan 1, 2023
Android/iOS video player based on FFmpeg n3.4, with MediaCodec, VideoToolbox support.

ijkplayer Platform Build Status Android iOS Video player based on ffplay Download Android: Gradle # required allprojects { repositories {

bilibili 31k Jan 3, 2023
Custom Android view with video player, loader and placeholder image

VideoPlayerView Custom Android view with video player, loader and placeholder image. To stay up-to-date with news about the library Usage Here is an e

Marcin Moskała 89 Nov 18, 2022
[] FFmpeg build for android random architectures with example jni

AndroidFFmpegLibrary This project aims to create working library providing playing video files in android via ffmpeg libraries. With some effort and N

AppUnite Sp. z o.o. Spk. 1.1k Dec 27, 2022
Emotion recognition by speech in android.

Vokaturi - Android Library Android port of the Vokaturi emotion recognition API. Overview Vokaturi is an emotion recognition software, that can unders

Owais 82 Nov 11, 2022
Free p2p cdn android github sdk to reduce video streaming costs of live and on demand video using webrtc by upto 90% and improve scalability by 6x - 🚀 Vadootv 🚀

Android p2p cdn sdk to distribute load and reduce costs(https://peervadoo.com) Vadootv is a p2p sdk integration to reduce your video streaming costs b

Vadootv 40 Oct 5, 2022
A better Android VideoView with more Media Controller customization. 一个更好用的Android VideoView

Android UniversalVideoView 中文版说明请点击这里 UniversalVideoView is a Android widget helps playing video easier, which is similar with the Android system nati

Linsea 978 Nov 30, 2022
Android Texture VideoView having a variety of scale types like the scale types of ImageView such as fitCenter, centerCrop, centerTopCrop and more

Android-ScalableVideoView Looking for the extra scale types of ImageView? Check out ScalableImageView. Android Texture VideoView having a variety of s

Yoshihito Ikeda 1.1k Jan 7, 2023
Android Video Crop

?? Before using this library, read information below ?? This library is not more supported. If you want to add new feature or fix a bug, grab source

Dmytro Danylyk 390 Jan 2, 2023
Android music player example.

Android music player example.

Chien 21 Jul 29, 2022