Control connected devices with the wave of a finger.

Overview

FUI

[Finger User Interface]

Control your devices with the wave of a finger.

Overview

Since 2009, coders have created thousands of experiments using Chrome, Android, AI, WebVR, AR and more. We showcase these projects and a variety of helpful tools and resources to inspire a diverse community of makers to explore, create, and share what’s possible with these technologies.

FUI (Finger User Interface) is a part of the TensorFlow Lite for Microcontroller Experiments, a collection of open source, interactive projects designed to demonstrate some fun ways to combine Arduino and TensorFlow Lite for Microcontrollers.

These projects were built with the Arduino Sense 33 BLE, TensorFlow Lite for Microcontrollers, standard web technologies ( HTML, CSS & Javascript ) and p5.js.


Experiment Description

Finger User Interface or FUI (pronounced Foo-ey) lets you control connected devices with the wave of a finger.

Other experiments to explore:


Tools


Install and Run

Flashing: Using the Arduino Nano Sense 33 BLE

  1. Install the Arduino IDE

  2. Setup Arduino board:

  • Plug in the board

  • Install the board by navigating to Tools > Board > Boards Manager and search for Arduino Mbed OS Nano Boards. Full instructions (including drivers required for Windows) here.

  • FAQ for connection problems can be found here.

  • After the board is installed, select it under to Tools > Board > Arduino Mbed OS Nano Boards > Arduino Nano 33 BLE Arduino board

  • Select the port by navigating to Tools -> Port -> dev/cu... (Arduino Nano 33 BLE) Arduino Port

  1. Install Arduino libraries
  • Navigate to Tools > Manage Libraries
  • Search for and install:
  • Arduino_LSM9DS1
  • ArduinoBLE
  • Arduino_TensorFlowLite Manage libraries TensorFlow Lite Library
  1. Open the sketch and flash
  • Download the latest relase of tf4micro motion kit here

  • Open the arduino/tf4micro-motion-kit and double click on file

  • Click the Right arrow in the top left corner to build and upload the sketch.
    Arduino Port

  • Warning: This process may take a few minutes. Also, warnings may populate but the upload should still succeed in spite of them.

  • If the sketch is installed, the LED on the board should flash red and green.

  1. Go to the URL related to the experiment. The URL can be found below and play!

Using the TensorFlow Microcontroller Kit by SparkFun

The board that comes with the TensorFlow Microcontroller Kit by SparkFun comes preflashed with a sketch that will work with some of the experiments right out of the box. If you are using one of the “TensorFlow Micro” kits and you just want to jump right into playing with the experiments then you can simply connect your arduino to a power source (USB or Battery) and connect to one of the following experiment URLs:


FAQ & Common Errors

What exactly is being transferred when I “connect”?
When you’re connecting the board to your computer, a pre-trained TensorFlow Lite machine learning model gets transferred over BLE onto the device. The sketches that are uploaded to the Arduino include a common TensorFlow Lite for Microcontrollers Experiments model architecture. The different experiment websites change the behavior of the sketch by changing the model to one specifically made for the experience.

What if I’m having issues connecting via bluetooth?
If you are having issues connecting try the following:

  1. Make sure your browser (Chrome or Edge) supports Bluetooth and it is enabled.
  2. Make sure your device (laptop, phone, etc) supports Bluetooth and that it is working and enabled.
  3. Refresh the web page, unplug the Arduino power cable and then plug it back in to reset, then try connecting again.

NOTE: If you’re using a managed device, like a computer from school or work, your device policy may prevent BLE pairing.

My board isn’t showing up on my computer, even though it’s plugged in. What should I do?
Try unplugging the Arduino power cable and then plug it back in to reset. Make sure you see the RGB LED blink red, green, blue in a sequential order.

The model isn’t getting my movements right. What do I do?
The way you move may be different from the data we used to pre-train the model. Different people move differently. That’s why we created Tiny Motion Trainer, which lets you train a custom model based on the way you move.

Do you have plans to support other boards?
We made these projects to work specifically with the Arduino Nano, and we currently don’t have plans to expand support. However, all of the code is open sourced, so you can remix or modify as needed.

Where should I go from here if I want to make my own model or project?
You can create your own model in several different ways. Check out these links:

"What sensors do the experiments use?"
The IMU is a LSM9DS1. It is a 3-axis accelerometer, 3-axis gyroscope and 3-axis magnetometer. This chip, made by ST Microelectronics, is a standard component supported by our library ArduinoLSM9DS1. Read more here: https://www.arduino.cc/en/Guide/NANO33BLESense

How do you shrink a TensorFlow model to fit on a microcontroller?
Post-training quantization is a conversion technique that can reduce model size while also improving CPU and hardware accelerator latency, with little degradation in model accuracy. You can quantize an already-trained float TensorFlow model when you convert it to TensorFlow Lite format using the TensorFlow Lite Converter. Read more here: https://www.tensorflow.org/lite/performance/post_training_quantization


Note

This is not an official Google product, but a collection of experiments that were developed at the Google Creative Lab. This is not a library or code repository that intends to evolve. Instead, it is a snapshot alluding to what’s possible at this moment in time.

We encourage open sourcing projects as a way of learning from each other. Please respect our and other creators’ rights, including copyright and trademark rights when present, when sharing these works and creating derivative work. If you want more info on Google's policy, you can find that here.


You might also like...
✔️The smallest header-only GUI library(4 KLOC) for all platforms
✔️The smallest header-only GUI library(4 KLOC) for all platforms

Welcome to GUI-lite The smallest header-only GUI library (4 KLOC) for all platforms. 中文 Lightweight ✂️ Small: 4,000+ lines of C++ code, zero dependenc

Graphical small-internet client for windows, linux, MacOS X and BSDs. Supports gemini, http, https, gopher, finger.
Graphical small-internet client for windows, linux, MacOS X and BSDs. Supports gemini, http, https, gopher, finger.

Graphical small-internet client for windows, linux, MacOS X and BSDs. Supports gemini, http, https, gopher, finger.

The program shows how bluetooth devices are connected, without using socket programming

The program shows how bluetooth devices are connected, without using socket programming, it shows how files are shared using principles of OOP

Identify I2C devices from a database of the most popular I2C sensors and other devices

I2C Detective Identify I2C devices from a database of the most popular I2C sensors and other devices. For more information see http://www.technoblogy.

Comparing the performance of Wave Digital Filter implementations

WDF Bakeoff Comparing performance between Wave Digital Filters implemented in C++ and Faust. Building First clone the repository and submodules: git c

If the button pressed esp will reset and App mode will on. App mode will on then led will on, network is connected led will off.

DHT22-to-Google-sheet-Reset-Using-ESP8266-LED-Switch If button pressed esp will reset and App mode will on. App mode will on then led will on, network

sc-ble-bridge is a utility that for every connected Steam Controller creates a virtual one acting as a bridge between SC and Steam

sc-ble-bridge The main goal of this utility is to provide workaround for steam-for-linux issue which makes Valve's Steam Controller unusable in BLE mo

Physical Tic-Tac-Toe smart board with PvP mode and two levels of AI. Built atop a custom PCB connected to an Arduino Mega 2560.

TicTacToe_SmartBoard The files in TicTacToe_SmartBoard are based on the files in https://wiki.illinois.edu/wiki/display/ECE110HLSF15/Tic-Tac-Toe+Smart

digital audio workstation, instrument and effect plugins, wave editor
digital audio workstation, instrument and effect plugins, wave editor

Stargate Stargate is a holistic audio production solution, everything you need to make music on a computer. To understand what Stargate is, and why, r

A C++ implementation of the graph algorithm of finding all strongly connected components with DFS

SinkSCC A C++ implementation of the graph algorithm of finding all strongly connected components with DFS. Details Each SCC of a graph can be treated

The HarmonicExciter is based on a half wave clipper, allow to add harmonics to the source by mix them in.
The HarmonicExciter is based on a half wave clipper, allow to add harmonics to the source by mix them in.

HarmonicExciter.lv2 The HarmonicExciter is based on a half wave clipper, allow to add harmonics to the source by mix them in. Dependencys libcairo2-de

Android example to get the rgb and disparity images from the OAK-D device connected to a phone.

depthai-android-jni-example (WIP) Android example to get the rgb and disparity images from the OAK-D device connected to a phone. DepthaiAndroidFast.m

Connected Santa Claus hat based on an ESP32, 8x8 matrices and LEDs stripped communicating over the internet with MQTT.
Connected Santa Claus hat based on an ESP32, 8x8 matrices and LEDs stripped communicating over the internet with MQTT.

Protobonnet What is it? A connected Santa Claus hat! What is it called? Le ProtoBonnet! Why? This hat has been created entirely for the Noël des proto

Wave Function Collapse library in C, plus a command-line tool
Wave Function Collapse library in C, plus a command-line tool

wfc Single-file Wave Function Collapse library in C, plus a command-line tool License: MIT Version: 0.01 This is an early version that supports the ov

Sandbox binary and source code for the Siggraph 2017 paper "Water Wave Packets" by Stefan Jeschke (NVIDIA) and Chris Wojtan (IST Austria)

----------------------------- Manual for wave packet viewer ----------------------------- System requirements: Windows8/8.1/10 with DirectX runtime e

A C++ port of Wave Function Collapse Tiling

Wave Function Collapse in C++ This is a C++ port of https://github.com/mxgmn/WaveFunctionCollapse. All sample images come from https://github.com/mxgm

Ramp is a HID attack program that steals all connected WiFi passwords within 13 seconds.
Ramp is a HID attack program that steals all connected WiFi passwords within 13 seconds.

Ramp Ramp is a HID attack program that steals all connected WiFi passwords within 13 seconds. Tested Windows 10 Warning Ramp has been created for the

Use the spline-interpolation method to a origin sine-wave data(40 MHz) captured by 250M sampling rate
Use the spline-interpolation method to a origin sine-wave data(40 MHz) captured by 250M sampling rate

Spline-Interpolation-Project Goals When we obtain some data that is discrete data, eg: from the the Analog-to-digital converter card, we need to resto

🐸 Coqui STT is an open source Speech-to-Text toolkit which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers

Coqui STT ( ?? STT) is an open-source deep-learning toolkit for training and deploying speech-to-text models. ?? STT is battle tested in both producti

Coqui.ai 1.5k Oct 4, 2022
Pure C ONNX runtime with zero dependancies for embedded devices

?? cONNXr C ONNX Runtime A onnx runtime written in pure C99 with zero dependencies focused on embedded devices. Run inference on your machine learning

Alvaro 135 Oct 3, 2022
A lightweight, portable pure C99 onnx inference engine for embedded devices with hardware acceleration support.

Libonnx A lightweight, portable pure C99 onnx inference engine for embedded devices with hardware acceleration support. Getting Started The library's

xboot.org 421 Sep 20, 2022
DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers.

Project DeepSpeech DeepSpeech is an open-source Speech-To-Text engine, using a model trained by machine learning techniques based on Baidu's Deep Spee

Mozilla 20.3k Sep 30, 2022
Real time eye tracking for embedded and mobile devices.

drishti Real time eye tracking for embedded and mobile devices in C++11. NEWS (2018/08/10) Native iOS, Android, and "desktop" variants of the real-tim

null 351 Sep 7, 2022
PocketSphinx is a lightweight speech recognition engine, specifically tuned for handheld and mobile devices, though it works equally well on the desktop

PocketSphinx 5prealpha This is PocketSphinx, one of Carnegie Mellon University's open source large vocabulary, speaker-independent continuous speech r

null 3.1k Oct 2, 2022
AI4Animation: Deep Learning, Character Animation, Control

This project explores the opportunities of deep learning for character animation and control as part of my Ph.D. research at the University of Edinburgh in the School of Informatics, supervised by Taku Komura. Over the last couple years, this project has become a modular and stable framework for data-driven character animation, including data processing, network training and runtime control, developed in Unity3D / Tensorflow / PyTorch.

Sebastian Starke 5.4k Oct 6, 2022
A project to control Petoi Bittle using human pose

Petoi Bittle Controlled by Pose A project to control Petoi Bittle by human pose Human pose is estimated using MoveNet and TensorFlow Lite YouTube Syst

iwatake 11 Dec 26, 2021
The official implementation of the research paper "DAG Amendment for Inverse Control of Parametric Shapes"

DAG Amendment for Inverse Control of Parametric Shapes This repository is the official Blender implementation of the paper "DAG Amendment for Inverse

Elie Michel 155 Sep 21, 2022
The Robotics Library (RL) is a self-contained C++ library for rigid body kinematics and dynamics, motion planning, and control.

Robotics Library The Robotics Library (RL) is a self-contained C++ library for rigid body kinematics and dynamics, motion planning, and control. It co

Robotics Library 616 Oct 2, 2022