Arduino Sketch and a Web Bluetooth API for loading models and running inference on the Nano Sense 33 BLE device.

Overview

TF4Micro Motion Kit

This repo contains the Arduino Sketch and a Web Bluetooth API for loading models and running inference on the device.

Install and Run on Arduino

Flashing: Using the Arduino Nano Sense 33 BLE

  1. Install the Arduino IDE

  2. Setup Arduino board:

  • Plug in the board

  • Install the board by navigating to Tools > Board > Boards Manager and search for Arduino Mbed OS Nano Boards. Full instructions (including drivers required for Windows) here.

  • FAQ for connection problems can be found here.

  • After the board is installed, select it under to Tools > Board > Arduino Mbed OS Nano Boards > Arduino Nano 33 BLE Arduino board

  • Select the port by navigating to Tools -> Port -> dev/cu... (Arduino Nano 33 BLE) Arduino Port

  1. Install Arduino libraries
  • Navigate to Tools > Manage Libraries
  • Search for and install:
  • Arduino_LSM9DS1
  • ArduinoBLE
  • Arduino_TensorFlowLite Manage libraries TensorFlow Lite Library
  1. Open the sketch and flash
  • Download the latest release here

  • Open the arduino/tf4micro-motion-kit and double click on <tf4micro-motion-kit.ino> file

  • Click the Right arrow in the top left corner to build and upload the sketch.
    Arduino Port

  • Warning: This process may take a few minutes. Also, warnings may populate but the upload should still succeed in spite of them.

  • If the sketch is installed, the LED on the board should flash red and green.

  1. Go to the URL related to the experiment. The URL can be found below and play!

Using the TensorFlow Lite for Microcontrollers Kit by SparkFun

The board that comes with the TensorFlow Lite for Microcontrollers Kit by SparkFun comes preflashed with a sketch that will work with some of the experiments right out of the box. If you are using one of the “TensorFlow Lite for Microcontrollers” kits and you just want to jump right into playing with the experiments then you can simply connect your arduino to a power source (USB or Battery) and connect to one of the following experiment URLs:


Web API Usage

Either use the button.js module to create a button that will connect and transfer your model togheter with your setting to the arduino over BLE.

Example:

import connectButton from "./tf4micro-motion-kit/web/button";

function handleGesture({index, value, velocity}){
   console.log(index, value, velocity);
}

function handleTransferProgress(progress){
  console.log(`Loaded ${Math.round(progress * 100)}%`);
}

connectButton(containerElement, {
   model: "./model.tflite",
   numClasses: 4,
   threshold: 0.169,
   numSamples: 10,
   captureDelay: 125,
   onGesture: handleGesture,
   onTransferProgress: handleTransferProgress,
   useMagnetometer: true
 }).addEventListener("click", () => {
   soundManager.userInit();
 });

or

Use the full api module (either by importing index.js or whatever functions you need from api.js).

Run Test Pages:

  1. Build and upload the Sketch in the arduino folder to your BLE 33 Nano Sense
  2. Run:
    • $ cd web
    • $ sh serv.sh

You need live-server, python2 or 3 installed

Then go to localhost:9090 in your browser and follow instructions


Note

This is not an official Google product, but a collection of experiments that were developed at the Google Creative Lab. This is not a library or code repository that intends to evolve. Instead, it is a snapshot alluding to what’s possible at this moment in time.

We encourage open sourcing projects as a way of learning from each other. Please respect our and other creators’ rights, including copyright and trademark rights when present, when sharing these works and creating derivative work. If you want more info on Google's policy, you can find that here.


Issues
  • Compilation errors: libmbed.a libcc_310_core.a libcc_310_ext.a libcc_310_trng.a not found

    Compilation errors: libmbed.a libcc_310_core.a libcc_310_ext.a libcc_310_trng.a not found

    Arduino IDE version: 1.8.12 (v2-beta also has the same issue). Board: Arduino Nano 33 BLE Sense Arduino Core: Arduino Mbed OS Nano Boards 2.1.0

    Tried the Arduino_TensorFlowLite library version 2.4.0-ALPHA and 2.1.1-ALPHA-precompiled and getting the same compilation errors:

    WARNING: library ArduinoBLE claims to run on samd, megaavr, mbed, apollo3 architecture(s) and may be incompatible with your current board which runs on mbed_nano architecture(s).
    arm-none-eabi-g++: error: ../../../../../../Users/carlos/Library/Arduino15/packages/arduino/hardware/mbed_nano/2.1.0/variants/ARDUINO_NANO33BLE/libs/libmbed.a: No such file or directory
    arm-none-eabi-g++: error: ../../../../../../Users/carlos/Library/Arduino15/packages/arduino/hardware/mbed_nano/2.1.0/variants/ARDUINO_NANO33BLE/libs/libcc_310_core.a: No such file or directory
    arm-none-eabi-g++: error: ../../../../../../Users/carlos/Library/Arduino15/packages/arduino/hardware/mbed_nano/2.1.0/variants/ARDUINO_NANO33BLE/libs/libcc_310_ext.a: No such file or directory
    arm-none-eabi-g++: error: ../../../../../../Users/carlos/Library/Arduino15/packages/arduino/hardware/mbed_nano/2.1.0/variants/ARDUINO_NANO33BLE/libs/libcc_310_trng.a: No such file or directory
    exit status 1
    Error compiling for board Arduino Nano 33 BLE.
    
    opened by carlosperate 2
  • fix: add package json to root

    fix: add package json to root

    This PR adds a package.json to fix install issues when using the git repo as a yarn / npm package source.

    The package.json uses the exports field which was introduced in 2020 to expose the /web folder.

    Should fix: https://github.com/googlecreativelab/astrowand/issues/1

    opened by rikard-io 1
  • Support Labels RxChar for local display

    Support Labels RxChar for local display

    First of all, thank you very much for this great and fun project.

    For context, I have ported the motion kit sketch to work with Adafruit nRF52 core to run on CLUE and Feather Sense boards. And it works great with lots of fun. CLUE has an builtin TFT, which can be very useful to display label on nrf52. However, currently the label is not available to devices (maybe I missed it, but I couldn't find them anywhere) and I need to "hardcorded" the label to use with "Air Snare" and "Finger User Interface" https://github.com/adafruit/tf4micro-motion-kit/blob/adafruit/arduino/adafruit/tf4micro-motion-kit/tf4micro-motion-kit.ino#L143

    It would be great to have an LabelRxChar so that server could pass it to nrf52 device. Something simple as comma separated string would be more than enough. Since the string can get to very long (with lost of labels) while characteristic length can only be up to 512. Therefore the detail of implementation is totally up to you.

    https://user-images.githubusercontent.com/249515/121933424-40ac5880-cd70-11eb-97ba-6b6d55b0425d.mov

    @ladyada

    PS: If you are ok with the comma separated char, I am happy to make an PR for it.

    opened by hathach 0
  • Update README.md

    Update README.md

    Fix Typos and some tiny grammer:

    Line 58: spelling "together" Line 87: remove period at end of sentence Line 95: edit "need live server" to "need a live server" Line 95: edit "and python2 or 3 installed" to "and either python 2 or 3 installed"

    opened by CallaJ 0
Releases(v005)
Owner
Google Creative Lab
Google Creative Lab
🪴💧 A Bluetooth Low Energy (BLE) soil moisture sensor.

b-parasite b-parasite is an open source Bluetooth Low Energy (BLE) soil moisture and ambient temperature/humidity sensor. Features Soil moisture senso

null 490 Jun 30, 2022
Bluetooth low energy (BLE) tracker for ESP32

BLEcker Bluetooth low energy (BLE) tracker for ESP32 This software is written for ESP32 boards to track BLE devices. It can be used for your smart hom

Vörös Ákos 39 Jun 23, 2022
ControllaBLE - A retro-controllers to Bluetooth BLE adapter

ControllaBLE - A retro-controllers to Bluetooth BLE adapter This is an ESP32 based controller adapter that outputs as a dual joypad through Bluetooth

null 13 Dec 1, 2021
Bluetooth Joystick : A wireless joystick with ESP-32 microcontroller and Dual Axis Joystick Module using the Bluetooth connectivity.

BluetoothJoystick Bluetooth Joystick : A wireless joystick with ESP-32 microcontroller and Dual Axis Joystick Module using the Bluetooth connectivity.

null 9 Feb 24, 2022
Exploits the Wii U's bluetooth stack to gain IOSU kernel access via bluetooth.

BluuBomb Exploits the Wii U's bluetooth stack to gain IOSU kernel access via bluetooth. For a more detailed write-up see WRITEUP.md. Not to be confuse

null 94 Jun 30, 2022
an implementation of the ansi c standard library that makes sense when ur stoned, tired, or otherwise inebriated

eli5 c stdlib an implementation of the ansi* c standard library that makes sense when ur stoned, tired, or otherwise inebriated * apparently this (htt

Ellen Körbes 7 Oct 13, 2021
VGG Runtime for loading design and running emulated apps.

VGG Runtime A design engine capable of loading design drafts as well as running design as an emulated app. Features Game-engine-like ECS architecture

Very Good Graphics 19 Jan 7, 2022
Gesture-Detecting-Macro-Keyboard - Glorified Bluetooth macro keyboard with machine learning (TensorFlow Lite for Microcontrollers) running on an ESP32.

Gesture detection tldr; Glorified Bluetooth macro keyboard with machine learning (TensorFlow Lite for Microcontrollers) running on an ESP32. Main feat

Jakob Krantz 63 Jun 19, 2022
Coqui Inference Engine is a library for efficiently deploying speech models.

Coqui Inference Engine Coqui Inference Engine is a library for efficiently deploying speech models. This project is at an early proof-of-concept stage

coqui 34 Feb 13, 2022
Otto-SetupAssist provides an Arduino sketch which assist you to build Otto robots.

Otto-SetupAssist Otto-SetupAssist provides an Arduino sketch which assist you to build Otto robots. This sketch provides two features: Move servos to

ROBOT.ICHIBA 1 Oct 20, 2021
Repository which holds the Arduino Sketch for the senseBox:bike

senseBox:bike Mobile Messstation mit der Temperatur, rel. Luftfeuchte, Feinstaub(PM10, PM25), Abstand nach links und rechts sowie die Bodenerschütteru

senseBox 2 Nov 22, 2021
A remote start arduino sketch, written for a Volkswagen Golf Gti MK4. Icons provided by Icons8

Introduction ?? This is an arduino sketch that enables the use of a sim-reader in order to remotely start a vehicle Description This Arduino sketch is

Sivert 1 Feb 18, 2022
WisBlock API takes care of all the LoRaWAN, BLE, AT command functionality

WisBlock-API Targeting low power consumption, this Arduino library for RAKwireless WisBlock Core modules takes care of all the LoRaWAN, BLE, AT comman

Bernd Giesecke 9 May 19, 2022
The ultimate battery tester with ESR measurement and discharge graph. Based on an Arduino Nano and a 1602 LCD.

Ultimate-Battery-Tester Version 1.0.0 Features Measures the ESR (equivalent series resistance) of the battery. This is an idicator of the health of th

Armin 6 Mar 28, 2022
Библиотека программного USB клавиатуры и мыши для Arduino Nano/UNO/Mega и прочих

EasyHID Библиотека программного USB клавиатуры и мыши для Arduino Nano/UNO/Mega и прочих Программный USB Работает на любой AVR Arduino 16 МГц Поддержк

Alex 42 Jun 25, 2022
BMI_automatics-faceshielding-device - this is a project utilizing STM32, MATLAB,Softmax classifier to achieve a portable face shielding(mask) device

BMI_automatics-faceshielding-device this is a project utilizing STM32, MATLAB,Softmax classifier to achieve a portable face shielding(mask) device in

Lyu Tian 2 Dec 23, 2021
Bluetooth LE Keyboard library for the ESP32 (Arduino IDE compatible)

ESP32 BLE Keyboard library This library allows you to make the ESP32 act as a Bluetooth Keyboard and control what it does. You might also be intereste

null 1.2k Jul 5, 2022
A sketch that not only parses NMEA sentences, but also allows sending UBX commands and decrypt answers from the ublox module

RAK4631-ublox-Commander This is a tokenizer and parser for raw NMEA sentences. This is not intended (yet anyway) for production, but as an exercice in

Kongduino 2 May 29, 2022
Sketch for the R2UppitySpinner PCB

R2 Uppity Spinner This is the sketch for the R2 Uppity Spinner. It depends on the Reeltwo library. You can download the latest release of the Reeltwo

ReelTwo 10 Mar 10, 2022