Compressed Delta OTA Updates for ESP32

Overview

Status

ESP32 Compressed Delta OTA Updates

About the Project

The project aims at enabling firmware update of ESP32 Over-the-Air with compressed delta binaries. Testing was done with ESP32-DevKitC v4 board.

Getting Started

Hardware Required

To run the OTA demo, you need an ESP32 dev board (e.g. ESP32-WROVER Kit) or ESP32 core board (e.g. ESP32-DevKitC). You can also try running on ESP32-S2 or ESP32-C3 dev boards and let me know how it worked out.

Prerequisites

  • ESP-IDF v4.3 and above

    You can visit the ESP-IDF Programmming Guide for the installation steps.

  • detools v0.49.0 and above

    Binary delta encoding in Python 3.6+. You can follow the instructions here for installation.

  • Partition Tool (parttool.py)

    parttool.py comes pre-installed with ESP-IDF; it can be used after the ESP-IDF python venv is initialised - See here.

Usage

  1. Build the example examples/http_delta_ota and flash the partition table partitions.csv.

    idf.py build && idf.py partition_table-flash

  2. To generate the patch, we need 2 application binaries, namely base_binary and updated_binary.

    detools create_patch -c heatshrink base_binary.bin updated_binary.bin patch.bin

  3. Currently, patch application demo is supported only for the ota_0 partition. So, we need to flash the base binary to the ota_0 partition.

    parttool.py --port "/dev/ttyUSB0" --baud 2000000 write_partition --partition-name=ota_0 --input="base_binary.bin"

  4. Open the project configuration menu (idf.py menuconfig) go to Example Connection Configuration ->

    1. WiFi SSID: WiFi network to which your PC is also connected to.
    2. WiFi Password: WiFi password
  5. In order to test the OTA demo -> examples/http_delta_ota :

    1. Flash the firmware idf.py -p PORT -b BAUD flash
    2. Run idf.py -p PORT monitor and note down the IP assigned to your ESP module. The default port is 80.
  6. After getting the IP address, send the patch binary through a HTTP Post request over cURL.

    curl -v -X POST --data-binary @- < patch.bin 192.168.224.196:80/ota

Demo Results

  • Base binary: assets/nvs_rw_value.bin (Example Link)
  • Updated binary: assets/nvs_rw_blob.bin (Example Link)
  • Patch binary: assets/patch_nvs.bin
  • After successfully patching and rebooting, press the EN button on your dev board to reboot it and increment the Restart counter variable.

Experiments

Chip Scenario Base binary Updated binary Compressed binary patch (Heatshrink) Patch-to-File %
ESP32 test_basic_enable_small_feature 168208 155136 11036 7.11%
ESP32 test_nvs_app_modification 190656 199824 16245 8.13%
ESP32 test_http_upgrade_with_ssl 672736 761184 138839 18.24%
ESP32 test_provisioning_upgrade_idf_patch_version 966656 924448 234096 25.32%
  • As Heatshrink uses static allocation with small look-ahead buffers, it has almost no impact on heap memory.

Test Scenarios: ESP-IDF 4.4-dev (Master branch)

  1. test_basic_enable_small_feature: Enabling a small feature in an update

    Base binary: Compile get-started/hello-world example

    Updated binary: Same, but disable CONFIG_VFS_SUPPORT_IO option in sdkconfig

  2. test_nvs_app_modification: Changing user application flow without changing the set of libraries used

    Base binary: Compile storage/nvs_rw_value example

    Updated binary: Compile storage/nvs_rw_blob example

  3. test_http_upgrade_with_ssl: Changing user application flow with new set of libraries added

    Base binary: Compile protocols/http_server/simple example

    Updated binary: Compile protocols/https_server/simple example

  4. test_provisioning_upgrade_idf_patch_version: Upgrading IDF to the next patch version

    Base binary: Using IDF 4.3, compile provisioning/wifi_prov_mgr example

    Update binary: Same, but with IDF 4.3.1; sdkconfig is generated from scratch.

To-do:

  • Experiments with more test scenarios, especially with bigger binaries

  • Add complete workflow example

    • Currently patch is applied to the ota_0 partition rather than the factory
  • Optimize Heatshrink compression parameters

    • Currently running on low memory usage mode (8, 7) and static allocation
  • Binary patch with LZMA compression in detools

    • LZMA is memory-heavy but will provide a greater compression ratio for the patch
  • Managing Flash wear and tear (e.g. Wear Levelling)

Acknowledgements & Resources

  • detool: Binary delta encoding in Python 3 and C 🡒 Source | Docs
  • heatshrink: An Embedded Data Compression Library 🡒 Source | Blog
  • Delta updates for embedded systems 🡒 Source | Docs

License

Distributed under the MIT License. See LICENSE for more information.

Issues
  • related: bsdiff port for esp32

    related: bsdiff port for esp32

    Hi,

    I was looking at this project to add support for OTA firmware delta compressed support to one project I'm working on and I really liked the work you have put here and started playing with the code.

    Ultimately we decided to switch to a slightly modified version of bsdiff/bspatch because the esp32 has already tinfl_decompress available (in the ROM!) and so DEFLATE can be used and using that with bsdiff was simpler than with detools. Nothing wrong with heatshrink but we already used tinfl_decompress for normal full otas so we figured reuse was a good way vs heatshrink to reduce [slightly] firmware size (at the partition level at least).

    If people are interested the port of bsdiff is available at https://github.com/Blockstream/esp32_bsdiff

    Feel free to close if you consider this spamming :-)

    Status: Completed Type: Documentation 
    opened by greenaddress 1
  • Can't upload code more than one time

    Can't upload code more than one time

    Hello.

    Your code is very good and useful. But I thing you have to remplace this lines file: [components/delta/delta.c]

    flash->src = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_APP_OTA_0, PARTITION_LABEL_SRC);
    lash->dest = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_APP_OTA_1, PARTITION_LABEL_DEST);
    

    By this lines

    flash->src = esp_ota_get_running_partition();
    flash->dest = esp_ota_get_next_update_partition(NULL);
    
    Status: Completed Type: Bug 
    opened by hdpklm 0
Owner
ESP32 Musings
ESP32-based projects
ESP32 Musings
Diwali Lighting using ESP32 with OTA update support

OTA_ESP32_8x50_ws2811 Diwali Lighting using ESP32 with OTA update support! This code is meant to drive 8 neopixel chains through separate pins. Being

Anindya Mitra 1 Feb 5, 2022
OTA Third Party Firmware Flasher for the original Wyze Plug (WLPP1) and Wyze Bulb (WLPA19).

Wyze Plug (and Bulb!) Flasher Use this software to install third party firmware on the original Wyze Plug (model WLPP1) and Wyze Bulb (model WLPA19) o

Elahd Bar-Shai 2 Nov 13, 2021
Disable OTA Update for iOS & iPadOS for 14 - 14.3 (Non Jailbreak devices)

OTADisabler-App Disable OTA Update for iOS & iPadOS for 14 - 14.3 (Non Jailbreak devices) Support Devices iOS 14.0-14.3 (confirmed on iOS 14.1 and abo

ichitaso 24 Jun 15, 2022
Delta Kernel for Redmi Note 10 Pro (Max) (sweet/sweetin)

Linux kernel ============ This file was moved to Documentation/admin-guide/README.rst Please notice that there are several guides for kernel develop

Static Design Studio 12 Mar 19, 2022
Custom kernel for sweet based on Delta, builds hosted on @sweet_epsilon on Telegram (GH releases are outdated)

Linux kernel ============ This file was moved to Documentation/admin-guide/README.rst Please notice that there are several guides for kernel develop

Udit Karode 14 Mar 29, 2022
Tools to measure libcurl performance delta between versions

relative Tools to measure libcurl performance delta between versions build-many This script iterates over all the curl versions listed in the top of t

curl 11 Aug 30, 2021
ESP32-Skid-Steer - Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on the model.

ESP32-Skid-Steer Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on

null 5 Feb 4, 2022
tiny_csg is a C++ library that generates meshes from brush-based level data and supports incremental updates (real-time CSG).

tiny_csg is a C++ library that generates meshes from brush-based level data and supports incremental updates (real-time CSG). It is intended to be used as a backend in 3d level editors and/or generators.

Luka Aleksić 26 Jun 3, 2022
Updates the Wii's current system time with the real world time.

Fix Wii System Time This is a homebrew tool I made for the Wii a while ago. It updates the current system time with the real world time via worldtimea

Puzzle 2 Dec 21, 2021
The first Roblox FPS unlocker for macOS. This is a very simple unpolished FPS unlocker with more updates to come

rbxfpsunlocker-osx The first Roblox FPS unlocker for macOS. This is a very simple unpolished FPS unlocker with more updates to come. Usage Download th

lanylow 28 May 19, 2022
This program adds, displays, updates, searches and deletes records as per user preference.

Store Management System This program adds, displays, updates, searches and deletes records as per user preference. Project Description And Example Inf

GANPAT UNIVERSITY 2 Feb 11, 2022
This is a manga compressed file auto-resize tool

This is a manga compressed file auto-resize tool Are you still bothered by the size of your comic compressed files? Still doing the tedious operation

null 3 Mar 15, 2022
yariv.h is a single C/C++ header to encode and decode SPIR-V shaders into a more compressed form I've called YARI-V.

YARI-V yariv.h is a single C/C++ header to encode and decode SPIR-V shaders into a more compressed form I've called YARI-V. YARI-V is an alternative e

Neil Henning 32 May 3, 2022
null 235 Jun 22, 2022
Allows for multiple SwitchBot buttons and curtains to be controlled via MQTT sent to ESP32. ESP32 will send BLE commands to switchbots and return MQTT responses to the broker. Also supports Meter/Temp Sensor

SwitchBot-MQTT-BLE-ESP32 Switchbot local control using ESP32. no switchbot hub used/required. works with any smarthub that supports MQTT https://githu

null 239 Jun 17, 2022
AnalogWrite for ESP32 and ESP32-S2 with LEDC PWM. Includes PWM Phase Control, DAC and Smart GPIO resource management.

analogWrite() ESP32 Installation Instructions This library was tested using using the ESP32 Arduino IDE Boards Manager installation method. Stable rel

null 22 Jun 18, 2022
ESP32 + GitHub Actions + Husarnet. A boilerplate project for ESP32 allowing in-field firmware update using GitHub Actions workflow.

esp32-internet-ota ESP32 + GitHub Actions + Husarnet. A boilerplate project for ESP32 allowing in-field firmware update using GitHub Actions workflow.

Husarnet 28 Apr 30, 2022
Espressif ESP32 implementation of ANSI-ESTA E1.11 DMX-512A

This library allows for transmitting and receiving ANSI-ESTA E1.11 DMX-512A using an Espressif ESP32. It provides control and analysis of the packet configuration and allows the user to read synchronously or asynchronously from the DMX bus. This library also includes tools for data error-checking to safely process DMX commands.

null 66 Jun 22, 2022
And ESP32 powered VU matrix using the INMP441 I2S microphone

ESP32-INMP441-Matrix-VU This is the repository for a 3D-printed, (optionally) battery-powered, WS2812B LED matrix that produces pretty patterns using

null 43 Jun 6, 2022