USB serial TTY for the Arduino Uno with ILI9341 LCD screen shield

Overview

ILI9341TTY

USB serial TTY for the Arduino Uno with ILI9341 LCD screen shield



[video]

Semigraphics, ANSI, unicode, and an Arduino LCD-screen terminal

I've been playing with retro-styled interfaces terminal lately. The unicode box and block drawing characters are wonderful for this, especially "symbols for legacy computing". These include characters from classic computers of the twentieth century, and the block-diagonal teletext characters. These can be combined with ANSI escape codes for styled and colored text. Here's an example of a plot rendering in Gnome terminal using the teletext block-diagonal characters and ANSI color codes:

plot_examples

How hard would it be to support these features on an Arduino terminal display? This question led me down a rabbit-hole of vt100 terminal emulators on the arduino.

Getting a basic Arduino terminal monitor is straigtforward, if building upon a graphics library that supports text rendering. Bodmer has a nice demo for the Arduino using the Adagruit GFX graphics libraries, and Philippe Lucidarme has a short-and-sweet demo for the Seeduino (video). Both of these use the ILI9341 touch-screen display, which is commonly available as an Arduino shield.

However, getting good performance with the Arduino is hard. Scrolling requires re-drawing large areas, and is slow. One work-around is to use the ILI9341's built-in vertical scrolling feature. Among the most feature-complete is Nick Matantsev's TinTTY VT100 emulator for Arduino+ILI9341. This includes fast vertical scrolling and a touchscreen keyboard. Nick cites as inspiration this VT100 emulator for the ILI9340 screen with Arduino, by Martin K. Schröder. Bodmer's vertical-scrolling terminal for Arduino+ILI9341 notes that performance can be improved further by optimizing the graphics drivers to write directly to the 8-bit PORTB/C/D registers on the Arduino Uno.

In the broader universe of DIY terminals, Fabrizio Di Vittorio has a full-featured graphics library for the ESP32, which includes an ANSI/VT terminal. James Moxham has explored Arduino terminals with a wide variety of LCD and LED displays. Scargill's tech blog has a VT-100-esque vertical terminal using a ILI9340 display for the ESP8266 (video). Matthew Sarnoff also has this beautiful VT100 implementation using an oscilliscope as a display. I was especially charmed by ht-deko's implementation of ANSI color and style codes for the STM32F103, which builds upon the work of cbm80amiga.

Can we get an Arduino terminal that supports enough semigraphics charactes? Let's implement a virtual terminal monitor for the Arduino Uno and the ILI9341 LCD display with the following objectives:

  1. Fast enough to be a reasonable output device
  2. Support ANSI escape codes for color and style
  3. Interpret incoming data as utf-8 encoded unicode
  4. Support the box, block, and legacy computing chracters
  5. Optimize the drawing commands
  6. Squeeze as much unicode onto the Arduino as possible

(sorry for the poor photo quality, digital camera interacts poorly with the screen's narrow view angle; contrast is better in person)

Montage5

Version 0.2

Version 0.2 includes some additional unicode characters, including Armenian, Georgian, and Katakana/Hiragana. There was an aborted, incomplete attempt to support devanagari (prefix diacritics would require a bit of extra code to handle). Supported unicode characters are given in v0.2/test_terminal/v0p2supported.txt

supported

How to test it

Download the sketch and upload it onto the Arduino Uno with an Adafruit-style ILI9341 LCD shield.

Python:

BAUDRATE = MYBAUDRATE
ARDUINO_PORT_FILE_NAME = "/dev/MYARDUINOSERIALPORT"

import serial
arduino = serial.Serial(port=ARDUINO_PORT_FILE_NAME, baudrate=BAUDRATE, timeout=.1)
time.sleep(3)

def print_arduino(s=''):
        arduino.write(s.encode('utf-8'))

reset = '\x1bc'
print_arduino(reset+'Hello world!')

Bash:

> /dev/MYARDUINOSERIALPORT ">
> stty -F /dev/MYARDUINOSERIALPORT MYBAUDRATE cr3 ff1 nl1 bs1
> # (wait for arduino to reset)
> echo -e "\x1bcHello World!" >> /dev/MYARDUINOSERIALPORT

(The flags cr3 ff1 nl1 bs1 will be explained later)

Copy the output of a command with unbuffer

> # (Apt install `expect` to get the `unbuffer` command.) 
> unbuffer ls -w1 --color=auto | tee /dev/MYARDUINOSERIALPORT

Send keystrokes with screen or picocom

> screen /dev/MYARDUINOSERIALPORT MYBAUDRATE

This will open a blank window. Try typing to see if text shows up on the Arduino. To exit, press Ctrl-A and release, then press k to kill the screen session, then press y to confirm. Screen sends Carriage Return (CR;\r) instead of Line Feed (LF; \n aka newline). There's no way to change this, but if you install picocom it will play nicely with newlines:

picocom /dev/MYARDUINOSERIALPORT --baud YOURBAUDRATE --omap crcrlf --echo
# (press Control+a Control+x to exit)

How mirror terminal output on the Arduino TTY

First, configure the serial device

> stty -F /dev/MYARDUINOSERIALPORT BAUDRATE ixon cr3 ff1 nl1 bs1

You can the output from a command to the Arduino TTY by redirecting standard output:

ls --color=yes > /dev/MYARDUINOSERIALPORT

We can also use tee to echo the output back to the curent terminal. The echoed output will appear on the main computer immediately, but the command will wait until serial data is finished drawing on the Arduino before exiting.

ls --color=yes | tee /dev/MYARDUINOSERIALPORT

Some commands behave differently when piped through tee, however. The command unbuffer (apt install expect) can trick them into behaving normally:

unbuffer ls --color=yes | tee /dev/MYARDUINOSERIALPORT

You can list the settings of a tty device using stty:

stty -F /dev/MYARDUINOSERIALPORT -a

stty can set the number rows and columns on a tty, but this won't work:

> stty -F /dev/MYARDUINOSERIALPORT MYBAUDRATE rows 20 cols 53
> stty -F /dev/MYARDUINOSERIALPORT -a
speed MYBAUDRATE baud; rows 0; columns 0; ...

What does work is setting the rows and columns on the current virtual terminal in linux. Note that this size will be reset by any window resize events:

stty cols 53 rows 20

Note how the output of ls is correctly wrapped after applying this command:

# (before applying stty cols 53 rows 20)
> unbuffer ls --color=yes | tee /dev/MYARDUINOSERIALPORT
prepare_fonts  self_contained_Uno9341TFT_TTY_horizontal_v14  test_terminal  writeup
> stty cols 53 rows 20
> unbuffer ls --color=yes | tee /dev/MYARDUINOSERIALPORT
prepare_fonts
self_contained_Uno9341TFT_TTY_horizontal_v14
test_terminal
writeup

You can pipe all output from the shell to the screen by starting a new instance of bash and using tee to send a copy of stdout to the Arduino:

> stty -F /dev/MYARDUINOSERIALPORT MYBAUDRATE ixon cr3 ff1 nl1 bs1
> stty cols 53 rows 20
> bash | tee /dev/MYARDUINOSERIALPORT
> # (all output will now be copied to the Arduino TTY )

Dealing with serial buffer overflow

If bytes arrive faster than the Arduino can react, the serial buffer will overflow. This leads to dropped bytes. To address this, one can

  1. Lower the baud rate
  2. Increase the serial buffer size [1,2,3]
  3. Ensure that the host machine limits the rate at which it sends data
  4. Implement software control flow, which sends XOFF (19) to pause and XON (17) to resume. These can be enabled on linux by providing the ixon argument to stty when configuring the serial connection. (Edit: this may not work for USB serial devices)
  5. Ask the host machine to add a small delay after some commands

At the moment, the only guaranteed solution is (1) or (3). Increasing the buffer size (2) can help with transient loads, but still requires the host application to carefully limit its overall data-rate. Software control flow appears to not work on Linux with USB serial, although there may be hacks that work for specific hardware. The additional delays added by (5) are generally too small to make a difference.

Increasing the serial buffer size

Find the file HardwareSerial.cpp in your Arduino installation (find / -name HardwareSerial.cpp 2>/dev/null), and open it for editing. You might need sudo depending on how Arduino was installed.

> vi /usr/share/arduino/hardware/arduino/cores/arduino/HardwareSerial.cpp

Find this section (lines 58-62 for me):

#if (RAMEND < 1000)
  #define SERIAL_BUFFER_SIZE 16
#else
  #define SERIAL_BUFFER_SIZE 64
#endif

Change the second #define SERIAL_BUFFER_SIZE 64 to something large, perhaps #define SERIAL_BUFFER_SIZE 256.

#if (RAMEND < 1000)
  #define SERIAL_BUFFER_SIZE 16
#else
  #define SERIAL_BUFFER_SIZE 256
#endif

Software control flow (probably won't work)

Ostensibly, it should be possible to request that the kernel use software control flow. In this mode, the Arduino should be able to send the byte XOFF (19) to pause transmission, and XON (17) to resume it. It is enabled by the flag ixon in stty:

> stty -f /dev/MYARDUINOSERIALPORT MUBAUDRATE ixon

HOWEVER, it seems like the XON/XOFF software flow control does not work. Google suggests that this is a common, hard-to-fix, issue [1,2,3,4,5]. "The short version is that there is no such thing as "flow control" in any USB-serial adapter based on USB-serial."

Requesting delays

stty allows you to request that the host add a small delay after certain commands

> man stty
# (output abridged..)
Output settings:
   bsN  backspace delay style, N in [0..1]
   crN  carriage return delay style, N in [0..3]
   ffN  form feed delay style, N in [0..1]
   nlN  newline delay style, N in [0..1]

These modes relate to flags in the c_oflag register of the termio struct in linux (more documentation here).

# For delays following a newline:
NL0	No delay for NLs
NL1	Delay further output after newline for 100 milliseconds

# For delays following a carriage return:
CR0	No delay for CRs
CR1	Delay after CRs depending on current column position
CR2	Delay 100 milliseconds after sending CRs
CR3	Delay 150 milliseconds after sending CRs

# For delays following a backspace character:
BS0	No delay for BSs
BS1	Delay 50 milliseconds after sending BSs

# Delays for form-feeds
FF0	No delay for FFs
FF1	Delay 2 seconds after sending FFs

To add the maximum delay after positioning commands, run

stty -F /dev/MYARDUINOSERIALPORT BAUDRATE ixon cr3 ff1 nl1 bs1

Notes

  • Unicode: Codepoints are decoding from incoming UTF-8 serial data. These are sent to a routine that finds the corresponding unicode block for eah code point. The first 128 code-points "Basic Latin" are mapped to ASCII. Otherwise, a custom subroutine for the given block is called to handle rendering.

  • Fonts: We provide a texture containing 512 basic glyphs, each 5 pixels wide and 11 pixels tall. Glyph indecies corresponding to 128-bit ASCII are kept the same. Other indecies are filled with various glyphs in no particular order. These are "base glyphs" that may be futher modified / transformed to render a particular unicode code point.

  • Soft fonts: The unicode blocks "Box Drawing", "Block Drawing" and "Legacy Computing Characters" are handled as a "soft font". These are implemented as custom subroutines that draw each character, rather than bitmapped fonts (although some do use bitmap data to help with drawing).

  • Font mapping: There are way too many characters in unicode. However, most of these consist of pre-composed variants of Latin characters, with various modifications. Many characters are identical across different alphabets. Others can be drawn as e.g. reflected versions of basic latin characters. For alphabetic blocks beyons "basic latin", we associate each unicode point with (1) a "base glyph" index and (2) a number indicating a transformation/modification to be performed, if any.

  • Colors: We use a state-machine to parse ANSI CSI escape codes for color. 8-bit RRRBBGGG foreground/background colors are stored in the global state registers fg and bg.

  • Blink: We support a single blink speed (ANSI "fast blink" and "slow blin" are treated the same). Blink is implemented by retaining two bit vectors, one which tracks whether a given row/column should be blinking, and another that tracks whether it is currently highlighted. While the arduino is not receiving serial data, it runs a background loop to toggle the highlight on blinking locations. Highlight is implemented using fast xor-paint. This allows the same drawing code to set/unset the highligt. Both the "blink" and "highlight" bit-vectors are scrolled when the screen scrolls.

  • Italic, Bold, Blackletter, and combinations thereof: There isn't enough room to store separate fonts, so we implement these in software as transformations. "Italic" applies a slant by shifting the bottom half of the character left one pixel. "Bold" applies a bit-convolution to thicken the lines (while avoiding merging vertical strokes). Blackletter is replaced with a "very bold" effect.

  • Box drawing unicode block: These characters are rendered by combinding basic subcomponents. For example, a bold box-drawing character is rendered by stamping a thin-line box-drawing character over top of a double-line box drawing character.

  • Combining diacritics: These are, frankly, a nightmare. But, an attempt has been made to implement them. No further comment.

  • Scrolling: I like the horizontal display, but there is no way to scroll it quickly on the ILI9341. To work around this, the "scroll" routine scrolls 8 lines at a time.

Faster rendering

The Adafruit drivers aren't optimized for speed. We stripped them down to the bare essentials needed to get text onto the screen. Here are a few suggestions, if you're writing your own:

  • Specialize the driver for your particular Arduino platform and screen. Strip away unrelated code.
  • What raw I/O operations is the driver actually doing? Are they all necessary? Check out the ILI9341 and AtMega328 datasheets. Expand all macros until you can see the raw writes to the Arduino's 8-bit IO ports. On the Arduino Uno, these are PORTB, PORTC, and PORTD.
  • See if you can sacrifice accuracyfor speed.
  • See if some common special cases can be handled quicker using an optimized sequence of commands.

Notes specific to the ILI9341 shield on the Uno

  • For the Uno, The ILI9341 shield uses PORTB and PORTD to send data, and PORTC to control the serial clock.
  • The ILI9341 shield uses an 8-bit parallel interface. Color data is 16-bit. Every pixel sent requires writing the high then low byte. The values of PORTB and PORTD must be changed twice to send a single pixel. This is expensive.
  • Instead, send only colors for which the HIGH and LOW bytes are the same. This allows you to set PORTB and PORTD once, then clock PORTC rapidly to fill in pixel data with a single color. Restricting to colors with the same HIGH and LOW bytes gives a 256 color pallet, with color channels packed as RRRBBGGG.
  • The interface requires changing bits on both PORTB and PORTD. If you're not using any other pins on these ports, it's OK to write to all pins on both ports. Although the serial line is technically using bits 0 and 1 of PORTD, these bits will be ignore as long as the USART serial interface is enabled. To prepare to send a 256-color byte b, one can simply write PORTB=PORTD=b; (assuming all pins are set to output mode, of course). This is by far fastest way to get color data from the Arduino onto the ILI9341.
  • Loop iteration costs are nontrivial when flood-filling pixels. See if you can unroll loops to reduce these .

Future work?

  • Switch to vertical oritentation to get fast scrolling
  • Write a proxy program on the host side to get XON/XOFF software flow control working properly
  • Switch the unicode mapping tables to a sparse format (binary search through a sorted list)
  • Stress test with more demanding programs (e.g. text editors) to detect bugs and improve compatability, performance.
Owner
M Rule
M Rule
Библиотека программного USB клавиатуры и мыши для Arduino Nano/UNO/Mega и прочих

EasyHID Библиотека программного USB клавиатуры и мыши для Arduino Nano/UNO/Mega и прочих Программный USB Работает на любой AVR Arduino 16 МГц Поддержк

Alex 45 Aug 1, 2022
Arduino M-BUS Master node for Arduino MKR M-BUS Shield

Arduino M-BUS Master node for Arduino MKR M-BUS Shield This software will read out a M-BUS device connected to an Arduino MKR board equipped with our

null 5 Jul 25, 2022
A simple tool using PC mouse via USART to control MCU and LCD/OLED (with LVGL library), if your screen cannot be controlled by touch.

LVGL_USB_Mouse A simple tool using PC mouse via USART to control MCU and LCD/OLED (with LVGL library), if your screen cannot be controlled by touch. 如

k_ying 5 May 5, 2022
A Fingerprint Door Lock using a microprocessor named Arduino UNO and programming through Arduino IDE

INSTRUCTIONS - The codes for the Fingerprint Door lock are present in the Code For Fingerprint Door Lock folder The instructions of how to operate the

Akhil Sahukaru 15 Mar 3, 2022
Serial Data Monitor is a multiplatform (Windows, Linux, Mac, ...) tool to interactively receive/edit/monitor data and send commands to an embedded system via the serial bus

See wiki for full documentation Serial Data Monitor Description Serial Data Monitor is a multiplatform (Windows, Linux, Mac, ...) tool to interactivel

monnoliv 4 Oct 29, 2021
The function is based on MQTT. When the original serial of ESP8266/ESP32 cannot be used, it can replace serial print.

MqttPrint and MqttMonitor The function is based on MQTT. When the original serial of ESP8266/ESP32 cannot be used, it can replace serial print. MqttPr

fw-box 3 Jan 10, 2022
John Walker 18 Aug 8, 2022
Baseline Arduino shield for temperature, humidity, soil moisture and ambient light

Introduction Green-Shield is an Arduino R3 shield for temperature, humidity, soil moisture and ambient light optimized for use in greenhouses where th

#Eduh 3 Dec 29, 2021
Synology DSM 7 USB serial drivers

Missing USB serial drivers for DSM 7 Supported drivers cp210x ch341 Supported platforms apollolake (linux-4.4.x) armada38x (linux-3.10.x-bsp) armadaxp

Robert Klep 92 Aug 3, 2022
Library of useful C++ snippets and reusable classes I've created as I build out Arduino Uno and ESP32 projects.

Arduino Snippets Library of useful C++ snippets and reusable classes I've created as I build out Arduino Uno and ESP32 projects. Button A simple butto

Max Lynch 7 Feb 5, 2022
Remote Arduino Uno-based TFT graphical display for VSR Mini Mega Alternator Regulator

VSMMAR_Remote_Display Remote Arduino Uno-based TFT graphical display for VSR Mini Mega Alternator Regulator This project is an optional accessory for

null 1 Nov 6, 2021
Play Nintendo Switch using an original N64 controller via an Arduino Uno!

N64 -> Arduino Uno -> Nintendo Switch Description By connecting an original N64 controller to an Arduino UNO R3 running this code, and plugging the US

null 17 Mar 15, 2022
The ultimate battery tester with ESR measurement and discharge graph. Based on an Arduino Nano and a 1602 LCD.

Ultimate-Battery-Tester Version 1.0.0 Features Measures the ESR (equivalent series resistance) of the battery. This is an idicator of the health of th

Armin 7 Jul 26, 2022
An Arduino library to control 2-bit (4 gray level) LCD displays

TwoBitDisplay (2-bpp LCD library) Project started 10/23/2021 Copyright (c) 2021 BitBank Software, Inc. Written by Larry Bank [email protected] The pur

Larry Bank 8 Mar 21, 2022
See your system information on LCD with Arduino!

Nodejs Arduino System Info See your system information on LCD with Arduino! You can see RAM usage, and CPU usage. Requipments: An Arduino Board. [ You

Akif9748 4 Jan 14, 2022
A tiny external monitor for PC using STM32 and ST7789. Connects to PC over USB and displays the captured screen on ST7789 (240x240) display.

STM32 Tiny Monitor A super tiny monitor for your PC, suitable for your pet ant. A python script sends the captured screen over USB to the STM32 microc

Avra Mitra 65 Aug 1, 2022
An all-in-one Spartan Edge Accelerator shield implementation for the gbaHD

gbaHD AIO Shield This is a Spartan Edge Accelerator shield which implements all the hardware connections needed for zwenergy's gbaHD, no wires require

null 37 Jul 30, 2022
ESP32 software USB host through general IO pins. We can connect up to 4 USB-LS HID (keyboard mouse joystick) devices simultaneously.

esp32_usb_soft_host esp32 USB-LS pure software host thru general IO pins. Up to 4 HID devices simultaneously. board ~$3 :https://www.aliexpress.com/pr

Samsonov Dima 274 Aug 10, 2022
ESP8266 software USB host through general IO pins. We can connect up to 2 USB-LS HID (keyboard mouse joystick) devices simultaneously.

esp8266_usb_soft_host Test for esp8266 usb host . Works with ESP-IDF v3.4 80MHz and 160MHz. WorkInProgress Test run with mouse & combined mouse & keyb

Samsonov Dima 27 Jul 30, 2022