USB serial TTY for the Arduino Uno with ILI9341 LCD screen shield
Semigraphics, ANSI, unicode, and an Arduino LCD-screen terminal
I've been playing with retro-styled interfaces terminal lately. The unicode box and block drawing characters are wonderful for this, especially "symbols for legacy computing". These include characters from classic computers of the twentieth century, and the block-diagonal teletext characters. These can be combined with ANSI escape codes for styled and colored text. Here's an example of a plot rendering in Gnome terminal using the teletext block-diagonal characters and ANSI color codes:
How hard would it be to support these features on an Arduino terminal display? This question led me down a rabbit-hole of vt100 terminal emulators on the arduino.
Getting a basic Arduino terminal monitor is straigtforward, if building upon a graphics library that supports text rendering. Bodmer has a nice demo for the Arduino using the Adagruit GFX graphics libraries, and Philippe Lucidarme has a short-and-sweet demo for the Seeduino (video). Both of these use the ILI9341 touch-screen display, which is commonly available as an Arduino shield.
However, getting good performance with the Arduino is hard. Scrolling requires re-drawing large areas, and is slow. One work-around is to use the ILI9341's built-in vertical scrolling feature. Among the most feature-complete is Nick Matantsev's TinTTY VT100 emulator for Arduino+ILI9341. This includes fast vertical scrolling and a touchscreen keyboard. Nick cites as inspiration this VT100 emulator for the ILI9340 screen with Arduino, by Martin K. Schröder. Bodmer's vertical-scrolling terminal for Arduino+ILI9341 notes that performance can be improved further by optimizing the graphics drivers to write directly to the 8-bit PORTB/C/D registers on the Arduino Uno.
In the broader universe of DIY terminals, Fabrizio Di Vittorio has a full-featured graphics library for the ESP32, which includes an ANSI/VT terminal. James Moxham has explored Arduino terminals with a wide variety of LCD and LED displays. Scargill's tech blog has a VT-100-esque vertical terminal using a ILI9340 display for the ESP8266 (video). Matthew Sarnoff also has this beautiful VT100 implementation using an oscilliscope as a display. I was especially charmed by ht-deko's implementation of ANSI color and style codes for the STM32F103, which builds upon the work of cbm80amiga.
Can we get an Arduino terminal that supports enough semigraphics charactes? Let's implement a virtual terminal monitor for the Arduino Uno and the ILI9341 LCD display with the following objectives:
- Fast enough to be a reasonable output device
- Support ANSI escape codes for color and style
- Interpret incoming data as utf-8 encoded unicode
- Support the box, block, and legacy computing chracters
- Optimize the drawing commands
- Squeeze as much unicode onto the Arduino as possible
(sorry for the poor photo quality, digital camera interacts poorly with the screen's narrow view angle; contrast is better in person)
Version 0.2 includes some additional unicode characters, including Armenian, Georgian, and Katakana/Hiragana. There was an aborted, incomplete attempt to support devanagari (prefix diacritics would require a bit of extra code to handle). Supported unicode characters are given in
How to test it
Download the sketch and upload it onto the Arduino Uno with an Adafruit-style ILI9341 LCD shield.
BAUDRATE = MYBAUDRATE ARDUINO_PORT_FILE_NAME = "/dev/MYARDUINOSERIALPORT" import serial arduino = serial.Serial(port=ARDUINO_PORT_FILE_NAME, baudrate=BAUDRATE, timeout=.1) time.sleep(3) def print_arduino(s=''): arduino.write(s.encode('utf-8')) reset = '\x1bc' print_arduino(reset+'Hello world!')
> stty -F /dev/MYARDUINOSERIALPORT MYBAUDRATE cr3 ff1 nl1 bs1 > # (wait for arduino to reset) > echo -e "\x1bcHello World!" >> /dev/MYARDUINOSERIALPORT
cr3 ff1 nl1 bs1 will be explained later)
Copy the output of a command with
> # (Apt install `expect` to get the `unbuffer` command.) > unbuffer ls -w1 --color=auto | tee /dev/MYARDUINOSERIALPORT
Send keystrokes with
> screen /dev/MYARDUINOSERIALPORT MYBAUDRATE
This will open a blank window. Try typing to see if text shows up on the Arduino. To exit, press
Ctrl-A and release, then press
k to kill the screen session, then press
y to confirm. Screen sends Carriage Return (CR;
\r) instead of Line Feed (LF;
\n aka newline). There's no way to change this, but if you install picocom it will play nicely with newlines:
picocom /dev/MYARDUINOSERIALPORT --baud YOURBAUDRATE --omap crcrlf --echo # (press Control+a Control+x to exit)
How mirror terminal output on the Arduino TTY
First, configure the serial device
> stty -F /dev/MYARDUINOSERIALPORT BAUDRATE ixon cr3 ff1 nl1 bs1
You can the output from a command to the Arduino TTY by redirecting standard output:
ls --color=yes > /dev/MYARDUINOSERIALPORT
We can also use
tee to echo the output back to the curent terminal. The echoed output will appear on the main computer immediately, but the command will wait until serial data is finished drawing on the Arduino before exiting.
ls --color=yes | tee /dev/MYARDUINOSERIALPORT
Some commands behave differently when piped through
tee, however. The command
unbuffer (apt install expect) can trick them into behaving normally:
unbuffer ls --color=yes | tee /dev/MYARDUINOSERIALPORT
You can list the settings of a tty device using
stty -F /dev/MYARDUINOSERIALPORT -a
stty can set the number rows and columns on a tty, but this won't work:
> stty -F /dev/MYARDUINOSERIALPORT MYBAUDRATE rows 20 cols 53 > stty -F /dev/MYARDUINOSERIALPORT -a speed MYBAUDRATE baud; rows 0; columns 0; ...
What does work is setting the
columns on the current virtual terminal in linux. Note that this size will be reset by any window resize events:
stty cols 53 rows 20
Note how the output of
ls is correctly wrapped after applying this command:
# (before applying stty cols 53 rows 20) > unbuffer ls --color=yes | tee /dev/MYARDUINOSERIALPORT prepare_fonts self_contained_Uno9341TFT_TTY_horizontal_v14 test_terminal writeup > stty cols 53 rows 20 > unbuffer ls --color=yes | tee /dev/MYARDUINOSERIALPORT prepare_fonts self_contained_Uno9341TFT_TTY_horizontal_v14 test_terminal writeup
You can pipe all output from the shell to the screen by starting a new instance of
bash and using
tee to send a copy of
stdout to the Arduino:
> stty -F /dev/MYARDUINOSERIALPORT MYBAUDRATE ixon cr3 ff1 nl1 bs1 > stty cols 53 rows 20 > bash | tee /dev/MYARDUINOSERIALPORT > # (all output will now be copied to the Arduino TTY )
Dealing with serial buffer overflow
If bytes arrive faster than the Arduino can react, the serial buffer will overflow. This leads to dropped bytes. To address this, one can
- Lower the baud rate
- Increase the serial buffer size [1,2,3]
- Ensure that the host machine limits the rate at which it sends data
- Implement software control flow, which sends
XOFF(19) to pause and
XON(17) to resume.
These can be enabled on linux by providing the(Edit: this may not work for USB serial devices)
ixonargument to stty when configuring the serial connection.
- Ask the host machine to add a small delay after some commands
At the moment, the only guaranteed solution is (1) or (3). Increasing the buffer size (2) can help with transient loads, but still requires the host application to carefully limit its overall data-rate. Software control flow appears to not work on Linux with USB serial, although there may be hacks that work for specific hardware. The additional delays added by (5) are generally too small to make a difference.
Increasing the serial buffer size
Find the file
HardwareSerial.cpp in your Arduino installation (
find / -name HardwareSerial.cpp 2>/dev/null), and open it for editing. You might need
sudo depending on how Arduino was installed.
> vi /usr/share/arduino/hardware/arduino/cores/arduino/HardwareSerial.cpp
Find this section (lines 58-62 for me):
#if (RAMEND < 1000) #define SERIAL_BUFFER_SIZE 16 #else #define SERIAL_BUFFER_SIZE 64 #endif
Change the second
#define SERIAL_BUFFER_SIZE 64 to something large, perhaps
#define SERIAL_BUFFER_SIZE 256.
#if (RAMEND < 1000) #define SERIAL_BUFFER_SIZE 16 #else #define SERIAL_BUFFER_SIZE 256 #endif
Software control flow (probably won't work)
Ostensibly, it should be possible to request that the kernel use software control flow. In this mode, the Arduino should be able to send the byte
XOFF (19) to pause transmission, and
XON (17) to resume it. It is enabled by the flag
ixon in stty:
> stty -f /dev/MYARDUINOSERIALPORT MUBAUDRATE ixon
HOWEVER, it seems like the XON/XOFF software flow control does not work. Google suggests that this is a common, hard-to-fix, issue [1,2,3,4,5]. "The short version is that there is no such thing as "flow control" in any USB-serial adapter based on USB-serial."
stty allows you to request that the host add a small delay after certain commands
> man stty # (output abridged..) Output settings: bsN backspace delay style, N in [0..1] crN carriage return delay style, N in [0..3] ffN form feed delay style, N in [0..1] nlN newline delay style, N in [0..1]
# For delays following a newline: NL0 No delay for NLs NL1 Delay further output after newline for 100 milliseconds # For delays following a carriage return: CR0 No delay for CRs CR1 Delay after CRs depending on current column position CR2 Delay 100 milliseconds after sending CRs CR3 Delay 150 milliseconds after sending CRs # For delays following a backspace character: BS0 No delay for BSs BS1 Delay 50 milliseconds after sending BSs # Delays for form-feeds FF0 No delay for FFs FF1 Delay 2 seconds after sending FFs
To add the maximum delay after positioning commands, run
stty -F /dev/MYARDUINOSERIALPORT BAUDRATE ixon cr3 ff1 nl1 bs1
Unicode: Codepoints are decoding from incoming UTF-8 serial data. These are sent to a routine that finds the corresponding unicode block for eah code point. The first 128 code-points "Basic Latin" are mapped to ASCII. Otherwise, a custom subroutine for the given block is called to handle rendering.
Fonts: We provide a texture containing 512 basic glyphs, each 5 pixels wide and 11 pixels tall. Glyph indecies corresponding to 128-bit ASCII are kept the same. Other indecies are filled with various glyphs in no particular order. These are "base glyphs" that may be futher modified / transformed to render a particular unicode code point.
Soft fonts: The unicode blocks "Box Drawing", "Block Drawing" and "Legacy Computing Characters" are handled as a "soft font". These are implemented as custom subroutines that draw each character, rather than bitmapped fonts (although some do use bitmap data to help with drawing).
Font mapping: There are way too many characters in unicode. However, most of these consist of pre-composed variants of Latin characters, with various modifications. Many characters are identical across different alphabets. Others can be drawn as e.g. reflected versions of basic latin characters. For alphabetic blocks beyons "basic latin", we associate each unicode point with (1) a "base glyph" index and (2) a number indicating a transformation/modification to be performed, if any.
Colors: We use a state-machine to parse ANSI CSI escape codes for color. 8-bit RRRBBGGG foreground/background colors are stored in the global state registers
Blink: We support a single blink speed (ANSI "fast blink" and "slow blin" are treated the same). Blink is implemented by retaining two bit vectors, one which tracks whether a given row/column should be blinking, and another that tracks whether it is currently highlighted. While the arduino is not receiving serial data, it runs a background loop to toggle the highlight on blinking locations. Highlight is implemented using fast xor-paint. This allows the same drawing code to set/unset the highligt. Both the "blink" and "highlight" bit-vectors are scrolled when the screen scrolls.
Italic, Bold, Blackletter, and combinations thereof: There isn't enough room to store separate fonts, so we implement these in software as transformations. "Italic" applies a slant by shifting the bottom half of the character left one pixel. "Bold" applies a bit-convolution to thicken the lines (while avoiding merging vertical strokes). Blackletter is replaced with a "very bold" effect.
Box drawing unicode block: These characters are rendered by combinding basic subcomponents. For example, a bold box-drawing character is rendered by stamping a thin-line box-drawing character over top of a double-line box drawing character.
Combining diacritics: These are, frankly, a nightmare. But, an attempt has been made to implement them. No further comment.
Scrolling: I like the horizontal display, but there is no way to scroll it quickly on the ILI9341. To work around this, the "scroll" routine scrolls 8 lines at a time.
The Adafruit drivers aren't optimized for speed. We stripped them down to the bare essentials needed to get text onto the screen. Here are a few suggestions, if you're writing your own:
- Specialize the driver for your particular Arduino platform and screen. Strip away unrelated code.
- What raw I/O operations is the driver actually doing? Are they all necessary? Check out the ILI9341 and AtMega328 datasheets. Expand all macros until you can see the raw writes to the Arduino's 8-bit IO ports. On the Arduino Uno, these are PORTB, PORTC, and PORTD.
- See if you can sacrifice accuracyfor speed.
- See if some common special cases can be handled quicker using an optimized sequence of commands.
Notes specific to the ILI9341 shield on the Uno
- For the Uno, The ILI9341 shield uses PORTB and PORTD to send data, and PORTC to control the serial clock.
- The ILI9341 shield uses an 8-bit parallel interface. Color data is 16-bit. Every pixel sent requires writing the high then low byte. The values of PORTB and PORTD must be changed twice to send a single pixel. This is expensive.
- Instead, send only colors for which the HIGH and LOW bytes are the same. This allows you to set PORTB and PORTD once, then clock PORTC rapidly to fill in pixel data with a single color. Restricting to colors with the same HIGH and LOW bytes gives a 256 color pallet, with color channels packed as RRRBBGGG.
- The interface requires changing bits on both PORTB and PORTD. If you're not using any other pins on these ports, it's OK to write to all pins on both ports. Although the serial line is technically using bits 0 and 1 of PORTD, these bits will be ignore as long as the USART serial interface is enabled. To prepare to send a 256-color byte
b, one can simply write
PORTB=PORTD=b;(assuming all pins are set to output mode, of course). This is by far fastest way to get color data from the Arduino onto the ILI9341.
- Loop iteration costs are nontrivial when flood-filling pixels. See if you can unroll loops to reduce these .
- Switch to vertical oritentation to get fast scrolling
- Write a proxy program on the host side to get
XON/XOFFsoftware flow control working properly
- Switch the unicode mapping tables to a sparse format (binary search through a sorted list)
- Stress test with more demanding programs (e.g. text editors) to detect bugs and improve compatability, performance.