Adds proper Apple Pencil support to Unity's InputSystem.

Overview

This project aims to create an Apple Pencil device for Unity's InputSystem package.

The problem: Unity does not fully integrate with Apple Pencil. Apple Pencil can provide events at 240hz, but only does 60hz by default with an extra API for getting all the events. Apple also provides an API for estimating values like pen pressure before the real values are ready (filling in the correct values later), as well as predicting future events in order to reduce visual latency when drawing, not to mention barrel-tap events with the Apple Pencil 2. As of writing this, Unity doesn't support any of that, which means that it's impossible to draw a smooth curve with Apple Pencil using Unity.

Unity's Input Situation: As of writing this, Unity has two built-in Input methods. The original Input (Old Input), and the newer InputSystem package which is still in development but is packaged with newer versions of Unity. The old Input system is capped to framerate, so even if we could get 240z from Apple Pencil we still wouldn't be able to consume the events since Input would throw most of them away. (Note - A very recent alpha build adds support for getting coalesced events with Old Input on Windows, but in my testing it is still weirdly giving me fewer events on lower framerates.) The newer InputSystem package is able to give us uncapped events using its lower level onEvent() API, but it doesn't call the full Apple API so we still only get 60hz on the device.

The solution: We need to write native code that can retrieve Apple Pencil events using the full API, and then make an InputSystem Device subclass that provides them in Unity. There are a few parts to this:

  1. A replacement UnityView+iOS.mm that needs to be copied into your built XCode project. This lets us modify Unity's touch event handlers to call our own code instead.
  2. A native code bridge, iOSApplePencil.hpp, which exposes methods we can call from Unity code. UnityView+iOS.mm sends touch events to the bridge, which puts them in a buffer that both it and Unity can use directly. The bridge then tells Unity when new events have been put in the buffer. (Note - The bridge is unfortunately written in C++ instead of Objective-C because I don't have a Mac and can't compile Objective-C code without going through Unity Cloud Build, meaning 20 minute compile times. Writing it in C++ allows me to test it locally using fake events by compiling it into a DLL.)
  3. An ApplePencil Device that acts as an InputSystem Pointer.

What's working currently: Currently I can get Apple Pencil events at 240hz on my iPad, with the original Apple Pencil (I don't have an Apple Pencil 2 to test with). Right now it only sends position, pressure and tilt to Unity.

To use: Follow these steps:

  1. In Player Settings in Unity, check the allow unsafe code option. This lets us get a pointer to a memory buffer that we can share with the native code bridge.

  2. If not already done, make sure you have the new InputSystem enabled in Player Settings (under Active Input Handling).

  3. Put ApplePencil.cs in your Unity project. To get the full 240hz, you'll need to use it with onEvent() like this:

     InputSystem.onEvent += OnInputEvent;
     void OnInputEvent(InputEventPtr eventPtr, InputDevice device) {
         if (!eventPtr.IsA<StateEvent>() && !eventPtr.IsA<DeltaStateEvent>())
             return;
         var pencil = device as ApplePencil;
         if (pencil == null)
             return;
         var pos = pencil.position.ReadValueFromEvent(eventPtr);
         var pressure = pencil.pressure.ReadValueFromEvent(eventPtr);
         // draw at a smooth 240hz
     }
    
  4. After building your iOS project in Unity, copy UnityView+iOS.mm and iOSApplePencil.hpp to {BUILD}/Classes/UI. I'm using Cloud Build and a post-export hook to do this automatically, but maybe you could do it with a Unity post-build hook too.

Owner
Oliver Smith
Oliver Smith
A bootloader and experimentation playground for Apple Silicon

m1n1: an experimentation playground for Apple Silicon (And to some extent a Linux bootloader) Building You need an aarch64-linux-gnu-gcc cross-compile

Asahi Linux 976 Dec 5, 2021
Upload arbitrary data via Apple's Find My network.

Send My Send My allows you to to upload abritrary data from devices without an internet connection by (ab)using Apple's Find My network. The data is b

Positive Security 1.4k Nov 23, 2021
Apple TV 2/3 Jailbreak

Blackb0x Apple TV 2/3 Jailbreak Untethered jailbreak tool that runs on modern versions of macOS. Devices supported: Apple TV 3,2 (A1469) (iOS 8.4.x un

null 216 Nov 21, 2021
An apple IIe emulator for the Raspberry Pi.

ApplePi "applepi" is an apple IIe emulator for the Raspberry Pi, but can be built (I think) on most Linux distributions. It runs easily on a Pi4B, and

J.B.Ward 8 Nov 28, 2021
Apple cctools and ld64 port for Linux, *BSD and macOS

Apple cctools and ld64 port for Linux, *BSD and macOS

Thomas Pöchtrager 556 Dec 1, 2021
Swift Framework for Apple Health Integration with Terra Enabling Developers LTD.

TerraSwift This library allows developers to connect to TERRA ENABLING DEVELOPERS LTD. through Swift (implemented with Swift5.0). This library uses He

null 5 Dec 6, 2021
Protobuf for Proxyman app - Include Apple Silicon & Intel architecture

Protobuf for Proxyman macOS app Protobuf for Proxyman app - Include Apple Silicon & Intel architecture How to build Open the project on the latest Xco

Proxyman 6 Nov 29, 2021
Get the current average CPU frequency on Apple M1 chips.

M1 CPU Frequency Get the current average CPU frequency on Apple M1 chips. Usage Download the precompiled binary from the releases and run it in the te

BitesPotatoBacks 1 Nov 21, 2021
My experiments with Linux for the Apple M1 platform, based on Asahi and Corellium patches

Linux kernel ============ There are several guides for kernel developers and users. These guides can be rendered in a number of formats, like HTML an

Danny Lin 9 Oct 24, 2021
Support for TrueType (.ttf) font files with Simple Directmedia Layer.

This library is a wrapper around the excellent FreeType 2.0 library

Simple Directmedia Layer 60 Nov 18, 2021
C/C++ language server supporting multi-million line code base, powered by libclang. Emacs, Vim, VSCode, and others with language server protocol support. Cross references, completion, diagnostics, semantic highlighting and more

Archived cquery is no longer under development. clangd and ccls are both good replacements. cquery cquery is a highly-scalable, low-latency language s

Jacob Dufault 2.3k Nov 25, 2021
Dynamic 3D cellular automata engine with lua scripting support

Cell3D Cell3D is a dynamic 3D cellular automata engine with lua scripting support Installation Dependencies: Lua 5.3 Raylib Simplest possible build co

null 1 May 25, 2021
a undetectable tool by modify odyssey, support sign disable & dylib injection, test on iphoneX(13.5.1 expolit by FreeTheSandbox), our qqgroup is 703156427

a undetectable ios root access tool by modify odyssey, support sign disable & dylib injection, test on iphoneX(13.5.1 expolit by FreeTheSandbox), our

null 58 Nov 22, 2021
Tiny FEL tools for allwinner SOC, support RISC-V D1 chip

XFEL Tiny FEL tools for allwinner SOC, support RISC-V D1 chip. How to build The xfel tools depends on the libusb-1.0 library, you need to install libu

xboot.org 34 Dec 3, 2021
Add Sidecar support to unsupported models

SidecarFixup Lilu Kernel extension for enabling Sidecar support on the following SMBIOS

Acidanthera 103 Nov 29, 2021
Backtrace support for Rust `no_std` and embedded programs.

mini-backtrace This crate provides backtrace support for no_std and embedded programs. This is done through by compiling LLVM's libunwind with certain

Amanieu d'Antras 14 Nov 20, 2021
Xournal++ is a handwriting notetaking software with PDF annotation support

Xournal++ is a handwriting notetaking software with PDF annotation support. Written in C++ with GTK3, supporting Linux (e.g. Ubuntu, Debian, Arch, SUSE), macOS and Windows 10. Supports pen input from devices such as Wacom Tablets.

Xournalpp 5.2k Nov 30, 2021
split89 keyboard - a 3d printed 89 key split TKL keyboard base powered by ATmega32U4 Pro Micro controllers with QMK Configurator support.

split89 keyboard - a 3d printed 89 key split TKL keyboard base powered by ATmega32U4 Pro Micro controllers with QMK Configurator support. This keyboar

null 33 Dec 5, 2021
This device repo aims to support booting AOSP on SDM845 devices supported by the mainline Linux kernel

device/generic/sdm845 (AOSP device config for SDM845 devices) This device repo aims to support booting AOSP on SDM845 devices supported by the mainlin

Caleb Connolly 10 Nov 26, 2021