vtx.foundation draft d03

VTX 1 · core · beta draft d03

VTX 1 Wire Specification: core

Oct 11, 2026, revised Nov 16, 202639 pagesALPN vtx1-d03MIT licence

§11Input

Input travels on channel 2. The client declares its input devices, and every device event names its device, carries a timestamp and carries seen (Section 6). Device classes and event types are registries, so new kinds of input can join without a new protocol version.

Devices. HELLO's inputs lists the client's devices, and DEVICES reports devices added or removed later. A descriptor is { id, class, ? name, … }; IDs are unique within a connection, and clients MAY coarsen name for privacy.

Class Descriptor fields Events
keyboard layout: the platform's layout name, such as de or us-intl KEY, COMPOSE
mouse buttons; trackpad POINTER, WHEEL; GESTURE from trackpads
touch contacts: the maximum simultaneous contacts POINTER, GESTURE
pen pressure, tilt, twist, hover (booleans) POINTER
gamepad mapping (standard or none), buttons, axes, haptics GAMEPAD
joystick buttons, axes, hats, haptics GAMEPAD
voice languages (BCP 47) the client can dictate in COMPOSE
a11y none A11Y
name@domain Defined by the extension EXT

Input frames (channel 2)

Type Name Direction Payload
0x10 KEY C→S { dev, t, kind, code, key, ? text, mods, ? unshifted, ? target, seen }
0x11 COMPOSE C→S { dev, t, phase, text, ? caret, ? confidence, ? seq, ? target, seen }
0x12 POINTER C→S { dev, t, kind, type, ? id, x, y, ? dx, ? dy, ? col, ? row, buttons, ? button, ? contact, ? pressure, ? tilt, ? twist, ? w, ? h, ? samples, ? target, seen }
0x13 WHEEL C→S { dev, t, dx, dy, unit, ? target, seen }
0x14 FOCUS C→S { ? target, focused, seen }
0x15 PASTE C→S { text, ? target, seq, ? more, seen }
0x16 INTENT C→S Section 9
0x17 DEVICES C→S { ? added: [* device], ? removed: [* uint] }
0x18 GESTURE C→S { dev, t, kind, phase, x, y, fingers, ? dx, ? dy, ? scale, ? rotation, ? vx, ? vy, ? direction, ? target, seen }
0x19 GAMEPAD C→S { dev, t, ? buttons, ? axes, ? hats, ? target, seen }
0x1A HAPTIC S→C { dev, effect, ? strong, ? weak, ? left, ? right, duration }
0x1B A11Y C→S { dev, t, action, ? target, seen }
0x1C EXT C→S { dev, t, type, data, ? target, seen }
0x1D RESIZE C→S { target, cols, rows, ? px, seen }
0x1E INPUT_ACK S→C { seq }

t is when the device produced the event, in microseconds on a monotonic client clock. Applications use it to measure latency, rebuild pen strokes and time gestures; it is never compared with the server's clock.

Keys

  • kind is press, repeat or release. Clients without input.keys send press only.
  • code is the physical key, using W3C UI Events KeyboardEvent.code values such as KeyA; left and right modifiers differ, as in ShiftLeft.
  • key is the logical key, using UI Events key values such as a, Enter or ArrowUp.
  • text is the text the key produced, if any.
  • mods is a bitmask: 1 Shift, 2 Alt, 4 Control, 8 Meta, 16 Caps Lock, 32 Num Lock, 64 Hyper, 128 AltGraph.
  • unshifted, optional, is the key value the key gives without Shift, such as a for A; the VT engine needs it for the Kitty keyboard protocol.

This removes the ambiguity of VT escape sequences, where Ctrl+I and Tab arrive as the same byte. Clients show their on-screen keyboard when a text field, or a grid with keys: "raw", has focus and no hardware keyboard is present.

Routing

  • Keys go to the focused node. Layer 2 fields handle text editing locally; the server sees the result in the next intent.
  • A grid with keys: "raw" receives every key except those the client reserves.
  • Every client MUST reserve at least one key combination that never reaches the server. It takes the user to the client's own controls and the trusted zone, much like SSH's ~. escape, and always releases pointer lock.
  • Gamepad state goes to the focused node only if it or an ancestor sets gamepad: true. Otherwise the client uses gamepads to navigate layer 2 itself, as game consoles do: the D-pad and left stick move focus, the south button activates and the east button goes back.

Composition and dictation. COMPOSE reports IME composition with phase start, update, commit or cancel. Layer 2 fields compose locally; grids receive the committed text, so CJK input works in legacy programs too. Dictation from a voice device uses the same events: interim results as update, the final result as commit, with an optional confidence from 0 to 1. In the core, the user always starts dictation from the client's own controls, and the audio never leaves the client. Streaming voice audio to the server needs the audio channel of Media 1 and a microphone permission.

Pointer

  • kind is move, down, up, enter, leave or cancel, plus lock and unlock when pointer lock starts or ends.
  • type is mouse, touch or pen; touch and pen contacts carry an id.
  • x, y are device-independent pixels from the display's top-left corner. Over a grid, col and row are added.
  • buttons is the W3C bitmask of pressed buttons: 1 primary (pen tip), 2 secondary (pen barrel), 4 middle, 8 back, 16 forward, 32 eraser. button names the one that changed.
  • Pens add pressure (0–1), tilt [x, y] and twist (0–359) in degrees, and contact: false while hovering in range. Touch contacts add their size as w, h in pixels.
  • samples holds the high-rate samples coalesced since the previous event, oldest first, each [t, x, y] or [t, x, y, pressure]. Clients SHOULD send them for pens and MAY for mice, so strokes and fast motion are not lost to the frame rate.
  • The client reports only what the target needs: a grid's mouse mode, or the pointer events the application asked for. Hover effects in layer 2 stay local.

Pointer lock. A node with pointer: "lock" asks for relative mouse input, as games and remote desktops need. On the user's first click in it, the client asks for the pointer.lock permission in the trusted zone, hides the cursor and sends a POINTER lock. While locked, move events carry the relative motion in dx, dy, and x, y stay fixed. The reserved key combination always releases the lock, and the client then sends unlock.

Gestures. A node lists the gestures it wants in its gestures prop: tap, double-tap, long-press, pan, pinch, rotate, swipe. The client recognises them with the platform's own recogniser, so they behave natively on every device, and sends GESTURE:

  • phase is start, update, end or cancel; tap, double-tap and long-press report end only.
  • x, y are the centroid; fingers is the number of contacts.
  • pan reports dx, dy and pinch reports scale, both cumulative since start; rotate reports rotation in degrees; swipe reports direction (up, down, left, right). At end, vx, vy give the velocity in pixels per second.
  • Contacts that form a recognised gesture are not also sent as raw touch events. Trackpad pinch and rotate arrive the same way, from the mouse device.
  • Layer 2 components handle scrolling, zooming and taps locally, without any of this.

Gamepads and joysticks

  • GAMEPAD carries the device's full current state. Clients send it when the state changes, at most 120 times a second per device.
  • With mapping: "standard", buttons and axes follow the W3C Gamepad standard mapping: 17 buttons and 4 axes. Other devices report them in device order.
  • Button values run from 0 to 1, so analogue triggers keep their travel. Axes run from −1 to 1. A hat is −1 when centred, or 0–7 clockwise from up.
  • With Sensors 1, GAMEPAD may travel as QUIC datagrams, where the newest state wins; Sensors 1 defines how, with every button change also sent on the stream.

Haptics. HAPTIC drives feedback on a device whose descriptor lists the effect: rumble with strong and weak (0–1), trigger-rumble with left and right (0–1), or vibrate on phones and tablets. duration is in milliseconds, at most 5,000. Clients MAY scale down or ignore effects, following the user's settings.

Accessibility actions. A11Y carries screen reader and switch-access commands: activate, increment, decrement, scroll-up, scroll-down, scroll-left, scroll-right and dismiss. Layer 2 components handle them locally; they reach the server for grids and, with the extensions, for surfaces, where Sensors 1 adds item roles, a pixel-surface overlay and the focus and set actions.

Wheel. dx, dy with unit px, line or page. Scrolling a layer 2 container is local; the wheel goes to the server only over a grid that reports the mouse, or when nothing local can scroll.

Paste. A user paste arrives as PASTE, never through the clipboard channel. A paste larger than max_frame is split: every part but the last carries more: true, and all parts share one seq. The server delivers the paste when its last part arrives and drops a partial paste when the connection drops; the client then sends every part again after the resume (Section 6). INTENT continues the same way. The VT engine cleans a paste and wraps it in bracketed paste (Section 8).

Resize. For a grid with size: fill, the client sends RESIZE with the cells that fit, and the size in pixels when known, whenever its space changes, at most once every 50 ms per grid while the user drags. The server answers with a patch of the grid's cols and rows, and the VT engine passes the size to the program.

Extensions. A device class outside the registry is named name@domain. The client offers it as a capability of the same name and sends EXT events only after the server accepts it; the extension defines type and data. Eye trackers, XR controllers or MIDI devices can join this way without a new protocol version.