Tokenizer

When to use

• Values in the user's own words: tags, invitees, keywords, anything you cannot enumerate in advance.
• A list the user builds by typing. Enter or a separator turns the text into a removable token, and pasting a comma- or line-separated list commits every part at once.
• Light rules on what gets in. max, duplicate refusal and validate turn a bad entry away and say why, while the text stays in the input to fix.
• Lists that are edited often. The chip row is one Tab stop, and Backspace removes the last token, so correcting a list never means hunting for a small target.

When not to use

• When values must come from a list you control. Use Combobox with multiple: its value can only be an item you gave it, which a tokenizer does not guarantee. They share the chip system, so the two look identical; what differs is where values may come from, and a prop should not switch that.
• For a single value or a sentence. Use Textarea for longer text; a token is a short value, not a paragraph.
• For a few fixed options. Use Checkbox or ToggleGroup when the choices are known and few.
• For tokens that carry a status. intent describes the field, never a token. A status belongs on a Chip or a Badge.
• For values edited in place. A token is added or removed, not edited inline.

Installation

pnpm add @tessinaui/ui

Usage

import { Tokenizer } from "@tessinaui/ui";

const [tags, setTags] = useState<string[]>([]);

<Tokenizer label="Tags" value={tags} onChange={setTags} placeholder="Add a tag…" />

Examples

Default

Sizes

The chips read one step down the chip scale, so a token never crowds the field it sits in.

Intents

intent is the state of the field — its border, its supporting text, and aria-invalid on the input. It never tints an individual token: a token is a value, not a status.

Rounded

One prop cascades to the field, its chips and their remove buttons, so rounded="none" leaves no curved corner anywhere.

Separators and paste

Typing a separator commits the part before it and keeps the rest in the input. Pasting text that contains separators commits every part as one batch — one state update and one announcement, not one per token.

Pasting into the middle of an in-progress token appends rather than splices; splicing would need caret bookkeeping for a case nobody does.

Maximum

Past the cap, commits are refused and announced. The input stays editable, so the text you typed is still there to fix.

Validation

validate runs after the empty, max and duplicate checks. Return false for the default message, or a string to supply your own — it is announced verbatim. Inside a pasted batch it sees the tokens committed earlier in that same batch.

Error message

errorMessage replaces supportingText while present, implies intent="error", joins the input's aria-describedby and is announced politely. Use it for problems with the whole field, such as a required list left empty; validate refuses a single token.

On a coloured surface

Inside a <Surface> the field inherits tone="on-color" and passes it to every chip. An explicit tone always wins.

Disabled

Skeleton

Wrap the real component when the tokens already exist — the placeholder is then its exact box by construction. Reach for TokenizerSkeleton only when there is nothing to wrap yet.

RTL

Pass dir="rtl" to mirror the field and the arrow key that enters the chip row. It also switches the built-in strings — each remove button's name and every announcement — to Arabic, with the right plural form for each count. Leave dir unset and the tokenizer inherits the page's direction but keeps its English strings; for another language, pass every strings entry.

API Reference

Tokenizer

Extends the HTML attributes of the wrapping <div>; className styles that wrapper and fieldClassName styles the bordered box inside it.

TokenizerSkeleton

Data attributes

Exports

Tokenizer · TokenizerSkeleton · tokenizerContainerVariants · TOKENIZER_STRINGS · TokenizerStrings · TokenizerProps · TokenizerSkeletonProps · TokenizerSize · TokenizerIntent · TokenizerRounded · TokenizerTone · TokenizerLabelPosition · TokenizerRejection · TokenizerAnnouncements

Accessibility

VoiceOver + Safari: passed 2026-09-14. The input reads "Tags, Press Enter or comma to add, edit text"; each chip's remove button reads "Remove design, button"; Backspace removal announces "research removed. 1 token remaining." NVDA and JAWS are untested.

Tab model. The chip row is one Tab stop, not one per chip — Tab reaches the row, Tab again reaches the input. Without that, a field holding eight tokens would be nine Tab stops before the next control.

Keyboard.

Announcements. A polite live region owned by the field — never by a chip, which would be unmounted by the very removal it needs to announce — reports:

Every string is overridable per entry through strings, or as a function through announcements; under dir="rtl" they default to Arabic. An empty commit and an Escape-clear announce nothing.

Naming. The label names the input. Each remove button is named "Remove" and described by its own token, so it announces as "Remove, design" rather than as an unnamed button. The chip row itself carries no role and no name — the field already has one, and a second would be a second name for one thing.