Tokenizer
A free-text multi-value input — type, press Enter, and the text becomes a removable chip. Not constrained to a list; Combobox is the control for that.
Playground
Installation
pnpm add @tessinaui/uiUsage
import { Tokenizer } from "@tessinaui/ui";const [tags, setTags] = useState<string[]>([]);
<Tokenizer label="Tags" value={tags} onChange={setTags} placeholder="Add a tag…" />Examples
Default
Sizes
The chips read one step down the chip scale, so a token never crowds the field it sits in.
Intents
intent is the state of the field — its border, its supporting text, and aria-invalid on the input. It never tints an individual token: a token is a value, not a status.
Rounded
One prop cascades to the field, its chips and their remove buttons, so rounded="none" leaves no curved corner anywhere.
Separators and paste
Typing a separator commits the part before it and keeps the rest in the input. Pasting text that contains separators commits every part as one batch — one state update and one announcement, not one per token.
Pasting into the middle of an in-progress token appends rather than splices; splicing would need caret bookkeeping for a case nobody does.
Maximum
Past the cap, commits are refused and announced. The input stays editable, so the text you typed is still there to fix.
Validation
validate runs after the empty, max and duplicate checks. Return false for the default message, or a string to supply your own — it is announced verbatim. Inside a pasted batch it sees the tokens committed earlier in that same batch.
On a coloured surface
Inside a <Surface> the field inherits tone="on-color" and passes it to every chip. An explicit tone always wins.
Disabled
Skeleton
Wrap the real component when the tokens already exist — the placeholder is then its exact box by construction. Reach for TokenizerSkeleton only when there is nothing to wrap yet.
RTL
API Reference
Tokenizer
Extends the HTML attributes of the wrapping <div>; className styles that wrapper and fieldClassName styles the bordered box inside it.
| Prop | Type | Default | Description |
|---|---|---|---|
value | string[] | — | The tokens, controlled. The in-progress text is never part of it |
defaultValue | string[] | [] | Starting tokens when uncontrolled |
onChange | (value: string[]) => void | — | Called with the whole next list after every commit and removal, never with a rejected candidate |
size | "xs" | "sm" | "md" | "lg" | "xl" | "md" | Density; the chips read one step down |
intent | "none" | "error" | "warning" | "success" | "info" | "none" | Semantic state of the field, never of a token |
rounded | "none" | "sm" | "md" | "lg" | "full" | "full" | Corner treatment; the chips follow it |
tone | "default" | "on-color" | inherited from <Surface> | Colour recipe for a coloured or dark surface |
disabled | boolean | false | Dims the field once — its chips do not dim again — disables every chip and remove button, and sets aria-disabled on the root |
dir | "ltr" | "rtl" | "ltr" | Text direction; also mirrors which arrow enters the chip row |
label | string | — | Visible label, associated with the input |
labelPosition | "outside-top" | "outside-left" | "outside-top" | Where the label sits. There is no inside: the field grows as chips wrap |
supportingText | string | — | Helper row under the field, wired to the input's aria-describedby |
placeholder | string | — | Placeholder, hidden once there is at least one token |
max | number | — | Hard cap on the number of tokens |
allowDuplicates | boolean | false | Let the same string in twice. Comparison is exact and case-sensitive |
validate | (token: string, tokens: string[]) => boolean | string | — | Gate a candidate. A returned string refuses and is announced verbatim |
separators | RegExp | null | /[,\n]/ | What splits typed and pasted text. null turns splitting off |
removeLabel | string | "Remove" | Accessible name of each remove button; the token is added via aria-describedby |
announcements | Partial<TokenizerAnnouncements> | — | Override any live-region string, for localisation |
required | boolean | — | Asterisk on the label, aria-required on the input |
id | string | generated | Id for the input; the label's htmlFor follows it |
fieldClassName | string | — | Class for the bordered field box |
TokenizerSkeleton
| Prop | Type | Default | Description |
|---|---|---|---|
size | "xs" | "sm" | "md" | "lg" | "xl" | "md" | Same scale as the Tokenizer |
rounded | "none" | "sm" | "md" | "lg" | "full" | "full" | Same map as the Tokenizer |
showLabel | boolean | false | Draw the label row |
labelPosition | "outside-top" | "outside-left" | "outside-top" | Mirrors the component |
showSupportingText | boolean | false | Draw the supporting-text row |
tokens | number | 3 | How many token placeholders to draw |
tokenChars | number | 6 | Length of a real token; the bar is sized from it |
dir | "ltr" | "rtl" | "ltr" | Text direction |
Data attributes
| Attribute | On | Value |
|---|---|---|
data-slot="tokenizer" | wrapper | — |
data-slot="tokenizer-field" | the bordered box | — |
data-slot="tokenizer-chips" | the chip row | — |
data-slot="tokenizer-input" | the text input | — |
data-slot="tokenizer-status" | the live region | — |
data-size | data-intent | data-tone | wrapper | the resolved value |
data-disabled | wrapper | present when disabled |
Exports
Tokenizer · TokenizerSkeleton · tokenizerContainerVariants · TokenizerProps · TokenizerSkeletonProps · TokenizerSize · TokenizerIntent · TokenizerRounded · TokenizerTone · TokenizerLabelPosition · TokenizerRejection · TokenizerAnnouncements
Accessibility
Tab model. The chip row is one Tab stop, not one per chip — Tab reaches the row, Tab again reaches the input. Without that, a field holding eight tokens would be nine Tab stops before the next control.
Keyboard.
| Key | Where | What it does |
|---|---|---|
Enter | input | Commits the trimmed text. An empty input is a no-op and is not prevented, so Enter still submits an enclosing form |
Backspace | input, empty | Removes the last token. Focus stays in the input |
Escape | input, with text | Clears the in-progress text and stops there, so an enclosing dialog does not also close |
ArrowLeft | input, caret at start | Moves focus to the last chip. ArrowRight in RTL |
ArrowLeft | ArrowRight | chip row | Move between chips. Home and End jump to the ends |
ArrowRight | chip row, last item | Returns focus to the input. ArrowLeft in RTL. The row does not wrap |
Backspace | Delete | a focused chip | Removes that chip and hands focus to the next one, then the previous, then the input |
Announcements. A polite live region owned by the field — never by a chip, which would be unmounted by the very removal it needs to announce — reports:
| Event | Announcement |
|---|---|
| one token added | "{token} added. {n} tokens." |
| several added at once | "{N} tokens added. {n} total." |
| token removed | "{token} removed. {n} tokens remaining." |
| duplicate refused | "{token} is already in the list." |
validate refused | "{token} is not a valid entry.", or your string verbatim |
max reached | "Limit reached. {max} tokens maximum." |
Every string is overridable through announcements. An empty commit and an Escape-clear announce nothing.
Naming. The label names the input. Each remove button is named "Remove" and described by its own token, so it announces as "Remove, design" rather than as an unnamed button. The chip row itself carries no role and no name — the field already has one, and a second would be a second name for one thing.
When to use which
Reach for Tokenizer when the values are the user's own words: tags, invitees, keywords, anything you cannot enumerate in advance.
Reach for Combobox with multiple when the values must come from a list you control. Its value can only ever be an item you gave it, which is exactly the guarantee a tokenizer does not make.
They share the chip system — the same scale, the same remove contract — so a row of tokens looks identical either way. What differs is where the values are allowed to come from, and that is not something a prop should switch between.