std.unicode

std.unicode provides strict Unicode primitives for counted byte and text views. Every str role contains valid UTF-8 and may contain U+0000; no text role is NUL-terminated by contract. std.ffi.CStr is the separate terminated byte family.

The higher-level algorithms are generated from Unicode 17.0.0 data.

Contract

Unicode functions operate on caller-owned byte or code-unit storage. _into functions write into caller-provided output and do not allocate. Functions that return ^str or ^[T] allocate owned output and can throw AllocError.

Malformed byte input and insufficient output buffers are reported as UnicodeError. A rune already proves a valid scalar value. Grapheme, normalization, and casing behavior follows Unicode 17.0.0 data; behavior for future Unicode versions is not implied by this page.

The module does not choose locale policy beyond explicit casing.Locale, does not perform filesystem or terminal encoding conversion, and does not define display width.

str Compatibility

Ordinary str.iter() yields rune through an affine, allocation-free cursor; there is no O(n) str.length property. Use std.unicode.utf8.text as the sole checked borrowed bytes-to-text gate. utf8.decode remains an explicit byte-stream operation when malformed input or byte-boundary errors must be reported. Scalar counting is an explicit RuneCursor loop. Use std.unicode.grapheme.iter when user-visible text boundaries matter.

UTF-8

// RFC 0061 target; implementation pending.
import std.bytes;
import std.unicode;

fn main() {
	let data = bytes.Bytes("Aé€");

	assert unicode.utf8.text(data: data) != null;
	assert (try unicode.utf8.decode(data: data, offset: 1)).value == 'é';
}

Use utf8.encode to encode one scalar value into a caller-provided buffer.

// RFC 0061 target; implementation pending.
import std.unicode.utf8;

fn main() {
	var out = [0 as u8; 4];
	let buffer: &[u8] in _ = &out[0..<out.length];
	let n = try utf8.encode(value: '😀', out: buffer);

	assert n == 4;
	assert out[0] == 0xf0 as u8;
}

UTF-16LE

import std.bytes;
import std.unicode;

fn main() {
	let utf16 = try unicode.utf8.to_utf16le(bytes.Bytes("A𐍈"));
	assert unicode.utf16le.valid(utf16);

	let units = try unicode.utf8.to_utf16le(bytes.Bytes("A𐍈"));
	let roundtrip = try unicode.utf16le.to_utf8(units);
	assert roundtrip == "A𐍈";
}

Normalization

unicode.norm implements NFC, NFD, NFKC, and NFKD. normalize returns owned owning ^str; normalize_into writes UTF-8 bytes into caller-owned output.

import std.bytes;
import std.unicode.norm;

fn main() {
	let nfc = try norm.normalize(bytes.Bytes("e\u{0301}"), .Nfc);
	let nfkc = try norm.normalize(bytes.Bytes("\u{fb01}"), .Nfkc);

	assert nfc == "\u{00e9}";
	assert nfkc == "fi";
	assert !try norm.normalized(bytes.Bytes("e\u{0301}"), .Nfc);
}

Grapheme Clusters

unicode.grapheme implements Unicode extended grapheme clusters. It is an explicit, allocation-free cursor over valid str; direct text iteration still yields scalars. Each result lends the cluster text and reports its byte range in the source.

// RFC 0061 target; implementation pending.
import std.unicode.grapheme;

fn main() {
	let text = "a\u{0301}b";
	var count: usize = 0;
	for cluster in grapheme.iter(text: text) {
		if count == 0 {
			assert cluster.text == "a\u{0301}";
			assert cluster.byte_start == 0;
			assert cluster.byte_size == 3 as usize;
		}
		count += 1;
	}
	assert count == 2;
}

Case Conversion And Folding

unicode.casing provides full Unicode case mappings and full default case folding.

Locale-sensitive conversion is explicit through casing.Locale. The supported locale modes are .Default, .Turkish, .Azeri, and .Lithuanian. fold is locale-independent and is intended for caseless matching, not display casing.

import std.bytes;
import std.unicode.casing;

fn main() {
	let upper = try casing.upper(bytes.Bytes("straße"));
	let tr = try casing.upper(bytes.Bytes("i"), .Turkish);
	let folded = try casing.fold(bytes.Bytes("Straße"));

	assert upper == "STRASSE";
	assert tr == "\u{0130}";
	assert folded == "strasse";
}

API Reference

Modules

Module	Description
unicode.utf8	Strict UTF-8 validation, counting, decode, encode, and UTF-16LE conversion.
unicode.utf16le	Strict UTF-16LE validation and UTF-8 conversion.
unicode.norm	Unicode 17.0.0 normalization forms.
unicode.grapheme	Unicode 17.0.0 extended grapheme cluster boundaries.
unicode.casing	Unicode 17.0.0 case conversion and full case folding.

Types

Type	Kind	Description
Decode	struct	Result of decoding one UTF-8 scalar: export let value: rune, export let byte_size: usize.
UnicodeError	error	Invalid for malformed encoded input or an invalid decode offset, Buffer for insufficient output.
norm.Form	enum	Normalization forms: .Nfc, .Nfd, .Nfkc, .Nfkd.
grapheme.Cursor	opaque affine struct	Allocation-free next-only traversal of one borrowed str.
grapheme.Cluster<region Source>	struct	Source-backed cluster with text, byte_start, and byte_size fields.
casing.Locale	enum	Locale mode for case conversion: .Default, .Turkish, .Azeri, .Lithuanian.

Functions

Function	Signature	Description
version	const version = "17.0.0"	Unicode data version used by higher-level algorithms.
scalar	scalar(value: u32) -> rune?	Checked nontrapping construction of a Unicode scalar from untrusted numeric input.
utf8.text	text<region Source>(data: Bytes in Source) -> str? in Source	Validates UTF-8 and lends the exact counted source as text; U+0000 is accepted.
utf8.decode	decode(data: Bytes, offset: usize) throws(UnicodeError) -> Decode	Decodes one scalar at byte offset.
utf8.encode	encode(value: rune, out: &[u8]) throws(UnicodeError) -> usize	Encodes one proven scalar into out; only Buffer can arise.
utf8.to_utf16le	to_utf16le(data: Bytes) throws(UnicodeError | AllocError) -> ^[u16]	Transcodes UTF-8 to owned UTF-16LE code units.
utf8.to_utf16le_into	to_utf16le_into(data: Bytes, out: &[u16]) throws(UnicodeError) -> usize	Transcodes into caller-provided UTF-16LE storage.
utf16le.valid	valid(data: [u16]) -> bool	Returns true when data is strict UTF-16LE.
utf16le.to_utf8	to_utf8(data: [u16]) throws(UnicodeError | AllocError) -> ^str	Transcodes UTF-16LE to an owned UTF-8 string.
utf16le.to_utf8_into	to_utf8_into(data: [u16], out: &[u8]) throws(UnicodeError) -> usize	Transcodes into caller-provided UTF-8 storage.
norm.normalize	normalize(data: Bytes, form: Form) throws(UnicodeError | AllocError) -> ^str	Normalizes UTF-8 to an owned string.
norm.normalize_into	normalize_into(data: Bytes, form: Form, out: &[u8]) throws(UnicodeError) -> usize	Normalizes into caller-owned UTF-8 storage.
norm.normalized	normalized(data: Bytes, form: Form) throws(UnicodeError) -> bool	Returns true if input is already in the selected form.
grapheme.iter	iter<region Source>(text: str in Source) -> ^Cursor in Source	Creates an allocation-free affine cursor yielding source-backed extended grapheme clusters.
casing.lower	lower(data: Bytes, locale: Locale = .Default) throws(UnicodeError | AllocError) -> ^str	Full lowercase conversion.
casing.lower_into	lower_into(data: Bytes, locale: Locale, out: &[u8]) throws(UnicodeError) -> usize	Lowercase conversion into caller-owned UTF-8 storage.
casing.upper	upper(data: Bytes, locale: Locale = .Default) throws(UnicodeError | AllocError) -> ^str	Full uppercase conversion.
casing.upper_into	upper_into(data: Bytes, locale: Locale, out: &[u8]) throws(UnicodeError) -> usize	Uppercase conversion into caller-owned UTF-8 storage.
casing.title	title(data: Bytes, locale: Locale = .Default) throws(UnicodeError | AllocError) -> ^str	Full titlecase scalar mapping.
casing.title_into	title_into(data: Bytes, locale: Locale, out: &[u8]) throws(UnicodeError) -> usize	Titlecase mapping into caller-owned UTF-8 storage.
casing.fold	fold(data: Bytes) throws(UnicodeError | AllocError) -> ^str	Full default Unicode case folding.
casing.fold_into	fold_into(data: Bytes, out: &[u8]) throws(UnicodeError) -> usize	Case folding into caller-owned UTF-8 storage.

Related Modules

Use std.bytes for borrowed byte views, and std.encoding for Base64 and hex text encodings.

Status

This page describes the RFC 0061 accepted target. The checked-in Unicode source still exposes the transitional u32 decode/encode and allocating grapheme helpers until migration. Higher-level algorithms remain pinned to Unicode 17.0.0. For the full exported surface, see Standard Library Inventory.