The Digital Text: From Characters to Foundational Representations

At the heart of Natural Language Processing lies a deceptively simple premise: language, in its digital form, is fundamentally a sequence of discrete symbols. Whether rendered as Unicode code points, ASCII bytes, or specialized encodings, text exists in machines not as continuous meaning but as a string of atomic units—characters, bytes, or tokens—that must be manipulated algorithmically. This chapter establishes that foundational premise: all NLP begins with discrete symbol manipulation. We will examine rigorously the preprocessing steps that transform raw character streams into canonical representations, and the statistical and geometric schemes that convert these symbols into numerical features. The journey from characters to vectors is not merely technical; it reveals the inherent tensions between sparsity and expressiveness, hand-crafted rules and learned patterns, and the curse of dimensionality versus the blessing of generalization. By the end of this chapter, the limitations of sparse, manually engineered features will be clear, setting the stage for the dense, learned representations that dominate modern NLP.

The Discrete Nature of Digital Text

Before any sophisticated processing can occur, text must be encoded into a machine-readable form. The standard today is Unicode, which assigns a unique code point to every character across writing systems. A Python string, for instance, is a sequence of Unicode code points:

```python text = "Café 🍕" print([ord(c) for c in text]) # [67, 97, 102, 233, 32, 127829] ```

But Unicode is only the first layer. At the byte level, text is encoded via UTF-8 (or UTF-16/32), where each code point becomes one or more bytes. For NLP, we typically operate on the decoded Unicode string, but understanding the underlying encoding matters for robustness—malformed UTF-8 can break tokenizers.

The key insight is that text is discrete. Even continuous signals like audio are discretized into frames; text is inherently symbolic. This discreteness allows exact matching, counting, and combinatorial manipulation—the bedrock of early NLP systems. However, it also creates the symbol grounding problem: how do discrete symbols relate to meaning? This chapter explores how we bridge that gap through representation.

Tokenization: Segmenting Meaning from Streams of Characters

Tokenization is the process of splitting a text string into atomic units—tokens—that serve as the basic symbols for downstream processing. While it may seem trivial (e.g., split on whitespace), tokenization is a surprisingly complex and language-dependent task with no universal solution.

Rule-Based and Whitespace Tokenization

The simplest approach splits on whitespace and punctuation:

```python import re text = "Hello, world! This is a test." tokens = re.findall(r'\b\w+\b', text.lower())

['hello', 'world', 'this', 'is', 'a', 'test']

```

This works reasonably for English but fails in many scenarios:

• **Contractions**: `"don't"` → `["don", "t"]` loses semantic unity.
• **Hyphenated words**: `"state-of-the-art"` → `["state", "of", "the", "art"]` vs. single token.
• **URLs/emails**: `"user@example.com"` splits incorrectly.
• **Morphologically rich languages**: German compound nouns like `"Donaudampfschiffahrtselektrizitätenhauptbetriebswerkbauunterbeamtengesellschaft"` (a real word) would be split into many meaningless fragments if split on non-alphanumeric.

Thus, more sophisticated rule-based tokenizers exist. The Moses tokenizer (widely used in machine translation) applies language-specific rules: separate punctuation, handle contractions, normalize characters, etc. For example:

``` Input: "I'm running to the station." Moses: "I 'm running to the station ." ```

Note the separation of `"I'm"` into `"I"` and `"'m"`. This is intentional: it reduces vocabulary size and treats the contraction as two morphemes. But is that linguistically optimal? Not always. For sentiment analysis, `"I'm"` might be better kept as one token because the apostrophe carries emotional nuance (vs. `"I am"`).

Subword Tokenization: Balancing the Vocabulary

The limitations of word-level tokenization—especially the out-of-vocabulary (OOV) problem—motivated the rise of subword tokenization. The core idea: split words into frequent subword units (e.g., `"unhappiness"` → `["un", "happiness"]` or `["un", "happy", "ness"]`). This yields a fixed, moderate vocabulary size while allowing any string to be represented as a sequence of known subwords.

Byte-Pair Encoding (BPE)

Introduced by Sennrich et al. (2016) for neural machine translation, BPE starts with a base vocabulary of characters (or bytes) and iteratively merges the most frequent adjacent pairs.

Algorithm: 1. Initialize vocabulary as all unique characters in the corpus. 2. Count all adjacent symbol pairs in the corpus. 3. Merge the most frequent pair into a new symbol, add to vocabulary. 4. Repeat steps 2–3 for a predetermined number of merges or until vocabulary reaches target size.

*Example*: Suppose we have corpus: `"low low low lower newer newest"`. Initially, each character is a token. After several merges, we might get:

• `"l"` + `"o"` → `"lo"`
• `"lo"` + `"w"` → `"low"`
• `"low"` + `"er"` → `"lower"`
• etc.

Final vocabulary might contain: `['l', 'o', 'w', 'er', 'n', 'e', 'w', 'r']` and merged tokens `['low', 'lo', 'lower', 'new', 'newer', 'newest']`. Then `"lowest"` (not in corpus) would be encoded as `['low', 'e', 's', 't']` or similar, depending on merge order.

Implementation nuance: BPE typically operates on bytes rather than Unicode characters to avoid encoding issues. The popular `subword-nmt` library and Hugging Face's tokenizers implement BPE.

WordPiece

Used in BERT (Devlin et al., 2019), WordPiece is similar but uses a different likelihood objective: it merges pairs that maximize the likelihood of the training data under a unigram language model. Concretely, it chooses the merge that maximizes:

$$\text{score}(x, y) = \frac{\text{freq}(x y)}{\text{freq}(x) \times \text{freq}(y)}$$

This tends to produce more linguistically coherent subwords than BPE's pure frequency.

SentencePiece

A modern framework (Kudo & Richardson, 2018) that treats text as a raw byte stream, eliminating the need for pre-tokenization. It supports both BPE and unigram language model tokenization. Crucially, it can handle any language without pre-processing, because it works at the byte level. For example, Chinese characters are just bytes; SentencePiece learns to merge common byte sequences that correspond to characters or sub-character units.

Subword Tokenization in Practice

Let's compare on a sentence: `"Tokenization is tricky!"`

• **Word-level (whitespace)**: `["Tokenization", "is", "tricky!"]` → vocabulary size huge, OOV for `"tricky"` if not seen.
• **BPE** (with 1k merges): might produce `["Token", "ization", "is", "tric", "ky", "!"]`. `"tricky"` splits into known subwords `"tric"` and `"ky"`.
• **WordPiece** (BERT-base): `["Token", "##ization", "is", "tric", "##ky", "!"]`. Note `"##"` prefix denotes subword continuation.

The `"##"` convention helps models know that `"##ization"` is not a standalone word but a suffix. This is important because the model should treat `"Token"` and `"##ization"` differently than if they were independent tokens.

Trade-offs:

• **Vocabulary size**: Subword vocabularies are typically 10k–100k, vs. millions for word-level.
• **Sequence length**: Subwords increase sequence length (more tokens per sentence), which hurts efficiency for long documents.
• **Linguistic validity**: Subwords may not align with morphological boundaries (e.g., `"##ly"` from `"quickly"`). But they are data-driven, so they capture frequent morphemes.

Practical advice: For morphologically rich languages (Finnish, Turkish), subword tokenization is essential. For English, word-level can work if vocabulary is huge, but subword is more robust. Always evaluate