Understanding Character Encoding: ASCII, Unicode, and UTF-8

Have you ever opened a document and seen gibberish like "café" instead of "café"? That's mojibake—garbled text caused by character encoding mismatches. Understanding character encoding is essential for any developer working with text.

The Problem: Computers Only Understand Numbers

Computers don't understand letters, symbols, or emoji. They only work with numbers—specifically, ones and zeros. To store and transmit text, we need a system that maps characters to numbers. That's what character encoding does.

ASCII: The Original Character Set

ASCII (American Standard Code for Information Interchange) was created in 1963 and became the foundation of text encoding. It uses 7 bits to represent 128 characters:

Some commonly used ASCII codes:

'A' = 65    'a' = 97    '0' = 48
'B' = 66    'b' = 98    '1' = 49
' ' = 32    '\n' = 10   '\t' = 9

The Limitations of ASCII

ASCII's 128 characters were enough for American English, but the world has far more than 128 characters:

• Accented letters: é, ñ, ü, ø
• Other alphabets: Cyrillic (Привет), Greek (Γειά), Arabic (مرحبا)
• Asian languages: Chinese (你好), Japanese (こんにちは), Korean (안녕)
• Symbols: €, £, ¥, ©, ®
• Emoji: 😀, 🎉, 🌍

Various "extended ASCII" encodings added characters 128-255, but each region created its own:

• ISO-8859-1 (Latin-1): Western European
• ISO-8859-5: Cyrillic
• Windows-1252: Microsoft's Western European
• GB2312: Simplified Chinese

This created chaos: the same byte value meant different characters in different encodings. Document created in one region became unreadable in another.

Unicode: One Encoding to Rule Them All

Unicode was created to solve this by assigning a unique number (called a code point) to every character in every language. The format is U+XXXX where XXXX is a hexadecimal number.

'A' = U+0041
'€' = U+20AC
'中' = U+4E2D
'😀' = U+1F600

Unicode currently defines over 150,000 characters covering:

• 161 modern and historic scripts
• Symbol sets (mathematical, musical, technical)
• Emoji (with more added regularly)

Code points range from U+0000 to U+10FFFF (over 1.1 million possible characters). You can browse common blocks, symbols and their code points in our Unicode character tables.

UTF-8: The Practical Implementation

Unicode defines what numbers represent what characters, but we still need to store those numbers as bytes. That's where UTF (Unicode Transformation Format) comes in.

UTF-8 is the dominant encoding on the web (used by over 98% of websites). It's a variable-length encoding:

UTF-8's brilliance:

1. ASCII compatible: ASCII characters use the same single byte, so old ASCII files are valid UTF-8
2. Self-synchronizing: You can find character boundaries by looking at byte patterns
3. No null bytes: Safe for C strings (except for U+0000 itself)
4. Efficient for Latin text: English text is the same size as ASCII

Example encodings:

'A' (U+0041) → 0x41 (1 byte)
'é' (U+00E9) → 0xC3 0xA9 (2 bytes)
'中' (U+4E2D) → 0xE4 0xB8 0xAD (3 bytes)
'😀' (U+1F600) → 0xF0 0x9F 0x98 0x80 (4 bytes)

UTF-8 vs UTF-16 vs UTF-32

UTF-16 uses 2 bytes for most common characters and 4 bytes (surrogate pairs) for others. It's used internally by Windows, Java, and JavaScript.

UTF-32 uses 4 bytes for everything—simple but wasteful. It's rarely used for storage or transmission.

Byte Order Mark (BOM)

Multi-byte encodings can store bytes in different orders:

• Big-endian: Most significant byte first
• Little-endian: Least significant byte first

The BOM is a special character (U+FEFF) at the start of a file that indicates the byte order:

UTF-8 BOM warning: While valid, the UTF-8 BOM can cause issues:

• Shell scripts may fail (interpreter can't read shebang)
• PHP may output characters before headers
• Concatenated files get BOM in the middle

Unless required (some Windows programs expect it), avoid UTF-8 BOM.

Fixing garbled text (mojibake)

Mojibake (from the Japanese for "character transformation") is what you see when bytes written in one encoding are read as if they were another. The original text is usually still there; it is just being displayed through the wrong lens. The classic examples all come from UTF-8 text being interpreted as Windows-1252 or ISO-8859-1:

The telltale sign is a pair or triple of characters where you expected one, usually starting with Ã, â or Â.

Why it happens

• No declared encoding. If a file, HTTP response or database column does not say what encoding it uses, the reader guesses. Windows programs often default to the system code page (Windows-1252 in Western Europe and the Americas), while the web defaults to UTF-8, so the same file looks fine in one place and garbled in another.
• A wrong declaration. A Content-Type: text/html; charset=ISO-8859-1 header on a UTF-8 page, or a MySQL connection opened as latin1 against utf8mb4 tables, tells the reader to use the wrong lens.
• Double encoding. Garbled text gets saved back to the database, then served as UTF-8, producing é and similar. Each round trip adds another layer.

How to fix it

You reverse the misread: turn the garbled string back into the bytes it came from, then decode those bytes with the right encoding. In Python:

broken = "café"
fixed = broken.encode('cp1252').decode('utf-8')   # 'café'

If encode('cp1252') raises an error on a character (Windows-1252 leaves five byte values undefined), try 'latin-1' instead. For double-encoded text, apply the fix twice. The ftfy library ("fixes text for you") automates all of this.

In JavaScript, TextDecoder can read raw bytes in either encoding, and can repair a string that was misread as Latin-1:

// Reading a file that is really Windows-1252
const text = new TextDecoder('windows-1252').decode(buffer);

// Repairing a string whose UTF-8 bytes were read as ISO-8859-1
const bytes = Uint8Array.from(broken, ch => ch.charCodeAt(0));
const fixed = new TextDecoder('utf-8').decode(bytes);

The repair works when every garbled character maps to a single byte (an ISO-8859-1 misread). If the garbage contains €, ™ or other characters from the 80–9F range of Windows-1252, the string was misread as Windows-1252 and those characters need mapping back to their byte values first. On the command line, iconv -f WINDOWS-1252 -t UTF-8 broken.txt > fixed.txt converts a whole file.

If you would rather not write code, paste the garbled text or upload the file into our Text Encoding Detector & Converter. It identifies the likely encoding and repairs the mojibake in the browser.

How to detect the encoding of a file

There is no metadata inside a plain text file that states its encoding, so detection works from the bytes:

1. Byte order mark. EF BB BF at the start means UTF-8, FF FE means UTF-16 little-endian, FE FF means UTF-16 big-endian.
2. Byte patterns. If every byte of 80 or above forms a valid UTF-8 sequence (a lead byte from C2 to F4 followed by the right number of 80–BF continuation bytes), the file is almost certainly UTF-8. Lone high bytes such as E9 or FC that are not followed by continuation bytes point to a single-byte encoding. Bytes in the 80–9F range are typical of Windows-1252 (curly quotes, €, ™) and unused in ISO-8859-1. Regular 00 bytes between ASCII letters suggest UTF-16.
3. Tools. file -i name.txt on Linux and macOS, Python's charset-normalizer or chardet, and the encoding detector all apply these heuristics for you. Detection can never be certain: a short file containing only ASCII is valid in every one of these encodings.

Prevent it in the first place

Declare the encoding wherever text crosses a boundary:

Content-Type: text/html; charset=utf-8

<meta charset="utf-8">

# Always specify encoding when opening files
with open('file.txt', 'r', encoding='utf-8') as f:
    content = f.read()

// Node.js
fs.readFileSync('file.txt', 'utf8');

Best Practices for Developers

1. Use UTF-8 everywhere: Files, databases, APIs, HTML
2. Declare encoding explicitly: Don't rely on defaults
3. Validate input: Check for valid UTF-8 sequences
4. Test with non-ASCII: Include accents, emoji, and various scripts in tests
5. Store encoding metadata: Know what encoding your data uses
6. Be careful with string length: "😀".length in JavaScript is 2 (UTF-16 code units), not 1

Database Configuration

-- MySQL
CREATE DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

-- PostgreSQL
CREATE DATABASE mydb ENCODING 'UTF8';

Note: MySQL's utf8 is actually 3-byte UTF-8 (no emoji). Use utf8mb4 for full UTF-8 support.

Summary

• ASCII: 128 characters, foundation of text encoding
• Unicode: Universal character set with 150,000+ characters
• UTF-8: Variable-length encoding, web standard, ASCII-compatible
• Use UTF-8 for everything unless you have a specific reason not to
• Declare encoding explicitly to avoid mojibake

Understanding character encoding helps you build robust applications that work correctly with text from around the world.

Need to convert between encodings? Try our Unicode Encoder, or use the Text Encoding Detector & Converter to identify a file's encoding and repair garbled text.