Intro

Single-byte character encodings

Multi-byte character encodings

Querying and setting a code page

Tested system locales and their code page

Code pages side by side

Disclaimer:
This document is limited in its viewpoints by focussing on computer files, Windows, languages of Western European origin, and relevance to TSE. It might have no relevance to peripherals, like printers, and especially has no relevance to keyboards.

People use characters to read and write. Computers use numbers. Therefore people assigned numbers to characters so that computers could use "characters".

A character set is a defined group of characters.

A character encoding is a character set for which each character has a defined number.

Historically countries and companies started out creating their own character encodings by the hundreds.
We still live with the remnants of that legacy, and that is a problem.

The following examples are huge simplifications. The rest of this document goes into the details that make the example behaviors happen.

Example:
Suppose an American writes a "√" (square root) character in a file using the Console variant of TSE.
If a non-American opens that file using the Console variant of TSE, then they see the "¹" (superscript one) character instead.
If anyone opens that file using the GUI or Linux variant of TSE, then they see the "û" (small letter u with circumflex) character instead.

Fortunately, over time some character encodings emerged as more commonly used, and nowadays the Unicode character encodings are a world standard.

Unfortunately, TSE does not natively support Unicode.

Example:
If you copy a file or content from the internet, or open a file that was created with a modern editor, then the odds are that the file was created using one of the Unicode encodings. This makes TSE show 2 to 4 inappropriate characters for some of the file's original characters.

In conclusion:
If TSE shows some weird or unexpected characters in a text, then read the rest of this document.

Because computer memory was extremely expensive, the initial trend was to use an as small as possible-sized number per character. Such a number is called a byte . Not coindidentally a byte is also the smallest addressable chunk of computer memory.

A byte size of 8 bits emerged as the most-used standard. Formally such a byte is called an "octet" after the Latin word for 8, but in English everybody calls it a "byte".

Bits can one-on-one be represented by binary numbers . An 8-bit binary number can express the decimal numbers 0 through 255. Byte-sized character numbers therefore enforce a limit of 256 characters.

For languages that started as Western European, this limit could contain all characters necessary for ordinary reading and writing.

The first 128 characters of each such a character encoding were standardized under the name "ASCII", and are the same for all such languages.

Initially each country and company created its own single-byte character encoding, in which the last 128 characters differed per country and language. This let to a huge amount of character encodings.

Nowadays, in Windows, in practice, for single-byte character encodings, that huge amount has been reduced to 3 mainly used ones.

Microsoft refers to each of its own character encodings as a code page, and identifies each code page by a code page number.

(There are other companies with their own "code pages" and their own terminology.)

Microsoft's mainly used single-byte code pages are:

• 437 English (United States)
  Typically used for the Windows command prompt, .bat and .cmd scripts, and the Console version of TSE.
  Without the need for accented characters, this character encoding has lots of (line) drawing-characters.
• 850 Other languages with a Western European origin, including English outside the United States.
  Typically used for the Windows command prompt, .bat and .cmd scripts, and the Console version of TSE.
  This character encoding gave up a part of the (line) drawing-characters to be able to provide accented characters for other languages.
• 1252 A Windows standard for languages with a Western European origin.
  Typically used for Windows GUI applications that still use a single-byte character encoding.
  This might be the single-byte character encoding that supports the most languages.
  Its character encoding gave up all (line) drawing-characters to support even more languages and to add some more commonly used not-alphabetic characters.
  According to Wikipedia Windows-1252 is the most-used single-byte character encoding in the world.

Editing a file that uses a single-byte character encoding will always show a character for each byte-value, but it might be a wrong character if the editor assumes a wrong character encoding.

A file's single-byte character encoding is not stored as a property of that file. An editor opening a file with single-byte character-encoding has no reliable way to determine which code page the file's creator used.

Sometimes the user can make a best guess based on context. Then other editors let the user on-the-fly change the code page they use for showing the file.

With TSE, if the user's guessed code page implements a single-byte character encoding, then the user can close the editor, type "chcp <guessed code page>" on the command prompt, and (re)start the Console variant of TSE to edit the file.

TSE natively only supports single-byte character encodings.

Using TSE to edit a file that uses a multi-byte character encoding will show multiple wrong characters per each multi-byte character.

TSE and single-byte code pages:

• The Console variant of TSE uses the local code page of its command prompt.
  If the local code page is a multi-byte one, then TSE only shows displayable ASCII characters (32 - 126) and a replacement character for each non-ASCII byte.
• The Linux variant of TSE uses code page 1252.
• The GUI variant of TSE uses code page 1252 for its default "Courier New" font and most other fonts.
• The GUI variant of TSE with font "Terminal" will use Windows' global code page if that is a single-byte code page.
  If the global code page is a multi-byte one, then TSE shows displayable ASCII characters (32 - 126) and a replacement character for each non-ASCII byte.

Be aware of this both helpful and misleading Windows command prompt feature:

• Whatever the code page, a Windows prompt command like "dir" will show multi-byte characters correctly when outputted directly to the screen.
• However, if the context has a single-byte codepage, then multi-byte characters in redirected output will be converted to a "?" (question mark).

Nowadays single-byte character encodings still exist for historical reasons, but modern technologies and applications default use one of Unicode's multi-byte character encodings.

For example:

• Windows itself now internally uses UTF-16LE.
• The Windows "system locale" setting has a beta checkmark for "Unicode", which makes "cmd" command prompts use UTF-8.
• Windows Office documents internally use UTF-8.
• The web mostly uses UTF-8.

"UTF-…" are names for Unicode character encodings.
Unicode has 1 character set and 5 main character encodings for that character set.
The character set is updated once a year in September.

A code point is Unicode's administrative number for a character.

Because the code point itself is not a character encoding, a code point is ubiquitously useful for uniquely identifying a character across character encodings, environments, and other contexts.

Unicode "UTF-…" character encodings can also be referred to by a Microsoft code page number:

Unicode name	Code page number	Bytes per character (*)
UTF-8	65001	1 to 4
UTF-16LE	1200	2 or 4
UTF-16BE	1201	2 or 4
UTF-32LE	12000	4
UTF-32BE	12001	4

(*) depending on the character.

GUI TSE and Linux TSE can use extensions to handle "Unicode files".

Unicode files are files in which the characters are encoded using one of the Unicode encodings.

Unicode files can optionally contain an invisible leading character (a "byte order mark", or "BOM"), that indicates which Unicode encoding was used.

Fortunately, even if no byte order mark was used, it is usually easy to with a very high reliability automatically deduce that a file is Unicode encoded and which Unicode encoding was used.

In Windows you can set a code page at 2 levels:

• A global code page for Windows, which makes it the default for all future command prompts and for the large amount of programs that ignore a local code page.
• A local code page for the current command prompt.

Local code page:

A Windows command prompt default uses the global code page, but can set a local code page.

You can query a command prompt's code page with the command "chcp" without parameters:
      chcp

You change a command prompt's code page with the command "chcp <code page number>":
      chcp 1252

Note that there is a huge amount of prompt commands and programs that ignore the local code page and always use the global one.

Console TSE will use the command prompt's code page.

GUI TSE will normally use code page 1252, whatever the global or local code page is.

You cannot change the code page that TSE uses from withing TSE.

Exception to the previous two points:
GUI TSE will use the global code page if GUI TSE's font is "Terminal".

When you start a .bat or .cmd file or a command from TSE, then you have the option to begin the command (file) with a "chcp <code page number>" command to set the desired character encoding for its output.

Again, be aware that lots of commands and programs ignore the local code page and use the global code page instead.

Global code page:

You cannot change Windows' global code page directly.

You have to change the Windows "system locale" setting, which has a "Language (Country)" format, and then Windows secretly sets its global code page based on that.

In Windows 10 you can change the system locale with Settings → Time & Language → Language → Administrative language settings → Change system locale.

Warning:
If you check-mark the "Beta: Use Unicode UTF-8 for worldwide language support" setting, then this overrules the system locale's code page, and sets the global code page to 65001 (Unicode's UTF-8 encoding).
This is an interesting option, but be aware of its impact on all command prompts and spawned command files.

Because many system locales share a same code page, often nothing happens when you change the system locale, but when the system locale changes the code page too, then you will be asked to restart Windows to activate the change.

In the GUI and Console variants of TSE you can query the global code page with the TSE macro statement "Query(CodePage)".
Do not set this internal TSE variable to a different value: It does not affect which code page TSE uses, and sabotages macros from having a convenient way to identify the global code page.

Click a code page's header to view its additional info on Wikipedia.

NBSP is the "no break space" character.
SHY is the "soft hyphen" character.
"₧" is one character, signifying the "pesetas" currency.