Intro

Single-byte character encodings

Multi-byte character encodings

Querying and setting a code page

Which code page TSE uses to display text

Tested system locales and their global code page

Example code pages and their maximum character size in bytes

Code pages side by side

Disclaimer:
This document is limited in its viewpoints by focussing on computer files, Windows, languages of Western European origin, and relevance to TSE. It might have no relevance to peripherals, like printers, and especially has no relevance to keyboards.

People use characters to read and write. Computers use numbers. Therefore people assigned numbers to characters so that computers could use "characters".

A character set is a defined group of characters.

A character encoding is a character set for which each character has a defined number.

Historically countries and companies started out creating their own character encodings by the hundreds.
We still live with the remnants of that legacy, and that is a problem.

The following examples are huge simplifications. The rest of this document goes into the details that make the example behaviors happen.

Example:
Suppose an American writes a "√" (square root) character in a file using the Console variant of TSE.
If a non-American opens that file using the Console variant of TSE, then they see the "¹" (superscript one) character instead.
If anyone opens that file using the GUI or Linux variant of TSE, then they see the "û" (small letter u with circumflex) character instead.

Fortunately, over time some character encodings emerged as more commonly used, and nowadays the Unicode character encodings are a world standard.

Unfortunately, TSE does not natively support Unicode.

Example:
If you copy a file or content from the internet, or open a file that was created with a modern editor, then the odds are that the file was created using one of the Unicode encodings. This makes TSE show 2 to 4 inappropriate characters for some of the file's original characters.

In conclusion:
If TSE shows some weird or unexpected characters in a text, then read the rest of this document.

Because computer memory was extremely expensive, the initial trend was to use an as small as possible-sized number per character. Such a number is called a byte . Not coincidentally a byte is also the smallest addressable chunk of computer memory.

A byte size of 8 bits emerged as the most-used standard. Formally such a byte is called an "octet" after the Latin word for 8, but in English everybody calls it a "byte".

Bits can one-on-one be represented by binary numbers . An 8-bit binary number can express the decimal numbers 0 through 255. Byte-sized character numbers therefore enforce a limit of 256 characters.

For languages that started as Western European, this limit could contain all characters necessary for ordinary reading and writing.

The first 128 characters of each such a character encoding were standardized under the name "ASCII", and are the same for all such languages.

Initially each country and company created its own single-byte character encoding, in which the last 128 characters differed per country and language. This let to a huge amount of character encodings.

Nowadays, in Windows, in practice, for single-byte character encodings, that huge amount has been reduced to 3 mainly used ones.

Microsoft refers to each of its own character encodings as a code page, and identifies each code page by a code page number.

(There are other companies with their own "code pages" and their own terminology.)

Microsoft's mainly used single-byte code pages are:

• 437 English (United States)
  Typically used for the Windows command prompt, .bat and .cmd scripts, and the Console version of TSE.
  Without the need for accented characters, this character encoding has lots of (line) drawing-characters.
• 850 Other languages with a Western European origin, including English outside the United States.
  Typically used for the Windows command prompt, .bat and .cmd scripts, and the Console version of TSE.
  This character encoding gave up a part of the (line) drawing-characters to be able to provide accented characters for other languages.
• 1252 A Windows standard for languages with a Western European origin.
  Typically used for Windows GUI applications that still use a single-byte character encoding.
  This might be the single-byte character encoding that supports the most languages.
  Its character encoding gave up all (line) drawing-characters to support even more languages and to add some more commonly used not-alphabetic characters.
  According to Wikipedia Windows-1252 is the most-used single-byte character encoding in the world.

Editing a file that uses a single-byte character encoding will always show a character for each byte-value, but it might be a wrong character if the editor assumes a wrong character encoding.

A file's single-byte character encoding is not stored as a property of that file. An editor opening a file with single-byte character-encoding has no reliable way to determine which code page the file's creator used.

Sometimes the user can make a best guess based on context. Then other editors let the user on-the-fly change the code page they use for showing the file.

With the Console variant of TSE, if the user's guessed code page implements a single-byte character encoding, then the user can close the editor, type "chcp <guessed code page>" on the command prompt, and (re)start Console TSE to edit the file.

TSE natively only supports single-byte character encodings.

Using TSE to edit a file that uses a multi-byte character encoding will show multiple wrong characters per each multi-byte character.

TSE and single-byte code pages:

• The Console variant of TSE uses the local code page of its command prompt.
  If the local code page is a multi-byte one, then TSE only shows displayable ASCII characters (32 - 126) and a replacement character for each non-ASCII byte.
• The GUI variant of TSE uses code page 1252 for its default "Courier New" font and most other fonts.
• The GUI variant of TSE with font "Terminal" will use Windows' global code page if that is a single-byte code page.
  If the global code page is a multi-byte one, then TSE shows displayable ASCII characters (32 - 126) and a replacement character for each non-ASCII byte.
• The Linux variant of TSE uses code page 1252.

Be aware of this both helpful and misleading Windows command prompt feature:

• If its code page limits a command prompt to a one-byte character set, a Windows prompt command like "dir" will still show names with multi-byte characters correctly when outputted directly to the screen.
• However, if that same output is redirected, then multi-byte characters in a name will be converted to a "?" (question mark).

Nowadays single-byte character encodings still exist for historical reasons, but modern technologies and applications default use one of Unicode's multi-byte character encodings.

For example:

• Windows nowadays uses UTF-16LE for its APIs.
• The Windows "system locale" setting has a beta checkmark for "Unicode", which makes "cmd" command prompts use UTF-8.
• Microsoft Office documents internally use UTF-8.
• The web mostly uses UTF-8.

"UTF-…" are names for Unicode character encodings.
Unicode has 1 character set and 5 main character encodings for that character set.
The character set is updated once a year in September.
It takes a long time for operating systems, browsers, editors, et cetera to implement a new Unicode version.

A code point is Unicode's administrative number for a character.

Because a Unicode code point is not a character encoding, Unicode code points are ubiquitously used for uniquely identifying a character across character encodings, environments, and other contexts.

Unicode "UTF-…" character encodings can also be referred to by a Microsoft code page number:

Unicode name	Code page number	Bytes per character (*)
UTF-8	65001	1 to 4
UTF-16LE	1200	2 or 4
UTF-16BE	1201	2 or 4
UTF-32LE	12000	4
UTF-32BE	12001	4

(*) depending on the character.

But note: Windows prompts and APIs do not recognize code pages 1200, 1201, 12000, and 12001. These code pages are only used by .net and in applications.

GUI TSE and Linux TSE can use extensions to handle "Unicode files".

Unicode files are files in which the characters are encoded using one of the Unicode encodings.

Unicode files can optionally contain an invisible leading character, named a "byte order mark", a.k.a. "BOM", that indicates which Unicode encoding is used.

Fortunately, even if no byte order mark is used, it is usually easy to automatically detect with a very high reliability whether a file is Unicode-encoded, and which Unicode encoding is used.

In Windows you can set a code page at 2 levels:

• A global code page for Windows, which makes it the default for all future command prompts and for the large amount of programs that ignore a local code page.
• A local code page for the current command prompt.

Local code page:

A Windows command prompt default uses the global code page, but can be set to a local code page.

You can query a command prompt's local code page with the command "chcp" without parameters:
      chcp

You change a command prompt's local code page with the command "chcp <code page number>":
      chcp 1252

Note that there is a huge amount of prompt commands and programs that ignore the local code page and always use the global one.

When you start a .bat or .cmd file or a command from TSE, then you have the option to begin the command (file) with a "chcp <code page number>" command to set the desired character encoding for its output.

Again, be aware that lots of commands and programs ignore the local code page and use the global code page instead.

Be aware of this both helpful and misleading Windows command prompt feature:

• If the local code page is a single-byte one, the "dir" command will still show names with multi-byte characters correctly if displayed directly to the screen.
• However, if that same output is redirected, then multi-byte characters are converted to a "?" (question mark).

Global code page:

You cannot change Windows' global code page directly.

You have to change the Windows "system locale" setting, which has a "Language (Country)" format, and then Windows implicitly sets its global code page based on that.

In Windows 10 you can change the system locale with: Settings → Time & Language → Language → Administrative language settings → Change system locale.

In Windows 11 you can change the system locale with: Settings → Time & Language → Beside "Windows display language" select the right-most ˅ → Select a "Language for non-Unicode programs".

Warning:
If you check-mark the "Beta: Use Unicode UTF-8 for worldwide language support" setting, then this overrules the system locale's code page, and sets the global code page to 65001 (Unicode's UTF-8 encoding).
This is an interesting option, but be aware of its impact on all command prompts and spawned command files.

Because many system locales share a same global code page, often nothing happens when you change the system locale, but when the system locale changes the code page too, then Windows 10 will ask you to restart Windows to activate the change.
Windows 11 will not ask, and sometimes does not need to be restarted, and sometimes does. I always do so to be safe.

Console TSE

• If the local code page is a single-byte one:
  • TSE displays text using the local code page.
• Else:
  • TSE displays text using ASCII for the lower 128 characters and displays a default character instead of the upper 128 characters.

GUI TSE

• If the font is Terminal:
  • If the global code page is a single-byte one:
    • TSE displays text using the global code page.
  • Else:
    • If the global code page is 65001 (UTF-8):
      • TSE displays text using ASCII for the lower 128 characters and displays a default character instead of the upper 128 characters.
    • Else:
      • TSE displays text using code page 850 (with a few characters blank or styled differently).
• Else:
  • TSE displays text using code page 1252.

Linux TSE

• TSE displays text using code page 1252.

Notes:

• There always is a local code page: Default it is the global one.
• You cannot change the code page that TSE uses from within TSE.
• In the Windows variants of TSE you can query the global code page with the TSE macro statement "Query(CodePage)".
  In older TSE versions you could change this internal TSE variable, which did nothing except sabotage running macros.
• Here the extreme exception was ignored, that if GUI TSE has font Terminal with font size 5, then TSE displays text using code page 437.

System Locale	Global Code Page
Danish (Denmark)	850
Dutch (Belgium)	850
Dutch (Netherlands)	850
English (Australia)	850
English (Canada)	850
English (New Zeeland)	850
English (United Kingdom)	850
English (United States)	437
French (Belgium)	850
French (Canada)	850
French (France)	850
German (Austria)	850
German (Germany)	850
German (Liechtenstein)	850
German (Luxembourg)	850
German (Switzerland)	850
Norwegian (Bokmål)	850
Norwegian (Nynorsk)	850
Portuguese (Brazil)	850
Portuguese (Portugal)	850
Spanish (Chile)	850
Spanish (Latin America)	850
Spanish (Mexican)	850
Spanish (Spain)	850
Spanish (United States)	850

This is an example table.
It was specific to my specific Windows computer at a specific date.
To list the available code pages on your Windows computer use the CodePageList tool.

This table is sorted on Max Char Size, and within Max Char Size on Code Page.

Code Page	Max Char Size	Description
37	1	(IBM EBCDIC - U.S./Canada)
437	1	(OEM - United States)
500	1	(IBM EBCDIC - International)
708	1	(Arabic - ASMO)
720	1	(Arabic - Transparent ASMO)
737	1	(OEM - Greek 437G)
775	1	(OEM - Baltic)
850	1	(OEM - Multilingual Latin I)
852	1	(OEM - Latin II)
855	1	(OEM - Cyrillic)
857	1	(OEM - Turkish)
858	1	(OEM - Multilingual Latin I + Euro)
860	1	(OEM - Portuguese)
861	1	(OEM - Icelandic)
862	1	(OEM - Hebrew)
863	1	(OEM - Canadian French)
864	1	(OEM - Arabic)
865	1	(OEM - Nordic)
866	1	(OEM - Russian)
869	1	(OEM - Modern Greek)
870	1	(IBM EBCDIC - Multilingual/ROECE (Latin-2))
874	1	(ANSI/OEM - Thai)
875	1	(IBM EBCDIC - Modern Greek)
1026	1	(IBM EBCDIC - Turkish (Latin-5))
1047	1	(IBM EBCDIC - Latin-1/Open System)
1140	1	(IBM EBCDIC - U.S./Canada (37 + Euro))
1141	1	(IBM EBCDIC - Germany (20273 + Euro))
1142	1	(IBM EBCDIC - Denmark/Norway (20277 + Euro))
1143	1	(IBM EBCDIC - Finland/Sweden (20278 + Euro))
1144	1	(IBM EBCDIC - Italy (20280 + Euro))
1145	1	(IBM EBCDIC - Latin America/Spain (20284 + Euro))
1146	1	(IBM EBCDIC - United Kingdom (20285 + Euro))
1148	1	(IBM EBCDIC - International (500 + Euro))
1149	1	(IBM EBCDIC - Icelandic (20871 + Euro))
1250	1	(ANSI - Central Europe)
1251	1	(ANSI - Cyrillic)
1252	1	(ANSI - Latin I)
1253	1	(ANSI - Greek)
1254	1	(ANSI - Turkish)
1255	1	(ANSI - Hebrew)
1256	1	(ANSI - Arabic)
1257	1	(ANSI - Baltic)
1258	1	(ANSI/OEM - Viet Nam)
10000	1	(MAC - Roman)
10004	1	(MAC - Arabic)
10005	1	(MAC - Hebrew)
10006	1	(MAC - Greek I)
10007	1	(MAC - Cyrillic)
10010	1	(MAC - Romania)
10017	1	(MAC - Ukraine)
10021	1	(MAC - Thai)
10029	1	(MAC - Latin II)
10079	1	(MAC - Icelandic)
10081	1	(MAC - Turkish)
10082	1	(MAC - Croatia)
20105	1	(IA5 IRV International Alphabet No.5)
20106	1	(IA5 German)
20107	1	(IA5 Swedish)
20108	1	(IA5 Norwegian)
20127	1	(US-ASCII)
20269	1	(ISO 6937 Non-Spacing Accent)
20273	1	(IBM EBCDIC - Germany)
20277	1	(IBM EBCDIC - Denmark/Norway)
20278	1	(IBM EBCDIC - Finland/Sweden)
20280	1	(IBM EBCDIC - Italy)
20284	1	(IBM EBCDIC - Latin America/Spain)
20285	1	(IBM EBCDIC - United Kingdom)
20290	1	(IBM EBCDIC - Japanese Katakana Extended)
20297	1	(IBM EBCDIC - France)
20420	1	(IBM EBCDIC - Arabic)
20423	1	(IBM EBCDIC - Greek)
20424	1	(IBM EBCDIC - Hebrew)
20833	1	(IBM EBCDIC - Korean Extended)
20838	1	(IBM EBCDIC - Thai)
20866	1	(Russian - KOI8)
20871	1	(IBM EBCDIC - Icelandic)
20880	1	(IBM EBCDIC - Cyrillic (Russian))
20905	1	(IBM EBCDIC - Turkish)
20924	1	(IBM EBCDIC - Latin-1/Open System (1047 + Euro))
21025	1	(IBM EBCDIC - Cyrillic (Serbian, Bulgarian))
21027	1	(Ext Alpha Lowercase)
21866	1	(Ukrainian - KOI8-U)
28591	1	(ISO 8859-1 Latin I)
28592	1	(ISO 8859-2 Central Europe)
28593	1	(ISO 8859-3 Latin 3)
28594	1	(ISO 8859-4 Baltic)
28595	1	(ISO 8859-5 Cyrillic)
28596	1	(ISO 8859-6 Arabic)
28597	1	(ISO 8859-7 Greek)
28598	1	(ISO 8859-8 Hebrew: Visual Ordering)
28599	1	(ISO 8859-9 Latin 5)
28603	1	(ISO 8859-13 Latin 7)
28605	1	(ISO 8859-15 Latin 9)
38598	1	(ISO 8859-8 Hebrew: Logical Ordering)
932	2	(ANSI/OEM - Japanese Shift-JIS)
936	2	(ANSI/OEM - Simplified Chinese GBK)
949	2	(ANSI/OEM - Korean)
950	2	(ANSI/OEM - Traditional Chinese Big5)
1361	2	(Korean - Johab)
10001	2	(MAC - Japanese)
10002	2	(MAC - Traditional Chinese Big5)
10003	2	(MAC - Korean)
10008	2	(MAC - Simplified Chinese GB 2312)
20000	2	(CNS - Taiwan)
20001	2	(TCA - Taiwan)
20002	2	(Eten - Taiwan)
20003	2	(IBM5550 - Taiwan)
20004	2	(TeleText - Taiwan)
20005	2	(Wang - Taiwan)
20261	2	(T.61)
20932	2	(JIS X 0208-1990 & 0212-1990)
20936	2	(Simplified Chinese GB2312)
51949	2	(EUC-Korean)
55000	2	(SMS GSM 7bit)
55001	2	(SMS GSM 7bit Spanish)
55002	2	(SMS GSM 7bit Portuguese)
55003	2	(SMS GSM 7bit Turkish)
55004	2	(SMS GSM 7bit Greek)
54936	4	(GB18030 Simplified Chinese)
57002	4	(ISCII - Devanagari)
57003	4	(ISCII - Bengali)
57004	4	(ISCII - Tamil)
57005	4	(ISCII - Telugu)
57006	4	(ISCII - Assamese)
57007	4	(ISCII - Odia (Oriya))
57008	4	(ISCII - Kannada)
57009	4	(ISCII - Malayalam)
57010	4	(ISCII - Gujarati)
57011	4	(ISCII - Punjabi (Gurmukhi))
65001	4	(UTF-8)
50220	5	(ISO-2022 Japanese with no halfwidth Katakana)
50221	5	(ISO-2022 Japanese with halfwidth Katakana)
50222	5	(ISO-2022 Japanese JIS X 0201-1989)
50225	5	(ISO-2022 Korean)
50227	5	(ISO-2022 Simplified Chinese)
50229	5	(ISO-2022 Traditional Chinese)
52936	5	(HZ-GB2312 Simplified Chinese)
65000	5	(UTF-7)

Click a code page's header to view its additional info on Wikipedia.

SP is the space character.
NBSP is the "no break space" character.
SHY is the "soft hyphen" character.
"₧" is one character, signifying the "pesetas" currency.