2 Definition Plain text is the lowest common denominator encoding for textual information, using an encoding standard such as ASCII or Unicode, and omitting all metadata, markup and formatting. Plain text is used in contrast with binary in describing the encoding of textual data. In fact, plain text is actually a type of binary encoding, since any computer file is fundamentally a sequence of bits, but plain text is usually seen as a special case and differentiated from other binary encodings, and this is the usage which is preferred here. A plain text file is a special case of a binary file which contains only text represented by alphabetic, numeric and punctuation characters. 'Plain text' is also often used to refer to electronic text files containing markup which only uses standard alphabetic numeric and punctuation characters, in formats such as SGML, XML or HTML. These types of text documents with markup are not however included in this definition of plain text and are dealt with in the Preservation Handbook for Marked-up Textual Data.. For the purposes of this document, the final definition we arrive at then is: Plain text data is electronic data which contains only text represented by alphabetic, numeric and punctuation characters and does not contain markup, annotations or metadata. Description ASCII Most texts described as plain text in the UK (and other predominantly Anglophone environments) use US-ASCII. While this is usually an adequate format for sharing documents between computers and applications when the text contains only simple, modern English prose, there are many cases where characters will not display properly when transferred between computers and applications. US-ASCII cannot represent a number of characters, including the pound, euro and trademark symbols, em- and en-dashes and characters such as thorn from early English prose. Accents and non-english characters are entirely lacking. Unix, Windows and Mac operating systems use different line break characters. This shows that even pieces of software that nominally use the same character sets cannot reliably interchange documents. These problems multiply when trying to cope with more than one character set. Code pages and ISO 8859 A number of regional and country-specific variants of ASCII exist to handle accents and characters used in European languages. These are called code pages, and have been standardised as ISO These are not suitable preservation formats because there is no reliable way to determine which code page a document is encoded in, and no standard way of including it within the file itself. Code pages also restrict the ability of documents and programs to contain and process text with a mixture of languages. The preferred preservation strategy is to convert from code pages and ISO 8859 to Unicode. Such conversion can damage files if the conversion tool translates from the wrong code page, so it is recommended that, unless the text can reliably be validated after conversion, the original also be preserved. EBCDIC EBCDIC is an alternative single-byte encoded character set used only on IBM mainframe computers, and a contemporary of ASCII. It was never widely used outside IBM machines and was never widely internationalised. Many reliable conversion tools exist. The recommendations for preservation are the same as for ASCII. Page 2 of 7 File: presplaintext_d2.doc File Author: File Date: 3/24/2005

3 Unicode "Unicode provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language." ( The aim of Unicode (standardised as the ISO family) is for all of the characters in all of the world's languages, including some languages of the past, to be mapped unambiguously onto a single numerical code. Unicode is certain to become the standard for character encoding in the future, and is already supported by current versions of Mac, Windows and Linux operating systems, along with Java, XML and MS Word. It already offers mappings for all widely spoken or well studied languages, coverage of less well known languages is growing fast and coverage of non-traditional languages (such as sign languages) is planned. US-ASCII and ISO begin with the same sequence as the default Unicode encoding (called UTF-8), which means that a US-ASCII file is also a UTF-8 file. There are other encodings of Unicode (called UTF-16 and UTF-32), but these are mainly used within applications, or where large volumes of text in languages with large character sets are being stored. Fortunately most applications which support Unicode (and all XML compliant applications) read all of these varieties of Unicode. The AHDS does not insist on the deposit of textual resources encoded with Unicode, but expects to recommend its use under most circumstances in the near future. Resources using Unicode are preferred now. It is also preferable in the case of resources which have been encoded in a non- Unicode format for a copy converted to Unicode to be deposited as the preservation copy. If the depositor has been working in a non-unicode format, and has converted the text to Unicode in order to create a version for deposit, then it is important that the non-unicode version be deposited as well. This will help to identify and fix problems which may have silently occurred in the conversion process. It may also be a useful format for a dissemination version of the resource, as it is not yet the case that all platforms and applications are Unicode-aware. Unicode Fonts Because the Unicode character set is very large and new languages are being added in an on-going fashion, many systems do not include a full set of Unicode fonts. Fonts covering portions of the Unicode character set are often included in language packs for various systems. Fonts for some languages may need to be installed manually by the user, especially for less widely used languages. Unlike other areas of Unicode support, this is likely to remain an issue, because the process of including minority, historic and artificial languages in Unicode is on-going. Custom Extensions All of the character sets above have been extended by various applications and users beyond their original use. While such extensions are not necessarily bad, their use makes the files unsuitable for preservation, as it is uncertain whether these extensions will be understood in the future. Unicode has a special method for handling such extensions, called the private use range. If the private use range is used, the meanings of each character within the range needs to be documented. In addition to this, it should be stressed that the successful preservation of the intellectual content of a resource is about more than these purely technical questions. The text needs to be represented in an unambiguous and consistent way. Above all, clear, concise and complete documentation is necessary. Additional Information Creating and Documenting Electronic Texts: a guide to good practice < > Last checked 17/08.05 Binary and text files (Wikipedia entry) < Last checked 17/08.05 Page 3 of 7 File: presplaintext_d2.doc File Author: File Date: 3/24/2005

4 Technical Environment Common Formats Note there are many different encodings for the many languages of the world. It is hoped that Unicode will replace all of these encodings, but at the moment there are different de facto standards in different regions and for different languages. It is not possible to cover all of this variation in this document. Format ASCII (7 and 8-bit encodings) Notes Suitability for preservation depends on method of encoding the text and adequacy of documentation. Other 7 and 8 bit encodings for different languages Conversion to Unicode preferable to create preservation copy. Retention of original may be necessary unless the Unicode version can be validated reliably. Unicode Suitable for preservation Additional Information Text Encoding Initiative: languages and character sets < > Last checked 01/10/2004 Unicode home page < > Last checked 01/10/2004 A brief introduction to code pages and Unicode < > Last checked 01/10/2004 The ISO 8859 Alphabet Soup < > Last checked 01/01/2004 A tutorial on character code issues < > Last checked 24/02/2005 Alan Wood s Unicode resources, < > Last checked 24/02/2005 Unicode Code Charts < > Last checked 24/02/2005 Unifier Converter (Windows) < > Last checked 24/02/2005 Sean Redmond s Greek - Unicode converter multi-platform CGI) < > Last checked 24/02/2005 Page 4 of 7 File: presplaintext_d2.doc File Author: File Date: 3/24/2005

5 Ingest Checklist Only certain formats are suitable for reliable interchange and migration and thus for preservation. For deposit of textual resources which are composed of text files with no or minimal markup, it should first be ascertained whether plain text is the appropriate format for the data. If it is important for the transmission of the intellectual content of the resource that any of the following be encoded, then it is not appropriate for them to be encoded in a plain text file: formatting information the visual appearance of the document the logical structure of the document annotations. It may be possible and appropriate for such information to be held in a separate file, and for a plain text data to be deposited as one component of a resource. The following guidelines should be followed for the ingest of textual data in plain text format: Level 1 (Essential) a manifest of all the components of the resource and its description documentation to be separate from the textual resource explicit documentation of the character set explicit documentation of the markup (if any) explicit differentiation of the markup (if any) from the text explicit documentation of the use of the Unicode private use range (if any) verification of the character set Level 2 (Preferred) a standard character set a standard markup formalism for text structure, annotation and markup. Level 3 (Best Practice) use of Unicode if a non-unicode character set is used, provision of an additional Unicode version for preservation purposes, in addition to the non-unicode version Inform Depositor Where it is necessary to convert to Unicode, the depositer should be asked to validate the migrated text. Page 5 of 7 File: presplaintext_d2.doc File Author: File Date: 3/24/2005

6 Preservation Significant Characteristics The following are the significant characteristics of plain text data for preservation purposes: the stream of alphanumeric characters; the character encoding used (e.g. Unicode UTF-8; Windows Code page 1250, etc); documentation of the languages and scripts which occur in the data; a list of any non-standard characters used, or techniques used to represent characters outside of the range of the encoding employed. It is necessary to assess whether plain text is an acceptable preservation format for the data in a particular case, and, if yes, then whether a relevant type of plain text encoding is employed. Plain text should be used in documents where access only to the text is required, and where it is not necessary to represent the structure, format of the document, and where there are no annotations. Furthermore, it is necessary to ascertain whether a relevant encoding is correctly applied and documented. If all of these conditions are met, then plain text is an acceptable preservation format. The AHDS aims to preserve the sequence of characters and words in a form such that the character set and language can be sufficiently well identified to allow the text to be rendered in a way that is meaningful to humans and to machines. Technique Ensure that information is available on the type of character encoding employed in the data, and that the information is accurate. It is preferable for a preservation copy to be made using Unicode and for the depositors to make such a copy during the construction phase of the resource-building project. If this is not done, long-term preservation of the resource cannot be guaranteed, but will be attempted on a best efforts basis. Most of the above discussion and recommendations apply to the contents of the textual resources themselves. However, it is essential that documentation of these resources be provided, usually in a separate file. It is recommended that documentation files adhere to the same recommendations that apply to textual resources, where this is relevant and possible. This means inter alia that XML, XHTML, SGML and 'plain text' following the above guidelines are preferred to binary and proprietary formats. A description of the file and text type should be provided on the AHDS Deposit form. Any further information about how the documentation can be read or interpreted should also be entered on the deposit form, including information about the language in which it is written if it is not English. Validation of Exported Data It is necessary to verify that the files received by the AHDS conform to the stated text formatting standard. Three techniques are useful for validating different aspects of the data: display a text file on screen. If necessary get an expert in the language to view it, or send an image or PDF file to the depositor for verification; run a script to check for characters outside of the expected range for the documented language or character set; use a program to automatically analyse and recognize language and encoding. Problems and Issues Resource creators using Unicode now should beware that some applications, such as word processors, editors and programs, may take Unicode documents and translate them into some other, probably 8-bit, character set. It may not be possible to translate back, so particular care should Page 6 of 7 File: presplaintext_d2.doc File Author: File Date: 3/24/2005

7 be taken to process Unicode documents only with software which has native support for Unicode, or which can preserve the encoding format. There are code pages for some languages which may be currently in use, but which are at a high risk of not being supported in the future. This is unlikely to be a problem for the major languages of Western Europe and North America which use Latin-based scripts, where compatibility with Unicode is relatively straightforward. However, for many other languages for which conversion to Unicode may be essential for preservation purposes. Data which makes use of these languages must be addressed on a case by case basis. Another potential problem relates to file suffixes. It may be convenient for the creator, user or for an application to have a file name that is meaningful, and this is permitted. It is however essential that information about file and text types should not be available only in file suffixes. Sometimes file suffixes are used to indicate the content of a computer file, or the program it should be associated with. People often speak of a "doc file", meaning a MS Word text file, or a "dot txt file" to refer to some sort of plain text. While this may sometimes be a useful shorthand in the spoken language, it is not a suitable way to identify file types. There are several problems. there are many different types of encoding encompassed by each of the suffixes; not all computer operating systems and applications treat file suffixes as special encodings; some computer systems and software will automatically and silently change or hide file suffixes; in some cases certain file suffixes will stop files working with certain applications; the meanings associated with particular suffixes are arbitrary and not subject to open or international standards, and so they mean different things to different people, operating systems and applications, and may vary in different countries and over time Filenames should be ASCII (or from the lower range of Unicode, which is identical to ASCII), since not all current operating systems yet fully support Unicode. Page 7 of 7 File: presplaintext_d2.doc File Author: File Date: 3/24/2005

The Unicode Standard Version 8.0 Core Specification To learn about the latest version of the Unicode Standard, see http://www.unicode.org/versions/latest/. Many of the designations used by manufacturers

Internationalization 'i18n' for short Deals with charactersets text direction local formats sorting order calendars The Web must be i18n 11 - Unicode CSC309 1 Character Encodings CharacterSet: a collection

International Language Support for Crestron Touchpanels Reference Guide This document was prepared and written by the Technical Documentation department at: Crestron Electronics, Inc. 15 Volvo Drive Rockleigh,

Chapter 2 Encoding Text with a Small Alphabet Given the nature of the Internet, we can break the process of understanding how information is transmitted into two components. First, we have to figure out

Chapter 2 Text Processing with the Command Line Interface Abstract This chapter aims to help demystify the command line interface that is commonly used in UNIX and UNIX-like systems such as Linux and Mac

Preservation Handbook Digital Audio Author Gareth Knight & John McHugh Version 1 Date 25 July 2005 Change History Page 1 of 8 Definition Sound in its original state is a series of air vibrations (compressions

Open EMS Suite O&M Agent Functional Overview Version 1.2 Nokia Siemens Networks 1 (18) O&M Agent The information in this document is subject to change without notice and describes only the product defined

Number Systems and Data Representation CS221 Inside today s computers, data is represented as 1 s and 0 s. These 1 s and 0 s might be stored magnetically on a disk, or as a state in a transistor, core,

Unlocking data creating knowledge Version 3.5 Release Notes November 2006 Introduction These release notes contain general information about the latest version of the Nesstar products and the new features

Signed Binary Arithmetic In the real world of mathematics, computers must represent both positive and negative binary numbers. For example, even when dealing with positive arguments, mathematical operations

MODULE 11 DOCUMENTS ON WEB OBJECTIVE QUESTIONS There are 4 alternative answers to each question. One of them is correct. Pick the correct answer. Do not guess. A key is given at the end of the module for

ASCII Code Data coding Morse code was the first code used for long-distance communication. Samuel F.B. Morse invented it in 1844. This code is made up of dots and dashes (a sort of binary code). It was

SMPP protocol analysis using Wireshark (SMS) Document Purpose Help analyzing SMPP traffic using Wireshark. Give hints about common caveats and oddities of the SMPP protocol and its implementations. Most

Information in a system point of view Information in a system point of view All information processed by computer systems is only a sequence of zeros and ones. Nothing more. In a system point of view ALL

10.2 Sign Magnitude Representation Sign Magnitude is straight-forward method for representing both positive and negative integers. It uses the most significant digit of the digit string to indicate the

Chapter 3 Data Representation in Computers After studying this chapter the student will be able to: *Learn about binary, octal, decimal and hexadecimal number systems *Learn conversions between two different

Tutorial: Exporting Open Green Map Data as CSV and KML files to use in Spreadsheet and GIS Software - For PC users - Version 1.0 Authors: Melissa Reese and Ciprian Samoila Last edit: 04 December 2009 www.opengreenmap.org

Use of Local languages in Computers By completing this module, you will be able to understand and learn the following: Fonts, Different kinds of fonts Enabling regional language support in Windows XP Indic

Tutorial 1 Getting Started with HTML5 HTML and CSS 6 TH EDITION Objectives Explore the history of the Internet, the Web, and HTML Compare the different versions of HTML Study the syntax of HTML tags and

Understanding Unicode and ODBC Data Access DataDirect Connect Series for ODBC Drivers Introduction This document provides a brief background on Unicode, its development, and how it is accommodated by Unicode

Archival Data Format Requirements July 2004 The Royal Library, Copenhagen, Denmark The State and University Library, Århus, Denmark Main author: Steen S. Christensen The Royal Library Postbox 2149 1016

Chapter 9. Documents 9-1 Chapter 9. Documents Supplementary or ancillary reference materials are usually included with archive products to improve their short- and long-term utility. These documents augment

Unicode, the Moving Target Roozbeh Pournader roozbeh@sharif.edu T E X Users Group Meeting, September 2002 Trivandrum, India The outline Introduction What is Unicode? What is OpenType? Other Friends New

Unicode Support in Enterprise COBOL Nick Tindall Stephen Miller Sam Horiguchi August 13, 2003 What is Unicode?! Industry standard for coded character set - defined by Unicode Consortium and ISO! Covers

UTF-16 and C/C++ language TANAKA Keishiro EBLE Markus Contents Why do we need UTF-16 in C/C++? How to support UTF-16 in C/C++ Practical experiences Question period Why do we need UTF-16 in C/C++? Classic

Chapter II Binary Data Representation The atomic unit of data in computer systems is the bit, which is actually an acronym that stands for BInary digit. It can hold only 2 values or states: 0 or 1, true

ELFRING FONTS UPC BAR CODES This package includes five UPC-A and five UPC-E bar code fonts in both TrueType and PostScript formats, a Windows utility, BarUPC, which helps you make bar codes, and Visual

11.1 Internet is a. a local computer network b. a world wide network of computers c. an interconnected network of computers d. a world wide interconnected network of computers which use a common protocol

XML CIS-3152, Spring 2013 Peter C. Chapin Markup Languages Plain text documents with special commands PRO Plays well with version control and other program development tools. Easy to manipulate with scripts

Number Representation Number System :: The Basics We are accustomed to using the so-called decimal number system Ten digits ::,,,3,4,5,6,7,8,9 Every digit position has a weight which is a power of Base

Home > Framework > Content Creation Advice Tips for Authors and Creators of Digital Content: Keep a record of which versions you have made publicly available and where. Use a numbering system that denotes

CUSTOMDP PROJECT THE FINAL DISSEMINATION EVENT Athens October 30 th and 31 st 2003 The CustomDP system as a cross media publishing platform learning objects, metadata and XML The CustomDP system for managing

ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT FORUM ON TAX ADMINISTRATION Guidance Note: Guidance for the Standard Audit File Tax Version 2.0 April 2010 CENTRE FOR TAX POLICY AND ADMINISTRATION

The Inevitable Unicode Project CONTENTS The Inevitable Unicode Project... 1 What is Unicode?... 2 Why should I adopt Unicode?... 2 Planning System Resources for Unicode System... 3 What should you know

Frequently Asked Questions on character sets and languages in MT and MX free format fields Version Final 17 January 2008 Preface The Frequently Asked Questions (FAQs) on character sets and languages that

White Paper PDF Primer PDF What is PDF and what is it good for? How does PDF manage content? How is a PDF file structured? What are its capabilities? What are its limitations? Version: 1.0 Date: October

Character Codes for Modern Computers This lecture covers the standard ways in which characters are stored in modern computers. There are five main classes of characters. 1. Alphabetic characters: upper

1001ICT Introduction To Programming Lecture Notes School of Information and Communication Technology Griffith University Semester 2, 2015 1 3 A First MaSH Program In this section we will describe a very

PHP Intro, Syntax, Variables, Echo, Data Types Introduction PHP scripts are executed on the server. What You Should Already Know Before you continue you should have a basic understanding of the following:

SECTION 5: Finalizing Your Workbook In this section you will learn how to: Protect a workbook Protect a sheet Protect Excel files Unlock cells Use the document inspector Use the compatibility checker Mark

CS 16: Assembly Language Programming for the IBM PC and Compatibles First, a little about you Your name Have you ever worked with/used/played with assembly language? If so, talk about it Why are you taking

Transit NXT Product Guide Service Pack 7 09/2013 Initial overview In this document, you will find out which product variant and which installation type suits your requirements. You are at the right place

Internationalizing the Domain Name System Šimon Hochla, Anisa Azis, Fara Nabilla Internationalize Internet Master in Innovation and Research in Informatics problematic of using non-ascii characters ease

Storing text OK, so that s how a computer stores numbers for us, but what about words, like the words in this book? Easy, a computer stores words as numbers too. What? That s really confusing; a computer

BINARY-CODED DECIMAL (BCD) Definition The binary-coded decimal (BCD) is an encoding for decimal numbers in which each digit is represented by its own binary sequence. Basics In computing and electronic

Release Notes W e are proud to introduce. is an improved and updated version of the unipaas V1.9 release. Read the information in this document to find out more about this latest unipaas version. For more