basant307/AI_Governance_Project

1# chardet2 3_Chardet_ is a character detection module written in pure JavaScript (TypeScript). Module uses occurrence analysis to determine the most probable encoding.4 5- Packed size is only **22 KB**6- Works in all environments: Node / Browser / Native7- Works on all platforms: Linux / Mac / Windows8- No dependencies9- No native code / bindings10- 100% written in TypeScript11- Extensive code coverage12 13## Installation14 15```16npm i chardet17```18 19## Usage20 21To return the encoding with the highest confidence:22 23```javascript24import chardet from 'chardet';25 26const encoding = chardet.detect(Buffer.from('hello there!'));27// or28const encoding = await chardet.detectFile('/path/to/file');29// or30const encoding = chardet.detectFileSync('/path/to/file');31```32 33To return the full list of possible encodings use `analyse` method.34 35```javascript36import chardet from 'chardet';37chardet.analyse(Buffer.from('hello there!'));38```39 40Returned value is an array of objects sorted by confidence value in descending order41 42```javascript43[44  { confidence: 90, name: 'UTF-8' },45  { confidence: 20, name: 'windows-1252', lang: 'fr' },46];47```48 49In browser, you can use [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array) instead of the `Buffer`:50 51```javascript52import chardet from 'chardet';53chardet.analyse(new Uint8Array([0x68, 0x65, 0x6c, 0x6c, 0x6f]));54```55 56## Working with large data sets57 58Sometimes, when data set is huge and you want to optimize performance (with a trade off of less accuracy),59you can sample only the first N bytes of the buffer:60 61```javascript62const encoding = await chardet.detectFile('/path/to/file', { sampleSize: 32 });63```64 65You can also specify where to begin reading from in the buffer:66 67```javascript68const encoding = await chardet.detectFile('/path/to/file', {69  sampleSize: 32,70  offset: 128,71});72```73 74## Working with strings75 76In both Node.js and browsers, all strings in memory are represented in UTF-16 encoding. This is a fundamental aspect of the JavaScript language specification. Therefore, you cannot use plain strings directly as input for `chardet.analyse()` or `chardet.detect()`. Instead, you need the original string data in the form of a Buffer or Uint8Array.77 78In other words, if you receive a piece of data over the network and want to detect its encoding, use the original data payload, not its string representation. By the time you convert data to a string, it will be in UTF-16 encoding.79 80Note on [TextEncoder](https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder/TextEncoder): By default, it returns a UTF-8 encoded buffer, which means the buffer will not be in the original encoding of the string.81 82## Supported Encodings:83 84- UTF-885- UTF-16 LE86- UTF-16 BE87- UTF-32 LE88- UTF-32 BE89- ISO-2022-JP90- ISO-2022-KR91- ISO-2022-CN92- Shift_JIS93- Big594- EUC-JP95- EUC-KR96- GB1803097- ISO-8859-198- ISO-8859-299- ISO-8859-5100- ISO-8859-6101- ISO-8859-7102- ISO-8859-8103- ISO-8859-9104- windows-1250105- windows-1251106- windows-1252107- windows-1253108- windows-1254109- windows-1255110- windows-1256111- KOI8-R112 113Currently only these encodings are supported.114 115## TypeScript?116 117Yes. Type definitions are included.118 119### References120 121- ICU project http://site.icu-project.org/122