basant307/AI_Governance_Project

1# fd-slicer2 3[![Build Status](https://travis-ci.org/andrewrk/node-fd-slicer.svg?branch=master)](https://travis-ci.org/andrewrk/node-fd-slicer)4 5Safe `fs.ReadStream` and `fs.WriteStream` using the same fd.6 7Let's say that you want to perform a parallel upload of a file to a remote8server. To do this, we want to create multiple read streams. The first thing9you might think of is to use the `{start: 0, end: 0}` API of10`fs.createReadStream`. This gives you two choices:11 12 0. Use the same file descriptor for all `fs.ReadStream` objects.13 0. Open the file multiple times, resulting in a separate file descriptor14    for each read stream.15 16Neither of these are acceptable options. The first one is a severe bug,17because the API docs for `fs.write` state:18 19> Note that it is unsafe to use `fs.write` multiple times on the same file20> without waiting for the callback. For this scenario, `fs.createWriteStream`21> is strongly recommended.22 23`fs.createWriteStream` will solve the problem if you only create one of them24for the file descriptor, but it will exhibit this unsafety if you create25multiple write streams per file descriptor.26 27The second option suffers from a race condition. For each additional time the28file is opened after the first, it is possible that the file is modified. So29in our parallel uploading example, we might upload a corrupt file that never30existed on the client's computer.31 32This module solves this problem by providing `createReadStream` and33`createWriteStream` that operate on a shared file descriptor and provides34the convenient stream API while still allowing slicing and dicing.35 36This module also gives you some additional power that the builtin37`fs.createWriteStream` do not give you. These features are:38 39 * Emitting a 'progress' event on write.40 * Ability to set a maximum size and emit an error if this size is exceeded.41 * Ability to create an `FdSlicer` instance from a `Buffer`. This enables you42   to provide API for handling files as well as buffers using the same API.43 44## Usage45 46```js47var fdSlicer = require('fd-slicer');48var fs = require('fs');49 50fs.open("file.txt", 'r', function(err, fd) {51  if (err) throw err;52  var slicer = fdSlicer.createFromFd(fd);53  var firstPart = slicer.createReadStream({start: 0, end: 100});54  var secondPart = slicer.createReadStream({start: 100});55  var firstOut = fs.createWriteStream("first.txt");56  var secondOut = fs.createWriteStream("second.txt");57  firstPart.pipe(firstOut);58  secondPart.pipe(secondOut);59});60```61 62You can also create from a buffer:63 64```js65var fdSlicer = require('fd-slicer');66var slicer = FdSlicer.createFromBuffer(someBuffer);67var firstPart = slicer.createReadStream({start: 0, end: 100});68var secondPart = slicer.createReadStream({start: 100});69var firstOut = fs.createWriteStream("first.txt");70var secondOut = fs.createWriteStream("second.txt");71firstPart.pipe(firstOut);72secondPart.pipe(secondOut);73```74 75## API Documentation76 77### fdSlicer.createFromFd(fd, [options])78 79```js80var fdSlicer = require('fd-slicer');81fs.open("file.txt", 'r', function(err, fd) {82  if (err) throw err;83  var slicer = fdSlicer.createFromFd(fd);84  // ...85});86```87 88Make sure `fd` is a properly initialized file descriptor. If you want to89use `createReadStream` make sure you open it for reading and if you want90to use `createWriteStream` make sure you open it for writing.91 92`options` is an optional object which can contain:93 94 * `autoClose` - if set to `true`, the file descriptor will be automatically95   closed once the last stream that references it is closed. Defaults to96   `false`. `ref()` and `unref()` can be used to increase or decrease the97   reference count, respectively.98 99### fdSlicer.createFromBuffer(buffer, [options])100 101```js102var fdSlicer = require('fd-slicer');103var slicer = fdSlicer.createFromBuffer(someBuffer);104// ...105```106 107`options` is an optional object which can contain:108 109 * `maxChunkSize` - A `Number` of bytes. see `createReadStream()`.110   If falsey, defaults to unlimited.111 112#### Properties113 114##### fd115 116The file descriptor passed in. `undefined` if created from a buffer.117 118#### Methods119 120##### createReadStream(options)121 122Available `options`:123 124 * `start` - Number. The offset into the file to start reading from. Defaults125   to 0.126 * `end` - Number. Exclusive upper bound offset into the file to stop reading127   from.128 * `highWaterMark` - Number. The maximum number of bytes to store in the129   internal buffer before ceasing to read from the underlying resource.130   Defaults to 16 KB.131 * `encoding` - String. If specified, then buffers will be decoded to strings132   using the specified encoding. Defaults to `null`.133 134The ReadableStream that this returns has these additional methods:135 136 * `destroy(err)` - stop streaming. `err` is optional and is the error that137   will be emitted in order to cause the streaming to stop. Defaults to138   `new Error("stream destroyed")`.139 140If `maxChunkSize` was specified (see `createFromBuffer()`), the read stream141will provide chunks of at most that size. Normally, the read stream provides142the entire range requested in a single chunk, but this can cause performance143problems in some circumstances.144See [thejoshwolfe/yauzl#87](https://github.com/thejoshwolfe/yauzl/issues/87).145 146##### createWriteStream(options)147 148Available `options`:149 150 * `start` - Number. The offset into the file to start writing to. Defaults to151   0.152 * `end` - Number. Exclusive upper bound offset into the file. If this offset153   is reached, the write stream will emit an 'error' event and stop functioning.154   In this situation, `err.code === 'ETOOBIG'`. Defaults to `Infinity`.155 * `highWaterMark` - Number. Buffer level when `write()` starts returning156   false. Defaults to 16KB.157 * `decodeStrings` - Boolean. Whether or not to decode strings into Buffers158   before passing them to` _write()`. Defaults to `true`.159 160The WritableStream that this returns has these additional methods:161 162 * `destroy()` - stop streaming163 164And these additional properties:165 166 * `bytesWritten` - number of bytes written to the stream167 168And these additional events:169 170 * 'progress' - emitted when `bytesWritten` changes.171 172##### read(buffer, offset, length, position, callback)173 174Equivalent to `fs.read`, but with concurrency protection.175`callback` must be defined.176 177##### write(buffer, offset, length, position, callback)178 179Equivalent to `fs.write`, but with concurrency protection.180`callback` must be defined.181 182##### ref()183 184Increase the `autoClose` reference count by 1.185 186##### unref()187 188Decrease the `autoClose` reference count by 1.189 190#### Events191 192##### 'error'193 194Emitted if `fs.close` returns an error when auto closing.195 196##### 'close'197 198Emitted when fd-slicer closes the file descriptor due to `autoClose`. Never199emitted if created from a buffer.200