opusdev/vector-similarity-api

1# safe-buffer [![travis][travis-image]][travis-url] [![npm][npm-image]][npm-url] [![downloads][downloads-image]][downloads-url] [![javascript style guide][standard-image]][standard-url]2 3[travis-image]: https://img.shields.io/travis/feross/safe-buffer/master.svg4[travis-url]: https://travis-ci.org/feross/safe-buffer5[npm-image]: https://img.shields.io/npm/v/safe-buffer.svg6[npm-url]: https://npmjs.org/package/safe-buffer7[downloads-image]: https://img.shields.io/npm/dm/safe-buffer.svg8[downloads-url]: https://npmjs.org/package/safe-buffer9[standard-image]: https://img.shields.io/badge/code_style-standard-brightgreen.svg10[standard-url]: https://standardjs.com11 12#### Safer Node.js Buffer API13 14**Use the new Node.js Buffer APIs (`Buffer.from`, `Buffer.alloc`,15`Buffer.allocUnsafe`, `Buffer.allocUnsafeSlow`) in all versions of Node.js.**16 17**Uses the built-in implementation when available.**18 19## install20 21```22npm install safe-buffer23```24 25## usage26 27The goal of this package is to provide a safe replacement for the node.js `Buffer`.28 29It's a drop-in replacement for `Buffer`. You can use it by adding one `require` line to30the top of your node.js modules:31 32```js33var Buffer = require('safe-buffer').Buffer34 35// Existing buffer code will continue to work without issues:36 37new Buffer('hey', 'utf8')38new Buffer([1, 2, 3], 'utf8')39new Buffer(obj)40new Buffer(16) // create an uninitialized buffer (potentially unsafe)41 42// But you can use these new explicit APIs to make clear what you want:43 44Buffer.from('hey', 'utf8') // convert from many types to a Buffer45Buffer.alloc(16) // create a zero-filled buffer (safe)46Buffer.allocUnsafe(16) // create an uninitialized buffer (potentially unsafe)47```48 49## api50 51### Class Method: Buffer.from(array)52<!-- YAML53added: v3.0.054-->55 56* `array` {Array}57 58Allocates a new `Buffer` using an `array` of octets.59 60```js61const buf = Buffer.from([0x62,0x75,0x66,0x66,0x65,0x72]);62  // creates a new Buffer containing ASCII bytes63  // ['b','u','f','f','e','r']64```65 66A `TypeError` will be thrown if `array` is not an `Array`.67 68### Class Method: Buffer.from(arrayBuffer[, byteOffset[, length]])69<!-- YAML70added: v5.10.071-->72 73* `arrayBuffer` {ArrayBuffer} The `.buffer` property of a `TypedArray` or74  a `new ArrayBuffer()`75* `byteOffset` {Number} Default: `0`76* `length` {Number} Default: `arrayBuffer.length - byteOffset`77 78When passed a reference to the `.buffer` property of a `TypedArray` instance,79the newly created `Buffer` will share the same allocated memory as the80TypedArray.81 82```js83const arr = new Uint16Array(2);84arr[0] = 5000;85arr[1] = 4000;86 87const buf = Buffer.from(arr.buffer); // shares the memory with arr;88 89console.log(buf);90  // Prints: <Buffer 88 13 a0 0f>91 92// changing the TypedArray changes the Buffer also93arr[1] = 6000;94 95console.log(buf);96  // Prints: <Buffer 88 13 70 17>97```98 99The optional `byteOffset` and `length` arguments specify a memory range within100the `arrayBuffer` that will be shared by the `Buffer`.101 102```js103const ab = new ArrayBuffer(10);104const buf = Buffer.from(ab, 0, 2);105console.log(buf.length);106  // Prints: 2107```108 109A `TypeError` will be thrown if `arrayBuffer` is not an `ArrayBuffer`.110 111### Class Method: Buffer.from(buffer)112<!-- YAML113added: v3.0.0114-->115 116* `buffer` {Buffer}117 118Copies the passed `buffer` data onto a new `Buffer` instance.119 120```js121const buf1 = Buffer.from('buffer');122const buf2 = Buffer.from(buf1);123 124buf1[0] = 0x61;125console.log(buf1.toString());126  // 'auffer'127console.log(buf2.toString());128  // 'buffer' (copy is not changed)129```130 131A `TypeError` will be thrown if `buffer` is not a `Buffer`.132 133### Class Method: Buffer.from(str[, encoding])134<!-- YAML135added: v5.10.0136-->137 138* `str` {String} String to encode.139* `encoding` {String} Encoding to use, Default: `'utf8'`140 141Creates a new `Buffer` containing the given JavaScript string `str`. If142provided, the `encoding` parameter identifies the character encoding.143If not provided, `encoding` defaults to `'utf8'`.144 145```js146const buf1 = Buffer.from('this is a tést');147console.log(buf1.toString());148  // prints: this is a tést149console.log(buf1.toString('ascii'));150  // prints: this is a tC)st151 152const buf2 = Buffer.from('7468697320697320612074c3a97374', 'hex');153console.log(buf2.toString());154  // prints: this is a tést155```156 157A `TypeError` will be thrown if `str` is not a string.158 159### Class Method: Buffer.alloc(size[, fill[, encoding]])160<!-- YAML161added: v5.10.0162-->163 164* `size` {Number}165* `fill` {Value} Default: `undefined`166* `encoding` {String} Default: `utf8`167 168Allocates a new `Buffer` of `size` bytes. If `fill` is `undefined`, the169`Buffer` will be *zero-filled*.170 171```js172const buf = Buffer.alloc(5);173console.log(buf);174  // <Buffer 00 00 00 00 00>175```176 177The `size` must be less than or equal to the value of178`require('buffer').kMaxLength` (on 64-bit architectures, `kMaxLength` is179`(2^31)-1`). Otherwise, a [`RangeError`][] is thrown. A zero-length Buffer will180be created if a `size` less than or equal to 0 is specified.181 182If `fill` is specified, the allocated `Buffer` will be initialized by calling183`buf.fill(fill)`. See [`buf.fill()`][] for more information.184 185```js186const buf = Buffer.alloc(5, 'a');187console.log(buf);188  // <Buffer 61 61 61 61 61>189```190 191If both `fill` and `encoding` are specified, the allocated `Buffer` will be192initialized by calling `buf.fill(fill, encoding)`. For example:193 194```js195const buf = Buffer.alloc(11, 'aGVsbG8gd29ybGQ=', 'base64');196console.log(buf);197  // <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>198```199 200Calling `Buffer.alloc(size)` can be significantly slower than the alternative201`Buffer.allocUnsafe(size)` but ensures that the newly created `Buffer` instance202contents will *never contain sensitive data*.203 204A `TypeError` will be thrown if `size` is not a number.205 206### Class Method: Buffer.allocUnsafe(size)207<!-- YAML208added: v5.10.0209-->210 211* `size` {Number}212 213Allocates a new *non-zero-filled* `Buffer` of `size` bytes.  The `size` must214be less than or equal to the value of `require('buffer').kMaxLength` (on 64-bit215architectures, `kMaxLength` is `(2^31)-1`). Otherwise, a [`RangeError`][] is216thrown. A zero-length Buffer will be created if a `size` less than or equal to2170 is specified.218 219The underlying memory for `Buffer` instances created in this way is *not220initialized*. The contents of the newly created `Buffer` are unknown and221*may contain sensitive data*. Use [`buf.fill(0)`][] to initialize such222`Buffer` instances to zeroes.223 224```js225const buf = Buffer.allocUnsafe(5);226console.log(buf);227  // <Buffer 78 e0 82 02 01>228  // (octets will be different, every time)229buf.fill(0);230console.log(buf);231  // <Buffer 00 00 00 00 00>232```233 234A `TypeError` will be thrown if `size` is not a number.235 236Note that the `Buffer` module pre-allocates an internal `Buffer` instance of237size `Buffer.poolSize` that is used as a pool for the fast allocation of new238`Buffer` instances created using `Buffer.allocUnsafe(size)` (and the deprecated239`new Buffer(size)` constructor) only when `size` is less than or equal to240`Buffer.poolSize >> 1` (floor of `Buffer.poolSize` divided by two). The default241value of `Buffer.poolSize` is `8192` but can be modified.242 243Use of this pre-allocated internal memory pool is a key difference between244calling `Buffer.alloc(size, fill)` vs. `Buffer.allocUnsafe(size).fill(fill)`.245Specifically, `Buffer.alloc(size, fill)` will *never* use the internal Buffer246pool, while `Buffer.allocUnsafe(size).fill(fill)` *will* use the internal247Buffer pool if `size` is less than or equal to half `Buffer.poolSize`. The248difference is subtle but can be important when an application requires the249additional performance that `Buffer.allocUnsafe(size)` provides.250 251### Class Method: Buffer.allocUnsafeSlow(size)252<!-- YAML253added: v5.10.0254-->255 256* `size` {Number}257 258Allocates a new *non-zero-filled* and non-pooled `Buffer` of `size` bytes.  The259`size` must be less than or equal to the value of260`require('buffer').kMaxLength` (on 64-bit architectures, `kMaxLength` is261`(2^31)-1`). Otherwise, a [`RangeError`][] is thrown. A zero-length Buffer will262be created if a `size` less than or equal to 0 is specified.263 264The underlying memory for `Buffer` instances created in this way is *not265initialized*. The contents of the newly created `Buffer` are unknown and266*may contain sensitive data*. Use [`buf.fill(0)`][] to initialize such267`Buffer` instances to zeroes.268 269When using `Buffer.allocUnsafe()` to allocate new `Buffer` instances,270allocations under 4KB are, by default, sliced from a single pre-allocated271`Buffer`. This allows applications to avoid the garbage collection overhead of272creating many individually allocated Buffers. This approach improves both273performance and memory usage by eliminating the need to track and cleanup as274many `Persistent` objects.275 276However, in the case where a developer may need to retain a small chunk of277memory from a pool for an indeterminate amount of time, it may be appropriate278to create an un-pooled Buffer instance using `Buffer.allocUnsafeSlow()` then279copy out the relevant bits.280 281```js282// need to keep around a few small chunks of memory283const store = [];284 285socket.on('readable', () => {286  const data = socket.read();287  // allocate for retained data288  const sb = Buffer.allocUnsafeSlow(10);289  // copy the data into the new allocation290  data.copy(sb, 0, 0, 10);291  store.push(sb);292});293```294 295Use of `Buffer.allocUnsafeSlow()` should be used only as a last resort *after*296a developer has observed undue memory retention in their applications.297 298A `TypeError` will be thrown if `size` is not a number.299 300### All the Rest301 302The rest of the `Buffer` API is exactly the same as in node.js.303[See the docs](https://nodejs.org/api/buffer.html).304 305 306## Related links307 308- [Node.js issue: Buffer(number) is unsafe](https://github.com/nodejs/node/issues/4660)309- [Node.js Enhancement Proposal: Buffer.from/Buffer.alloc/Buffer.zalloc/Buffer() soft-deprecate](https://github.com/nodejs/node-eps/pull/4)310 311## Why is `Buffer` unsafe?312 313Today, the node.js `Buffer` constructor is overloaded to handle many different argument314types like `String`, `Array`, `Object`, `TypedArrayView` (`Uint8Array`, etc.),315`ArrayBuffer`, and also `Number`.316 317The API is optimized for convenience: you can throw any type at it, and it will try to do318what you want.319 320Because the Buffer constructor is so powerful, you often see code like this:321 322```js323// Convert UTF-8 strings to hex324function toHex (str) {325  return new Buffer(str).toString('hex')326}327```328 329***But what happens if `toHex` is called with a `Number` argument?***330 331### Remote Memory Disclosure332 333If an attacker can make your program call the `Buffer` constructor with a `Number`334argument, then they can make it allocate uninitialized memory from the node.js process.335This could potentially disclose TLS private keys, user data, or database passwords.336 337When the `Buffer` constructor is passed a `Number` argument, it returns an338**UNINITIALIZED** block of memory of the specified `size`. When you create a `Buffer` like339this, you **MUST** overwrite the contents before returning it to the user.340 341From the [node.js docs](https://nodejs.org/api/buffer.html#buffer_new_buffer_size):342 343> `new Buffer(size)`344>345> - `size` Number346>347> The underlying memory for `Buffer` instances created in this way is not initialized.348> **The contents of a newly created `Buffer` are unknown and could contain sensitive349> data.** Use `buf.fill(0)` to initialize a Buffer to zeroes.350 351(Emphasis our own.)352 353Whenever the programmer intended to create an uninitialized `Buffer` you often see code354like this:355 356```js357var buf = new Buffer(16)358 359// Immediately overwrite the uninitialized buffer with data from another buffer360for (var i = 0; i < buf.length; i++) {361  buf[i] = otherBuf[i]362}363```364 365 366### Would this ever be a problem in real code?367 368Yes. It's surprisingly common to forget to check the type of your variables in a369dynamically-typed language like JavaScript.370 371Usually the consequences of assuming the wrong type is that your program crashes with an372uncaught exception. But the failure mode for forgetting to check the type of arguments to373the `Buffer` constructor is more catastrophic.374 375Here's an example of a vulnerable service that takes a JSON payload and converts it to376hex:377 378```js379// Take a JSON payload {str: "some string"} and convert it to hex380var server = http.createServer(function (req, res) {381  var data = ''382  req.setEncoding('utf8')383  req.on('data', function (chunk) {384    data += chunk385  })386  req.on('end', function () {387    var body = JSON.parse(data)388    res.end(new Buffer(body.str).toString('hex'))389  })390})391 392server.listen(8080)393```394 395In this example, an http client just has to send:396 397```json398{399  "str": 1000400}401```402 403and it will get back 1,000 bytes of uninitialized memory from the server.404 405This is a very serious bug. It's similar in severity to the406[the Heartbleed bug](http://heartbleed.com/) that allowed disclosure of OpenSSL process407memory by remote attackers.408 409 410### Which real-world packages were vulnerable?411 412#### [`bittorrent-dht`](https://www.npmjs.com/package/bittorrent-dht)413 414[Mathias Buus](https://github.com/mafintosh) and I415([Feross Aboukhadijeh](http://feross.org/)) found this issue in one of our own packages,416[`bittorrent-dht`](https://www.npmjs.com/package/bittorrent-dht). The bug would allow417anyone on the internet to send a series of messages to a user of `bittorrent-dht` and get418them to reveal 20 bytes at a time of uninitialized memory from the node.js process.419 420Here's421[the commit](https://github.com/feross/bittorrent-dht/commit/6c7da04025d5633699800a99ec3fbadf70ad35b8)422that fixed it. We released a new fixed version, created a423[Node Security Project disclosure](https://nodesecurity.io/advisories/68), and deprecated all424vulnerable versions on npm so users will get a warning to upgrade to a newer version.425 426#### [`ws`](https://www.npmjs.com/package/ws)427 428That got us wondering if there were other vulnerable packages. Sure enough, within a short429period of time, we found the same issue in [`ws`](https://www.npmjs.com/package/ws), the430most popular WebSocket implementation in node.js.431 432If certain APIs were called with `Number` parameters instead of `String` or `Buffer` as433expected, then uninitialized server memory would be disclosed to the remote peer.434 435These were the vulnerable methods:436 437```js438socket.send(number)439socket.ping(number)440socket.pong(number)441```442 443Here's a vulnerable socket server with some echo functionality:444 445```js446server.on('connection', function (socket) {447  socket.on('message', function (message) {448    message = JSON.parse(message)449    if (message.type === 'echo') {450      socket.send(message.data) // send back the user's message451    }452  })453})454```455 456`socket.send(number)` called on the server, will disclose server memory.457 458Here's [the release](https://github.com/websockets/ws/releases/tag/1.0.1) where the issue459was fixed, with a more detailed explanation. Props to460[Arnout Kazemier](https://github.com/3rd-Eden) for the quick fix. Here's the461[Node Security Project disclosure](https://nodesecurity.io/advisories/67).462 463 464### What's the solution?465 466It's important that node.js offers a fast way to get memory otherwise performance-critical467applications would needlessly get a lot slower.468 469But we need a better way to *signal our intent* as programmers. **When we want470uninitialized memory, we should request it explicitly.**471 472Sensitive functionality should not be packed into a developer-friendly API that loosely473accepts many different types. This type of API encourages the lazy practice of passing474variables in without checking the type very carefully.475 476#### A new API: `Buffer.allocUnsafe(number)`477 478The functionality of creating buffers with uninitialized memory should be part of another479API. We propose `Buffer.allocUnsafe(number)`. This way, it's not part of an API that480frequently gets user input of all sorts of different types passed into it.481 482```js483var buf = Buffer.allocUnsafe(16) // careful, uninitialized memory!484 485// Immediately overwrite the uninitialized buffer with data from another buffer486for (var i = 0; i < buf.length; i++) {487  buf[i] = otherBuf[i]488}489```490 491 492### How do we fix node.js core?493 494We sent [a PR to node.js core](https://github.com/nodejs/node/pull/4514) (merged as495`semver-major`) which defends against one case:496 497```js498var str = 16499new Buffer(str, 'utf8')500```501 502In this situation, it's implied that the programmer intended the first argument to be a503string, since they passed an encoding as a second argument. Today, node.js will allocate504uninitialized memory in the case of `new Buffer(number, encoding)`, which is probably not505what the programmer intended.506 507But this is only a partial solution, since if the programmer does `new Buffer(variable)`508(without an `encoding` parameter) there's no way to know what they intended. If `variable`509is sometimes a number, then uninitialized memory will sometimes be returned.510 511### What's the real long-term fix?512 513We could deprecate and remove `new Buffer(number)` and use `Buffer.allocUnsafe(number)` when514we need uninitialized memory. But that would break 1000s of packages.515 516~~We believe the best solution is to:~~517 518~~1. Change `new Buffer(number)` to return safe, zeroed-out memory~~519 520~~2. Create a new API for creating uninitialized Buffers. We propose: `Buffer.allocUnsafe(number)`~~521 522#### Update523 524We now support adding three new APIs:525 526- `Buffer.from(value)` - convert from any type to a buffer527- `Buffer.alloc(size)` - create a zero-filled buffer528- `Buffer.allocUnsafe(size)` - create an uninitialized buffer with given size529 530This solves the core problem that affected `ws` and `bittorrent-dht` which is531`Buffer(variable)` getting tricked into taking a number argument.532 533This way, existing code continues working and the impact on the npm ecosystem will be534minimal. Over time, npm maintainers can migrate performance-critical code to use535`Buffer.allocUnsafe(number)` instead of `new Buffer(number)`.536 537 538### Conclusion539 540We think there's a serious design issue with the `Buffer` API as it exists today. It541promotes insecure software by putting high-risk functionality into a convenient API542with friendly "developer ergonomics".543 544This wasn't merely a theoretical exercise because we found the issue in some of the545most popular npm packages.546 547Fortunately, there's an easy fix that can be applied today. Use `safe-buffer` in place of548`buffer`.549 550```js551var Buffer = require('safe-buffer').Buffer552```553 554Eventually, we hope that node.js core can switch to this new, safer behavior. We believe555the impact on the ecosystem would be minimal since it's not a breaking change.556Well-maintained, popular packages would be updated to use `Buffer.alloc` quickly, while557older, insecure packages would magically become safe from this attack vector.558 559 560## links561 562- [Node.js PR: buffer: throw if both length and enc are passed](https://github.com/nodejs/node/pull/4514)563- [Node Security Project disclosure for `ws`](https://nodesecurity.io/advisories/67)564- [Node Security Project disclosure for`bittorrent-dht`](https://nodesecurity.io/advisories/68)565 566 567## credit568 569The original issues in `bittorrent-dht`570([disclosure](https://nodesecurity.io/advisories/68)) and571`ws` ([disclosure](https://nodesecurity.io/advisories/67)) were discovered by572[Mathias Buus](https://github.com/mafintosh) and573[Feross Aboukhadijeh](http://feross.org/).574 575Thanks to [Adam Baldwin](https://github.com/evilpacket) for helping disclose these issues576and for his work running the [Node Security Project](https://nodesecurity.io/).577 578Thanks to [John Hiesey](https://github.com/jhiesey) for proofreading this README and579auditing the code.580 581 582## license583 584MIT. Copyright (C) [Feross Aboukhadijeh](http://feross.org)585