basant307/AI_Governance_Project

1# cacache [![npm version](https://img.shields.io/npm/v/cacache.svg)](https://npm.im/cacache) [![license](https://img.shields.io/npm/l/cacache.svg)](https://npm.im/cacache) [![Travis](https://img.shields.io/travis/npm/cacache.svg)](https://travis-ci.org/npm/cacache) [![AppVeyor](https://ci.appveyor.com/api/projects/status/github/npm/cacache?svg=true)](https://ci.appveyor.com/project/npm/cacache) [![Coverage Status](https://coveralls.io/repos/github/npm/cacache/badge.svg?branch=latest)](https://coveralls.io/github/npm/cacache?branch=latest)2 3[`cacache`](https://github.com/npm/cacache) is a Node.js library for managing4local key and content address caches. It's really fast, really good at5concurrency, and it will never give you corrupted data, even if cache files6get corrupted or manipulated.7 8On systems that support user and group settings on files, cacache will9match the `uid` and `gid` values to the folder where the cache lives, even10when running as `root`.11 12It was written to be used as [npm](https://npm.im)'s local cache, but can13just as easily be used on its own.14 15## Install16 17`$ npm install --save cacache`18 19## Table of Contents20 21* [Example](#example)22* [Features](#features)23* [Contributing](#contributing)24* [API](#api)25  * [Using localized APIs](#localized-api)26  * Reading27    * [`ls`](#ls)28    * [`ls.stream`](#ls-stream)29    * [`get`](#get-data)30    * [`get.stream`](#get-stream)31    * [`get.info`](#get-info)32    * [`get.hasContent`](#get-hasContent)33  * Writing34    * [`put`](#put-data)35    * [`put.stream`](#put-stream)36    * [`rm.all`](#rm-all)37    * [`rm.entry`](#rm-entry)38    * [`rm.content`](#rm-content)39    * [`index.compact`](#index-compact)40    * [`index.insert`](#index-insert)41  * Utilities42    * [`clearMemoized`](#clear-memoized)43    * [`tmp.mkdir`](#tmp-mkdir)44    * [`tmp.withTmp`](#with-tmp)45  * Integrity46    * [Subresource Integrity](#integrity)47    * [`verify`](#verify)48    * [`verify.lastRun`](#verify-last-run)49 50### Example51 52```javascript53const cacache = require('cacache')54const fs = require('fs')55 56const cachePath = '/tmp/my-toy-cache'57const key = 'my-unique-key-1234'58 59// Cache it! Use `cachePath` as the root of the content cache60cacache.put(cachePath, key, '10293801983029384').then(integrity => {61  console.log(`Saved content to ${cachePath}.`)62})63 64const destination = '/tmp/mytar.tgz'65 66// Copy the contents out of the cache and into their destination!67// But this time, use stream instead!68cacache.get.stream(69  cachePath, key70).pipe(71  fs.createWriteStream(destination)72).on('finish', () => {73  console.log('done extracting!')74})75 76// The same thing, but skip the key index.77cacache.get.byDigest(cachePath, integrityHash).then(data => {78  fs.writeFile(destination, data, err => {79    console.log('tarball data fetched based on its sha512sum and written out!')80  })81})82```83 84### Features85 86* Extraction by key or by content address (shasum, etc)87* [Subresource Integrity](#integrity) web standard support88* Multi-hash support - safely host sha1, sha512, etc, in a single cache89* Automatic content deduplication90* Fault tolerance (immune to corruption, partial writes, process races, etc)91* Consistency guarantees on read and write (full data verification)92* Lockless, high-concurrency cache access93* Streaming support94* Promise support95* Fast -- sub-millisecond reads and writes including verification96* Arbitrary metadata storage97* Garbage collection and additional offline verification98* Thorough test coverage99* There's probably a bloom filter in there somewhere. Those are cool, right? 🤔100 101### Contributing102 103The cacache team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.104 105All participants and maintainers in this project are expected to follow [Code of Conduct](CODE_OF_CONDUCT.md), and just generally be excellent to each other.106 107Please refer to the [Changelog](CHANGELOG.md) for project history details, too.108 109Happy hacking!110 111### API112 113#### <a name="ls"></a> `> cacache.ls(cache) -> Promise<Object>`114 115Lists info for all entries currently in the cache as a single large object. Each116entry in the object will be keyed by the unique index key, with corresponding117[`get.info`](#get-info) objects as the values.118 119##### Example120 121```javascript122cacache.ls(cachePath).then(console.log)123// Output124{125  'my-thing': {126    key: 'my-thing',127    integrity: 'sha512-BaSe64/EnCoDED+HAsh=='128    path: '.testcache/content/deadbeef', // joined with `cachePath`129    time: 12345698490,130    size: 4023948,131    metadata: {132      name: 'blah',133      version: '1.2.3',134      description: 'this was once a package but now it is my-thing'135    }136  },137  'other-thing': {138    key: 'other-thing',139    integrity: 'sha1-ANothER+hasH=',140    path: '.testcache/content/bada55',141    time: 11992309289,142    size: 111112143  }144}145```146 147#### <a name="ls-stream"></a> `> cacache.ls.stream(cache) -> Readable`148 149Lists info for all entries currently in the cache as a single large object.150 151This works just like [`ls`](#ls), except [`get.info`](#get-info) entries are152returned as `'data'` events on the returned stream.153 154##### Example155 156```javascript157cacache.ls.stream(cachePath).on('data', console.log)158// Output159{160  key: 'my-thing',161  integrity: 'sha512-BaSe64HaSh',162  path: '.testcache/content/deadbeef', // joined with `cachePath`163  time: 12345698490,164  size: 13423,165  metadata: {166    name: 'blah',167    version: '1.2.3',168    description: 'this was once a package but now it is my-thing'169  }170}171 172{173  key: 'other-thing',174  integrity: 'whirlpool-WoWSoMuchSupport',175  path: '.testcache/content/bada55',176  time: 11992309289,177  size: 498023984029178}179 180{181  ...182}183```184 185#### <a name="get-data"></a> `> cacache.get(cache, key, [opts]) -> Promise({data, metadata, integrity})`186 187Returns an object with the cached data, digest, and metadata identified by188`key`. The `data` property of this object will be a `Buffer` instance that189presumably holds some data that means something to you. I'm sure you know what190to do with it! cacache just won't care.191 192`integrity` is a [Subresource193Integrity](#integrity)194string. That is, a string that can be used to verify `data`, which looks like195`<hash-algorithm>-<base64-integrity-hash>`.196 197If there is no content identified by `key`, or if the locally-stored data does198not pass the validity checksum, the promise will be rejected.199 200A sub-function, `get.byDigest` may be used for identical behavior, except lookup201will happen by integrity hash, bypassing the index entirely. This version of the202function *only* returns `data` itself, without any wrapper.203 204See: [options](#get-options)205 206##### Note207 208This function loads the entire cache entry into memory before returning it. If209you're dealing with Very Large data, consider using [`get.stream`](#get-stream)210instead.211 212##### Example213 214```javascript215// Look up by key216cache.get(cachePath, 'my-thing').then(console.log)217// Output:218{219  metadata: {220    thingName: 'my'221  },222  integrity: 'sha512-BaSe64HaSh',223  data: Buffer#<deadbeef>,224  size: 9320225}226 227// Look up by digest228cache.get.byDigest(cachePath, 'sha512-BaSe64HaSh').then(console.log)229// Output:230Buffer#<deadbeef>231```232 233#### <a name="get-stream"></a> `> cacache.get.stream(cache, key, [opts]) -> Readable`234 235Returns a [Readable Stream](https://nodejs.org/api/stream.html#stream_readable_streams) of the cached data identified by `key`.236 237If there is no content identified by `key`, or if the locally-stored data does238not pass the validity checksum, an error will be emitted.239 240`metadata` and `integrity` events will be emitted before the stream closes, if241you need to collect that extra data about the cached entry.242 243A sub-function, `get.stream.byDigest` may be used for identical behavior,244except lookup will happen by integrity hash, bypassing the index entirely. This245version does not emit the `metadata` and `integrity` events at all.246 247See: [options](#get-options)248 249##### Example250 251```javascript252// Look up by key253cache.get.stream(254  cachePath, 'my-thing'255).on('metadata', metadata => {256  console.log('metadata:', metadata)257}).on('integrity', integrity => {258  console.log('integrity:', integrity)259}).pipe(260  fs.createWriteStream('./x.tgz')261)262// Outputs:263metadata: { ... }264integrity: 'sha512-SoMeDIGest+64=='265 266// Look up by digest267cache.get.stream.byDigest(268  cachePath, 'sha512-SoMeDIGest+64=='269).pipe(270  fs.createWriteStream('./x.tgz')271)272```273 274#### <a name="get-info"></a> `> cacache.get.info(cache, key) -> Promise`275 276Looks up `key` in the cache index, returning information about the entry if277one exists.278 279##### Fields280 281* `key` - Key the entry was looked up under. Matches the `key` argument.282* `integrity` - [Subresource Integrity hash](#integrity) for the content this entry refers to.283* `path` - Filesystem path where content is stored, joined with `cache` argument.284* `time` - Timestamp the entry was first added on.285* `metadata` - User-assigned metadata associated with the entry/content.286 287##### Example288 289```javascript290cacache.get.info(cachePath, 'my-thing').then(console.log)291 292// Output293{294  key: 'my-thing',295  integrity: 'sha256-MUSTVERIFY+ALL/THINGS=='296  path: '.testcache/content/deadbeef',297  time: 12345698490,298  size: 849234,299  metadata: {300    name: 'blah',301    version: '1.2.3',302    description: 'this was once a package but now it is my-thing'303  }304}305```306 307#### <a name="get-hasContent"></a> `> cacache.get.hasContent(cache, integrity) -> Promise`308 309Looks up a [Subresource Integrity hash](#integrity) in the cache. If content310exists for this `integrity`, it will return an object, with the specific single integrity hash311that was found in `sri` key, and the size of the found content as `size`. If no content exists for this integrity, it will return `false`.312 313##### Example314 315```javascript316cacache.get.hasContent(cachePath, 'sha256-MUSTVERIFY+ALL/THINGS==').then(console.log)317 318// Output319{320  sri: {321    source: 'sha256-MUSTVERIFY+ALL/THINGS==',322    algorithm: 'sha256',323    digest: 'MUSTVERIFY+ALL/THINGS==',324    options: []325  },326  size: 9001327}328 329cacache.get.hasContent(cachePath, 'sha521-NOT+IN/CACHE==').then(console.log)330 331// Output332false333```334 335##### <a name="get-options"></a> Options336 337##### `opts.integrity`338If present, the pre-calculated digest for the inserted content. If this option339is provided and does not match the post-insertion digest, insertion will fail340with an `EINTEGRITY` error.341 342##### `opts.memoize`343 344Default: null345 346If explicitly truthy, cacache will read from memory and memoize data on bulk read. If `false`, cacache will read from disk data. Reader functions by default read from in-memory cache.347 348##### `opts.size`349If provided, the data stream will be verified to check that enough data was350passed through. If there's more or less data than expected, insertion will fail351with an `EBADSIZE` error.352 353 354#### <a name="put-data"></a> `> cacache.put(cache, key, data, [opts]) -> Promise`355 356Inserts data passed to it into the cache. The returned Promise resolves with a357digest (generated according to [`opts.algorithms`](#optsalgorithms)) after the358cache entry has been successfully written.359 360See: [options](#put-options)361 362##### Example363 364```javascript365fetch(366  'https://registry.npmjs.org/cacache/-/cacache-1.0.0.tgz'367).then(data => {368  return cacache.put(cachePath, 'registry.npmjs.org|cacache@1.0.0', data)369}).then(integrity => {370  console.log('integrity hash is', integrity)371})372```373 374#### <a name="put-stream"></a> `> cacache.put.stream(cache, key, [opts]) -> Writable`375 376Returns a [Writable377Stream](https://nodejs.org/api/stream.html#stream_writable_streams) that inserts378data written to it into the cache. Emits an `integrity` event with the digest of379written contents when it succeeds.380 381See: [options](#put-options)382 383##### Example384 385```javascript386request.get(387  'https://registry.npmjs.org/cacache/-/cacache-1.0.0.tgz'388).pipe(389  cacache.put.stream(390    cachePath, 'registry.npmjs.org|cacache@1.0.0'391  ).on('integrity', d => console.log(`integrity digest is ${d}`))392)393```394 395##### <a name="put-options"></a> Options396 397##### `opts.metadata`398 399Arbitrary metadata to be attached to the inserted key.400 401##### `opts.size`402 403If provided, the data stream will be verified to check that enough data was404passed through. If there's more or less data than expected, insertion will fail405with an `EBADSIZE` error.406 407##### `opts.integrity`408 409If present, the pre-calculated digest for the inserted content. If this option410is provided and does not match the post-insertion digest, insertion will fail411with an `EINTEGRITY` error.412 413`algorithms` has no effect if this option is present.414 415##### `opts.integrityEmitter`416 417*Streaming only* If present, uses the provided event emitter as a source of418truth for both integrity and size. This allows use cases where integrity is419already being calculated outside of cacache to reuse that data instead of420calculating it a second time.421 422The emitter must emit both the `'integrity'` and `'size'` events.423 424NOTE: If this option is provided, you must verify that you receive the correct425integrity value yourself and emit an `'error'` event if there is a mismatch.426[ssri Integrity Streams](https://github.com/npm/ssri#integrity-stream) do this for you when given an expected integrity.427 428##### `opts.algorithms`429 430Default: ['sha512']431 432Hashing algorithms to use when calculating the [subresource integrity433digest](#integrity)434for inserted data. Can use any algorithm listed in `crypto.getHashes()` or435`'omakase'`/`'お任せします'` to pick a random hash algorithm on each insertion. You436may also use any anagram of `'modnar'` to use this feature.437 438Currently only supports one algorithm at a time (i.e., an array length of439exactly `1`). Has no effect if `opts.integrity` is present.440 441##### `opts.memoize`442 443Default: null444 445If provided, cacache will memoize the given cache insertion in memory, bypassing446any filesystem checks for that key or digest in future cache fetches. Nothing447will be written to the in-memory cache unless this option is explicitly truthy.448 449If `opts.memoize` is an object or a `Map`-like (that is, an object with `get`450and `set` methods), it will be written to instead of the global memoization451cache.452 453Reading from disk data can be forced by explicitly passing `memoize: false` to454the reader functions, but their default will be to read from memory.455 456##### `opts.tmpPrefix`457Default: null458 459Prefix to append on the temporary directory name inside the cache's tmp dir. 460 461#### <a name="rm-all"></a> `> cacache.rm.all(cache) -> Promise`462 463Clears the entire cache. Mainly by blowing away the cache directory itself.464 465##### Example466 467```javascript468cacache.rm.all(cachePath).then(() => {469  console.log('THE APOCALYPSE IS UPON US 😱')470})471```472 473#### <a name="rm-entry"></a> `> cacache.rm.entry(cache, key, [opts]) -> Promise`474 475Alias: `cacache.rm`476 477Removes the index entry for `key`. Content will still be accessible if478requested directly by content address ([`get.stream.byDigest`](#get-stream)).479 480By default, this appends a new entry to the index with an integrity of `null`.481If `opts.removeFully` is set to `true` then the index file itself will be482physically deleted rather than appending a `null`.483 484To remove the content itself (which might still be used by other entries), use485[`rm.content`](#rm-content). Or, to safely vacuum any unused content, use486[`verify`](#verify).487 488##### Example489 490```javascript491cacache.rm.entry(cachePath, 'my-thing').then(() => {492  console.log('I did not like it anyway')493})494```495 496#### <a name="rm-content"></a> `> cacache.rm.content(cache, integrity) -> Promise`497 498Removes the content identified by `integrity`. Any index entries referring to it499will not be usable again until the content is re-added to the cache with an500identical digest.501 502##### Example503 504```javascript505cacache.rm.content(cachePath, 'sha512-SoMeDIGest/IN+BaSE64==').then(() => {506  console.log('data for my-thing is gone!')507})508```509 510#### <a name="index-compact"></a> `> cacache.index.compact(cache, key, matchFn, [opts]) -> Promise`511 512Uses `matchFn`, which must be a synchronous function that accepts two entries513and returns a boolean indicating whether or not the two entries match, to514deduplicate all entries in the cache for the given `key`.515 516If `opts.validateEntry` is provided, it will be called as a function with the517only parameter being a single index entry. The function must return a Boolean,518if it returns `true` the entry is considered valid and will be kept in the index,519if it returns `false` the entry will be removed from the index.520 521If `opts.validateEntry` is not provided, however, every entry in the index will522be deduplicated and kept until the first `null` integrity is reached, removing523all entries that were written before the `null`.524 525The deduplicated list of entries is both written to the index, replacing the526existing content, and returned in the Promise.527 528#### <a name="index-insert"></a> `> cacache.index.insert(cache, key, integrity, opts) -> Promise`529 530Writes an index entry to the cache for the given `key` without writing content.531 532It is assumed if you are using this method, you have already stored the content533some other way and you only wish to add a new index to that content. The `metadata`534and `size` properties are read from `opts` and used as part of the index entry.535 536Returns a Promise resolving to the newly added entry.537 538#### <a name="clear-memoized"></a> `> cacache.clearMemoized()`539 540Completely resets the in-memory entry cache.541 542#### <a name="tmp-mkdir"></a> `> tmp.mkdir(cache, opts) -> Promise<Path>`543 544Returns a unique temporary directory inside the cache's `tmp` dir. This545directory will use the same safe user assignment that all the other stuff use.546 547Once the directory is made, it's the user's responsibility that all files548within are given the appropriate `gid`/`uid` ownership settings to match549the rest of the cache. If not, you can ask cacache to do it for you by550calling [`tmp.fix()`](#tmp-fix), which will fix all tmp directory551permissions.552 553If you want automatic cleanup of this directory, use554[`tmp.withTmp()`](#with-tpm)555 556See: [options](#tmp-options)557 558##### Example559 560```javascript561cacache.tmp.mkdir(cache).then(dir => {562  fs.writeFile(path.join(dir, 'blablabla'), Buffer#<1234>, ...)563})564```565 566#### <a name="tmp-fix"></a> `> tmp.fix(cache) -> Promise`567 568Sets the `uid` and `gid` properties on all files and folders within the tmp569folder to match the rest of the cache.570 571Use this after manually writing files into [`tmp.mkdir`](#tmp-mkdir) or572[`tmp.withTmp`](#with-tmp).573 574##### Example575 576```javascript577cacache.tmp.mkdir(cache).then(dir => {578  writeFile(path.join(dir, 'file'), someData).then(() => {579    // make sure we didn't just put a root-owned file in the cache580    cacache.tmp.fix().then(() => {581      // all uids and gids match now582    })583  })584})585```586 587#### <a name="with-tmp"></a> `> tmp.withTmp(cache, opts, cb) -> Promise`588 589Creates a temporary directory with [`tmp.mkdir()`](#tmp-mkdir) and calls `cb`590with it. The created temporary directory will be removed when the return value591of `cb()` resolves, the tmp directory will be automatically deleted once that 592promise completes.593 594The same caveats apply when it comes to managing permissions for the tmp dir's595contents.596 597See: [options](#tmp-options)598 599##### Example600 601```javascript602cacache.tmp.withTmp(cache, dir => {603  return fs.writeFile(path.join(dir, 'blablabla'), 'blabla contents', { encoding: 'utf8' })604}).then(() => {605  // `dir` no longer exists606})607```608 609##### <a name="tmp-options"></a> Options610 611##### `opts.tmpPrefix`612Default: null613 614Prefix to append on the temporary directory name inside the cache's tmp dir. 615 616#### <a name="integrity"></a> Subresource Integrity Digests617 618For content verification and addressing, cacache uses strings following the619[Subresource620Integrity spec](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity).621That is, any time cacache expects an `integrity` argument or option, it622should be in the format `<hashAlgorithm>-<base64-hash>`.623 624One deviation from the current spec is that cacache will support any hash625algorithms supported by the underlying Node.js process. You can use626`crypto.getHashes()` to see which ones you can use.627 628##### Generating Digests Yourself629 630If you have an existing content shasum, they are generally formatted as a631hexadecimal string (that is, a sha1 would look like:632`5f5513f8822fdbe5145af33b64d8d970dcf95c6e`). In order to be compatible with633cacache, you'll need to convert this to an equivalent subresource integrity634string. For this example, the corresponding hash would be:635`sha1-X1UT+IIv2+UUWvM7ZNjZcNz5XG4=`.636 637If you want to generate an integrity string yourself for existing data, you can638use something like this:639 640```javascript641const crypto = require('crypto')642const hashAlgorithm = 'sha512'643const data = 'foobarbaz'644 645const integrity = (646  hashAlgorithm +647  '-' +648  crypto.createHash(hashAlgorithm).update(data).digest('base64')649)650```651 652You can also use [`ssri`](https://npm.im/ssri) to have a richer set of functionality653around SRI strings, including generation, parsing, and translating from existing654hex-formatted strings.655 656#### <a name="verify"></a> `> cacache.verify(cache, opts) -> Promise`657 658Checks out and fixes up your cache:659 660* Cleans up corrupted or invalid index entries.661* Custom entry filtering options.662* Garbage collects any content entries not referenced by the index.663* Checks integrity for all content entries and removes invalid content.664* Fixes cache ownership.665* Removes the `tmp` directory in the cache and all its contents.666 667When it's done, it'll return an object with various stats about the verification668process, including amount of storage reclaimed, number of valid entries, number669of entries removed, etc.670 671##### <a name="verify-options"></a> Options672 673##### `opts.concurrency`674 675Default: 20676 677Number of concurrently read files in the filesystem while doing clean up.678 679##### `opts.filter`680Receives a formatted entry. Return false to remove it.681Note: might be called more than once on the same entry.682 683##### `opts.log`684Custom logger function:685```686  log: { silly () {} }687  log.silly('verify', 'verifying cache at', cache)688```689 690##### Example691 692```sh693echo somegarbage >> $CACHEPATH/content/deadbeef694```695 696```javascript697cacache.verify(cachePath).then(stats => {698  // deadbeef collected, because of invalid checksum.699  console.log('cache is much nicer now! stats:', stats)700})701```702 703#### <a name="verify-last-run"></a> `> cacache.verify.lastRun(cache) -> Promise`704 705Returns a `Date` representing the last time `cacache.verify` was run on `cache`.706 707##### Example708 709```javascript710cacache.verify(cachePath).then(() => {711  cacache.verify.lastRun(cachePath).then(lastTime => {712    console.log('cacache.verify was last called on' + lastTime)713  })714})715```716