awacke1/OpenAI-CLIP-JavaScript
2
1<!DOCTYPE html>2<html>3 <head>4 <meta charset="utf-8">5 <meta name="viewport" content="width=device-width">6 <title>OpenAI CLIP Image Search in JavaScript (Using ONNX Web Runtime)</title>7 <script src="enable-threads.js"></script>8 <script src="./vips/vips.js"></script>9 </head>10 <body>11 <style>12 body * {13 font-family: monospace;14 }15 </style>16 <script src="https://cdn.jsdelivr.net/npm/onnxruntime-web@1.12.0/dist/ort.js"></script>17 <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@3.17.0/dist/tf.min.js"></script> <!-- NOTE: tfjs is currently only used for image preprocessing stuff. -->18 19 <div>20 <h1 style="font-size:1rem;">Sort/search images using OpenAI's CLIP in your browser</h1>21 <p>This web app sorts/searches through images in a directory on your computer using OpenAI's CLIP model, and the new File System Access API. <a href="https://github.com/josephrocca/clip-image-sorter">Here's the Github repo</a> for this web app, and <a href="https://github.com/josephrocca/openai-clip-js">here's the Github repo</a> for the web-ported CLIP models. Feel free to open an issue or <a href="https://twitter.com/rocca27" target="_blank">DM me on Twitter</a> if you have any questions about this demo.</p>22 <p>All processing happens in your browser, on your device - i.e. your images are <b>not</b> uploaded to a server for processing.</p>23 <p id="browserCompatibilityWarning" style="padding:0.25rem; background:rgb(255, 227, 160); display:none;"><b>Note</b>: This page uses new browser features (File System Access API, and credentialless COEP) that are currently only available in some browsers. As of writing, it works in Chrome, Edge and Brave. Other browsers like Firefox and Safari are often a bit slower in implementing cutting-edge features.</p>24 <script>25 if(Date.now() < 1648725949710+1000*60*60*24*365) { // display until start of April 202326 browserCompatibilityWarning.style.display = "";27 }28 </script>29 <hr>30 31 <div id="modelNameSelectCtn" style="padding:0.5rem; background:lightgrey; margin:0.5rem;">32 <b>Step 1:</b> Choose model:33 <select onchange="window.MODEL_NAME=this.value;">34 <option value="clip_vit_32">CLIP ViT-B/32 (recommended)</option>35 <option value="clip_vit_32_uint8">CLIP ViT-B/32 (quantized - inaccurate embeddings)</option>36 <option value="lit_b16b">LiT B16B</option>37 </select>38 </div>39 40 <div id="initCtnEl" style="padding:0.5rem; background:lightgrey; margin:0.5rem;">41 <b>Step 2:</b> Download and initialize the models.42 <br>43 Download image model: <progress id="imageModelLoadingProgressBarEl" value="0"></progress> <span id="imageModelLoadingMbEl"></span>44 <br>45 Download text model: <progress id="textModelLoadingProgressBarEl" value="0"></progress> <span id="textModelLoadingMbEl"></span>46 <br>47 Initialize workers: <progress id="workerInitProgressBarEl" value="0"></progress>48 <div style="display:none;"> <!-- more workers (dividing threads between them) doesn't seem to make things faster -->49 Number of image embedding workers/threads: <input id="numThreadsEl" type="range" min="1" max="4" value="1" oninput="numThreadsDisplayEl.textContent=this.value"> <span id="numThreadsDisplayEl"></span> <script>numThreadsEl.max = navigator.hardwareConcurrency; numThreadsDisplayEl.textContent=numThreadsEl.value;</script>50 </div>51 <br>52 <button id="initWorkersBtn" onclick="modelNameSelectCtn.style.pointerEvents='none'; modelNameSelectCtn.style.opacity=0.5; initializeWorkers()">initialize workers</button>53 </div>54 55 <div id="pickDirCtnEl" style="opacity:0.5; pointer-events:none; padding:0.5rem; background:lightgrey; margin:0.5rem;">56 <b>Step 3:</b> Pick a directory of images (images in subdirectories will be included).57 <br>58 <button id="pickDirectoryBtn" onclick="pickDirectory({source:'local'})">pick directory</button> or <button id="useRedditImagesBtn" onclick="pickDirectory({source:'reddit'})">use ~200k reddit images</button> (remove nsfw:<input id="removeRedditNsfwEl" type="checkbox" checked>)59 <br>60 <div id="redditLoadProgressCtn" style="display:none;">Download progress: <progress id="redditProgressBarEl" value="0"></progress> <span id="redditProgressMbEl"></span></div>61 <div id="existingEmbeddingsProgressCtn" style="display:none;">Loading existing embeddings: <span id="existingEmbeddingsLoadedEl">none</span></div>62 </div>63 64 <div id="computeEmbeddingsCtnEl" style="opacity:0.5; pointer-events:none; padding:0.5rem; background:lightgrey; margin:0.5rem;">65 <b>Step 4:</b> Compute image embeddings. <span style="opacity:0.5;">(they will be saved as <ModelName>_embeddings.tsv in the selected directory)</span>66 <br>67 <button id="computeEmbeddingsBtn" onclick="computeImageEmbeddings(); this.disabled=true;">compute image embeddings</button>68 <br>69 <span id="computeEmbeddingsProgressEl">0</span> images embedded (<span id="computeEmbeddingsSpeedEl">?</span> ms per image) <span id="preexistingEmbeddingsEl"></span>70 </div>71 72 <div id="existingEmbeddingsFoundCtnEl" style="display:none; padding:0.5rem; background:lightgrey; margin:0.5rem;">73 <b>Step 5:</b> <b>Existing embeddings found.</b>74 <br>75 Only needed if you've added or changed images: <button onclick="existingEmbeddingsFoundCtnEl.style.display='none'; computeEmbeddingsCtnEl.style.display=''; disableCtn(searchCtnEl); computeEmbeddingsBtn.click()">(re)compute image embeddings</button>76 <input id="onlyEmbedNewImagesCheckbox" type="checkbox" checked> Only new images?77 </div>78 79 <div id="searchCtnEl" style="opacity:0.5; pointer-events:none; padding:0.5rem; background:lightgrey; margin:0.5rem;">80 <b>Step 6:</b> Enter a search term.81 <br>82 <input id="searchTextEl" style="width:300px;" value="" placeholder="Enter search text here..." onkeyup="if(event.which==13) searchSort()">83 <button id="searchBtn" onclick="searchSort()">search</button>84 </div>85 </div>86 87 <hr>88 <b>Results</b> <span style="opacity:0.5;">(hover for cosine similarities)</span>89 <div id="resultsEl" style="margin-top:1rem; min-height:100vh;"><span style="opacity:0.5;">Click the search button to compute the results.</span></div>90 91 <script>92 /////////////93 // STEP 1 //94 /////////////95 window.MODEL_NAME = "clip_vit_32";96 window.modelData = {97 clip_vit_32: {98 image: {99 modelUrl: (quantized) => `https://huggingface.co/rocca/openai-clip-js/resolve/main/clip-image-vit-32-${quantized ? "uint8" : "float32"}.onnx`,100 embed: async function(blob, session) {101 let rgbData = await getRgbData(blob);102 const feeds = {input: new ort.Tensor('float32', rgbData, [1,3,224,224])};103 const results = await session.run(feeds);104 const embedVec = results["output"].data; // Float32Array105 return embedVec;106 }107 },108 text: {109 modelUrl: (quantized) => `https://huggingface.co/rocca/openai-clip-js/resolve/main/clip-text-vit-32-${quantized ? "uint8" : "float32-int32"}.onnx`,110 embed: async function(text, session) {111 if(!window.textTokenizerClip) {112 let Tokenizer = (await import("https://deno.land/x/clip_bpe@v0.0.6/mod.js")).default;113 window.textTokenizerClip = new Tokenizer(); 114 }115 let textTokens = window.textTokenizerClip.encodeForCLIP(text);116 textTokens = Int32Array.from(textTokens);117 const feeds = {input: new ort.Tensor('int32', textTokens, [1, 77])};118 const results = await session.run(feeds);119 return [...results["output"].data];120 },121 }122 },123 lit_b16b: {124 image: {125 modelUrl: () => 'https://huggingface.co/rocca/lit-web/resolve/main/embed_images.onnx',126 embed: async function(blob, session) {127 128 // TODO: Maybe remove tf from this code so you can remove the whole tfjs dependency129 blob = await bicubicResizeAndCenterCrop(blob);130 let inputImg = new Image();131 await new Promise(r => inputImg.onload=r, inputImg.src=URL.createObjectURL(blob));132 let img = tf.browser.fromPixels(inputImg);133 img = tf.sub(tf.div(tf.expandDims(img), 127.5), 1);134 let float32RgbData = img.dataSync();135 136 const feeds = {'images': new ort.Tensor('float32', float32RgbData, [1,224,224,3])};137 const results = await session.run(feeds);138 return results["Identity_1:0"].data;139 },140 },141 text: {142 modelUrl: () => 'https://huggingface.co/rocca/lit-web/resolve/main/embed_text_tokens.onnx',143 embed: async function(text, session) {144 if(!window.bertTextTokenizerLit) {145 window.bertTextTokenizerLit = await import("./bert-text-tokenizer.js").then(m => new m.BertTokenizer());146 await window.bertTextTokenizerLit.load();147 }148 let textTokens = window.bertTextTokenizerLit.tokenize(text);149 textTokens.unshift(101); // manually put CLS token at the start150 textTokens.length = 16;151 textTokens = [...textTokens.slice(0, 16)].map(e => e == undefined ? 0 : e); // pad with zeros to length of 16152 textTokens = Int32Array.from(textTokens);153 const feeds = {'text_tokens': new ort.Tensor('int32', textTokens, [1,16])};154 const results = await session.run(feeds);155 return [...results["Identity_1:0"].data];156 }157 }158 },159 };160 let imageWorkers = [];161 let onnxImageSessions = [];162 let onnxTextSession;163 let textTokenizer;164 async function initializeWorkers() {165 initWorkersBtn.disabled = true;166 numThreadsEl.disabled = true;167 168 let useQuantizedModel = false;169 170 if(MODEL_NAME.endsWith("_uint8")) {171 MODEL_NAME = MODEL_NAME.replace(/_uint8$/g, "");172 useQuantizedModel = true;173 }174 175 let imageOnnxBlobPromise = downloadBlobWithProgress(window.modelData[MODEL_NAME].image.modelUrl(useQuantizedModel), function(e) {176 let ratio = e.loaded / e.total;177 imageModelLoadingProgressBarEl.value = ratio;178 imageModelLoadingMbEl.innerHTML = Math.round(ratio*e.total/1e6)+" MB";179 });180 181 let textOnnxBlobPromise = downloadBlobWithProgress(window.modelData[MODEL_NAME].text.modelUrl(useQuantizedModel), function(e) {182 let ratio = e.loaded / e.total;183 textModelLoadingProgressBarEl.value = ratio;184 textModelLoadingMbEl.innerHTML = Math.round(ratio*e.total/1e6)+" MB";185 });186 187 let [imageOnnxBlob, textOnnxBlob] = await Promise.all([imageOnnxBlobPromise, textOnnxBlobPromise])188 189 let imageModelUrl = window.URL.createObjectURL(imageOnnxBlob);190 let textModelUrl = window.URL.createObjectURL(textOnnxBlob);191 192 let numImageWorkers = Number(numThreadsEl.value);193 194 // Inference latency is about 5x faster with wasm threads, but this requires these headers: https://web.dev/coop-coep/ I'm using this as a hack (in enable-threads.js) since Github pages doesn't allow setting headers: https://github.com/gzuidhof/coi-serviceworker195 if(self.crossOriginIsolated) {196 ort.env.wasm.numThreads = Math.ceil(navigator.hardwareConcurrency / numImageWorkers) / 2; // divide by two to utilise only half the CPU's threads because trying to use all the cpu's threads actually makes it slower197 }198 199 workerInitProgressBarEl.max = numImageWorkers + 2; // +2 because of text model and bpe library200 201 let imageModelExecutionProviders = ["wasm"]; // webgl is not compatible with this model (need to tweak conversion data/op types)202 203 for(let i = 0; i < numImageWorkers; i++) {204 let session = await ort.InferenceSession.create(imageModelUrl, { executionProviders: imageModelExecutionProviders }); 205 onnxImageSessions.push(session);206 imageWorkers.push({207 session,208 busy: false,209 });210 workerInitProgressBarEl.value = Number(workerInitProgressBarEl.value) + 1;211 }212 console.log("Image model loaded.");213 214 onnxTextSession = await ort.InferenceSession.create(textModelUrl, { executionProviders: ["wasm"] }); // webgl is not compatible with this model (need to tweak conversion data/op types)215 console.log("Text model loaded.");216 workerInitProgressBarEl.value = Number(workerInitProgressBarEl.value) + 1;217 218 window.URL.revokeObjectURL(imageModelUrl);219 window.URL.revokeObjectURL(textModelUrl);220 221 window.vips = await Vips(); // for bicubicly resizing images (since that's what CLIP expects)222 window.vips.EMBIND_AUTOMATIC_DELETELATER = false;223 224 workerInitProgressBarEl.value = Number(workerInitProgressBarEl.value) + 1;225 226 disableCtn(initCtnEl);227 enableCtn(pickDirCtnEl);228 }229 230 231 /////////////232 // STEP 2 //233 /////////////234 let directoryHandle;235 let embeddingsFileHandle;236 let embeddings;237 let dataSource;238 async function pickDirectory(opts={}) {239 dataSource = opts.source;240 241 if(dataSource === "local") {242 if(!window.showDirectoryPicker) return alert("Your browser does not support some modern features (specifically, File System Access API) required to use this web app. Please try updating your browser, or switching to Chrome, Edge, or Brave.");243 directoryHandle = await window.showDirectoryPicker();244 embeddingsFileHandle = await directoryHandle.getFileHandle(`${window.MODEL_NAME}_embeddings.tsv`, {create:true});245 246 pickDirectoryBtn.disabled = true;247 useRedditImagesBtn.disabled = true;248 pickDirectoryBtn.textContent = "Loading...";249 }250 251 let redditEmbeddingsBlob;252 if(dataSource === "reddit") {253 if(window.MODEL_NAME !== "clip_vit_32") return alert("Sorry, there are only pre-computed Reddit image embeddings for the CLIP ViT-B/32 model at the moment.");254 if(!removeRedditNsfwEl.checked && !confirm("Are you sure you'd like to see NSFW Reddit images?")) return;255 if(removeRedditNsfwEl.checked) alert("Note that NSFW images are filtered from Reddit using CLIP, and CLIP can make mistakes, so some NSFW images may still be shown.");256 257 pickDirectoryBtn.disabled = true;258 useRedditImagesBtn.disabled = true;259 useRedditImagesBtn.textContent = "Loading...";260 redditLoadProgressCtn.style.display = "";261 262 redditEmbeddingsBlob = await downloadBlobWithProgress("https://huggingface.co/datasets/rocca/top-reddit-posts/resolve/main/clip_embeddings_top_50_images_per_subreddit.tsv.gz", function(e) {263 let ratio = e.loaded / e.total;264 redditProgressBarEl.value = ratio;265 redditProgressMbEl.innerHTML = Math.round(ratio*213)+" MB";266 });267 }268 269 try {270 existingEmbeddingsProgressCtn.style.display = "";271 272 embeddings = {};273 let file, opts;274 if(dataSource === "local") {275 file = await embeddingsFileHandle.getFile();276 opts = {};277 }278 if(dataSource === "reddit") {279 file = redditEmbeddingsBlob;280 opts = {decompress:"gzip"};281 }282 283 let i = 0;284 for await (let line of makeTextFileLineIterator(file, opts)) {285 if(!line || !line.trim()) continue; // <-- to skip final new line (not sure if this is needed)286 let [filePath, embeddingVec] = line.split("\t");287 embeddings[filePath] = JSON.parse(embeddingVec);288 i++;289 if(i % 1000 === 0) {290 existingEmbeddingsLoadedEl.innerHTML = i;291 await sleep(10);292 }293 }294 } catch(e) {295 embeddings = undefined;296 console.log("No existing embedding found, or the embeddings file was corrupted:", e);297 existingEmbeddingsProgressCtn.style.display = "none";298 }299 300 pickDirectoryBtn.textContent = "Done.";301 useRedditImagesBtn.textContent = "Done.";302 303 disableCtn(pickDirCtnEl);304 enableCtn(computeEmbeddingsCtnEl);305 enableCtn(searchCtnEl);306 307 if(embeddings && Object.keys(embeddings).length > 0) {308 computeEmbeddingsCtnEl.style.display = "none";309 existingEmbeddingsFoundCtnEl.style.display = "";310 }311 312 if(dataSource === "reddit") {313 disableCtn(existingEmbeddingsFoundCtnEl);314 }315 316 }317 318 319 /////////////320 // STEP 3 //321 /////////////322 let totalEmbeddingsCount = 0;323 let imagesEmbedded;324 let recentEmbeddingTimes = []; // how long each embed took in ms, newest at end325 let recomputeAllEmbeddings;326 let imagesBeingProcessedNow = 0;327 let needToSaveEmbeddings = false;328 async function computeImageEmbeddings() {329 imagesEmbedded = 0;330 totalEmbeddingsCount = Object.keys(embeddings).length;331 332 recomputeAllEmbeddings = !onlyEmbedNewImagesCheckbox.checked;333 let gotSomeExistingEmbeddings = totalEmbeddingsCount > 0;334 335 if(onlyEmbedNewImagesCheckbox.checked && gotSomeExistingEmbeddings) {336 preexistingEmbeddingsEl.innerHTML = `(loaded ${Object.keys(embeddings).length} existing embeddings)`; 337 }338 339 if(recomputeAllEmbeddings || !gotSomeExistingEmbeddings) {340 embeddings = {}; // <-- maps file path (relative to top/selected directory) to embedding341 }342 343 try {344 await recursivelyProcessImagesInDir(directoryHandle);345 await saveEmbeddings();346 } catch(e) {347 console.error(e);348 alert(e.message);349 }350 351 disableCtn(computeEmbeddingsCtnEl);352 enableCtn(searchCtnEl);353 }354 async function recursivelyProcessImagesInDir(dirHandle, currentPath="") {355 for await (let [name, handle] of dirHandle) {356 const {kind} = handle;357 let path = `${currentPath}/${name}`;358 if (handle.kind === 'directory') {359 await recursivelyProcessImagesInDir(handle, path);360 } else {361 let isImage = /\.(png|jpg|jpeg|webp)$/.test(path);362 if(!isImage) continue;363 364 let alreadyGotEmbedding = !!embeddings[path];365 if(alreadyGotEmbedding && !recomputeAllEmbeddings) continue;366 367 if(needToSaveEmbeddings) {368 await saveEmbeddings();369 needToSaveEmbeddings = false;370 }371 372 while(imageWorkers.filter(w => !w.busy).length === 0) await sleep(1);373 374 let worker = imageWorkers.filter(w => !w.busy)[0];375 worker.busy = true;376 imagesBeingProcessedNow++;377 378 (async function() {379 let startTime = Date.now();380 381 let blob = await handle.getFile();382 const embedVec = await modelData[MODEL_NAME].image.embed(blob, worker.session);383 384 embeddings[path] = [...embedVec];385 worker.busy = false;386 387 imagesEmbedded++;388 totalEmbeddingsCount++;389 390 computeEmbeddingsProgressEl.innerHTML = imagesEmbedded;391 392 let saveInterval = totalEmbeddingsCount > 50_000 ? 10_000 : 1000; // since saves take longer if there are lots of embeddings393 if(imagesEmbedded % saveInterval === 0) {394 needToSaveEmbeddings = true;395 }396 397 recentEmbeddingTimes.push(Date.now()-startTime);398 if(recentEmbeddingTimes.length > 100) recentEmbeddingTimes = recentEmbeddingTimes.slice(-50);399 if(recentEmbeddingTimes.length > 10) computeEmbeddingsSpeedEl.innerHTML = Math.round(recentEmbeddingTimes.slice(-20).reduce((a,v) => a+v, 0)/20);400 401 imagesBeingProcessedNow--;402 })();403 }404 }405 while(imagesBeingProcessedNow > 0) await sleep(10);406 }407 408 409 /////////////410 // STEP 4 //411 /////////////412 async function searchSort() {413 searchBtn.disabled = true;414 415 if(dataSource === "local") {416 for(let imgEl of [...document.querySelectorAll("img")]) {417 URL.revokeObjectURL(imgEl.src);418 }419 }420 421 resultsEl.innerHTML = "Loading...";422 await sleep(50);423 424 let searchTextEmbedding = await modelData[MODEL_NAME].text.embed(searchTextEl.value, onnxTextSession);425 let similarities = {};426 for(let [path, embedding] of Object.entries(embeddings)) {427 similarities[path] = cosineSimilarity(searchTextEmbedding, embedding);428 }429 let similarityEntries = Object.entries(similarities).sort((a,b) => b[1]-a[1]).slice(0, 5000);430 431 if(dataSource === "reddit" && removeRedditNsfwEl.checked) {432 let nsfwTextEmbedding = await modelData[MODEL_NAME].text.embed(atob('cG9ybiBuYWtlZCBwZW5pcyB2YWdpbmEgbnVkZSBzZXggZGljayBwdXNzeSBzZXh1YWwgcG9ybm9ncmFwaGljIGFzcyBib29icw=='), onnxTextSession); // nsfw words (hidden with `btoa`)433 let nsfwSimilarities = {};434 for(let [path, similarity] of similarityEntries) {435 let embedding = embeddings[path];436 nsfwSimilarities[path] = cosineSimilarity(nsfwTextEmbedding, embedding);437 }438 similarityEntries = similarityEntries.filter(e => nsfwSimilarities[e[0]] < 0.2093);439 }440 441 let resultHtml = "";442 let numResults = 0;443 for(let [path, score] of similarityEntries.slice(0, 500)) {444 if(dataSource === "local") {445 let handle = await getFileHandleByPath(path);446 let url = URL.createObjectURL(await handle.getFile());447 resultHtml += `<img src="${url}" style="max-height:400px;" title="${path}: ${score}"/>`;448 }449 if(dataSource === "reddit") {450 let imageUrl = `https://i.redd.it/${path.split("__")[1]}`;451 let postUrl = `https://reddit.com/comments/${path.split("__")[0].split("/")[1]}`;452 resultHtml += `<a href="${postUrl}" target="_blank"><img src="${imageUrl}" onload="this.style.height='';this.style.width='';this.style.border='';" style="max-height:400px; height:300px; width:300px; border:1px solid black;" title="${path}: ${score}" /></a>`;453 }454 numResults++;455 }456 457 if(!resultHtml) {458 resultsEl.innerHTML = "No results found after filtering NSFW.";459 } else {460 resultsEl.innerHTML = resultHtml;461 }462 463 searchBtn.disabled = false;464 }465 466 467 468 469 /////////////////////////////470 // FUNCTIONS / UTILITIES //471 /////////////////////////////472 473 async function getFileHandleByPath(path) {474 let handle = directoryHandle;475 let chunks = path.split("/").slice(1);476 for(let i = 0; i < chunks.length; i++) {477 let chunk = chunks[i];478 if(i === chunks.length-1) {479 handle = await handle.getFileHandle(chunk);480 } else {481 handle = await handle.getDirectoryHandle(chunk);482 }483 }484 return handle;485 }486 487 async function getRgbData(blob) {488 // let blob = await fetch(imgUrl, {referrer:""}).then(r => r.blob());489 490 let resizedBlob = await bicubicResizeAndCenterCrop(blob);491 let img = await createImageBitmap(resizedBlob);492 493 let canvas = new OffscreenCanvas(224, 224);494 let ctx = canvas.getContext("2d");495 ctx.drawImage(img, 0, 0);496 let imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);497 498 let rgbData = [[], [], []]; // [r, g, b]499 // remove alpha and put into correct shape:500 let d = imageData.data;501 for(let i = 0; i < d.length; i += 4) { 502 let x = (i/4) % canvas.width;503 let y = Math.floor((i/4) / canvas.width)504 if(!rgbData[0][y]) rgbData[0][y] = [];505 if(!rgbData[1][y]) rgbData[1][y] = [];506 if(!rgbData[2][y]) rgbData[2][y] = [];507 rgbData[0][y][x] = d[i+0]/255;508 rgbData[1][y][x] = d[i+1]/255;509 rgbData[2][y][x] = d[i+2]/255;510 // From CLIP repo: Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711))511 rgbData[0][y][x] = (rgbData[0][y][x] - 0.48145466) / 0.26862954;512 rgbData[1][y][x] = (rgbData[1][y][x] - 0.4578275) / 0.26130258;513 rgbData[2][y][x] = (rgbData[2][y][x] - 0.40821073) / 0.27577711;514 }515 rgbData = Float32Array.from(rgbData.flat().flat());516 return rgbData;517 }518 519 async function bicubicResizeAndCenterCrop(blob) {520 let im1 = vips.Image.newFromBuffer(await blob.arrayBuffer());521 522 // Resize so smallest side is 224px:523 const scale = 224 / Math.min(im1.height, im1.width);524 let im2 = im1.resize(scale, { kernel: vips.Kernel.cubic });525 526 // crop to 224x224:527 let left = (im2.width - 224) / 2;528 let top = (im2.height - 224) / 2;529 let im3 = im2.crop(left, top, 224, 224)530 531 let outBuffer = new Uint8Array(im3.writeToBuffer('.png'));532 im1.delete(), im2.delete(), im3.delete();533 return new Blob([outBuffer], { type: 'image/png' });534 }535 536 537 function downloadBlobWithProgress(url, onProgress) {538 return new Promise((res, rej) => {539 var blob;540 var xhr = new XMLHttpRequest();541 xhr.open('GET', url, true);542 xhr.responseType = 'arraybuffer';543 xhr.onload = function(e) {544 blob = new Blob([this.response]); 545 };546 xhr.onprogress = onProgress;547 xhr.onloadend = function(e){548 res(blob);549 }550 xhr.send();551 });552 }553 554 async function saveEmbeddings(opts={}) {555 let writable = await embeddingsFileHandle.createWritable();556 let textBatch = "";557 let i = 0;558 for(let [filePath, embeddingVec] of Object.entries(embeddings)) {559 let vecString = opts.compress ? JSON.stringify(embeddingVec.map(n => n.toFixed(3))).replace(/"/g, "") : JSON.stringify(embeddingVec);560 textBatch += `${filePath}\t${vecString}\n`;561 i++;562 if(i % 1000 === 0) {563 await writable.write(textBatch);564 textBatch = "";565 }566 }567 await writable.write(textBatch);568 await writable.close();569 }570 571 // Tweaked version of example from here: https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamDefaultReader/read572 async function* makeTextFileLineIterator(blob, opts={}) {573 const utf8Decoder = new TextDecoder("utf-8");574 let stream = await blob.stream();575 576 if(opts.decompress === "gzip") stream = stream.pipeThrough(new DecompressionStream("gzip"));577 578 let reader = stream.getReader();579 580 let {value: chunk, done: readerDone} = await reader.read();581 chunk = chunk ? utf8Decoder.decode(chunk, {stream: true}) : "";582 583 let re = /\r\n|\n|\r/gm;584 let startIndex = 0;585 586 while (true) {587 let result = re.exec(chunk);588 if (!result) {589 if (readerDone) {590 break;591 }592 let remainder = chunk.substr(startIndex);593 ({value: chunk, done: readerDone} = await reader.read());594 chunk = remainder + (chunk ? utf8Decoder.decode(chunk, {stream: true}) : "");595 startIndex = re.lastIndex = 0;596 continue;597 }598 yield chunk.substring(startIndex, result.index);599 startIndex = re.lastIndex;600 }601 if (startIndex < chunk.length) {602 // last line didn't end in a newline char603 yield chunk.substr(startIndex);604 }605 }606 607 function cosineSimilarity(A, B) {608 if(A.length !== B.length) throw new Error("A.length !== B.length");609 let dotProduct = 0, mA = 0, mB = 0;610 for(let i = 0; i < A.length; i++){611 dotProduct += A[i] * B[i];612 mA += A[i] * A[i];613 mB += B[i] * B[i];614 }615 mA = Math.sqrt(mA);616 mB = Math.sqrt(mB);617 let similarity = dotProduct / (mA * mB);618 return similarity;619 }620 621 function sleep(ms) {622 return new Promise(r => setTimeout(r, ms));623 }624 625 function enableCtn(el) {626 el.style.opacity = 1;627 el.style.pointerEvents = "";628 }629 function disableCtn(el) {630 el.style.opacity = 0.5;631 el.style.pointerEvents = "none";632 }633 634 // From the PyTorch model running on CUDA:635 // Text: "a portrait of an astronaut with the American flag" 636 // Embedding: [-1.6626e-01, 5.2277e-02, -1.5332e-01, 4.4946e-01, 2.0667e-01, -2.9565e-01, 4.0588e-02, -4.1016e-01, -1.5027e-01, 3.1934e-01, -6.9702e-02, -2.5488e-01, 1.2335e-01, -9.5337e-02, 2.4109e-01, -4.8950e-02, 2.6074e-01, 5.3835e-04, 2.1033e-01, 3.7012e-01, 4.5679e-01, 3.9795e-01, 3.1641e-01, 3.9551e-01, 1.3931e-02, -4.3060e-02, 4.8798e-02, 3.7158e-01, 1.1731e-01, -3.7256e-01, -2.7295e-01, 3.3130e-01, 5.4980e-01, -2.9816e-02, -2.5806e-01, -1.0016e-01, 8.0750e-02, -6.7139e-02, -2.4072e-01, 2.4353e-01, -3.2202e-01, -1.0327e-01, 1.1566e-01, 6.2646e-01, 1.8262e-01, 2.7539e-01, -1.1816e-01, 4.9512e-01, 8.9539e-02, 5.6299e-01, 2.1313e-01, -1.5625e-01, 1.9958e-01, -5.0049e-01, -2.5854e-01, -4.0430e-01, -1.1298e-01, -6.6338e-03, 2.5391e-01, -5.0629e-02, 2.2253e-01, -2.7295e-01, -5.8289e-03, -4.8804e-01, -7.7820e-02, -3.5187e-02, -3.7537e-02, 4.3213e-01, 3.8300e-02, 2.1045e-01, -3.0347e-01, -9.8999e-02, -1.7407e-01, 2.8882e-01, 1.1322e-01, -1.0883e-01, 1.7065e-01, -2.1191e-01, 1.7920e-01, -1.2805e-01, -4.6924e-01, 1.1957e-01, -1.1829e-01, -1.1902e-01, -2.4353e-01, -9.6008e-02, 2.2913e-01, -1.0948e-02, -1.5686e-01, -2.0483e-01, -2.4756e-01, 9.1125e-02, -9.5557e-01, -4.2511e-02, 4.6356e-02, 4.3481e-01, 2.3633e-01, -3.3252e-01, 3.7231e-01, -5.5695e-02, 7.1777e-02, -1.0370e-01, -2.1912e-01, -1.3733e-01, 1.2048e-01, 1.7151e-01, -1.2659e-01, 2.3523e-01, 2.6001e-01, -4.0381e-01, 1.1761e-01, -4.1626e-02, 1.0974e-01, -5.5206e-02, 4.9713e-02, 5.2197e-01, 3.9124e-02, 5.7959e-01, 9.9609e-02, -3.3740e-01, -2.7295e-01, -7.3389e-01, 1.0962e-01, -3.2178e-01, 6.5869e-01, 2.3460e-03, 2.6733e-02, 3.2471e-02, -2.4500e-01, 7.9041e-02, 1.5405e-01, -3.5547e-01, -1.5625e-01, 4.2695e+00, -1.6113e-01, 1.6467e-01, -3.8794e-01, 2.1545e-02, 1.5771e-01, 2.0068e-01, -3.4741e-01, 2.5244e-01, 1.2201e-01, -3.9795e-02, 3.2471e-01, 2.6562e-01, 1.2915e-01, 1.4465e-02, -2.1265e-01, -2.5055e-02, 5.6689e-01, -6.5125e-02, 1.1652e-01, -5.1025e-01, 1.0712e-01, 8.8867e-02, 1.6882e-01, -6.5125e-02, 4.5929e-02, 2.9517e-01, 6.5479e-01, -9.0881e-02, -4.6417e-02, -3.0713e-01, -5.1537e-03, 4.9390e-01, 5.4492e-01, -3.1812e-01, -6.0577e-02, 3.6591e-02, 7.6782e-02, -3.5181e-01, 4.9487e-01, -4.1040e-01, 4.3970e-01, 1.6553e-01, -3.7329e-01, 3.3594e-01, -1.9263e-01, -1.8225e-01, -1.9623e-02, -4.0454e-01, 1.6187e-01, 1.5259e-01, 2.5122e-01, 1.1993e-01, -8.4595e-02, -4.1016e-01, -1.8225e-01, 1.8555e-01, -3.9124e-02, 2.5122e-01, -2.2949e-01, 5.3125e-01, -1.0504e-01, -2.1439e-02, -2.2559e-01, 5.9357e-02, -3.9160e-01, -3.3716e-01, -9.0393e-02, -1.7493e-01, -2.5952e-01, 2.3401e-01, 3.8013e-01, 1.2927e-01, 1.5491e-01, 1.1920e-01, -1.5906e-01, 5.7487e-03, 7.6172e-02, 1.5552e-01, 2.1790e-01, 9.9304e-02, -3.8025e-02, -1.1829e-01, -1.3293e-01, -2.2278e-01, -2.1472e-01, 3.0957e-01, -1.0254e-01, -2.0264e-01, -2.6840e-02, 8.8379e-02, -8.2092e-02, 1.3647e-01, -2.1399e-01, -2.5684e-01, 1.3745e-01, 6.1371e-02, -1.2988e-01, 6.2683e-02, 1.2964e-01, -3.3112e-02, 1.4111e-01, -2.6440e-01, 4.7379e-03, -1.0815e-01, -4.4971e-01, -3.5583e-02, 1.1469e-01, 6.7871e-02, -5.8350e-02, 1.0297e-01, -6.3086e-01, -8.3350e-01, 4.3481e-01, -1.7383e-01, -1.5491e-01, -5.5176e-01, 5.9766e-01, -2.6880e-01, 5.6976e-02, -2.6318e-01, -4.0466e-02, 2.4927e-01, -1.4893e-01, -2.0032e-01, -2.7515e-01, -1.2598e-01, 3.2440e-02, -1.2939e-01, 6.8018e-01, -3.6060e-01, -3.3496e-01, 9.8267e-02, -1.0010e-01, 1.9653e-01, -3.3032e-01, -2.0032e-01, 1.9116e-01, 8.2214e-02, 3.2397e-01, 1.3708e-01, 3.0899e-02, 1.5454e-01, 3.2422e-01, 1.0693e-01, 3.1714e-01, 3.7280e-01, -2.3401e-01, -5.0171e-02, 7.6758e-01, 2.2046e-01, 3.3997e-02, 1.0231e-02, -2.3999e-01, -1.3672e-01, 5.2002e-01, -6.5552e-02, 6.9275e-02, 2.3706e-01, -3.8257e-01, 1.4990e-01, -2.2656e-01, 4.3018e-01, -4.4678e-01, 2.5977e-01, 3.8208e-01, 9.5215e-02, -2.0239e-01, -2.3169e-01, -3.1396e-01, 3.2806e-02, -2.7905e-01, -1.0361e+00, 3.8147e-02, -1.0278e-01, 2.8369e-01, -3.8623e-01, -2.3132e-01, 1.5784e-01, 4.2734e+00, -1.5820e-01, -1.4758e-01, 2.0032e-01, -1.9678e-01, -7.5586e-01, 1.0712e-01, -2.1777e-01, -4.0552e-01, -1.0919e-01, 1.1932e-01, 7.3486e-01, -1.7700e-01, 5.0732e-01, -9.7351e-02, -4.7192e-01, 5.9052e-02, -7.0117e-01, -1.8774e-01, 2.4329e-01, 3.5083e-01, -2.3633e-01, 2.2205e-01, -2.4719e-03, 4.4507e-01, -1.1005e-01, 5.6494e-01, -1.4697e-01, -8.5022e-02, -2.6050e-01, -8.3679e-02, 3.0615e-01, 1.0181e-01, -2.2266e-01, -1.2054e-01, 1.0094e-02, -1.6382e-01, 3.7671e-01, 1.1371e-01, 8.8959e-03, -1.2866e-01, 7.5500e-02, -3.1177e-01, 9.0698e-02, 2.4139e-02, 2.7124e-01, 4.1382e-01, 9.3872e-02, -3.5815e-01, -1.0602e-01, -2.7637e-01, 1.5613e-01, 2.6025e-01, -1.3428e-01, -2.1509e-01, -3.0005e-01, 7.7515e-02, -2.2253e-01, -1.2634e-01, 1.0114e-01, 2.9395e-01, 4.2065e-01, -4.6425e-03, -6.3721e-01, -4.0308e-01, -5.1849e-02, -9.1309e-02, 1.0577e-01, -1.6800e-02, -4.4823e-03, 2.4231e-01, -1.3635e-01, 1.7041e-01, -9.9243e-02, -1.2439e-01, 1.5247e-01, 1.4717e-02, -1.6785e-01, -3.0615e-01, 2.6074e-01, 1.0938e-01, 4.9487e-01, 1.0529e-01, 3.1799e-02, 7.5928e-02, -1.1212e-01, -3.1201e-01, 5.8740e-01, -1.3171e-01, -1.1090e-01, 5.8887e-01, -1.1420e-01, -2.0056e-01, 1.0425e-01, 2.7710e-01, -5.8098e-03, 5.7324e-01, -1.4417e-01, 1.4575e-01, -2.7466e-01, -2.1313e-01, -1.7627e-01, 1.5466e-01, 3.8013e-01, -1.4612e-01, -2.7246e-01, -1.8604e-01, 1.0394e-01, 1.6016e-01, -1.1017e-01, 1.8140e-01, -3.0078e-01, 6.0303e-01, -1.3904e-01, 1.7322e-01, 2.2510e-01, 2.3303e-01, -5.0879e-01, -2.3462e-01, -2.0544e-01, -2.4768e-01, -2.4121e-01, -7.2754e-01, 7.2754e-01, -4.4312e-02, -1.4198e-02, -1.1475e-01, -1.3684e-01, -3.5278e-01, 1.2347e-01, -2.9602e-02, 4.0550e-03, 1.1951e-01, -6.5575e-03, -7.1228e-02, -3.8062e-01, 6.5125e-02, -1.6541e-01, -3.0289e-02, 8.6609e-02, -1.2134e-01, -1.2164e-01, 3.0319e-02, -1.5173e-01, -7.9834e-02, 1.4148e-01, 2.7319e-01, -2.1545e-01, -1.6382e-01, -2.9419e-01, -2.6611e-01, -7.5102e-04, 1.3135e-01, 9.8389e-02, 3.1812e-01, 5.5115e-02, -5.3253e-02, 4.3823e-02, 3.6957e-02, -1.3599e-01, 1.1023e-01, 9.8267e-02, 2.1643e-01, 9.3567e-02, 1.2718e-02, 1.6406e-01, -1.0338e-02, 1.9019e-01, 1.4392e-01, 3.3081e-02, 1.0138e-01, -1.6943e-01, 8.6136e-03, 1.4478e-01, 1.0941e-02, 1.3635e-01, -8.1543e-01, -3.4912e-01, 7.4959e-03, 2.1997e-01, -2.5681e-02, 2.3206e-01, 3.7622e-01, 3.6401e-01, -1.6357e-01, -2.0984e-01, -1.3220e-01, -6.7322e-02, 2.0117e-01, -4.7583e-01, 6.8054e-02, 2.2437e-01, 2.6709e-01, -5.4626e-02, -4.0741e-02, 5.2002e-02, -1.8872e-01, 3.1372e-01, -1.3574e-01, -2.6538e-01];637 </script>638 </body>639</html>640