Browser Plugin

[do-me]

As partially discussed in #11 , there are more concrete plans to create a browser plugin (maybe even supported by web-GPU). @lizozom what are your exact plans?

[varunneal]

@xenova has implemented features in a transformers.js PR that will make it possible to cache models in a browser extension. I'll be working from the PR to turn Semantic Finder into an extension. This will be a bit of work but I hope to make it as elegant as the MkDocs homepage that @do-me referenced.

[varunneal]

Here's an update on the browser extension: https://github.com/VarunNSrivastava/transformers.js.

It's a very rudimentary proof of concept. Feel free to take a look.

For bugfixing / improvements:

• the content of the webpage isn't always sent for whatever reason (reloading the page and/or the extension usually fixes this)
• Right now I'm only scraping the

  paragraph

  content of a webpage. Of course we can do much better but some natural difficulties pop up.

In terms of next steps, I was considering

• focusing on building/recreating a really good sentence parser (auto-segmentation) in javascript. Ideally, it would be robust to segmenting the entirety of the document.body of a webpage
• making a really beautiful GUI for the extension (anime.js is one library I was looking at)
• eventually: PDF, image parsing

[do-me]

I'm really curious to try, but it threw some errors:

(base) dome@dome:/mnt/c/Users/dome/Downloads/transformers.js/examples/extension$ npm run dev

> [email protected] dev
> webpack --watch

assets by path *.js 2.27 MiB
  asset popup.js 1.45 MiB [emitted] (name: popup)
  asset background.js 435 KiB [emitted] (name: background)
  asset content.js 410 KiB [emitted] (name: content)
asset icons/icon.png 12.7 KiB [emitted] [from: public/icons/icon.png] [copied]
asset manifest.json 977 bytes [emitted] [from: public/manifest.json] [copied]
asset popup.css 665 bytes [emitted] [from: src/popup.css] [copied]
asset popup.html 261 bytes [emitted]
runtime modules 2.91 KiB 14 modules
modules by path ./node_modules/ 478 KiB 56 modules
modules by path ./src/ 30.4 KiB
  modules by path ./src/*.vue 19 KiB
    ./src/popup.vue 865 bytes [built] [code generated]
    ./src/popup.vue?vue&type=template&id=34cc76a3&scoped=true 211 bytes [built] [code generated]
    ./src/popup.vue?vue&type=script&lang=js 256 bytes [built] [code generated]
    + 13 modules
  modules by path ./src/*.js 10.9 KiB
    ./src/background.js 3.87 KiB [built] [code generated]
    ./src/popup.js 121 bytes [built] [code generated]
    ./src/content.js 598 bytes [built] [code generated]
    ./src/utils.js 1.16 KiB [built] [code generated]
    + 2 modules
  ./src/natural/sentence_tokenizer_parser.js 576 bytes [built] [code generated]

ERROR in ./src/semantic.js 3:0-53
Module not found: Error: Can't resolve '@xenova/transformers' in '/mnt/c/Users/dome/Downloads/transformers.js/examples/extension/src'
resolve '@xenova/transformers' in '/mnt/c/Users/dome/Downloads/transformers.js/examples/extension/src'
  Parsed request is a module
  using description file: /mnt/c/Users/dome/Downloads/transformers.js/examples/extension/package.json (relative path: ./src)
    Field 'browser' doesn't contain a valid alias configuration
    resolve as module
      /mnt/c/Users/dome/Downloads/transformers.js/examples/extension/src/node_modules doesn't exist or is not a directory
      looking for modules in /mnt/c/Users/dome/Downloads/transformers.js/examples/extension/node_modules
        single file module
          using description file: /mnt/c/Users/dome/Downloads/transformers.js/examples/extension/package.json (relative path: ./node_modules/@xenova/transformers)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /mnt/c/Users/dome/Downloads/transformers.js/examples/extension/node_modules/@xenova/transformers doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /mnt/c/Users/dome/Downloads/transformers.js/examples/extension/node_modules/@xenova/transformers.js doesn't exist
            .json
              Field 'browser' doesn't contain a valid alias configuration
              /mnt/c/Users/dome/Downloads/transformers.js/examples/extension/node_modules/@xenova/transformers.json doesn't exist
            .wasm
              Field 'browser' doesn't contain a valid alias configuration
              /mnt/c/Users/dome/Downloads/transformers.js/examples/extension/node_modules/@xenova/transformers.wasm doesn't exist
        /mnt/c/Users/dome/Downloads/transformers.js/examples/extension/node_modules/@xenova/transformers doesn't exist
      /mnt/c/Users/dome/Downloads/transformers.js/examples/node_modules doesn't exist or is not a directory
      looking for modules in /mnt/c/Users/dome/Downloads/transformers.js/node_modules
        single file module
          using description file: /mnt/c/Users/dome/Downloads/transformers.js/package.json (relative path: ./node_modules/@xenova/transformers)
            no extension
              /mnt/c/Users/dome/Downloads/transformers.js/node_modules/@xenova/transformers doesn't exist
            .js
              /mnt/c/Users/dome/Downloads/transformers.js/node_modules/@xenova/transformers.js doesn't exist
            .json
              /mnt/c/Users/dome/Downloads/transformers.js/node_modules/@xenova/transformers.json doesn't exist
            .wasm
              /mnt/c/Users/dome/Downloads/transformers.js/node_modules/@xenova/transformers.wasm doesn't exist
        /mnt/c/Users/dome/Downloads/transformers.js/node_modules/@xenova/transformers doesn't exist
      /mnt/c/Users/dome/Downloads/node_modules doesn't exist or is not a directory
      /mnt/c/Users/dome/node_modules doesn't exist or is not a directory
      /mnt/c/Users/node_modules doesn't exist or is not a directory
      /mnt/c/node_modules doesn't exist or is not a directory
      /mnt/node_modules doesn't exist or is not a directory
      /node_modules doesn't exist or is not a directory
 @ ./src/background.js 5:0-41 94:24-34

webpack 5.80.0 compiled with 1 error in 2555 ms

When I load it in Chrome:

[varunneal]

Looking at the error message, it might be that the relative import

    "@xenova/transformers": "../../dist/transformers.js",

in extension/src/package.json is causing a problem. To fix it, when in src, you can try

npm uninstall @xenova/transformers
npm install ../../dist/transformers.js"

Let me know if these don't work but the point is you want to be using the local version of transformers.js, and there are lots of hacky ways to get it working.

[do-me]

@VarunNSrivastava I just saw a notification on my phone that you sent the build folder somewhere but I cannot find the message anymore - could you send it again here maybe or ping me again wherever you sent the message please?

[varunneal]

I wanted to send you the build file as a zip but I realized that it would not correctly encapsulate local dependencies (e.g. the local version of transformers.js). I would have also liked to send it to you as a .crx file but this is no longer really supported.

Let me know if you can't get it built locally, otherwise we can troubleshoot other solutions.

[do-me]

Ah ok so I didn't dream it! ;)
I'll try on a different machine today and let you know if it works, thanks.

[do-me]

Got it running by manually copying the files from transformers dist folder to /transformers.js/examples/extension/node_modules/@xenova.

I really like the simplistic UI - not intrusive but very powerful!

I tried for the wikipedia page of Albert Einstein, but I guess the splitting function has some issues:

What are your defaults?

If you want to keep it simple I guess it's very hard to come up with universally well-working settings, considering different text lenght nd the tradeoff of indexing time vs. chunk lenght / accuracy (if you're interested in words or rather paragraphs). Maybe it would be an option to:

a) define a waiting time that is still acceptable for users like 5-10 sec (even though this heavily depends on the users machine...)
b) see, how many chunks can be indexed in this time
c) aim at splitting the text in this number of chunks so the waiting time is not exceeded
d) if the text is very long, respect the max token count the model can still process without truncation

Anyway, it's awesome to see this working in a browser and on any page!

As a side note or a play of thought: just like in the web-version, it would also be cool if a web page could already provide the indexed file to the extension so the search would be almost instant. E.g. I was thinking that this browser plugin could fuel mkdocs semantic search too and maybe even attract more developers to work on it.

Mkdocs-material also has a good tutorial section about their approach to tokenization with some regex examples.

[varunneal]

I'll pour through the through those docs, thanks for the link.

The splitting function is my biggest priority. What I'm doing right now is

• using readability to get the cleaned page content
• using a huge regex to process "sentences"

I'm going to work through a few more approaches. I've played around with python NLP packages that should be quite good for sentence segmentation (spaCy and NLTK).

In my head, the default should be a good sentence-level segmentation, which optional configs to make it character/token based.

It would be really cool to provide the index to the extension in the site. Great potential source of exploration.

[do-me]

For anyone reading through this, there are some comments in the PR #33.

FYI: Mozilla announced an open ecosystem for browser plugins on mobile.

[varunneal]

After the most recent PR #34, my biggest priority would be adding iframes support (including/especially pdfs). I have a hacky solution for pdfs in right now but it's not working consistently.

After that, after some bugtesting, we could probably do an initial ship on the extensions store.

[do-me]

Yes, supporting pdfs would be so useful! I can't remember how many times I knew "I read something somewhere" but couldn't find the paragraph anymore. Also it's very user-friendly to have the original pdf layout in the browser instead of copy-pasting the text in the browser version of SemanticFinder!

Let me know when you think we should go ahead with the first version and spreading the word. I agree we should thoroughly test for bugs first.

[varunneal]

If anyone has any tips, tricks, or resources for getting page content that would be great. The current solution (using mozilla's readability.js library) is great but it doesn't access iframes and some other content. I've tried some hacky solutions but nothing is very satisfactory so far.