Automatically generate Dart wrappers and doc from MDN

[jodinathan]

Hi,

I am a Dart developer and I would like to create a JS package for Dart devs to use.
The idea is to read some MDN API, json file or even crawl the MDN website and generate the Dart files, for example:

@JS()
library window;

@JS()
abstract class HtmlElement {}

@JS()
abstract class HtmlDocument {
  external HtmlElement? getElemenyById(String buf);
}

@JS()
external HtmlDocument get document;

The above snippet is a Dart interface to use the window object along with document and getElementById.
What I am thinking in doing is automatize the process by reading stuff from MDN, generate wrappers like the above and publish a package to the community.

Is this ok?
If yes, do you have some API I could fetch these infos or something like that?

[sideshowbarker]

See https://developer.mozilla.org/en-US/docs/MDN/Tools/Unsupported_GET_API for info about the only thing available.

For example, to get a JSON representation of the https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement article, you can using https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/index.json

Is this ok?

Yes, it’s OK to scrape the production site to get either that JSON data or the HTML source, or any other data you want to get.

That said, you don’t actually need to remotely scrape the production site — you can instead run a “preview server” locally to get what you need. The details are in step 6 of https://github.com/mdn/content/blob/main/README.md#more-substantial-changes — but the gist of it is, you clone this repo, and then run this sequence of commands:

# Switch to a separate terminal.
cd ~/repos/mdn/content
yarn
yarn start

That will start a web server on your local system, accessible at http://localhost:5000. So if you want to get the JSON data for the HTMLElement element locally, you can use http://localhost:5000/en-US/docs/Web/API/HTMLElement/index.json.

[jodinathan]

This is just great, thanks!
I am trying to use a local clone but I am having an issue that the server is shutting down when I try to fetch the compatibility info and it has a 404.

mdn/content-main/node_modules/@mdn/yari/server/index.js:291
      bcdData.find((data) => data.url.toLowerCase() === bcdDataURL).data
                                                                   ^

TypeError: Cannot read property 'data' of undefined

how can I avoid shutting the server after an exception like that?

[peterbe]

First of all; Hi! nice to meet you.
Second; https://www.peterbe.com/plog/how-to-get-all-of-mdn-web-docs-running-locally

This error here is intriguing. As we can see, that code isn't safe. It's assuming that something always comes out of that .find(). But how do you even get there? That server/index.js code, although clearly sloppy as we just discovered, isn't meant to be robust or recovering perfectly.
What are you doing to trigger this?

[jodinathan]

calling /bcd.json.
I was able to avoid calling the compatibility data if the dataURL is not present in the compatibility body item

[jodinathan]

was able to avoid the exception by checking if there is an dataURL in compatibility at the body info.

I can't seem to be able to automatically identify the return values or methods and getters.
Any idea what I could do?

[sideshowbarker]

I would personally be out of my depth trying to do any troubleshooting with this. The @mdn/core-dev might be able to help you (though just to be clear, it seems like you what you want to do isn’t something that’s a supported purpose of the infrastructure, and so the project’s not necessarily resourced by giving much detailed guidance or help on something like this).

[greggman]

I thought I'd point out what rather than scrapping MDN you'd probably be better off using the official docs for each API. Most of them have a machine readable IDL file or section(s) that is/are specifically designed for generating code. In fact most of the browsers themselves read those files to generate some of the code in the browser itself.

I don't know if the IDL files are sitting somewhere or just buried in the specs but for example the WebGL2 IDL is here -- it says it's scrapped from the spec.

[sideshowbarker]

Yeah what @greggman describes in https://github.com/mdn/content/discussions/5713#discussioncomment-855953 definitely seems like the best way to approach this.

I don't know if the IDL files are sitting somewhere

https://github.com/web-platform-tests/wpt/tree/master/interfaces has them all; for example:

https://github.com/web-platform-tests/wpt/blob/master/interfaces/html.idl

[jodinathan]

yeah, I am playing with both, MDN website and IDLs.
MDN to get names and desc and IDL to get the type information.

@sideshowbarker that IDL list link seems very helpful. I was digging pages with regex to only get the IDL.
Weird how difficult is to find only IDL stuff.

The json that I am creating should be full of valuable information.
Maybe we could use it here in MDN to add type information in a machine readable way.
As I am machine reading MDN, it is finding some errors like MS stuff not marked as non standard etc.
I will also try to list them up once I finish this.

[jodinathan]

hello
I have built the package: https://pub.dev/packages/js_bindings.
It uses WebIDLs from w3c and MDN for documentation.

Checkout out how nice the MediaStream file got with the docs: https://github.com/jodinathan/js_bindings/blob/main/lib/bindings/mediacapture_streams.dart

thanks a lot for the info, was very helpful

[sideshowbarker]

Checkout out how nice the MediaStream file got with the docs: jodinathan/js_bindings@main/lib/bindings/mediacapture_streams.dart

Beautiful 🎉