Stumptown-like related_content for sidebars

[peterbe]

In Stumptown we wanted to get away from KumaScript rendered blobs of cryptic macro code. Instead, the sidebar would just be an (infinitely deep) array of arrays. As simple as:

[  
  {title: "Top", url: "/en-US/docs/MDN", content: [    
    {title: "About MDN", url: "/en-US/docs/MDN/About"},    
    {title: "How to contribute", url: "/en-US/docs/MDN/Contribute"},    
    ...
  ]
]

Today, in about an hour's worth of work I put together a working prototype of this in Yari.
The idea is that you do this to the content: https://gist.github.com/peterbe/321aba029edcafb47639e9024e621cef
And the business logic that implements this won't be KumaScript ejs but instead pure Node JS code. This means it's really easy to code on (Prettier, syntax highlighting, caching, unit testing) and hopefully a lot more performant too since it's so simple instead of building.

Also, when shipped as part of the document state, it's just an array of objects rather than a fully encoded blob of HTML.
So it'll load faster. And we can potentially do more things that were hard in KumaScript. For example, instead of hardcoding things we can reach out to Yari APIs to get sorted lists based on traversing the filesystem.

It's just an experiment and I'm doing it carefully and gradually. It degrates to a fallback of the old KumaScript HTML sidebar macros if the content hasn't been updated.
There's definitely some work left on the rendering part so that it displays exactly like we want it to.
Here's what the prototype looks like:

Code here: mdn/yari@mdn:master...peterbe:related_content-experiment

[peterbe]

One of the genuinely cool things that Stumptown "invented" was the fresh simplicity. As a "purer" data structure, without weird hacks and quirks, sidebars become a lot easier to maintain and extend.

And from a user-point of view, it helps to navigate and consume the site if sidebars behave and look like you're expecting them too. When some sidebars are very different, it forces the brain to parse a new format that you might not be used to.

We can start slowly and build up exactly what we need. For example, we might want extra stuff beyond just url and title. For example, on some sidebars we might want to include a little icon (e.g. experimental beaker or deprecated thumbs-down)

[peterbe]

Usually, the logic for something to appear in the sidebars is either by looking through document tags, or alphabetical order, or perhaps a curated sort order.

Sidebars are a content-writers domain but I think it should be as simple as picking a named style or simply jotting down a specific order. Computers can do the rest.

[Ryuno-Ki]

You're showing a nested example in your code snippet. Would it need to be that way? (As in: breadcrumbs)
Or could the sidebar also refer to another area (say, from the Web API to Learn area).

[ghost]

It may be a good idea until they ask us for a series of customizations such as icons, colors, particular behaviors. But until then it's definitely okay to model better and in JSON definitely.

---

One thing I've seen Java folks do is use XML which is more extensible for a variety of reasons. At first it seemed stupid to save data in XML but it has its advantages because you can create custom names like in JSON but in addition you have the descendancy relationship (element that contains 1 or more elements, instead in JSON you can create a children key but it is tiring to manage, you see the DOM as it is complex).

[peterbe]

@escattone and I chatted about this offline (email) and basically concluded that we don't need to anything with this right now. We'll just leave it as a prototype.
But, as soon as we (dev team + content team) decide to want to tackle our sidebars, this prototype should graduate to a real PR. That'll make it much easier to evolve the legacy sidebars.

[ghost]

I think that if possible the data should be only in the pages and in the BCD and as much as possible the other components should be the result of computation between 5 sources that are:

• FRONT MATTER in YAML,
• content in HTML,
• position in the repository in "NODE",
• BCD in JSON,
• Git in "TEXT",

[wbamberg]

I'm not sure how we define a sidebar in this model. Is it just a JSON object? If so I think it would be hard to maintain this kind of sidebar. With our sidebars we often want to say something like: "under here, list all the object's properties in alphabetical order" (which translates to "list all the subpages of this page that have a 'Property' tag") and it would be nice to be able to do this without explicitly listing every property page in the sidebar specification, otherwise we have to remember to update the list when property pages are added/removed.

So the strength of KS is that it's able to do these kinds of quite powerful things. On the other hand:

• because KS is so powerful, there are some very complex macros whose behaviour is really hard to understand

• because every KS macro is a complete top-to-bottom implementation totally independent of any other implementations, there's a lot of code duplication for things like "under here, add a link to every subpage with tag X". And all these macros are written and maintained by non-professional developers, and many don't even have basic tests.

• because KS macros generate HTML directly, when/if we change the way we want to render them (as for example when we wanted to replace nests <ul> elements with <details>), then this is a big change.

• it's very hard (impossible really) when looking at a KS sidebar macro, to understand what kind of sidebar it will build. It would be lovely to have an expressive syntax for this.

It would be great to invent a syntax that's just expressive enough for our uses but simple enough for the heavy lifting to be done in common code that's properly maintained. This is where I was trying to get to with mdn/stumptown-content#302. I don't really know if this is practical though.

---

Either way though, we should treat sidebars as a content design problem first. That is, before deciding how we would like to implement sidebars, decide which different sidebars we should have and what they should contain. Because I think only by doing that can we know that a particular implementation can support the sidebars we want. That's how I went about it when we were thinking of how to do JS sidebars in stumptown.

At the moment, some of our KS sidebar macros produce pretty good sidebars, and some don't: for example, the CSS sidebar. But the usability problems with the CSS sidebar are not the fault of KS.

[escattone]

@wbamberg Will!!!! It is so awesome to have you (as well as @Elchi3 -- both of you true MDN masters), back on MDN. Welcome back dear friends!

[peterbe]

Wonderful! A lot of my prototype work and my own opinions are based on learning from you.
One thing that's clear to me though is that right now (well, as of my prototype) the sidebars are (still) part of the yari repo. This doesn't solve, or even attempt to solve, the problem that sidebars is a content problem but it sits inside yari.
And following that, independent of all your points about needs ahead of implementation, I think a pure Node JS implementation is easier to reason around compared to a .ejs file. It's easier to debug too since it's presentation-agnostic whereas the .ejs implementations bundle logic and HTML.

My prototype is most probably on the right track but it's but a small portion of the true challenge which is bigger than .ejs vs. Node JS. Did I summarize that OK?

[wbamberg]

Sorry to be slow replying. I think maybe I understand this a bit better now. And I hope this makes some kind of sense, it seemed more coherent in my head...

For me a goal is to make "sidebar implementations" - that is, the thing written by information architects/tech writers that defines what should appear in a particular sidebar - as simple to write and as human-readable as possible, so IAs/TWs can concentrate on their job, without having to write (and duplicate) too much code. And to factor out very repetitive operations into common well-tested modules.

At the moment a sidebar implementation is written by content authors, in KS, and its output is HTML. In this proposal it seems like a sidebar implementation is written by content authors in Node and its output is a nested JSON structure. Is that right?

Especially the second piece - emitting JSON rather than HTML - would be a big benefit for sidebar authors. All the sidebar code that laboriously pieces together big HTML structures is fragile and very much tends to be duplicated across all 30-odd sidebar implementations.

And as we've discussed in the past, this gives the website the ability to decide how a sidebar should be represented in HTML.

---

Going a bit further I wonder if it would be possible for individual items in this JSON output to be just references to pages in the content/files tree, rather than {title, url} tuples? Something like:

[  
  {
    "page": "en-us/docs/mdn",
    "children": [    
    "en-us/docs/mdn/about",    
    "en-us/docs/mdn/contribute",    
    ]
  }
]

...because resolving a page reference into a title+URL tuple is a common operation that could be done once, rather than making every sidebar implementation do it.

(Unless sidebars want to include things that aren't links to pages - for example, "Interfaces" in the sidebar at https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API.)

This also connects with localization. If sidebar implementations include titles, then they have to return the localized version of the title. At the moment localization is also handled separately by each sidebar implementation. Complex code repeated again and again:

https://github.com/mdn/yari/blob/c3f07a086389f9f4e215b5ff59c9dacca4548fc9/kumascript/macros/APIRef.ejs#L159-L168
https://github.com/mdn/yari/blob/c3f07a086389f9f4e215b5ff59c9dacca4548fc9/kumascript/macros/DefaultAPISidebar.ejs#L55-L61
https://github.com/mdn/yari/blob/c3f07a086389f9f4e215b5ff59c9dacca4548fc9/kumascript/macros/CSSRef.ejs#L189-L198
https://github.com/mdn/yari/blob/c3f07a086389f9f4e215b5ff59c9dacca4548fc9/kumascript/macros/JSRef.ejs#L165-L172

...with little differences. I have no idea if these differences are important or not...

Alternatively many of our sidebar implementations use giant maps to look up localized strings, like in https://github.com/mdn/yari/blob/main/kumascript/macros/LearnSidebar.ejs. This has other problems: translators must edit code (i.e. .ejs files) to add new translations. They're usually not familiar with the code side of things, so often introduce bugs. On the other side, the reviewers of the PRs they make are not experts in the languages in question, so can't properly review their PRs.

So it would be great if sidebar authors could just say "this page" and then The System could resolve that into a URL and the correct title for the locale. We also might want to have an optional short_title in the front matter, because full titles are sometimes too long to fit in a sidebar.

---

I used to think about stuff like this as a three-step process:

• step 1 is where content authors (humans) write documentation, including sidebars. The tools and formats here should try to be as usable by humans as possible: so things like: we should try to eliminate as much repetitive work as possible, and the output formats should be as human-readable as possible. So for instance, the output of this step could be something that like "links to this collection of pages"
• step 2 is where the output of step 1 is "compiled" by software into a format that's not committed to HTML, yet, but is made easily consumable by machines, and more self-contained. So this step could take "links to this collection of pages" and output "this collection of title/url tuples, properly localized".
• step 3 is where the output of step2 is rendered as HTML

When I was thinking about the sidebar-authoring part, I played with a really declarative approach to writing sidebars, so you could specify a sidebar by writing something like : mdn/stumptown-content#313 (comment) . Then step 2 turns that into JSON, and step 3 renders to HTML. Maybe that's going too far though :).

But even if we want to keep the write-sidebars-in-JS model, there may be scope for adding high-level APIs to make sidebar implementations less painful and with less duplicated code (like "get all subpages of this page that are of that page type"). Of course there are already special APIs for things like "getSubpages". But these are quite low-level and generic, and we could probably do better.

[peterbe]

You got it! But you're taking it a step further than my prototype. And I like it. But we have to find a nice middle-ground.
What's best for the humans to tackle "step 1"? YAML or Node?
Before you answer that; consider:

1. I don't think it matters what you use to build in the conveniences of being able to just say the slug (rather than having to figure out the exact URL, title, localized title, etc.)
2. In Node, the sidebar author, would have access to anything and everything that the "Yari API" offers. This might come in handy if you need to get a dynamic list based on some complex query. Or just the convenience of being able to do something like arrayMethods = ['slice', 'push', shift'].map(p => Document.findByURL('en-us/web/javascript/array/ + p))

I love the idea already about making it as "lazy and convenient" as possible. YAML or Node, you should only need to write down the slugs or URLs and the rest of the system should figure out all the titles (in all locales) and URLs and stuff. Being able to just type web/css/background-image and have the system figure out the right case for it.

In my prototype which is purely Node, at the moment, it didn't even begin to offer these conveniences (my code was more of a direct port of the .ejs code to .js). See the code for related-content/mdn.js for example. It still hardcodes all the titles when technically all you'd really need from a human is just a list of slugs.

Do you think it's worth extending my Node-based prototype to incorporate these ideas of convenience? Or, do you think it's not right enough and that it's better to chase a solution where the authoring is driven from YAML. Perhaps even from YAML within the mdn/content repo instead.

Be damned the many imperfections of my prototype, I do sense it at least covers "step 2" and "step 3". At least in principle. Now we just need to hash out "step 1". Right?

[peterbe]

I've started tracking my experimentation in this PR: mdn/yari#3928