Marking up code blocks for unsupported languages

[hamishwillee]

What should our rules be for marking up code blocks when the languages are not one of the supported set (as listed in https://github.com/mdn/yari/blob/main/build/syntax-highlight.js#L15).

Options:

1. Add support for all language(s).
2. Use plain to indicate that the type is explicitly set as nothing/unsupported
3. Use a substitute language where you get somewhat useful highlighting - e.g. bash as a substitute for powershell, and use plain otherwise.
4. Markup in the actual language of the text, whether or not it renders. Use plain if the "code" doesn't have a language, or at least one supported by prism.

The ideal would be to do "1" but it isn't clear what the cost of adding a new language is. If it is a per-page cost then you'd reasonably just add all the languages, whereas if the cost applies to every page, you'd have to consider pages separately. Haven't got an answer on how this works - trying in mdn/yari#6763

Either way, currently I use "3" because really, the syntax for all the shells is much the same (so you get some benefit using bash for command, powershell, etc etc.). But I'm leaning towards suggesting we use "4").

Why? My assumption is that this is the best future proofing and indication of intent. It can't be wrong, and it should all work nicely if we were to add linters for different language types later on.

[Note, posted this because of https://github.com/mdn/content/pull/19259

[queengooborg]

Considering that the file for importing and loading Prism is in the build/ folder, and skimming the source code shows that syntax highlighting is performed during the build step, I'm voting for option 1 as there would be no (or very minimal) cost on the reader's end.

[queengooborg]

I've opened mdn/yari#6831 to add many more languages (and add some comments to list out the languages loaded by default, so the function can be used as a reference of available languages)!

[wbamberg]

Whatever we do, please let's document the identifiers that authors should be allowed to use, and update the list when there is a need for more.

We currently do this here: https://developer.mozilla.org/en-US/docs/MDN/Writing_guidelines/Howto/Markdown_in_MDN#example_code_blocks. I'm not really in favour of people being able to add whatever identifiers they feel like. There isn't a cost to readers in this but there is a cost to maintainers.

[hamishwillee]

Ah, so you think we should use "2" as opposed to "4" - i.e. have a discreet subset and mark the things we don't support with plain. I think that's OK too. Possibly also have a note explaining how/where you might request updates to yari.

[hamishwillee]

@wbamberg If we're not allowing everything then we need to

• agree a policy for what we will allow/want,
• accept and document an initial set that matches that.
• add the policy and process for getting new ones added to the docs.

Note, github uses a very large set: https://github.com/github/linguist/blob/master/lib/linguist/languages.yml
We should use a subset of those.

Not sure what the policy should be. Any ideas? I would argue around

• dominant language of a supported reference set (e.g. if we document typescript or wasm we would have it)
• makes code markup significantly easier to read than plain for some significant set of docs
• discuss on a case by case basis by posting justification as a PR to yari.

The list proposed in @queengooborg initial PR is below. I would argue for wasm and jsx based on first point and powershell because this gets added by users enough to be irritating. I'm not sure about yaml - it is pretty well served by "plain". pug pages would definitely benefit from this, but we're only talking 3 or 4 of them - is that wortht he cost?

  "powershell",
   "jsx",
    "typescript",
    "diff",
    "ini",
    "ignore",
    "less",
    "regex",
    "scss",
    "sql",
    "uri",
    "wasm",
    "yaml",
    "pug",

[Josh-Cena]

So according to @queengooborg adding a language is virtually zero-cost for the user. In this case I don't see any reason why we can't mark up all code blocks with the right language. In my editor, the code will be displayed with the corresponding syntax highlighting; for static analysis tools (not just limited to JS linters), they will be able to grep all code blocks to be linted; for users, they enjoy richer syntax highlighting, or failing that, at least with no adverse effects.

I'm going towards "4". We should do the following:

• Be as correct as possible (i.e. not arbitrarily tell people "no, we don't use typescript on MDN, please just mark it as js")
• Whenever there's a need for a new language, go add it in Yari
• However, before the language becomes available in Yari or before it's on its way to becoming available, do not allow people to tag code blocks with languages that are never to be enabled (usually because Prism doesn't support it—e.g. vue/svelte).

I don't have strong opinions on telling people what language should be allowed to use. If we have a diverse language set to start with, that definitely works. What I don't want is for people to work around the platform limitations and mark up powershell code blocks as bash.

[wbamberg]

This sounds great to me, Hamish, thank you for thinking it out. I would call it "identifier" rather than alias, because to me "alias" implies an alternate name, and I think we want to have only one name. Otherwise +1

[Josh-Cena]

Also, FWIW, js and javascript are already both supported—as soon as we load one language, all aliases of it defined by Prism are available out-of-the-box.

[hamishwillee]

@wbamberg @Josh-Cena

• Awesome. I modified the text above to say "identifier" rather than alias.

• Also, FWIW, js and javascript are already both supported—as soon as we load one language, all aliases of it defined by Prism are available out-of-the-box.

  Good point. But I think we should settle on documenting just one identifier because it may make things easier if we want to lint the sections, and it certainly makes things easier for reviewers to have just one valid case. Not to mention possible future migration to something other than prism. Lastly, I can't see any real benefit in stating you use one of these identifiers to indicate a language.

  Above I have updated this to add sub rules

  • Identifier should match Github - we're trying to be close to GFM.
  • If there are two possible identifiers, the most common contraction should be documented (and enforced) on MDN.
  • The identifier should be added to the alias list even if prism already supports it "natively". This allows the list of identifiers to be inferred from the alias list.

I'll try apply this rule to https://github.com/mdn/yari/pull/6831/files and see how it turns out.

[teoli2003]

Looks good to me. @hamishwillee: the answer is a bit lost in the discussion right now. Can you copy it into a new Suggested answer below? Then, if the consensus stays, mark it as the official answer. That way, we won't need to go through the whole discussion to refer to it.

[hamishwillee]

Done!

[hamishwillee]

The consensus appears to be:

• We allow any language supported by prism except for the blocklist (blockslist is currently just WebIDL).
• We don't add "all languages" by default. Someone has to "want" them and be willing to document them.
• Users can request a new language via a Yari PR, noting:
  • the language and identifier used on MDN.
    • Identifier should match Github - we're trying to be close to GFM.
    • If there are two possible identifiers, the most common contraction should be documented (and enforced) on MDN.
    • The identifier should be added to the alias list even if prism already supports it "natively". This allows the list of identifiers to be inferred from the alias list.
  • a link to the language definition (i.e. for ini it is not obvious what the language might be)
  • a non-spurious reason⁺⁺
  • Link to any places known to be affected in MDN.
• The PR author is responsible for also adding alias, language and link to language definition to the list.
• Unsupported languages do not get submitted - they must be set as plain or wait for corresponding PR to be accepted.

⁺⁺The "non-spurious" reason should be pretty lax. I just want PR reviewers to have a right to reject PRs where no thought has gone into adding the language. Should we have "Fortran"? Maybe, but not because someone decides it might be fun.