Problem with MDN bot synchronizing localized documents

[fwqaaq]

In here, yin1999 and I have had a discussion and confirmed that there is indeed an wrong.

For unrelated documents without localized translation, it should not be automatically generated. Actually, for synchronization purpose, it moved the deleted documents to a new location (in the same location as the new English documents). However, they don't correspond to each other.

I think it's not just zh-CN just like this.

zh-CN:

• files/zh-cn/web/html/inline_elements/index.md ->files/zh-cn/glossary/inline-level_content/index.md

• ...cn/web/html/block-level_elements/index.md → ...-cn/glossary/block-level_content/index.md

ja:

https://github.com/mdn/translated-content/pull/13205/files

[schalkneethling]

Hey @fwqaaq, thank you for raising this problem. I have notified the engineering team.

[LeoMcA]

I've tried to understand the context - but I'm almost certainly missing some, so apologies, and bear with me - but it seems like the root of the problem here is a missing redirect in the mdn/content#25891 PR:

/en-US/docs/Web/HTML/Block-level_elements to /en-US/docs/Glossary/Block-level_content was set up, which was properly reflected in zh-CN, with /zh-CN/docs/Web/HTML/Block-level_elements (and existing redirects pointing at that) being updated to /zh-CN/docs/Glossary/Block-level_content in mdn-bot's commit

However, /en-US/docs/Web/HTML/Inline_elements to /en-US/docs/Glossary/Inline-level_content wasn't set up as a redirect (which is why https://developer.mozilla.org/en-US/docs/Web/HTML/Inline_elements currently 404s), so the sync script got confused and redirected to orphan pages, which had to be manually fixed in zh-CN.

Since redirects like /en-US/docs/Web/HTML/Inline_elemente to /en-US/docs/Glossary/Inline-level_content were created, then I suppose the sync script should pick up the change for /zh-CN/docs/Web/HTML/Inline_elemente, but perhaps not, because we then have a conflict between what the zh-CN redirects file is telling us, and what we're inferring from the en-US redirects file, it wouldn't be possible to set up a locale specific redirect without risk of it being overwritten by changes later in en-US - this'll require a bit of thought and diving into the sync code.

We'll need to open another PR to add that missing /en-US/docs/Web/HTML/Inline_elements redirect.

@OnkarRuikar I'm wondering how it didn't make it into the original PR (but /en-US/docs/Web/HTML/Block-level_elements did): I haven't dealt much with redirects, but I believe we have some tooling to automatically set them up when moving files. Is it not comprehensive enough for complex cases like this, so requires manual editing? Do we need to/could we set up some CI to check we're not missing a redirect, rather relying on manual review of a rather complex file?

[OnkarRuikar]

I haven't dealt much with redirects, but I believe we have some tooling to automatically set them up when moving files. Is it not comprehensive enough for complex cases like this, so requires manual editing?

Yes, we use yarn content move tool to move the documents instead of git mv. The tool generally works well, but looks like in this case it has a bug.

@OnkarRuikar I'm wondering how it didn't make it into the original PR (but /en-US/docs/Web/HTML/Block-level_elements did)

The tool made a wrong entry:

/en-US/docs/Glossary/inline-level	/en-US/docs/Glossary/inline-level
# note the same path on both sides

Instead it should have made:

/en-US/docs/Web/HTML/Inline_elements   /en-US/docs/Glossary/inline-level

Thinking it made a redundant entry I deleted it manually to avoid possible infinite redirects. We need to investigate and fix the code.
The bug may be due the fact that the file had two typos in its name elemente and elmements before. This may be a special case.

Do we need to/could we set up some CI to check we're not missing a redirect, rather relying on manual review of a rather complex file?

We could have a nightly cron job (like we have for markdown linting), that will regenerate/validate/fix the redircts file based on entire git history. But I don’t think the redirects file at the moment conforms strictly to the move/delete history, some manual edits may have been done, which makes it hard to implement such validator.
Also at the moment the redirects file has some broken redirectTo URLs and a lot of broken URL fragments. We need to fix them first. Fixing them is a manual task so it will take a long time.

---

Regarding the original argument in the PR,

Yeah, I agree with what you said. If it's entirely new content, I believe it shouldn't be moved.

inline, block used to be HTML terms now they are related to CSS. HTML has moved to different concept is a different story.
As they are same terms the pages were “moved” to CSS glossary, but then they got heavily edited. I guess deleting the HTML pages and adding redirects to CSS pages would also have worked.

Does the translated content sync tool handles add->delete+redirect differently than move+redirect?
If it makes simple then can we make yarn content move command to do copy->delete+redirect instead of move+redirect all the time?

[fwqaaq]

As they are same terms the pages were “moved” to CSS glossary, but then they got heavily edited. I guess deleting the HTML pages and adding redirects to CSS pages would also have worked.

Yeah, I think your analysis are right. Maybe It most likely a problem with adding redirect. But I don't know the mdn-bot implementation, it's just a guess on my part.

[yin1999]

Yeah, I think your analysis are right. Maybe It most likely a problem with adding redirect. But I don't know the mdn-bot implementation, it's just a guess on my part.

As I've added sync case changes + move adjacent files supports for the bot. I think I could tell the the core implementation of "sync content", the steps should be like this (see the source code here):

1. Get the slug of a document

2. Slug checking:

   1. Check if the slug is been redirected to other slug, if so, run git mv to move document and adjacent files
   2. Check if the slug is not exist in en-US, if so, mark this document as orphaned and run git mv to move document and adjacent files to /orphaned folder
   3. Check if the slug case has changed, if not, just end the process for this document.

3. Modify the document's content, to reflect the slug change.

So, if there is no redirect entry in en-US for some slug, the document would be move to the /orphaned.

[fwqaaq]

Thank you for your answer. Maybe I get it.

1. First, using yarn move content in the en-US documents caused redirection issues.
2. Second, mdn-bot moving the offending the document using git mv.

Maybe so?