Search results do not convert/remove footnote references correctly

[pftnhr]

I installed an empty instance of yellow locally with the only extension yellow-search to find out at what point the strange behaviour occurs that the entity of the footnote reference [^1] is no longer converted to 1 in the search results. The placeholder for images, as well as the resulting <img> tag, will continue to be completely hidden. Perhaps the footnote references should also be completely hidden from the search results.

The limit is 4,088 characters between the second --- of the header of the *.md and the first footnote ([^1]: Footnote). It does not matter whether the characters are placeholders for images, footnotes, etc. or normal text.

There is no error message. Neither in debug mode nor in the yellow-website.log:

2023-08-29 09:24:58 info Install Datenstrom Yellow 0.8.22, PHP 8.2.0, Apache 2.4.54, Mac
2023-08-29 09:24:58 info Install extension 'Update 0.8.94'
2023-08-29 09:24:58 info Install extension 'Core 0.8.115'
2023-08-29 09:24:58 info Install extension 'Edit 0.8.75'
2023-08-29 09:24:58 info Install extension 'Generate 0.8.52'
2023-08-29 09:24:58 info Install extension 'Image 0.8.19'
2023-08-29 09:24:58 info Install extension 'Markdown 0.8.24'
2023-08-29 09:24:58 info Install extension 'Serve 0.8.24'
2023-08-29 09:24:58 info Install extension 'Stockholm 0.8.14'
2023-08-29 09:24:58 info Install extension 'English 0.8.43'
2023-08-29 09:24:58 info Install extension 'German 0.8.43'
2023-08-29 09:24:58 info Install extension 'Swedish 0.8.43'
2023-08-29 09:25:24 info Install extension 'Blog 0.8.26'
2023-08-29 09:25:25 info Add user 'Test'
2023-08-29 09:26:41 info Install extension 'Search 0.8.26'

I'm not very good at describing what doesn't work properly, but I'm happy to answer questions.

[GiovanniSalmeri]

I suspect that the cause could be in search.php:151:

        return substrb($page->rawData, 0, $page->metaDataOffsetBytes).substrb($output, 0, 4096);

which actually keeps only the first 4096 bytes. The footnotes are normally at the bottom of the markdown file, so if the text is longer they are not included in this portion of the file. But a footnote reference of the form [^xxx] is left as it is by the markdown parser if the corresponding footnote is not found. Therefore this strange result. Did I guess right? 🤔

[pftnhr]

I set the value to four times and everything is as it should be. So I guess your guess was right. But I'll wait for @annaesvensson and/or @markseuffert's opinion before marking the answer.

[annaesvensson]

Thank you for the bug report and the additional information. Maybe reference numbers should be taken out of the text description entirely, they don't serve any purpose, as far as I can see. We will take a closer look at the strange behaviour.

[pftnhr]

Ok, it has already been done. Giovanni's guess seems to be right, as I no longer have the error with 16,384 bytes.

[annaesvensson]

This bug was caused by the code to extract relevant search results, the intention being that if you display only some text characters from the top of a page, you don't render the entire page and throw most of it away afterwards. Rendering only 4096 bytes showed a significant difference during performance testing. Unfortunately this kind of performance improvement doesn't work for Markdown footnotes, as Giovanni explained. Looks like we have to spend more CPU time to handle possible footnotes. The search extension and core extension have been updated, text descriptions no longer contain reference numbers as suggested by Robert.

Should be fixed now, please try it out.

[pftnhr]

I have tried it and can confirm that it's now fixed.