Problem downloading packages with GitHub Actions running into a timeout

[albert-github]

In the issue: #1228 I mentioned that on GitHub Actions I got a timeout when executing:

$wc.DownloadFile("https://ctan.math.illinois.edu/systems/win32/miktex/setup/windows-x64/miktexsetup-5.2.0+b8f430f-x64.zip","miktexsetup-5.2.0+b8f430f-x64.zip")
unzip miktexsetup-5.2.0+b8f430f-x64.zip
miktexsetup_standalone --verbose --local-package-repository="C:/miktex-repository" --remote-package-repository="https://ctan.math.illinois.edu/systems/win32/miktex/tm/packages/" --package-set=essential download

See https://github.com/doxygen/doxygen/actions/runs/3534887466/jobs/5932282689#step:10:13).

Based on the comment in the mentioned issue:

I cannot reproduce the miktexsetup_standalone issue, i.e., this works for me:

miktexsetup_standalone.exe --remote-package-repository=https://ctan.math.illinois.edu/systems/win32/miktex/tm/packages/ --local-package-repository=./miktex/download --package-set=essential --quiet download

I also tried this locally and it did work. The package biber-darwin-x86_64.tar.lzma takes a bit longer as it is the largest package:

downloading https://ctan.math.illinois.edu/systems/win32/miktex/tm/packages/biber-darwin-x86_64.tar.lzma (expecting 86865097 bytes)...

Based on this I looked into the last good run on GitHub Actions (https://github.com/doxygen/doxygen/actions/runs/3506865429/jobs/5874999704) and I see here that there is no biber-darwin-x86_64.tar.lzma (or other biber... package here).

So my guess is that one or more large packages have entered the set of essential packages and that either the site now rejects to download these packages as it takes a bit longer or that in the miktexsetup_standalone.exe there is some timeout / size limitation on the files.
Question also is: is the biber package an essential package? and when downloading to Windows why also download the biber versions for darwin and Linux (packages are probably large due to the executables that are contained in the packages).

I also looked in the documentation of miktex_standalone (https://docs.miktex.org/manual/miktexsetup.html) but I didn't see a possibility to increase the timeout.

Does anybody have an idea?

[albert-github]

I think it is good that I mention here the complete message I get:

Run ./miktexsetup_standalone --verbose \
  ./miktexsetup_standalone --verbose \
                --local-package-repository="C:/miktex-repository" \
                --remote-package-repository="https://ctan.math.illinois.edu/systems/win32/miktex/tm/packages/" \
                --package-set=essential \
                download
  shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
this is MiKTeX Setup Service 5.4 (MiKTeX 22.10)
starting downloader...
miktexsetup_standalone: Timeout was reached
miktexsetup_standalone: Data: code="28", url="https://api2.miktex.org/hello"
Error: Process completed with exit code 1.

according to #1228 the problem with "https://api2.miktex.org/hello" should be solved (unfortunately a number of relevant comments were removed).
Though we still face the problem.

[albert-github]

Relevant to this discussion is also #1237 though this has been close with:

Thank you. I don't know what is going wrong and so it is not possible to fix the issue. A future version of the setup utility might provide more trace information.

So it is not only happening on GitHub Actions but also on Azure VM's in GWC and on these platforms MiKTeX cannot be used anymore ...

In the cited comment there was some reference to possible extra debug facilities in the future so:

• is there a time schedule for this future version?

[jjolidon]

As of now, I believe miktex.org is simply not reachable from the Azure Pipelines agents. Downloading the command line setup from https://miktex.org/download/ctan/systems/win32/miktex/setup/windows-x64/miktexsetup-5.2.0+b8f430f-x64.zip simply fails with a timeout.

I tried running Test-NetConnection from the agent, here is the output:

WARNING: TCP connect to (92.205.128.66 : 443) failed
WARNING: Ping to 92.205.128.66 failed with status: TimedOut
VERBOSE: miktex.org
VERBOSE: miktex.org
VERBOSE: Perform operation 'Invoke CimMethod' with following parameters, ''methodName' = QueryIsolationType,'namespaceName' = root\standardcimv2,'className' = MSFT_NetAddressFilter'.
VERBOSE: Operation 'Invoke CimMethod' complete.

ComputerName           : miktex.org
RemoteAddress          : 92.205.128.66
RemotePort             : 443
InterfaceAlias         : Ethernet 3
SourceAddress          : 10.1.0.40
PingSucceeded          : False
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : False


Finishing: Check connectivity to miktex.org

[jjolidon]

I've opened a ticket with the Azure DevOps team, I believe the issue with GitHub Actions may be related. https://developercommunity.visualstudio.com/t/miktexorg-not-reachable-from-build-agen/10231770?q=miktex+not+reachable

[albert-github]

I don't think the problem lies with Azure or GitHub, it might be a subtle interaction between MiKTeX and Azure / GitHub but is all a bit looking int darkness as there is no debug possibility (at least for me). In #1237 (comment) there is a reference of potential extra verbose facilities in the future but no time schedule yet #1233 (comment).

As a side note for doxygen on GitHub we switched (temporarily ?) to tlmgr through teatimeguest/setup-texlive-action@v2 (see https://github.com/teatimeguest/setup-texlive-action) maybe something similar exists for Azure.

[jjolidon]

Yeah, so I hope the Azure team has good networking troubleshooting tools and will be able to understand what's wrong. The issue is pure networking, on Azure at least, as the Test-NetConnection is really just doing a SYN-ACK check on the specified port (HTTPS). This should work for any public website. ICMP also fails, so the issue is not only with TCP, though.

Since Microsoft owns GitHub I believe the GitHub Actions are executed on the Azure stack, which is why I believe it is related. Visibly TexLive is not impacted, only miktex.org.

I just finished testing the range of CTAN mirrors from Azure DevOps and the good news is - at least near to me, i.e. in Europe - none is unreachable.