MPIEXEC Error in Writing Normal Equations

[ParisaShafiei]

Dear developers,

In order to write down the normal equations in network processing, I have followed the procedure mentioned in the cookbook. However, when I launch my config file, as soon as it comes to the last step of processing, i.e. writing normal equations, it exited with the error shown in the attached.

What is the solution to this problem? Did something go wrong with groopsMPI installation?

Thank you in advance for your time and help.

With kind regards,
Parisa

[zhedumi]

Hi Parisa,

On first glance it is hard to say what lead to this error. Could you provide the groops script for it so i can check in detail?

best regards,
Patrick

[akvas]

Hi Parisa,

signal 9 may have a number of causes (cf. Wikipedia). Essentially, this is your operating system telling the process to terminate immediately. One reason for this may be that the memory on your machine was full and GROOPS could not allocate any more.

As a first step, I would run the script again and check the memory consumption on the machine with a tool like top that should be installed on most Linux distributions by default.

Cheers,
Andreas

[ParisaShafiei]

Hi @zhedumi and @akvas,

This is what I have done so far:

1. using ~650 IGS stations in my network processing ------ writing normal equations failed
2. decreasing the number of stations to 498 ------ writing normal equations failed
3. decreasing the number of stations to 360 ------ managed to write the normal equations

Now, I checked the memory and CPU consumption with top and I noticed when I came to writing normal equations, the CPU consumption percentage went up, reached 100% and then killed processing.

Thanks for your time and help.

With kind regards,
Parisa

[akvas]

Hi Parisa,

in your first screenshot, you can see that of the 256 GB available memory, already 253 GB are used. Probably more memory is then allocated which stops GROOPS. Your successful tests with a decreased number of stations also suggest that it is a memory consumption issue. @zhedumi is there a setting to reduce the number of parameters before accumulating?

Cheers,
Andreas

[zhedumi]

Hi Andreas,

Yeah with selectParametrization it should be possible to disable all other parameters except the ones you actually want if I remember correct. Problem that the accumulation of the normals is not in the estimation processStep but in the writeNormalEquations processingStep which will if epochwise parameters are selected will always create them along.

Best regards,
Patrick

[tmayerguerr]

Hi Parisa,

A way to reduce the memory consumption is to add before a processingStep->selectNormalsBlockStructure with keepEpochNormalsInMemory = no .

regards
Torsten

[ParisaShafiei]

Hi Torsten and Andreas,

Thanks for your feedback.

I checked my config file and I had added "selectNormalsBlockStructure" with keepEpochNormalsInMemory = no in the middle of my processing steps. But based on what you told me, I have added it in the first part of my processing steps as you see in the following screenshot. However, I have still the same issue in getting the normal equations which gives me the same error that I had before.

With kind regards,
Parisa

[ParisaShafiei]

Dear @zhedumi,

Here, I have attached my GROOPS script file for writing down the normal equations in network processing. I would be pleased if you could take a look at it. I am wondering why I still cannot get the normal equations when the number of stations reaches to more than 360. Even though I added the setting not to keep the normals in the memory, I could not succeed.

Like always, thank you so much for your time and help.

With kind regards,
Parisa
030GnssProcessing-test-for-normal-equations.zip

[zhedumi]

Hi Parisa,

Sorry for the late replay. I honestly thought I had already answered this more thorougly reason. Yeah it still has way to many parameters as seen in the selectParametrization part of the log. Also I somehow had in mind that your accumulation fails during the estimation process but you are already past that in ur GNSS processing.
So I think you can do two solutions:
First try to disable all epochwise parameters. I see in your script that you enable transmitterAntenna parameters but this does not mean that it disables all other parameters. It will just set the parameter to enable additionally to any other parameters. So you could just also add disable parameters with everything else in there.

1. Enable only the transmitterAntenna and disable all other parameters before writing the normalEquations. This will write your transmitterAntenna parameters alone.
2. Enable transmitterAntenna parameters and enable any not epochwise parameters and then write normalEquations with remainigParameters your transmitterAntenna. Just make sure that any epoch wise parameters are disabled
3. Write your equations stationwise PPP-wise: Make a forEachReceiverSeparately and select your station in selectParametrization and then write it out with writeNormalEquation with remaining parameters your transmitterAntennas. This is probably the optimal way in my oppinion since it will do parameterElimination of the other parameters before writing the normalequations

Best regards,
Patrick

[ParisaShafiei]

Hi Patrick,

I should say thank you that you went through my problem and the log file.

Now, I found what was the problem in writing down the normal equations.

First of all, as a side note I'd like to add that in my previous processing, I had chosen a wrong paramterization for the transmitter antennas. So, I changed it to radialBasis based on what is mentioned in Norbert's PhD thesis:

transmitterAntennas -> anrtennaCenterVariations: radialBasis -> triangleVertex -> level=20, maxDegree=66, cap=14 degree

Secondly, I added "selectNormalBlockStructure" at the first part of my processingSteps and removed all the default values in it. (Please see the attached screenshot).

In the end, based on what you told me, by disabling all the other parameters except the parameter.transmitterAntenna, I managed to get normal equations for my whole network consisting of ~ 600 stations whitout any problems :)

So, big thanks to you as well (@tmayerguerr and @akvas ) for your help and support.

With kind regards,
Parisa