CRUD operations eat all the working memory

[MichaelUlbigPTB]

UseCase and effect

We are trying to send data to FIWARE via CRUD(Post/Patch) operations. We have tried it via the requests package from python and via a curl executions in bash script. The scripts patch about 10 data files every 10 seconds. The effect we see is linear increase of active memory usage over time.

Script Example

#!/bin/bash

BASE_URL="http://example:port/ngsi-ld/v1/entities"

update_l1_values() {
  local device_id=$1
  local apparent_power_l1=$2
  local current_l1=$3
  local phase_voltage_l1=$4
  local reactive_power_l1=$5

   local data=$(cat <<EOF
{
  "https://smartdatamodels.org/dataModel.Energy/apparentPower": {
    "type": "Property",
    "value": {
      "L1": $apparent_power_l1
    }
  },
  "https://smartdatamodels.org/dataModel.Energy/current": {
    "type": "Property",
    "value": {
      "L1": $current_l1
    }
  },
  "https://smartdatamodels.org/dataModel.Energy/phaseVoltage": {
    "type": "Property",
    "value": {
      "L1": $phase_voltage_l1
    }
  },
  "https://smartdatamodels.org/dataModel.Energy/reactivePower": {
    "type": "Property",
    "value": {
      "L1": $reactive_power_l1
    }
  }
}
EOF
  )

  curl -X PATCH "$BASE_URL/$device_id/attrs" -H 'Content-Type: application/json' -d "$data"
}

generate_random_value() {
  local value=$((RANDOM % 251))
  echo "$value"
}


device_ids=(
  "urn:ngsi-ld:TPM:1"
  "urn:ngsi-ld:TPM:2"
  "urn:ngsi-ld:TPM:3"
  "urn:ngsi-ld:TPM:4"
  "urn:ngsi-ld:TPM:5"
  "urn:ngsi-ld:TPM:6"
  "urn:ngsi-ld:TPM:7"
  "urn:ngsi-ld:TPM:8"
  "urn:ngsi-ld:TPM:9"
  "urn:ngsi-ld:TPM:10"
  "urn:ngsi-ld:TPM:11"
  "urn:ngsi-ld:TPM:12"
  "urn:ngsi-ld:TPM:13"
)

while true; do
  for device_id in "${device_ids[@]}"; do
    new_apparent_power_l1=$(generate_random_value)
    new_current_l1=$(generate_random_value)
    new_phase_voltage_l1=$(generate_random_value)
    new_reactive_power_l1=$(generate_random_value)

    update_l1_values "$device_id" "$new_apparent_power_l1" "$new_current_l1" "$new_phase_voltage_l1" "$new_reactive_power_l1"
    echo "$(date +'%Y-%m-%d %H:%M:%S') - Updated L1 values for $device_id"
  done

  sleep 10  # Pause for 10 seconds before next iteration
done 

FIWARE Details

 {
  "orionld version": "post-v1.5.1",
  "orion version": "1.15.0-next",
  "uptime": "0 d, 2 h, 43 m, 49 s",
  "git_hash": "nogitversion",
  "compile_time": "Sun Jan 28 08:20:37 UTC 2024",
  "compiled_by": "root",
  "compiled_in": "",
  "release_date": "Sun Jan 28 08:20:37 UTC 2024",
  "doc": "https://fiware-orion.readthedocs.org/en/master/"
}

Our troubleshooting so far

We have tried so far to excplicitly kill the connections with "Connection" : "Close" as well as response.close(). We also sed log=NONE and disabled the logging from the docker container as well. We also know created the bash script just to check if our python script is the problem. But we can see the same effect.

The onyl thing which seems so far to prevent Orion from crashing is to open up a session over the requests package in combination with a with statement:

with requests.Session() as fiware_session:
  response = fiware_session.patch(url = url, json = json_data_copy, headers = {'Content-Type': 'application/ld+json', 'Connection':'Close'})

But also using sessions is eating our active memory it just doesn't die at the end.
Is this a known problem? What are we missing? We are aware of IoT-Agents but we are not sure if this is complient with our security concept and also we try to keep it as simple as possible.

[MichaelUlbigPTB]

We have found the issue 1459.
We also put the context in the body and moved it to the header.
It helped to slow down the memory usage increase but the problem still remains.

We are still confused by this effect and would like an explanation or maybe solutlion.

[kzangeli]

So, you were sending the @context ion the payload body? Over and over again ...
Yeah, that explains the RAM shortage.

The broker stores all contexts in a RAM cache (in a hash table for fast lookups) and each and every request with an "inline" @contest ... well, it's a new context => more RAM usage.

However, I understand you still have a problem with memory.
Are you using subscriptions?
A non.responsive notification-receiver would typically provoke something like what you're seeing.

[MichaelUlbigPTB]

We are using subscriptions but the problem started long before that and we did not notice any change of the behavior.

We are sending the @context via header like:

headers = {
                'Link': '<https://raw.githubusercontent.com/smart-data-models/dataModel.Energy/master/context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"',                
                'Connection':'Close'
            }

And the payload body from our initial post looks like:

{
    "id": "urn:ngsi_ld:entity:test",
    "type": "ThreePhaseAcMeasurement",
    "refDevice": {
        "type": "Property",
        "value": "urn:ngsi_ld:entity:test_device"
    },
    "totalActivePower": {
        "type": "Property",
        "value": null,
        "observedAt": "",
        "unitCode": "kW",
        "measurementType": {
            "type": "Property",
            "value": "online"
        }
    },

    "totalReactivePower": {
        "type": "Property",
        "value": null,
        "observedAt": "",
        "unitCode": "kvar",
        "measurementType": {
            "type": "Property",
            "value": "online"
        }
    },

    "totalApparentPower": {
        "type": "Property",
        "value": null,
        "observedAt": "",
        "unitCode": "kVA",
        "measurementType": {
            "type": "Property",
            "value": "online"
        }
    }
}

And the payload body from our following patch looks like:

{
  'refDevice': {
    'type': 'Property', 
    'value': 'urn:ngsi_ld:entity:test'
  }, 
  'totalActivePower': {
    'type': 'Property', 
    'value': 3600.0, 
    'observedAt': '2024-02-06T12:47:39+00:00', 
    'unitCode': 'kW', 
    'measurementType': {
      'type': 'Property', 
      'value': 'online'
    }
  }, 
  'totalReactivePower': {
    'type': 'Property', 
    'value': 49.25265884399414, 
    'observedAt': '2024-02-06T12:47:39+00:00', 
    'unitCode': 'kvar',
    'measurementType': {
      'type': 'Property', 
      'value': 'online'
    }
  }, 
  'totalApparentPower': {
    'type': 'Property', 
    'value': 3600.336904291486, 
    'observedAt': '2024-02-06T12:47:39+00:00', 
    'unitCode': 'kVA', 
    'measurementType': {
      'type': 'Property', 
      'value': 'online'
    }
  }
}

But we still see the same behavior as before.