Short Time History: Persistence as compound vs. numeric values

[IngMiad]

Hello @kzangeli,

I have another question regarding short term history: When is an attribute saved as compound value and when as dedicated number / boolean / ... in the database?

The background is that I have two different OPC UA real-life datasources that are read via IoT-Agent OPC UA and provided to Orion-LD. Orion-LD persists the data from the two sources in different ways:

1. The first data source is a weather station which uses the WeatherObserved data model (see here: https://raw.githubusercontent.com/smart-data-models/dataModel.Weather/master/WeatherObserved/model.yaml#/WeatherObserved). This data is persisted as numeric values in the time-series-database as expected.
2. The other is a process control system which uses a self-made data model. This data is persisted as compound values in the database for boolean and numeric values. I would also expect that this data is persisted as numeric / boolean values in the time-series-database just like the first WeatherObserved entity.

When I have a look at the data models I don't understand why data from the first source is persisted in a different way compared to data from the second source. Here is a direct example for two different attributes that are persisted as a number in case of the WeatherObserved entity and as a compound value in the case of the process control system:

    # FROM FIRST DATASOURCE
	WeatherObserved:
      description: "Weather conditions"     
      properties:
        windSpeed:
          description: Intensity of the wind
          minimum: 0
          type: number
          x-ngsi:
            model: http//schema.org/Number
            type: Property  
		...

	# FROM SECOND DATASOURCE
    TurbineBearing:
      description: "Process management system component 'Turbine bearing'"
      properties:
        0MAD10CT001_Temp_Lager_P84_1_U:
          description: "0MAD10CT001/Temp_Lager_P84_1.U: Turbine bearing Nr. 10 Measuring circuit Temperature Nr. 001  (Analog value measurement)"
          type: "number"
          x-ngsi:
            model: "https://schema.org/Number"
            type: "Property"
		...

Both data models use a property name, description, type and x-ngsi-model and x-ngsi-type - the first is using a minimum additionally, but this is not true for alle WeatherObserved attributes.

Here is a full example of both entity models for further reference:

    #####################
    ## WeatherObserved ##
    #####################
    # FROM FIRST DATASOURCE
	WeatherObserved:
      description: "Weather conditions"
      required:
        - id
        - type
        #- dateObserved
        #- location      
      properties:
        atmosphericPressure:
          description: The atmospheric pressure observed measured in Hecto Pascals
          minimum: 0
          type: number
          x-ngsi:
            model: https://schema.org/Number
            type: Property
            units: Hecto pascals
        diffuseIrradiation:
          description: Diffuse irradiance is the part of the solar irradiance that is scattered by the atmosphere
          minimum: 0
          type: number
          x-ngsi:
            model: https://schema.org/Number
            type: Property
            units: w/m2
        directIrradiation:
          description: Direct irradiance is the part of the solar irradiance that directly reaches a surface
          minimum: 0
          type: number
          x-ngsi:
            model: https://schema.org/Number
            type: Property
            units: w/m2
        id:
          anyOf:
            - description: Identifier format of any NGSI entity
              maxLength: 256
              minLength: 1
              pattern: ^[\w\-\.\{\}\$\+\*\[\]`|~^@!,:\\]+$
              type: string
              x-ngsi:
                type: Property
            - description: Identifier format of any NGSI entity
              format: uri
              type: string
              x-ngsi:
                type: Property
          description: Unique identifier of the entity
          x-ngsi:
            type: Property
        precipitation:
          description: 'Amount of water rain registered. '
          minimum: 0
          type: number
          x-ngsi:
            model: https://schema.org/Number
            type: Property
            units: Liters per square meter
        refDevice:
          anyOf:
            - description: Identifier format of any NGSI entity
              maxLength: 256
              minLength: 1
              pattern: ^[\w\-\.\{\}\$\+\*\[\]`|~^@!,:\\]+$
              type: string
              x-ngsi:
                type: Property
            - description: Identifier format of any NGSI entity
              format: uri
              type: string
              x-ngsi:
                type: Property
          description: A reference to the device(s) which captured this observation
          x-ngsi:
            model: https://schema.org/URL
            type: Relationship
        relativeHumidity:
          description: Humidity in the Air. Observed instantaneous relative humidity (water vapour in air)
          maximum: 1
          minimum: 0
          type: number
          x-ngsi:
            type: Property
        solarRadiation:
          description: The solar radiation observed measured in Watts per square
          minimum: 0
          type: number
          x-ngsi:
            model: https://schema.org/Number
            type: Property
            units: w/m2
        temperature:
          description: Temperature of the item
          type: number
          x-ngsi:
            type: Property
        type:
          description: NGSI Entity type. It has to be WeatherObserved
          enum:
            - WeatherObserved
          type: string
          x-ngsi:
            type: Property
        windDirection:
          description: Direction of the wind bet
          maximum: 360
          minimum: 0
          type: number
          x-ngsi:
            model: http://schema.org/Number
            type: Property
        windSpeed:
          description: Intensity of the wind
          minimum: 0
          type: number
          x-ngsi:
            model: http//schema.org/Number
            type: Property     


    ####################
    ## TurbineBearing ##
    ####################
	# FROM SECOND DATASOURCE
    TurbineBearing:
      description: "Process management system function 'Turbine bearing' with KKS function code: 'MAD' (de: 'Turbinenlagerung')"
      required:
        - "id"
        - "type"
        - "dateObserved"
        - "isPartOf"
      properties:
        id:
          anyOf:
            - description: "Property. Identifier format of any NGSI entity"
              maxLength: 256
              minLength: 1
              pattern: '^[\w\-\.\{\}\$\+\*\[\]`|~^@!,:\\]+$'
              type: "string"
            - description: "Property. Identifier format of any NGSI entity"
              format: "uri"
              type: "string"
          description: "Unique identifier of the entity"
          x-ngsi:
            type: "Property"
        type:
          description: "NGSI Entity type. It has to be 'TurbineBearing' (a specific PCS part)"
          enum:
            - "TurbineBearing"
          type: "string"
          x-ngsi:
            type: "Property"
        dateObserved:
          description: "Date of the observed entity defined by the user."
          type: "string"
          x-ngsi:
            type: "Property"
        isPartOf:
          description: "This component is a sub component of another system"
          type: "string"
          x-ngsi:
            model: "https://schema.org/isPartOf"
            type: "Relationship"
        kksFunction:
          description: "KKS function of property (level 1)"
          type: "string"
          x-ngsi:
            type: "Property"
        0MAD10CT007_Temp_Lager_P84_7_U:
          description: "0MAD10CT007/Temp_Lager_P84_7.U: Turbine bearing Nr. 10 Measuring circuit Temperature Nr. 007  (Analog value measurement)"
          type: "number"
          x-ngsi:
            model: "https://schema.org/Number"
            type: "Property"
        0MAD10CT007_Temp_Lager_P84_7_VSTATUS:
          description: "0MAD10CT007/Temp_Lager_P84_7.VSTATUS: Turbine bearing Nr. 10 Measuring circuit Temperature Nr. 007  (VSTATUS)"
          type: "boolean"
          x-ngsi:
            model: "https://schema.org/Boolean"
            type: "Property"
        0MAD10CT008_Temp_Lager_P84_8_U:
          description: "0MAD10CT008/Temp_Lager_P84_8.U: Turbine bearing Nr. 10 Measuring circuit Temperature Nr. 008  (Analog value measurement)"
          type: "number"
          x-ngsi:
            model: "https://schema.org/Number"
            type: "Property"
        0MAD10CT008_Temp_Lager_P84_8_VSTATUS:
          description: "0MAD10CT008/Temp_Lager_P84_8.VSTATUS: Turbine bearing Nr. 10 Measuring circuit Temperature Nr. 008  (VSTATUS)"
          type: "boolean"
          x-ngsi:
            model: "https://schema.org/Boolean"
            type: "Property"
        0MAD10CT001_Temp_Lager_P84_1_U:
          description: "0MAD10CT001/Temp_Lager_P84_1.U: Turbine bearing Nr. 10 Measuring circuit Temperature Nr. 001  (Analog value measurement)"
          type: "number"
          x-ngsi:
            model: "https://schema.org/Number"
            type: "Property"
		...

[kzangeli]

Hello Inga,

the broker simply stores the values as they come in.
It doesn't make any decisions, it just follows orders.
In your case it seems like some values are compounds and others are not.
If compound (Array or Object), they will be stored as Array or Object.
If simple data type, then that way.
I can only assume that that's what you're experiencing.

[kzangeli]

And, the broker is agnostic to data models.
It only looks at the incoming payload body.
If you provide those bodies, I would be able to explain this in more detail

[IngMiad]

Hello @kzangeli,

thank you for the response. I had time to investigate the incoming payloads using wireshark and the logs of the OPC UA IoT-Agent. The logs of the IoT-Agent seem to be more useful for investigation.

First data source with numeric values:

...
trans=5fe32dba-7267-4ff4-8276-31d875e3c7a7 | op=IoTAgentNGSI.Request | from=n/a | srv=n/a | subsrv=n/a | msg=Options: {
    "url": "http://orion-ld:1026/ngsi-ld/v1/entityOperations/upsert/?options=update",
    "method": "POST",
    "headers": {
        "fiware-service": "opcua_meteo",
        "fiware-servicepath": "/weather",
        "Content-Type": "application/ld+json",
        "NGSILD-Tenant": "opcua_meteo",
        "NGSILD-Path": "/weather"
    },
    "json": [
        {
            "@context": "http://ld-context/ld-context/datamodels.context-ngsild.jsonld",
            "temperature": {
                "type": "Property",
                "value": 46252,
                "observedAt": "2024-09-26T15:46:16.925Z"
            },
            "id": "urn:ngsi-ld:WeatherObserved:meteostation-01",
            "type": "WeatherObserved"
        }
    ]
} | comp=IoTAgent
...

Second data source with compound values:

...
trans=89b57f04-8b90-42f1-bd47-11fdd947af41 | op=IoTAgentNGSI.Request | from=n/a | srv=n/a | subsrv=n/a | msg=Options: {
    "url": "http://orion-ld:1026/ngsi-ld/v1/entityOperations/upsert/?options=update",
    "method": "POST",
    "headers": {
        "fiware-service": "opcua_pcs",
        "fiware-servicepath": "/pcs",
        "Content-Type": "application/ld+json",
        "NGSILD-Tenant": "opcua_pcs",
        "NGSILD-Path": "/pcs"
    },
    "json": [
        {
            "@context": "http://ld-context/ld-context/datamodels.context-ngsild.jsonld",
            "id": "urn:ngsi-ld:PCS-STJ:DuctSystemHotAir",
            "type": "DuctSystemHotAir",
            "0HNB97CF901_LuMe_WL_TRFUnten_U": {
                "type": "Property",
                "value": {
                    "@type": "https://schema.org/Number",
                    "@value": 0.2824963629245758
                },
                "observedAt": "2024-09-26T15:45:59.444Z"
            }
        }
    ]
} | comp=IoTAgent
...

I guess the difference is then that the IoT-Agent is providing the data from the second datasource in an array with type + value for some reason. This way I should check for the difference at IoT-Agent side. Thank you very much for clarification!

Greetings
Inga