existing entities are overwritten when creating a device

[jkriwet]

Hi,

i dont know if this is intented behavior, but if a device is created, referencing an existing entity from the context broker, this entity is overwritten and all the attributes are lost.

The following steps can be done to replicate the issue:
1.) Create an entity with an id and post it to the context broker, for example the following entity

 {
    "id": "urn:ngsi-ld:CoreSensor:001",
    "type": "CoreSensor",
    "temperature": {
        "type": "Property",
        "value": 35
    },
    "core_load": {
        "type": "Property",
        "value": 100
    },
    "@context": [
        "http://context/ngsi-context.jsonld"
    ]
}

2.) Create and post a device, referencing this entity

{'devices': [
    {
        "device_id": "CoreSensor:001",
        "entity_name": "urn:ngsi-ld:CoreSensor:001",
        "entity_type": "CoreSensor",
        "timezone": "Europe/Berlin",
        "transport": "MQTT",
        "protocol": "IoTA-JSON",
        "attributes": [
            {
              'name': 'temperature',
              'type': 'Property'
            }
        ]
    }]}

3.) Get the entity from the context broker:

• Expected behavior: I get the entity with the previous defined attributes and since the device has not made any changes to the attributes, i basically get back the same thing as i posted in step 1.)
• Actual behavior: I get back the following

{
 'id': 'urn:ngsi-ld:CoreSensor:001', 
 'type': 'CoreSensor'
}

Following are the logs of the IOT Agent:

time=2023-06-01T15:40:32.076Z | lvl=DEBUG | corr=5a2245bc-248f-4443-aa01-46afc02b3683 | trans=5a2245bc-248f-4443-aa01-46afc02b3683 | op=IoTAgentNGSI.DeviceService | from=n/a | srv=test | subsrv=/first_test | msg=Registering device into NGSI Service:
{
    "id": "CoreSensor:001",
    "type": "CoreSensor",
    "name": "urn:ngsi-ld:CoreSensor:001",
    "service": "test",
    "subservice": "/first_test",
    "active": [
        {
            "name": "temperature",
            "type": "Property",
            "object_id": "temperature"
        }
    ],
    "staticAttributes": [],
    "lazy": null,
    "commands": [],
    "timezone": "Europe/Berlin",
    "protocol": "IoTA-JSON",
    "transport": "MQTT",
    "internalId": null,
    "explicitAttrs": true,
    "subscriptions": []
} | comp=IoTAgent
time=2023-06-01T15:40:32.076Z | lvl=DEBUG | corr=5a2245bc-248f-4443-aa01-46afc02b3683 | trans=5a2245bc-248f-4443-aa01-46afc02b3683 | op=IoTAgentNGSI.Registration | from=n/a | srv=test | subsrv=/first_test | msg=Registration with Context Provider is not needed. Device without lazy atts or commands | comp=IoTAgent
time=2023-06-01T15:40:32.076Z | lvl=DEBUG | corr=5a2245bc-248f-4443-aa01-46afc02b3683 | trans=5a2245bc-248f-4443-aa01-46afc02b3683 | op=IoTAgentNGSI.Devices-LD | from=n/a | srv=test | subsrv=/first_test | msg=config.timestamp undefined true | comp=IoTAgent
time=2023-06-01T15:40:32.076Z | lvl=DEBUG | corr=5a2245bc-248f-4443-aa01-46afc02b3683 | trans=5a2245bc-248f-4443-aa01-46afc02b3683 | op=IoTAgentNGSI.Devices-LD | from=n/a | srv=test | subsrv=/first_test | msg=deviceData: {"id":"CoreSensor:001","type":"CoreSensor","name":"urn:ngsi-ld:CoreSensor:001","service":"test","subservice":"/first_test","active":[{"name":"temperature","type":"Property","object_id":"temperature"}],"staticAttributes":[],"lazy":null,"commands":[],"timezone":"Europe/Berlin","protocol":"IoTA-JSON","transport":"MQTT","internalId":null,"explicitAttrs":true,"subscriptions":[]} | comp=IoTAgent
time=2023-06-01T15:40:32.076Z | lvl=DEBUG | corr=5a2245bc-248f-4443-aa01-46afc02b3683 | trans=5a2245bc-248f-4443-aa01-46afc02b3683 | op=IoTAgentNGSI.Devices-LD | from=n/a | srv=test | subsrv=/first_test | msg=Creating initial entity in the Context Broker:
 {
    "url": "http://scorpio:9090/ngsi-ld/v1/entityOperations/upsert/",
    "method": "POST",
    "json": [
        {
            "@context": "http://context/ngsi-context.jsonld",
            "id": "urn:ngsi-ld:CoreSensor:001",
            "type": "CoreSensor"
        }
    ],
    "headers": {
        "fiware-service": "test",
        "fiware-servicepath": "/first_test",
        "NGSILD-Tenant": "test",
        "NGSILD-Path": "/first_test",
        "Content-Type": "application/ld+json"
    }
} | comp=IoTAgent

It can be seen that an upsert request is performed, coupled with the message that the "initial entity in the Context Broker" is created. This overwrites the previous entity. However, this makes the IOT Agent less useful in a scenario where entities already exist that you want to reference via the IOT Agent.

I suggest the following approach, which I unfortunately can't implement myself because I can't code JavaScript :)

1. For each device creation it is first checked if the referenced entity already exists.
   2.a) If the entity exists, it is updated with the attributes defined in the device. Matching attributes remain unchanged, new attributes are updated and the value is set to None.
   2.b) If the Entity does not exist, it will be created. However, the defined attributes are also passed directly, and the values are set None.

This would have the advantage of being able to connect already created entities to the IOT Agent without losing any information. On the other hand an update with the attributes, which stand in the device configuration, would have the advantage that even if still no message was received, a query of the entity shows, which attributes of this are actually intended, even if the values for it are in each case still None.

[mapedraza]

Did you check this part of the documentation?:
https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/api.md#measurement-persistence-options

Specially the config parameter appendMode

[jkriwet]

Thank you for your comment! Yes i have tried playing with those options, including the appendMode option. But this doesnt change the behaviour for me.

From what i understood those 3 options are primarily responsible for controlling how the Iot Agent behaves, when it receives a message, right?

What i am investigating is at a point in time, where there has no message yet been send or received. The 3 options have no influence on the initial creation of the entity via an upsert request, which overwrites the existing entity.

[mapedraza]

In theory, using Append Mode should not overwrite the entities in the broker. As far as I can see, you are using scorpio broker. Probably @jason-fox could give you more info about the NGSI-LD behaviour regarding how the IoT Agent interacts with the broker itself and the expected behaviour

[jkriwet]

But just in general, no matter the combination of the the 3 options, what the IOTAgent does, is an upsert request, creating the "initial entity", which has at that point not received any message. But with that upsert request, the context broker overwrites an existing entity, if there may be an existing entity with the provided id. Again from the log:

msg=Creating initial entity in the Context Broker:
 {
    "url": "http://scorpio:9090/ngsi-ld/v1/entityOperations/upsert/",
    "method": "POST",
    "json": [
        {
            "@context": "http://context/ngsi-context.jsonld",
            "id": "urn:ngsi-ld:CoreSensor:001",
            "type": "CoreSensor"
        }
    ],
    "headers": {
        "fiware-service": "test",
        "fiware-servicepath": "/first_test",
        "NGSILD-Tenant": "test",
        "NGSILD-Path": "/first_test",
        "Content-Type": "application/ld+json"
    }
} | comp=IoTAgent

So i dont think this is "false behaviour" from the context broker, is it? I personally think the request from the iot agent could be of the form:

msg=Creating initial entity in the Context Broker:
  {
    "url": "http://scorpio:9090/ngsi-ld/v1/entityOperations/upsert?options=update",
    "method": "POST",
    "json": [
        {
            "@context": "http://context/ngsi-context.jsonld",
            "id": "urn:ngsi-ld:CoreSensor:001",
            "type": "CoreSensor"
        }
    ],
    "headers": {
        "fiware-service": "test",
        "fiware-servicepath": "/first_test",
        "NGSILD-Tenant": "test",
        "NGSILD-Path": "/first_test",
        "Content-Type": "application/ld+json"
    }
} | comp=IoTAgent

To make sure, that when creating the initial entity, existing entities are not overwritten. This is of course only then an issue, if this would be the expected or intended behaviour of the iot agent.

Maybe i shortly explain what i use that for and why this is an issue for me and maybe this is also more of a niche usecase and i just have to work around that if this is not what the iot agent is supposed to be used for :D

I have an onthology describing a system i am modelling. I now have all the entities and relationships of the system (according to the given onthology) in the context broker, so many entities with different propertys and relationships.

Only a subset of those property attributes i now want to "connect" to the iot agent. So the best workflow would be to create all the entities in the context broker with all the information and then connect the subset of the entities each with their subset of attributes to the iot agent.

A workaround at the moment is to just change the order, first create the devices for the iot agent and then create the entitys, then everything works as expected.

But i was just wondering if this is in general expected behaviour of the iotagent, of if not maybe having upsert?options=update when provisioning the devices would be a good idea? Basically everything would stay the same, if an entity under the given id doesnt exist, it will be created, the given attributes are all written to the entity, the only thing that changes would be, that when an entity under this id already exists, attributes which arent referenced in the iot agent arent deleted.