Inconsistencies for transient entities handling in JSON format

[falbert-ptc]

Both 4.0 and 4.01 seems to have inconsistencies regarding handling of transient entities in the JSON format.

Null ID for transient entities in JSON:

• In 4.0, it is stated that if an entity is transient, then id must be null in JSON (http://docs.oasis-open.org/odata/odata-json-format/v4.0/errata03/os/odata-json-format-v4.0-errata03-os-complete.html#_Toc453766634).
• In 4.01, it is stated that you don’t even need to return an id in the response (https://docs.oasis-open.org/odata/odata-json-format/v4.01/odata-json-format-v4.01.html#_Toc38457743).

However, it seems to be incompatible with at least 2 other use cases:

1. In the case of recursive navigations ($expand=Nav($levels=max)), it is stated you must replace the entity instance by a reference, using its id:

• http://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/part1-protocol/odata-v4.0-errata03-os-part1-protocol-complete.html#_Toc453752287
• http://docs.oasis-open.org/odata/odata-json-format/v4.0/errata03/os/odata-json-format-v4.0-errata03-os-complete.html#_Toc453766656
• When the entities are transient, it is impossible since none of them will have an id.

2. In the case of navigationLink, it is stated it must contain the read URL appended with the navigation property name:

• http://docs.oasis-open.org/odata/odata-json-format/v4.0/errata03/os/odata-json-format-v4.0-errata03-os-complete.html#_Toc453766637
• When the entities are transient, they do not have a read URL.

In the case of the XML format, this is not an issue since even transient entities have an id. The client can use those transient ids when parsing and make sense of the references.

[ralfhandl]

@falbert-ptc Thanks for reporting this, we will discuss it in one of the next OData TC meetings.

[ralfhandl]

Regarding use case #1 - recursive navigation between transient entities: I don't think we have a problem here because 11.2.5.2.1.1 Expand Option $levels only requires that services

MUST solve circular dependencies by injecting an entity reference somewhere in the circular dependency

Transient entities don't have an identity, so there can't be circles consisting only of transient entities.

@falbert-ptc Do you have use cases for circular references involving transient entities?

[ralfhandl]

Regarding use case #2 - navigation links of transient entities: with 4.02 4.3 Transient Entities may have an explicit read URL, which means they also have default navigation links (read URL plus navigation segment).

If they don't have a read URL, they may have explicit navigation links.

@falbert-ptc: Do you have use cases for transient entities with navigation links?

[falbert-ptc]

We do have use cases with circular references between transient entities.

We have an endpoint that returns structured data (parent-child relationship) of transient entities. Each transient entity represent an information aggregate of multiple data sources, and cannot be addressed individually outside of the current response. Each transient entity has multiple navigations to said data sources.

While rare, there are cases where there could be circular references (direct or indirect) between an ascendant and a descendant in the structure.
In that case, we are supposed to insert an entity reference.

The issue is with the JSON format, where transient entities are specifically stated to not have an entity id. So it's impossible to have an entity reference or a read URL + navigation segment.
Were that not the case, the above 2 use cases would not be a problem.
For XML, transient entities have an entity id so there is no issue with what the protocol prescribes w.r.t. navigation links or entity references.

Note that in my research I've only looked at 4.0 and 4.01. I wasn't aware that 4.02 had been published. Although, I did not find any update to the JSON format in 4.02 (https://docs.oasis-open.org/odata/odata/v4.02/).

[ralfhandl]

@falbert-ptc Thanks for the explanation!

How would you recognize circular references between transient entities? Do these entities have an internal id that you can use to determine whether two of them are really the same and not just coincidentally have the same representation?

[ralfhandl]

For XML, transient entities have an entity id

With XML, do you mean the OData Atom Format Version 4.0?

If yes, do you use this in "production" software?

[falbert-ptc]

Correct.
Each transient entity has a unique ID within that response.
This is generated in the format described in the Atom format (odata:transient:{some-generated-unique-identifier-to-not-break-atom-parsers})

That way, the client can use those unique ID when finding an entity reference.

[falbert-ptc]

For XML, transient entities have an entity id

With XML, do you mean the OData Atom Format Version 4.0?

If yes, do you use this in "production" software?

Yes, that's what I meant.
We do use it in production, commercial software. Although our support for XML is less extensive than for JSON. Because of market demand we are focusing on JSON.

[ralfhandl]

Each transient entity has a unique ID within that response.

Could you use this to construct a non-canonical URI to use in the @odata.id, pretend that your transient entities are not transient, and return 410 Gone if someone actually tries to request such a URI?

That way you could use entity references to express cycles within a response.

[ralfhandl]

We do use it in production, commercial software.

Did you encounter any gaps in the Atom format?

[falbert-ptc]

Each transient entity has a unique ID within that response.

Could you use this to construct a non-canonical URI to use in the @odata.id, pretend that your transient entities are not transient, and return 410 Gone if someone actually tries to request such a URI?

That way you could use entity references to express cycles within a response.

Yes, that would be the easiest and preferred way.
The issue is the JSON format currently specifies that the "@odata.id" value must be null if it's a transient entity: "If the entity is transient (i.e. cannot be read or updated), the odata.id annotation MUST appear and have the null value." .
We had this reported as a defect in our software (where odata.id wasn't null for transient entities in JSON). When we fixed it to set it to null, it made it compliant with the spec but we didn't realize the impacts on the 2 use cases mentioned in this ticket.

[ralfhandl]

We had this reported as a defect in our software (where odata.id wasn't null for transient entities in JSON).

Which (kind of) client (software/app) did stumble over this, and how could they figure out that the entities were transient?

I'd like to know how much chaos we may cause if we relax the specification and allow "temporary" or unresolvable URIs in addition to null for transient entities.

[falbert-ptc]

We do use it in production, commercial software.

Did you encounter any gaps in the Atom format?

What we're finding challenging with Atom is that there are less examples, or examples are a lot more abstract (as opposed to the Order/Customer/etc. examples). One use case that comes to mind is action invocation: we found it hard to understand the correct way to pass action parameters in the request body.
But I can't say that we've found gaps.

Because this is not a widely requested feature, our investment has mostly been on the JSON format.

[ralfhandl]

read URL + navigation segment

With 4.02 we explicitly allow transient entities to have read URLs, which means they also can have default computed navigation links that can be omitted from the payload. This of course requires the service to correctly answer requests to {navigationLink}/{navigationProperty}.

Already with 4.0 it was possible to have explicit navigation links in transient entities, if one doesn't mind the resulting verbosity of the JSON responses.

[falbert-ptc]

We had this reported as a defect in our software (where odata.id wasn't null for transient entities in JSON).

Which (kind of) client (software/app) did stumble over this, and how could they figure out that the entities were transient?

I'd like to know how much chaos we may cause if we relax the specification and allow "temporary" or unresolvable URIs in addition to null for transient entities.

Honestly, I do not remember the specifics.
In our records, I could not find a mention of the actual client software that reported the issue. I think it was a middleware like Boomi, but I'm really not sure.
Also, at that time our OData implementation would not correctly generate a unique transient entity ID (the entity ID in Atom XML was also not unique and wasn't prefixed with "odata:transient:"). I think the client was trying to use that ID as a read URL, but it would fail because it could not determine it was a transient entity.

I think the problem of an unresolvable URI exists today in the Atom format because it always includes the entity id.
... <a:id>odata:transient:MyType:123456</a:id> <a:link rel="edit" href="odata:transient:MyType:123456"/> <a:category scheme="http://docs.oasis-open.org/odata/ns/scheme" term="#Domain.MyType"/> ...

{ "@odata.context": "http://<serviceRoot>/$metadata#Domain.MyType", "@odata.type": "#Domain.MyType", "@odata.id": null, ... }

But, if the entity id is in the specific "odata:transient:" format, then the client would know that it is not resolvable.

[falbert-ptc]

Another way to say this is: if I could reuse the same generated transient ID in both XML and JSON, that would resolve all our problems :).
My main problem is with the verb "MUST" for the entity ID being null in the JSON format. If it was a "MAY", that wouldn't be an issue.
Or something along the lines "the odata.id annotation MUST appear and MAY have either the null value or a unique transient id following the pattern odata:transient:{some-generated-unique-identifier}"

[ralfhandl]

@falbert-ptc Please check & review PR #1942

[ralfhandl]

@falbert-ptc after discussing this in the Technical Committee we came up with a new proposal: use @Core.ContentID to "identify" transient entities, but keep their id null.

Entity references already can contain instance annotations, so having an entity reference of the form

{
  "@context": "http://host/service/$metadata#$ref",
  "@id": null,
  "@Core.ContentID": "some-value-already-used-in-this-response"
}

would be allowed already.

Would that solve your problem?

[falbert-ptc]

Hi @ralfhandl

Are you suggesting to use this instance annotation in the response to any type of request, not just deep insert or deep update ?

It would also make this annotation essentially mandatory, since you cannot predict whether a particular transient entity will be found again in the payload.
Per the spec, this annotation is not currently required for entity references, so the client would have no guaranteed way of uniquely identifying a transient entity in a payload.

It is also unclear how that would work in the Atom XML format. The Atom format always contains an id with a value (whether the entity is transient or not). Is that annotation supposed to supersede the id ? What should the client use (and if they have different values) ?

Thank you

[ralfhandl]

Hi @falbert-ptc,

It would also make this annotation essentially mandatory

It would make the @Core.ContentID annotation mandatory in transient entities that are actually referenced by an entity reference in the same response.

This means

• the service produces transient entities that form association cycles completely consisting of transient entities
• the service can uniquely identify (some of) these transient entities within a response
• the service supports $levels=max

In this case the service can

1. proactively inject @Core.ContentID into all identifiable transient entities, or
2. only do this for association chains where $levels=max is applied, or
3. only do this for transient entities that are actually referenced by an entity reference, or
4. pretend that these entities are not transient and respond with 404 Not Found or 410 Gone if they are directly accessed and cannot be recreated on the fly, or
5. do something I haven't thought of.

Which of these options would work for your use case?

[falbert-ptc]

Hi @ralfhandl

From a client perspective, it means it would now be expected for a client to find and process this annotation in the response since a client cannot know in advance if the resulting entities are transient (no way to know from the EDM), nor if there will be circular references. Another way to say this is that for the JSON format, if the id is null, the client is expected to find and process this Core.ContentID annotation to be able to interpret circular references in nested entities. For the Atom XML format, client can use the entity id or this annotation, assuming they have an identical value (which for now is a pretty big assumption).

This also adds new use cases for this annotation, which up until now was optional and only for deep insert use cases.
This would be a breaking change (i.e. new requirements) on both servers and clients to always support this annotation if they deal with transient entities. It means for 4.0 and 4.01, there would be no way of supporting recursive navigations for transient entities.

Additionally, like I said before, there is currently no mechanism for the client to know if an entity is transient and if they have to process this annotation or not. It means, any client will have to always look for this annotation. This is also a bit inconsistent with the statement about resolving circular dependencies with entity references, since an entity reference MAY have an annotation, but is not required to. And the current definition of the entity reference does not specify this annotation as being a core/required element of what identifies an entity (albeit transient).

Finally, it's also unclear how the annotation would work in the Atom XML format.

TL;DR:

• making use of this annotation in 4.02, why not (although I'm not sure the policy about breaking changes in minor versions). But it has to be fully fleshed out: annotation becomes required in certain use cases, its value is consistent with the entity id for XML, other spec elements about entity reference and recursive navigations are updated accordingly.
• we still need a mechanism in 4.0 and 4.01 to solve this issue for the JSON format. Allowing a non-null ID with the odata:transient prefix (like in XML) is so much easier, unless I'm missing something.

[ralfhandl]

Next idea:

• purpose of @id:null is to prevent clients from trying to access a transient entity
• another way of doing this is to have @readLink:null, which clients can't use to construct valid HTTP requests - in this case the @id value should not matter

Cases

• traditional transient entity (can't reference, can't read, can't expect to still exist once received, exists since 4.0)
  • @id:null
• re-readable transient entity (can't reference, can re-read, can expect to get a read response, added in 4.02)
  • @id:null, @readLink:"https://something/that/will/re-read/this"
• referencable transient entity (can reference, can't read, can't expect to still exist once received)
  • @id:"something-unique", @readLink:null
  • @id SHOULD follow the pattern odata:transient:{some-generated-unique-identifier-to-not-break-atom-parsers}
  • services MAY add something like @Vendor.transient:true to tell vendor-specific clients that this is not a "permanent unreadable entity"

@falbert-ptc Would that work for you?

[falbert-ptc]

Hi @ralfhandl ,
Yes, your suggestion makes sense.
Thank you