index.bs

<pre class="metadata">
Shortname: webxr-mesh-detection
Title: WebXR Mesh Detection Module
Group: immersivewebcg
Status: CG-DRAFT
Level: 1
ED: https://github.com/immersive-web/real-world-meshing/
Repository: immersive-web/real-world-meshing
Mailing List Archives: https://lists.w3.org/Archives/Public/public-immersive-web/

!Participate: <a href="https://github.com/immersive-web/real-world-meshing/issues/new">File an issue</a> (<a href="https://github.com/immersive-web/real-world-meshing/issues">open issues</a>)
!Participate: <a href="https://lists.w3.org/Archives/Public/public-immersive-web/">Mailing list archive</a>
!Participate: <a href="irc://irc.w3.org:6665/">W3C's #immersive-web IRC</a>

Editor: Rik Cabanier, Meta https://meta.com, cabanier@meta.com

Abstract:
</pre>

<pre class="link-defaults">
</pre>

<pre class="anchors">
spec: WebXR Device API - Level 1; urlPrefix: https://immersive-web.github.io/webxr/#
    for: XRFrame;
        type: dfn; text: active; url: xrframe-active
        type: dfn; text: session; url: dom-xrframe-session
        type: dfn; text: time; url: xrframe-time
    for: XRSession;
        type: dfn; text: list of frame updates; url: xrsession-list-of-frame-updates
        type: dfn; text: mode; url: xrsession-mode
        type: dfn; text: XR device; url: xrsession-xr-device
    for: XRSpace;
        type: dfn; text: effective origin; url: xrspace-effective-origin
        type: dfn; text: native origin; url: xrspace-native-origin
        type: dfn; text: origin offset; url: xrspace-origin-offset
        type: dfn; text: session; url: xrspace-session
    type: dfn; text: capable of supporting; url: capable-of-supporting
    type: dfn; text: feature descriptor; url: feature-descriptor
    type: dfn; text: identity transform; url: identity-transform
    type: dfn; text: inline XR device; url: inline-xr-device
    type: dfn; text: quantization; url: quantization
    type: dfn; text: rounding; url: rounding
    type: dfn; text: XR device; url: xr-device
    type: dfn; text: xr compositor
spec: WebXR Anchors Module; urlPrefix: https://immersive-web.github.io/anchors/#
    type: dfn; text: create new anchor object; url: create-new-anchor-object
    type: dfn; text: update anchors; url: update-anchors
    type: interface; text: XRAnchor; url: xr-anchor
</pre>

<pre class=biblio>
{
  "webxr-anchors-module": {
    "authors": [
      "Piotr Bialecki"
    ],
    "href": "https://immersive-web.github.io/anchors/",
    "title": "WebXR Anchors Module",
    "status": "DR"
  }
}
</pre>

<style>
  .non-normative::before {
    content: "This section is non-normative.";
    font-style: italic;
  }
  .tg {
    border-collapse: collapse;
    border-spacing: 0;
  }
  .tg th {
    border-style: solid;
    border-width: 1px;
    background: #90b8de;
    color: #fff;
    font-family: sans-serif;
    font-weight: bold;
    border-color: grey;
  }
  .tg td {
    padding: 4px 5px;
    background-color: rgb(221, 238, 255);
    font-family: monospace;
    border-style: solid;
    border-width: 1px;
    border-color: grey;
    overflow: hidden;
    word-break: normal;
  }
  .unstable::before {
    content: "This section is not stable";
    display: block;
    font-weight: bold;
    text-align: right;
    color: red;
  }
  .unstable {
    border: thin solid pink;
    border-radius: .5em;
    padding: .5em;
    margin: .5em calc(-0.5em - 1px);
    background-image: url("data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' width='300' height='290'><text transform='rotate(-45)' text-anchor='middle' font-family='sans-serif' font-weight='bold' font-size='70' y='210' opacity='.1' fill='white'>Unstable</text></svg>");
    background-repeat: repeat;
    background-color: #282828;
  }
  @media (prefers-color-scheme: light) {
    .unstable {
      background-image: url("data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' width='300' height='290'><text transform='rotate(-45)' text-anchor='middle' font-family='sans-serif' font-weight='bold' font-size='70' y='210' opacity='.1'>Unstable</text></svg>");
      background-color: #FFF4F4;
    }
  }
  .unstable h3:first-of-type {
    margin-top: 0.5rem;
  }
  .unstable.example:not(.no-marker)::before {
    content: "Example " counter(example) " (Unstable)";
    float: none;
  }
</style>

Introduction {#intro}
============

<section class="non-normative">

</section>

Initialization {#anchor-feature-initialization}
==================

Feature descriptor {#anchor-feature-descriptor}
------------------

In order for the applications to signal their interest in using mesh detection during a session, the session must be requested with appropriate [=feature descriptor=]. The string <dfn>mesh-detection</dfn> is introduced by this module as a new valid feature descriptor for mesh detection feature.

A device is [=capable of supporting=] the mesh-detection feature if the device's tracking system exposes a [=native mesh detection=] capability. The [=inline XR device=] MUST NOT be treated as [=capable of supporting=] the mesh-detection feature.

When a session is created with mesh-detection feature enabled, the [=update meshes=] algorithm MUST be added to the [=list of frame updates=] of that session.

<div class="example">
The following code demonstrates how a session that requires mesh detection could be requested:

<pre highlight="js">
const session = await navigator.xr.requestSession("immersive-ar", {
  requiredFeatures: ["mesh-detection"]
});
</pre>

</div>

Meshs {#meshes-section}
======

XRMesh {#mesh}
-------

<script type="idl">
[Exposed=Window] interface XRMesh {
    [SameObject] readonly attribute XRSpace meshSpace;

    readonly attribute FrozenArray<Float32Array> vertices;
    readonly attribute Uint32Array indices;
    readonly attribute DOMHighResTimeStamp lastChangedTime;
    readonly attribute DOMString? semanticLabel;
};
</script>

An {{XRMesh}} represents a single instance of 3D geometry detected by the underlying XR system.

The {{XRMesh/meshSpace}} is an {{XRSpace}} that establishes the coordinate system of the mesh. The [=XRSpace/native origin=] of the {{XRMesh/meshSpace}} tracks mesh's center. The underlying XR system defines the exact meaning of the mesh center. The Y axis of the coordinate system defined by {{XRMesh/meshSpace}} MUST represent the mesh's normal vector.

Each {{XRMesh}} has an associated <dfn for=XRMesh>native entity</dfn>.

Each {{XRMesh}} has an associated <dfn for=XRMesh>frame</dfn>.

The {{XRMesh/vertices}} is an array of vertices that describe the shape of the mesh. They are expressed in the coordinate system defined by {{XRMesh/meshSpace}}.

The {{XRMesh/indices}} is an array of indices that describe the index of each vertex of the mesh.

The {{XRMesh/lastChangedTime}} is the last time some of the mesh attributes have been changed.

Note: The pose of a mesh is not considered a mesh attribute and therefore updates to mesh pose will not cause the {{XRMesh/lastChangedTime}} to change. This is because mesh pose is a property that is derived from two different entities - {{XRMesh/meshSpace}} and the {{XRSpace}} relative to which the pose is to be computed via {{XRFrame/getPose()}} function.

Obtaining detected meshes {#obtaining-meshes}
=========================

XRMeshSet {#mesh-set}
----------

<script type="idl">
[Exposed=Window] interface XRMeshSet {
  readonly setlike<XRMesh>;
};
</script>

An {{XRMeshSet}} is a collection of {{XRMesh}}s. It is the primary mechanism of obtaining the collection of meshes detected in an {{XRFrame}}.

<script type="idl">
partial interface XRFrame {
  readonly attribute XRMeshSet detectedMeshes;
};
</script>

{{XRFrame}} is extended to contain {{XRFrame/detectedMeshes}} attribute which contains all meshes that are still tracked in the frame. The set is initially empty and will be populated by the [=update meshes=] algorithm. If this attribute is accessed when the frame is not [=XRFrame/active=], the user agent MUST throw {{InvalidStateError}}.

{{XRSession}} is also extended to contain associated <dfn for=XRSession>set of tracked meshes</dfn>, which is initially empty. The elements of the set will be of {{XRMesh}} type.

<div class="algorithm" data-algorithm="update-meshes">
In order to <dfn>update meshes</dfn> for |frame|, the user agent MUST run the following steps:
    1. Let |session| be a |frame|'s [=XRFrame/session=].
    1. Let |device| be a |session|'s [=XRSession/XR device=].
    1. Let |trackedMeshs| be a result of calling into |device|'s [=native mesh detection=] capability to obtain tracked meshes at |frame|'s [=XRFrame/time=].
    1. For each |native mesh| in |trackedMeshs|, run:
        1. If desired, treat the |native mesh| as if it were not present in |trackedMeshs| and continue to the next entry. See [[#privacy-security]] for criteria that could be used to determine whether an entry should be ignored in this way.
        1. If |session|'s [=XRSession/set of tracked meshes=] contains an object |mesh| that [=corresponds to=] |native mesh|, invoke [=update mesh object=] algorithm with |mesh|, |native mesh|, and |frame|, and continue to the next entry.
        1. Let |mesh| be the result of invoking the [=create mesh object=] algorithm with |native mesh| and |frame|.
        1. Add |mesh| to |session|'s [=XRSession/set of tracked meshes=].
    1. Remove each object in |session|'s [=XRSession/set of tracked meshes=] that was neither created nor updated during the invocation of this algorithm.
    1. Set |frame|'s {{XRFrame/detectedMeshes}} to [=XRSession/set of tracked meshes=].
</div>

<div class="algorithm" data-algorithm="create-mesh-object">
In order to <dfn>create mesh object</dfn> from a [=native mesh object=] |native mesh| and {{XRFrame}} |frame|, the user agent MUST run the following steps:
    1. Let |result| be a new instance of {{XRMesh}}.
    1. Set |result|'s [=XRMesh/native entity=] to |native mesh|.
    1. Set |result|'s {{XRMesh/meshSpace}} to a new {{XRSpace}} object created with [=XRSpace/session=] set to |frame|'s {{XRFrame/session}} and [=XRSpace/native origin=] set to track |native mesh|'s native origin.
    1. Invoke [=update mesh object=] algorithm with |result|, |native mesh|, and |frame|.
    1. Return |result|.

A mesh object, |result|, created in such way is said to <dfn>correspond to</dfn> the passed in native mesh object |native mesh|.
</div>

<div class="algorithm" data-algorithm="update-mesh-object">
In order to <dfn>update mesh object</dfn> |mesh| from a [=native mesh object=] |native mesh| and {{XRFrame}} |frame|, the user agent MUST run the following steps:
    1. Set |mesh|'s [=XRMesh/frame=] to |frame|.
    1. Set |mesh|'s {{XRMesh/vertices}} to the new array of vertices representing |native mesh|'s vertices, performing all necessary conversions to account for differences in native mesh representation.
    1. Set |mesh|'s {{XRMesh/indices}} to the new array of indices representing |native mesh|'s vertices, performing all necessary conversions to account for differences in native mesh representation.
    1. If desired, reduce the level of detail of the |mesh|'s {{XRMesh/vertices}} and {{XRMesh/indices}} as described in [[#privacy-security]].
    1. Set |mesh|'s {{XRMesh/lastChangedTime}} to [=XRFrame/time=].
</div>

Native device concepts {#native-device-concepts}
======================

Native mesh detection {#native-mesh-detection-section}
----------------------

<section class="non-normative">

The mesh detection API provides information about 3D surfaces detected in users' environment. It is assumed in this specification that user agents can rely on <dfn>native mesh detection</dfn> capabilities provided by the underlying platform for their implementation of mesh-detection features. Specifically, the underlying XR device should provide a way to query all meshes that are tracked at a time that corresponds to the [=XRFrame/time=]of a specific {{XRFrame}}.

Moreover, it is assumed that the tracked meshes, known as <dfn>native mesh objects</dfn>, maintain their identity across frames - that is, given a mesh object <code>P</code> returned by the underlying system at time <code>t0</code>, and a mesh object <code>Q</code> returned by the underlying system at time <code>t1</code>, it is possible for the user agent to query the underlying system about whether <code>P</code> and <code>Q</code> correspond to the same logical mesh object. The underlying system is also expected to provide a [=native origin=] that can be used to query the location of a pose at time <code>t</code>, although it is not guaranteed that mesh pose will always be known (for example for meshes that are still tracked but not localizable at a given time). In addition, the native mesh object should expose a polygon describing approximate shape of the detected mesh.

In addition, the underlying system should recognize native meshes as native entities for the purposes of {{XRAnchor}} creation. For more information, see [[webxr-anchors-module#native-anchor]] section.

</section>

Privacy & Security Considerations {#privacy-security}
=================================

<section class="non-normative">

The mesh detection API exposes information about users' physical environment. The exposed mesh information (such as mesh's polygon) may be limited if the user agent so chooses. Some of the ways in which the user agent can reduce the exposed information are: decreasing the level of detail of the mesh's polygon in [=update mesh object=] algorithm (for example by decreasing the number of vertices, or by [=rounding=] / [=quantization|quantizing=] the coordinates of the vertices), or removing the mesh altogether by behaving as if the mesh object was not present in <code>trackedMeshs</code> collection in [=update meshes=] algorithm (this could be done for example if the detected mesh is deemed to small / too detailed to be surfaced and the mechanisms to reduce details exposed on meshes are not implemented by the user agent). The poses of the meshes (obtainable from {{XRMesh/meshSpace}}) could also be [=quantization|quantized=].

Since concepts from mesh detection API can be used in methods exposed by [[webxr-anchors-module]] specification, some of the privacy & security considerations that are relevant to WebXR Anchors Module also apply here. For details, see [[webxr-anchors-module#privacy-security]] section.

Due to how mesh detection API extends WebXR Device API, the section [[webxr#security]] is also applicable to the features exposed by the WebXR Mesh Detection Module.

</section>

Acknowledgements {#ack}
================

The following individuals have contributed to the design of the WebXR Mesh Detection specification: