Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a Stream Migration spec #406

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
+315 −0
Open

Add a Stream Migration spec #406

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
4fc768f
Initial Pass
MarcoPolo Apr 13, 2022
b0db416
Add authors and interest group
MarcoPolo Apr 13, 2022
dac9a8d
Clarifications
MarcoPolo Apr 14, 2022
37a86ff
Update spec to clarify this is a protocol that prefixes other protocols
MarcoPolo Apr 18, 2022
6f03926
Don't pass arguments in protocol id
MarcoPolo Apr 21, 2022
406171f
Update note on protocol ID. Add should on knowing responder supports …
MarcoPolo Apr 21, 2022
99f055e
Rename stream A to stream 1 to match uints
MarcoPolo Apr 25, 2022
c0fd69f
Update stream migration spec
MarcoPolo Apr 25, 2022
eef233e
Add some links
MarcoPolo Apr 25, 2022
49c0597
Simply use the lower peer id
MarcoPolo Apr 25, 2022
4e1f106
Remove designated stream migrator
MarcoPolo Apr 25, 2022
a380dc4
Add ack_migrate
MarcoPolo May 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 56 additions & 17 deletions connections/stream-migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ The goal of the protocol is to move traffic from one stream to another
seamlessly. The final state of the new stream should be the same as the initial
state of the old stream.

The protocol works as a prefix before another protocol. If we are creating a
stream for some user protocol `P`, we coordinate the stream-migration protocol
first, and then negotiate protocol `P` later. The initial stream-migration
negotiation is so that both sides agree on an ID for the stream. This way when a
peer decides to migrate the stream, it can reference which stream it wants to
migrate and both peers know which stream is being referenced.

![stream-migration](./stream-migration/stream-migration.svg)

<details>
Expand All @@ -43,18 +50,44 @@ skinparam sequenceMessageAlign center
entity Initiator
entity Responder

note over Initiator, Responder: Assume at least 2 connections.
' note over Initiator, Responder: Assume at least 2 connections.

Initiator -> Responder: Open connection
... <i>Establish both sides support multistream-select</i> ...

Initiator -> Responder: Open multiplexed stream

Initiator -> Responder: Send ""<stream-migration protocol id>/<stream-id>""\ne.g. ""streamMigration/1.0.0/A""
Initiator -> Responder: Send ""<user-protocol-id>"" (as-in multistream select)

alt Responder supports stream-migration
Initiator <- Responder: Echo back ""<user-protocol-id>"" (as-in multistream select)
note over Initiator
The initiator knows the peer supports stream-migration since it echoed back
the user protocol.
end note

Initiator <-> Responder: <b>Connection 1; stream A</b>
Initiator <-> Responder: <b>Connection 2</b>

note over Responder
The responder knows that this stream is called <b>stream A</b>
end note

... <i>Continue negotiating another protocol</i> ...
else Responder does <i>not</i> supports stream-migration
MarcoPolo marked this conversation as resolved.
Show resolved Hide resolved
Initiator <- Responder: send back "na" for the stream-migration protocol (like multistream-select)
Initiator <- Responder: Echo back ""<user-protocol-id>"" (as-in multistream select)
end

... <i>Nodes use the stream as normal<i> ...

== Stream Migration ==

note over Initiator, Responder: Migrate <b>Stream A</b> to <b>Stream B</b>

Initiator -> Responder: Open new stream on <b>Connection 2</b>. Call this <b>Stream B</b>

Initiator -> Responder: <b>Stream B:</b> Migrate stream with <b>id=A</b> to this stream
Initiator -> Responder: ""<stream-migration protocol id>/<this-stream-id>/from/<original-stream-id>""\ne.g. ""streamMigration/1.0.0/B/from/A""

Initiator <- Responder: <b>Stream B:</b> Ack Migrate

note over Responder
Expand Down Expand Up @@ -100,6 +133,21 @@ plantuml stream-migration.md -o stream-migration -tsvg

Note: some of these steps may be pipelined.


### Stream IDs

In the above diagram stream IDs have the labels `A` and `B`. In practice this
ID will be represented as an int defined by the initiator. Both sides can
MarcoPolo marked this conversation as resolved.
Show resolved Hide resolved
globally identify this stream by the namespacing the ID with the initiator's
MarcoPolo marked this conversation as resolved.
Show resolved Hide resolved
peer ID. e.g. `12D3Foo.1`.
### Stream migration protocol id

The stream migration protocol id should follow the format of
`/streamMigration/1.0.0/<streamID>`. The stream should be an int.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/libp2p/stream-migration/1.0.0

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/libp2p/stream-migration/, we can still version this later if necessary.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The protocol name can't include the stream ID. The stream ID is the payload of this protocol.

Copy link
Contributor Author

@MarcoPolo MarcoPolo Apr 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The protocol name can't include the stream ID. The stream ID is the payload of this protocol.

Why not? This way you wouldn't need a separate payload message.

The initiator would send the stream-migration protocol ID+stream id, then can send the underlying protocol right away without having to wait for a response.

The responder would read the stream migration protocol and ack it (by echoing back as in multistream select. Note this doc doesn't say this part yet.), then continue negotiating the underlying protocol. If the other node for some reason doesn't support stream-migration (even if we thought it did), it would echo back na, and continue as before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, it could be because we cache the protocols seen on the other side, correct?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, exactly. Even worse, we send an Identify Delta messages for every new protocol that we add.

Also, logically speaking, the stream ID is not part of the protocol ID, but it's a payload of the stream migration protocol. Counting the bytes on the wire, it makes no (or at least not more than a few byte) difference if it's in the protocol name or in the payload.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this was mentioned before: For the sake of evolvability of the protocol in the future, can the stream ID be send embedded in a Protobuf? That way we can add new fields in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call out. I did this in the poc, but I’ll update the spec to include this.


When migrating an existing stream, the protocol id should follow the format of
`/streamMigration/1.0.0/<streamID>/from/<streamID>`.

### Resets

If either stream is "reset" before both ends are closed, both streams must be
Expand Down Expand Up @@ -132,19 +180,17 @@ note over Responder: <b>Stream A</b> is closed for writing

note over Initiator, Responder: Migrate <b>Stream A</b> to <b>Stream B</b>

Initiator -> Responder: <b>Stream A:</b> Start stream migration, this stream is <b>id=A</b>

Initiator -> Responder: Open new stream on <b>Connection 2</b>. Call this <b>Stream B</b>

Initiator -> Responder: <b>Stream B:</b> Migrate stream with <b>id=A</b> to this stream
Initiator -> Responder: ""streamMigration/1.0.0/B/from/A""

Initiator <- Responder: <b>Stream B:</b> Ack Migrate

note over Initiator
Close <b>stream A</b> for writing.
Will only write to <b>stream B</b> from now on.
end note


note over Initiator
We have already seen the ""EOF"" on
<b>Stream A</b> from <i>Responder</i>
Expand All @@ -160,8 +206,8 @@ note over Responder
end note
Initiator <- Responder: <b>Stream B:</b> ""EOF""


note over Initiator, Responder: Stream A is now migrated to Stream B

@enduml
```
To generate:
Expand Down Expand Up @@ -205,12 +251,5 @@ where to migrate to.

Some questions that will probably be resolved when a PoC is implemented.

- How many streams are there total? 2 or 3? There is the old stream and new
stream, but is there a migration protocol specific stream? Or can the
migration be started from the old stream? If yes, how do we prevent
application data from looking like a migration without framing?
- In simultaneous open how do we pick who's the initiator? I think we can rely
on the `/libp2p/simultaneous-connect` to do the correct thing here.
- Do we need a new connection that we migrate towards? Or can we re-use an
existing connection?
- Can we migrate multiplexed streams to another multiplexed stream destination?
on the `/libp2p/simultaneous-connect` to do the correct thing here.
Loading