mudhorn/meshtastic
1
0
Fork 0
mirror of https://github.com/meshtastic/meshtastic.git synced 2024-12-27 14:39:54 -08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Merge pull request #240 from dfloer/master

Browse source
This commit is contained in:
Foster Irwin 2022-02-12 07:23:41 -07:00 committed by GitHub
parent 6c9c14eeb0 df41fea622
commit c4309caaab
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 63 additions and 179 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

242
docs/developers/device/mesh-alg.md
View file

@ -4,219 +4,103 @@ title: Mesh broadcast algorithm
sidebar_label: Mesh algorithm
---
## Current algorithm
## Current Algorithm
The routing protocol for Meshtastic is really quite simple (and suboptimal). It is heavily influenced by the mesh routing algorithm used in [RadioHead](https://www.airspayce.com/mikem/arduino/RadioHead/) (which was used in very early versions of this project). It has four conceptual layers.
### A note about protocol buffers
### A Note About Protocol Buffers
Because we want our devices to work across various vendors and implementations, we use [Protocol Buffers](https://github.com/meshtastic/Meshtastic-protobufs) pervasively. For information on how the protocol buffers are used with respect to API clients see [sw-design](/software/other/sw-design.md), for purposes of this document you mostly only
need to consider the MeshPacket and Subpacket message types.
### Layer 1: Non-reliable zero hop messaging
### Later 0: LoRa Radio
This layer is conventional non-reliable LoRa packet transmission. The transmitted packet has the following representation on the ether:
All data is converted into LoRa symbols which are sent to the radio for transmission. The details are described elsewhere, but it is worth noting that in addition to the converted packet bytes described below, there is also a preamble sent at the start of any data packet.
- A 256-bit LoRa preamble (to allow receiving radios to synchronize clocks and start framing). We use a longer than minimum (8 bit) preamble to maximize the amount of time the LoRa receivers can stay asleep, which dramatically lowers power consumption.
This preamble allows receiving radios to synchronize clocks and start framing. We use a preable length of 32, which is longer than the minimum preamble length of 8, to maximize the amount of time the LoRa receivers can stay asleep, which dramatically lowers power consumption.
After the preamble and the syncWord of 0x2b, the 16 byte packet header is transmitted. This header is described directly by the PacketHeader class in the C++ source code. But indirectly it matches the first portion of the "MeshPacket" protobuf definition. But notably: this portion of the packet is sent directly as the following 16 bytes (rather than using the protobuf encoding). We do this to both save airtime and to allow receiving radio hardware the option of filtering packets before even waking the main CPU.
### Layer 1: Unreliable Zero Hop Messaging
- to (4 bytes): the unique NodeId of the destination (or 0xffffffff for NodeNum_BROADCAST)
- from (4 bytes): the unique NodeId of the sender
- id (4 bytes): the unique (with respect to the sending node only) packet ID number for this packet. We use a large (32 bit) packet ID to ensure there is enough unique state to protect any encrypted payload from attack.
- flags (4 bytes): Only a few bits are currently used - 3 bits for the "HopLimit" (see below) and 1 bit for "WantAck"
This layer is conventional non-reliable LoRa packet transmission. The transmitted packet has the following representation before encoding for transmission:
After the packet header, the actual packet is placed onto the the wire. These bytes are merely the encrypted packed protobuf encoding of the SubPacket protobuf. A full description of our encryption is available in [crypto](/docs/developers/device/encryption). It is worth noting that only this SubPacket is encrypted, headers are not. Which leaves open the option of eventually allowing nodes to route packets without knowing the keys used to encrypt.
| Offset | Length | Type | Usage |
|--------|--------|------|-------|
| 0x00 | 1 byte | Integer | syncWord, always `0x2B`. |
| 0x01 | 4 bytes | Integer | Packet header: Destination. The destination's unique NodeID. `0xFFFFFFFF` for broadcast. |
| 0x05 | 4 bytes | Integer | Packet Header: Sender. The sender's unique NodeID. |
| 0x09 | 4 bytes | Integer | Packet Header: The sending node's unique packet ID for this packet. |
| 0x0D | 32 bits | Bits | Packet Header: Flags. See the [header flags](#packet-header-flags) for usage. |
| 0x11 .. 0xFD | Varies, maximum of 237 bytes. | Bytes | Actual packet data. Unused bytes are not transmitted. |
| 0xFE .. 0xFF | 2 Bytes | Bytes | Unused. |
NodeIds are constructed from the bottom four bytes of the macaddr of the Bluetooth address. Because the OUI is assigned by the IEEE, and we currently only support a few CPU manufacturers, the upper byte is defacto guaranteed unique for each vendor. The bottom 3 bytes are guaranteed unique by that vendor.
#### Packet Header Flags
To prevent collisions all transmitters will listen before attempting to send. If they hear some other node transmitting, they will reattempt transmission in x milliseconds. This retransmission delay is random between 20 and 22 seconds. (these two numbers are currently hardwired, but really should be scaled based on expected packet transmission time at current channel settings).
| Index | # of Bits | Usage |
|-------|-----------|-------|
| 0 | 3 | HopLimit (see note in Layer 3) |
| 3 | 1 | WantAck |
| 4 .. 32 | 28 | Currently unused |
### Layer 2: Reliable zero hop messaging
#### Usage Details
This layer adds reliable messaging between the node and its immediate neighbors (only).
- **Packet Header:** is described directly by the `PacketHeader` class in the C++ source code. But indirectly it matches the first portion of the `MeshPacket` protobuf definition. Note that the packet header is not encoded using a protobuf, but is sent as raw bytes. This both saves airtime and allows receiving radio hardware to optionally filter packets before waking the main CPU.
The default messaging provided by layer-1 is extended by setting the "want-ack" flag in the MeshPacket protobuf. If want-ack is set the following documentation from mesh.proto applies:
- **Packet Header - NodeIDs:** are constructed from the bottom four bytes of the MAC address of the Bluetooth address. Because the OUI is assigned by the IEEE, and we currently only support a few CPU manufacturers, the upper byte is defacto guaranteed unique for each vendor. The bottom 3 bytes are guaranteed unique by that vendor.
"""This packet is being sent as a reliable message, we would prefer it to arrive
at the destination. We would like to receive an ACK packet in response.
- **Packet Header - Unique ID:** The ID is a large, 32 bit ID to ensure there is enough unique state to protect an encrypted payload from attack.
Broadcast messages treat this flag specially: Since ACKs for broadcasts would
rapidly flood the channel, the normal ACK behavior is suppressed. Instead,
the original sender listens to see if at least one node is rebroadcasting this
packet (because naive flooding algorithm). If it hears that, the odds (given
typical LoRa topologies) are very high that every node should
eventually receive the message. So FloodingRouter.cpp generates an implicit
ACK which is delivered to the original sender. If after some time we don't
hear anyone rebroadcast our packet, we will timeout and retransmit, using the
regular resend logic."""
- **Payload:** An encrypted and packed protobuf encoding of the SubPacket protobuf. Only the SubPacket is encrypted, while headers are not. This allows the option of eventually allowing nodes to route packets without knowing anything about the encrypted payload. For more information, see the [encryption](/docs/developers/device/encryption) and [protobufs](/docs/developers/protobufs/api) documentation. Any data past the maximum length is truncated.
If a transmitting node does not receive an ACK (or a NAK) packet within FIXME milliseconds, it will use layer-1 to attempt a retransmission of the sent packet. A reliable packet (at this 'zero hop' level) will be resent a maximum of three times. If no ACK or NAK has been received by then the local node will internally generate a NAK (either for local consumption or use by higher layers of the protocol).
#### Collision Avoidance
### Layer 3: (Naive) flooding for multi-hop messaging
All transmitters must listen before attempting to transmit. If another node is heard transmitting, the listening node will reattempt transmission after a calculated delay. The delay depends on various settings and is based on Semtech's [LoRa Modem Design Guide](/documents/LoRa_Design_Guide.pdf), section 4 and implemented in `RadioInterface::getPacketTime`. The following tables contains some calculated values for how long it takes to transmit a packet, which is used to calculate the delay.
Given our use-case for the initial release, most of our protocol is built around [flooding](<https://en.wikipedia.org/wiki/Flooding_(computer_networking)>). The implementation is currently 'naive' - i.e. it doesn't try to optimize flooding other than abandoning retransmission once we've seen a nearby receiver has ACKed the packet. Therefore, for each source packet up to N retransmissions might occur (if there are N nodes in the mesh).
| Payload Bytes | Spreading Factor | Bandwidth | Coding Rate | Time |
|---------------|------------------|-----------|-------------|------|
| 0 | 7 | 125 kHz | 4/5 | 13 ms |
| 237 | 7 | 125 kHz | 4/5 | 100 ms |
| 0 | 7 | 500 kHz | 4/5 | 4 ms |
| 237 | 7 | 500 kHz | 4/5 | 25 ms |
| 0 | 10 | 250 kHz | 4/7 | 51 ms |
| 237 | 10 | 250 kHz | 4/7 | 391 ms |
| 0 | 11 | 250 kHz | 4/6 | 101 ms |
| 237 | 11 | 250 kHz | 4/6 | 633 ms |
| 0 | 9 | 31.25 kHz | 4/8 | 201 ms |
| 237 | 9 | 31.25 kHz | 4/8 | 2.413 **seconds** |
| 0 | 12 | 125 kHz | 4/8 | 402 ms |
| 237 | 12 | 125 kHz | 4/8 | 3.482 **seconds** |
Each node in the mesh, if it sees a packet on the ether with HopLimit set to a value other than zero, it will decrement that HopLimit and attempt retransmission on behalf of the original sending node.
The actual delay calculation is a random value between 100ms (a hardcoded minimum) and the time taken to transmit just the header, which is given in the table as a 0 Byte payload.
### Layer 4: DSR for multi-hop unicast messaging
### Layer 2: Reliable Zero Hop Messaging
This layer is not yet fully implemented (and not yet used). But eventually (if we stay with our own transport rather than switching to QMesh or Reticulum)
we will use conventional DSR for unicast messaging. Currently, (even when not requiring 'broadcasts') we send any multi-hop unicasts as 'broadcasts' so that we can
leverage our (functional) flooding implementation. This is suboptimal but it is a very rare use-case, because the odds are high that most nodes (given our small networks and 'hiking' use case) are within a very small number of hops. When any node witnesses an ACK for a packet, it will realize that it can abandon its own
broadcast attempt for that packet.
This layer adds reliable messaging between the node and its immediate neighbors only.
## Misc notes on remaining tasks
The default messaging provided by Layer 1 is extended by setting the `WantAck` flag in the MeshPacket protobuf. If `WantAck` is set, the following documentation from mesh.proto applies:
This section is currently poorly formatted, it is mostly a mere set of TODO lists and notes for @geeksville during his initial development. After release 1.0 ideas for future optimization include:
> This packet is being sent as a reliable message, we would prefer it to arrive at the destination. We would like to receive an ACK packet in response.
>
> Broadcast messages treat this flag specially: Since ACKs for broadcasts would rapidly flood the channel, the normal ACK behavior is suppressed. Instead, the original sender listens to see if at least one node is rebroadcasting this
packet (because naive flooding algorithm). If it hears that, the odds (given typical LoRa topologies) are very high that every node should eventually receive the message. So FloodingRouter.cpp generates an implicit ACK which is delivered to the original sender. If after some time we don't hear anyone rebroadcast our packet, we will timeout and retransmit, using the regular resend logic.
- Make flood-routing less naive (because we have GPS and radio signal strength as heuristics to avoid redundant retransmissions)
- If nodes have been user marked as 'routers', preferentially do flooding via those nodes
- Fully implement DSR to improve unicast efficiency (or switch to QMesh/Reticulum as these projects mature)
If a transmitting node does not receive an ACK (or NAK) packet within FIXME milliseconds, it will use Layer 1 to attempt a retransmission of the sent packet. A reliable packet (at this 'zero hop' level) will be resent a maximum of three times. If no ACK or NAK has been received by then the local node will internally generate a NAK (either for local consumption or use by higher layers of the protocol).
great source of papers and class notes: <http://www.cs.jhu.edu/~cs647/>
Similarly to the delay used for [collision avoidance](#collision-avoidance), a delay is used before attempting to retransmit a dropped packet. The delay is calculated based on the time it takes to transmit just the header, which is a packet with a 0 Byte payload, and is a random value between `9 * header time` and `10 * header time`, in milliseconds.
flood routing improvements
### Layer 3: (Naive) Flooding for Multi-Hop Messaging
- DONE if we don't see anyone rebroadcast our want_ack=true broadcasts, retry as needed.
Given our use-case for the initial release, most of our protocol is built around [flooding](<https://en.wikipedia.org/wiki/Flooding_(computer_networking)>). The implementation is currently 'naive' and doesn't try to optimize flooding, except by abandoning retransmission once a node has seen a nearby receiver ACK the packet it's trying to flood. This means that up to N retransmissions of a packet could occur in an N node mesh.
reliable messaging tasks (stage one for DSR):
If any mesh nodes see a packet with a HopLimit other than zero, it will decrement that HopLimit and attempt retransmission on behalf of the original sending node.
- DONE generalize naive flooding
- DONE add a max hops parameter, use it for broadcast as well (0 means adjacent only, 1 is one forward etc...). Store as three bits in the header.
- DONE add a 'snoopReceived' hook for all messages that pass through our node.
- DONE use the same 'recentmessages' array used for broadcast msgs to detect duplicate retransmitted messages.
- DONE in the router receive path?, send an ACK packet if want_ack was set and we are the final destination. FIXME, for now don't handle multi-hop or merging of data replies with these ACKs.
- DONE keep a list of packets waiting for ACKs
- DONE for each message keep a count of # retries (max of three). Local to the node, only for the most immediate hop, ignorant of multi-hop routing.
- DONE delay some random time for each retry (large enough to allow for ACKs to come in)
- DONE once an ACK comes in, remove the packet from the retry list and deliver the ACK to the original sender
- DONE after three retries, deliver a NAK packet to the original sender (i.e. the phone app or mesh router service)
- DONE test one hop ACK/NAK with the python framework
- DONE Do stress test with ACKs
:::note
A node being in router mode does **not** currently change this behavior, but is intended to be used for future versions of the meshing algorithm. For an explanation of what router mode currently does, see [settings](/docs/software/settings/router#enabledisable-router-mode).
:::
DSR tasks
### Layer 4: DSR for Multi-Hop Unicast Messaging
- DONE oops I might have broken message reception
- DONE Don't use broadcasts for the network pings (close open GitHub issue)
- DONE add ignoreSenders to radioconfig to allow testing different mesh topologies by refusing to see certain senders
- DONE test multi-hop delivery with the python framework
This layer is not yet fully implemented and is currently unused. Eventually conventional [DSR](https://en.wikipedia.org/wiki/Dynamic_Source_Routing) will be used for for unicast messaging, unless transport is switched to [QMesh](https://github.com/faydr/QMesh) or [Reticulum](https://github.com/markqvist/Reticulum).
optimizations / low priority:
Currently, we send any multi-hop unicasts as broadcasts so that we can leverage the existing flooding implementation, even when broadcasts are not required. While this is suboptimal, it is a very rare use-case, because current networks are small and are geared towards a hiking use case, which means that most nodes are withing a very small number of hops.
- read this [this](http://pages.cs.wisc.edu/~suman/pubs/nadv-mobihoc05.pdf) paper and others and make our naive flood routing less naive
- read @cyclomies long email with good ideas on optimizations and reply
- DONE Remove NodeNum assignment algorithm (now that we use 4 byte node nums)
- DONE make android app warn if firmware is too old or too new to talk to
- change nodenums and packetids in protobuf to be fixed32
- low priority: think more careful about reliable retransmit intervals
- make ReliableRouter.pending threadsafe
- bump up PacketPool size for all the new ACK/NAK/routing packets
- handle 51 day rollover in doRetransmissions
- use a priority queue for the messages waiting to send. Send ACKs first, then routing messages, then data messages, then broadcasts?
when we send a packet
- do "hop by hop" routing
- when sending, if destnodeinfo.next_hop is zero (and no message is already waiting for an arp for that node), startRouteDiscovery() for that node. Queue the message in the 'waiting for arp queue' so we can send it later when then the arp completes.
- otherwise, use next_hop and start sending a message (with ack request) towards that node (starting with next_hop).
when we receive any packet
- sniff and update tables (especially useful to find adjacent nodes). Update user, network and position info.
- if we need to route() that packet, resend it to the next_hop based on our nodedb.
- if it is broadcast or destined for our node, deliver locally
- handle routereply/routeerror/routediscovery messages as described below
- then free it
routeDiscovery
- if we've already passed through us (or is from us), then it ignore it
- use the nodes already mentioned in the request to update our routing table
- if they were looking for us, send back a routereply
- NOT DOING FOR NOW -if max_hops is zero, and they weren't looking for us, drop (FIXME, send back error - I think not though?)
- if we receive a discovery packet, and we don't have next_hop set in our nodedb, we use it to populate next_hop (if needed) towards the requester (after decrementing max_hops)
- if we receive a discovery packet, and we have a next_hop in our nodedb for that destination we send a (reliable) we send a route reply towards the requester
when sending any reliable packet
- if timeout doing retries, send a routeError (NAK) message back towards the original requester. all nodes eavesdrop on that packet and update their route caches.
when we receive a routereply packet
- update next_hop on the node, if the new reply needs fewer hops than the existing one (we prefer shorter paths). FIXME, someday use a better heuristic
when we receive a routeError packet
- delete the route for that failed recipient, restartRouteDiscovery()
- if we receive routeerror in response to a discovery,
- FIXME, eventually keep caches of possible other routes.
TODO:
- optimize our generalized flooding with heuristics, possibly have particular nodes self mark as 'router' nodes.
- DONE reread the radiohead mesh implementation - hop to hop acknowledgment seems VERY expensive but otherwise it seems like DSR
- DONE read about mesh routing solutions (DSR and AODV)
- DONE read about general mesh flooding solutions (naive, MPR, geo assisted)
- DONE reread the disaster radio protocol docs - seems based on Babel (which is AODVish)
- REJECTED - seems dying - possibly dash7? <https://www.slideshare.net/MaartenWeyn1/dash7-alliance-protocol-technical-presentation> <https://github.com/MOSAIC-LoPoW/dash7-ap-open-source-stack> - does the open source stack implement multi-hop routing? flooding? their discussion mailing list looks dead-dead
- update duty cycle spreadsheet for our typical use case
a description of DSR: <https://tools.ietf.org/html/rfc4728>, good slides here: <https://www.slideshare.net/ashrafmath/dynamic-source-routing>,
good description of batman protocol: <https://www.open-mesh.org/projects/open-mesh/wiki/BATMANConcept>
interesting paper on LoRa mesh: <https://portal.research.lu.se/portal/files/45735775/paper.pdf>
It seems like DSR might be the algorithm used by RadioheadMesh. DSR is described in <https://tools.ietf.org/html/rfc4728>
<https://en.wikipedia.org/wiki/Dynamic_Source_Routing>
broadcast solution:
Use naive flooding at first (FIXME - do some math for a 20 node, 3 hop mesh. A single flood will require a max of 20 messages sent)
Then move to MPR later (<http://www.olsr.org/docs/report_html/node28.html>). Use altitude and location as heuristics in selecting the MPR set
compare to db sync algorithm?
What about never flooding GPS broadcasts? Instead, only have them go one hop in the common case. But if any node X is looking at the position of Y on their GUI, then send a unicast to Y asking for position update. Y replies.
If Y were to die, at least the neighbor nodes of Y would have their last known position of Y.
## approach 1
- send all broadcasts with a TTL
- periodically(?) do a survey to find the max TTL that is needed to fully cover the current network.
- to do a study, first send a broadcast (maybe our current initial user announcement?) with TTL set to one (so therefore no one will rebroadcast our request)
- survey replies are sent unicast back to us (and intervening nodes will need to keep the route table that they have built up based on past packets)
- count the number of replies to this TTL 1 attempt. That is the number of nodes we can reach without any rebroadcasts
- repeat the study with a TTL of 2 and then 3. stop once the # of replies stops going up.
- it is important for any node to do listen before talk to prevent stomping on other rebroadcasters...
- For these little networks I bet a max TTL would never be higher than 3?
## approach 2
- send a TTL1 broadcast, the replies let us build a list of the nodes (stored as a bitvector?) that we can see (and their RSSIs)
- we then broadcast out that bitvector (also TTL1) asking "can any of ya'll (even indirectly) see anyone else?"
- if a node can see someone I missed (and they are the best person to see that node), they reply (unidirectionally) with the missing nodes and their RSSIs (other nodes might sniff (and update their db) based on this reply, but they don't have to)
- given that the max number of nodes in this mesh will be like 20 (for normal cases), I bet globally updating this db of "nodenums and who has the best RSSI for packets from that node" would be useful
- once the global DB is shared, when a node wants to broadcast, it just sends out its broadcast . the first level receivers then make a decision "am I the best to rebroadcast to someone who likely missed this packet?" if so, rebroadcast
## approach 3
- when a node X wants to know other nodes positions, it broadcasts its position with want_replies=true. Then each of the nodes that received that request broadcast their replies (possibly by using special timeslots?)
- all nodes constantly update their local db based on replies they witnessed.
- after 10s (or whatever) if node Y notices that it didn't hear a reply from node Z (that Y has heard from recently) to that initial request, that means Z never heard the request from X. Node Y will reply to X on Z's behalf.
- could this work for more than one hop? Is more than one hop needed? Could it work for sending messages (i.e. for a message sent to Z with want-reply set).
## approach 4
look into the literature for this idea specifically.
- don't view it as a mesh protocol as much as a "distributed db unification problem". When nodes talk to nearby nodes, they work together
to update their nodedbs. Each nodedb would have a last change date and any new changes that only one node has would get passed to the
other node. This would nicely allow distant nodes to propagate their position to all other nodes (eventually).
- handle group messages the same way, there would be a table of messages and time of creation.
- when a node has a new position or message to send out, it does a broadcast. All the adjacent nodes update their db instantly (this handles 90% of messages I'll bet).
- Occasionally, a node might broadcast saying "anyone have anything newer than time X?" If someone does, they send the diffs since that date.
- essentially everything in this variant becomes broadcasts of "request db updates for >time X - for _all_ or for a particular nodenum" and nodes sending (either due to request or because they changed state) "here's a set of db updates". Every node is constantly trying to
build the most recent version of reality, and if some nodes are too far, then nodes closer in will eventually forward their changes to the distributed db.
- construct non-ambiguous rules for who broadcasts to request db updates. Ideally the algorithm should nicely realize node X can see most other nodes, so they should just listen to all those nodes and minimize the # of broadcasts. The distributed picture of nodes RSSI could be useful here?
- possibly view the BLE protocol to the radio the same way - just a process of reconverging the node/msgdb database.
If a node is attemptimg to broadcast a packet and receives an ACK for that packet, it can stop trying to broadcast that packet.