Allow direct message paths when denyf * is set

[ZjMNZHgG5jMXw]

Problem

Flood messages cannot pass repeaters when region denyf * is set. Repeaters with this setting would not only deny forwarding channel messages and flood adverts, but also all messages which are used to negotiate and establish new paths for direct messages. In areas where all repeaters deny forwarding flood traffic, no direct message paths can be (re)established.

What changed

The repeater setting region denyf * now allows flood-forwarding

• REQ,
• RESPONSE,
• TXT_MSG,
• ACK,
• ANON_REQ,
• PATH,
• TRACE,
• MULTIPART,
• CONTROL,

and denies flood-forwarding

• GRP_TXT,
• GRP_DATA, and
• ADVERT.

The payload type CUSTOM won't be forwarded, this logic has not been touched by this patch.

Why

The mesh network should always allow and support direct messages. The intention of a repeater admin who chooses to set region denyf * is likely wanting to restrict message types which typically clog the network by being flooded, such as GRP_TXT, ADVERT, and, since equally potential, GRP_DATA. The intention of the admin is likely not to shutdown the network for direct messages which they could easier achieve by switching off the repeater. Setting region denyf * in the current implementation (without this patch) would leave repeaters in an unusable state for forwarding (flood and) direct messages.

Expected outcome

Repeaters with region denyf * set are usable for direct messages, including forwarding messages and establishing paths for direct messages by flood-forwarding all message types which are used for establishing direct message paths. The repeaters should stop forwarding channel traffic and flood adverts.

Compatibility

Repeaters modified by this patch are compatible with unmodified repeaters of all versions, since region denyf * only takes out channel messages and flood adverts. These messages are fire-and-forget for companions and repeaters. Also, repeaters may choose to drop messages at their discretion already today. For companions repeaters would appear as absent for channel messages and flood adverts.

Testing

This patch has been successfully tested on a Heltec v3 repeater with region denyf * and region allowf *. For the test, a nearby observer node was watching the traffic created by a companion and forwarded by the repeater.

[skyepn]

I agree that the mesh network should always allow and support direct messages at a minimum, but it may still be useful to have the ability to turn off direct message flood routing in the case of a bad actor or misbehaving software.

In many large meshes the main problem is too much group text traffic (bots, etc) between regions.
More fine grained flood control will be needed for different scenarios.

I would propose replacing denyf arg * with bitwise flags for functional groups/packet classes, something like:
ALL (would override any other flag)
TXT (direct messages)
GRP (group messages)
ADVERT
etc tbd

I don't have detailed knowledge of which packet types are required for what functionality, but being able to block flood for only certain classes of packets would be a big improvement, and grouping the packet types into functional classes would make administration much more intuitive.

[ZjMNZHgG5jMXw]

I don't have detailed knowledge of which packet types are required for what functionality, but being able to block flood for only certain classes of packets would be a big improvement, and grouping the packet types into functional classes would make administration much more intuitive.

I agree, this could be a good next step. 👍

[skyepn]

Slightly off topic but using * as the argument is creating a lot of confusion.
People don't expect * to mean "no region set" they assume it means "any region set". And then it's still confusing because it's not clear if it means any region set, or any region or no region.
denyf unset or similar would be a lot more intuitive.

[Jemihi]

Please merge

[ZjMNZHgG5jMXw]

@ripplebiz, is there anything I can do to attract your attention to this PR for merging into the next release? Repeaters using the PR's code with denyf * activated are already in wide use throughout Germany. The code could therefore be considered tested. I would also appreciate your comment if you deem the concept or implementation flawed and thus vote to close the PR. In this case, I would be interested in the alternatives which might be planned or in implementation for future releases. Thanks for your great work!

[zindello]

I'll put my hand up and say I'd love to see this implemented sooner rather than later too. We have a repeater on the edge of our local mesh that has an asymmetric link with another - very noisy mesh. We've no way to block that noise other than to turn off global flood - but everything that relies on that repeater to link back to us is now no longer able to be administered remotely.

Very frustrating.

If this gets rolled into the next firmware I'll also put my hand up to implement it in pyMC as well.

[ripplebiz]

Apologies for taking so long getting back to this PR.
My reading of the main problem is that the companion firmware support for scopes is still incomplete. ie. If via the app UI, user could set scope for adverts, scope for DM's, requests, etc then this workaround wouldn't be needed(?)

Just seems messy to break up the rule to only apply to certain packet types.

[zindello]

Does the login process form the repeater flood any return traffic?

If the companion could be updated to send the appropriate traffic via direct OR with the relevant scope attached then that would solve the issue as well, but if there is any flood traffic in the responses to the companion as well.

Seems like a lot more work.

There are absolutely use cases where you might not want flood group messages to propagate through the repeaters, but you DO want adverts to propagate so that you can then establish a direct messaging conversation, or remote admin a new repeater on the network. If denyf blocks ALL flood, these packets will never propagate through the network.

I personally would actually like to see a fine grained repeater level ACL system, based on neighbour hint, source, packet type or any combination, but that's outside the scope of this discussion.

[Jemihi]

Denyf * (which is crucial to set) blocks not only channel unscoped repeats, it also blocks other unscoped repeats, which should not be blocked.

This PR fixes this problem. In current release firmware denyf * is not usable, which is the reason nobody I know uses the official release and instead uses mattzz' evo-firmware.

If this fix is not adressing all problems or so, a new issue should be opened and be fixed.

[ZjMNZHgG5jMXw]

Thanks for your reply, @ripplebiz!

If via the app UI, user could set scope for adverts, scope for DM's, requests, etc then this workaround wouldn't be needed(?)

Cross-region direct messages would still require this or a similar (as in using a different command) PR even with default region support. This way, people from different cities, states, and countries could still talk while repeater admins won't be concerned setting up regions (or rather message corridors) for every possible communication path. For instance, Hamburg and Leipzig are well connected and there are people who want to use this connection to exchange direct messages. The combined channel traffic plus flood adverts from both cities would clog the mesh, this much we know from experience, so it makes sense to keep the two cities in distinct regions without supporting one common super-region.

As an outlook, companions currently have trouble to find the addresses of peers, since cross-region flood adverts are suppressed when denyf * is active. In the future, this issue could be resolved by a new directory node which consumes and sends upon request direct message adverts (as opposed to zero-hop adverts and flood adverts). People from Hamburg could query their directory node for a directory node and a room in Leipzig. In the room, they could meet people which they subsequently could contact by requesting the direct message advert from Leipzig's directory node. The advantage of this setup is (a) it's not overly complicated to understand for users, (b) it's working well with separate regions providing local support groups, and not least (c) it's nudging people towards direct messages where flood messages would hurt the mesh network most: long-distance messaging requiring otherwise (potentially) over-sized regions including too many companions.

Also, since mentioned several times as the alternative, default regions would be a complex beast to tame! Two parties wouldn't be able to connect for direct messaging if their default regions do not intersect for both parties at the same time. I tried to scribble it down with colour-coded scopes for communication parties (letters) and their default regions (blobs).

Anyway, that's why I like and would argue for going forward with enabled cross-region direct messages.

[ZjMNZHgG5jMXw]

Staring at it again: in the last case, A isn't even connected to C, since C is out of reach for A sending messages in their default scope to C. It doesn't help that C uses the super-region as default region, both parties would have to play this game. Eventually, A, B, C will learn that it's the most robust choice to use the super-region as default. However, this creates a tragedy of the commons situation regarding the mesh's bandwidth, that is, it recreates the very issue which I thought should be resolved by regions.

[skyepn]

However, this creates a tragedy of the commons situation regarding the mesh's bandwidth, that is, it recreates the very issue which I thought should be resolved by regions.

This seems to be a critical flaw and why I think the issue has to be solved at the repeater level by implementing a basic packet filter + rate limits + routing table

[speedloc]

hi why flood ADVERTs dont forward if region was set? can you implement it please? if region of repeater was receive / send by companion and is ok, forward flood advert, if not exit() ?! if region denyf * was set and for exsample region allowf #de => flood allowed, adverts must be work. but dont do it :(
regards

[speedloc]

v1.15 bring default scope, hope this commit was updated to send advert with default scope than^^

[speedloc]

v1.15 do forward * F denied and PMs packages automatical, if default region/scope was set in companion/client. patch dont need anymore i mean

[ripplebiz]

Am closing this one because default scope addresses this, but also because it just complicates the configuration, and denying unscoped flood traffic should mean exactly that. I can't help but think these 'exceptions' will just confuse admins down the track.

[msmuenchen]

Default scope does not address the problem from a user perspective - DM path discovery packets should be treated differently from other packets.

Let's look at the German mesh for example. People are encouraged to use the most regional scope possible as default to reduce the load on the wider network - mostly, that means their city (e.g. de-by-muc for Munich). Now, when someone from Munich wants to reach someone in nearby Landshut (where people use de-by-la)... path discovery will not work because the repeaters on the region borders don't repeat the "wrong" scoped packets.

Using the "larger" scopes (e.g. de-by or de) is not possible in practice because these are blocked on too many repeaters.

[ZjMNZHgG5jMXw]

While the criticism of regions might still be valid, I lost my interest in arguing for this PR. In Hamburg, we observed a surge of Anon Req, Request, and Response packets, often with absurdly long paths, thus, obviously without any relevance for my part of the world, and probably caused by repeater/room confusions due to ID/1-byte hash collisions. In contrast, cross-regional direct messages were rather not happening and connections to distant rooms could not happen with all these 1-byte hash collisions.

The constant hammering of obviously irrelevant packets will eventually drive me away from MeshCore. Until then I got another PR running for repeater owners who want to encourage people in their area to switch to modern defaults, that is, regions and multi-byte hashes enabled. I hope enough people will adopt these new(?) features with or without the new PR so the senseless hammering packet flood stops eventually. Once the mesh works locally, we can think about exchanging messages on a more global scale again.