Subscribe homie/+/# is effectively what’s happening though as it’s every device
Possibly limited to the Homey homie tree
homie/homeyname/+/#
@robertklep Can you reproduce the actual problem?
If you restart the hub, does it cause any flow to be disabled?
I’ve got 1 device in Homey (Homey itself), so it’s hard to test 
I’ll check too later when I get back home. I have a few devices …Got to go
I have not seen this issue but don’t use incoming MQTT flow triggers really
Interesting though
I hardly have any devices in Homey, and not running the app, so not sure why I said that I could reproduce the actual problem… 
Not sure if this helps but whenever I try to discover new devices in Tasmota MQTT I get the too many flows error occur , they then all get disabled and I then have to manually re-enable them again. All these flows use MQTT client …
I have tried hard to utilise the in-built 433MHZ feature in Homey (see my other posts) but I gave up in the end and now use a SonOff RF bridge thats been flashed with Tasmota and communicate to all my 433Hhz devices via MQTT…
When listening for MQTT events from my 433Mhz PIR sensors I use this method to capture all the incoming messages .
See flow (below)…
Basically I just get the system to scan and listen for a specific piece of text in all of the incoming MQTT data (identifying the unique device identity) and then make trigger depending on that devices identity…
This is the only on MQTT communication I use in my flows or system but I get the flow overload error when ever I get Tasmota MQTT to discover new devices . Not sure if this helps but just thought I’d share it …
The cause of the problem is line 37-40 in messagehandling.js
It triggers a flow card for each incoming message.
@scanno Can we match the incoming message topic against ‘ref.triggers.broker.getTopicArray().getTriggerTopics()’ before actually executing the trigger?
So the rate limiter is tied to .trigger() being called, and not the run listener for that trigger that may (subsequently) decide that the trigger isn’t meant for it? Not sure if that makes sense…
FWIW, I already tested this in my PoC and I wasn’t able to trigger the rate limiter:
const trigger = this.homey.flow.getTriggerCard('test');
trigger.registerRunListener((args, state) => {
console.log('listener triggered', args, state);
return Promise.resolve(false);
});
for (let i = 0; i < 500; i++) {
trigger.trigger({ idx: i }, {});
}
I can reproduce the problem by adding a flow card trigger on a broad wildcard topic like ‘#’ or ‘homie/#’. Such a flow will be disabled in no time due the many triggers, but that’s expected behaviour.
Found something else, I see these log messages a lot:
20210409-18:18:25 MQTT Offline
20210409-18:18:25 De MQTT broker is ONBESCHIKBAAR
20210409-18:18:29 Broker State: DISCONNECTED
20210409-18:18:35 MQTT Closed
20210409-18:18:35 Broker State: DISCONNECTED
20210409-18:18:37 MQTT Reconnect
20210409-18:18:37 Broker State: RECONNECTING
20210409-18:18:42 De MQTT broker is weer BESCHIKBAAR
20210409-18:18:45 MQTT client connected
20210409-18:18:46 Broker State: CONNECTED
Each time the broker is reconneced, all retained messages on the broker (thousands in my case) are send to the client. This causes an endless stream of the same messages.
@Zimo To answer your initial question about the log:
LOG explained:
- a message is received on topic: homie/homey-5dd…1/town/conditioncode
- a flow card trigger is registered to topic: openHab/Okno3/BlindsControl
The next line says: We are not waiting for this topic
meaning: The flow card is not triggered by this message, because the pattern does not match.
It does not explain why the flow gets disabled though…
NOTE: Your log is from the MQTT client, not from the Hub.
That would not solve all problems… You need to iterate through the trigger cards anyway to see if a topic matches.
But a topic match can be checked before even calling the trigger? The flowcard trigger topics are known on beforehand. I know it shouldn’t matter because the trigger response will be false anyway, but who knows…
You only prevent iterating through the trigger cards when there is no registered topic. You still have to go through them all to do the matching when the topic is registered.
The best solution would be to have a list of topics and the corresponding trigger card id’s and then be able to trigger that specific card.
Sorry Harrie for dumb question, is there a way to get better logs then copy-pasting log-lines from app interface? Because it trims logs and I can see only last 30 seconds after hub restart and there is much more log lines which I cannot capture…
Was just curious as to how this went. Was the problem figured out ?
@Russell_S Well…
I made a pull request with some improvements to the MQTT Client.
@scanno Needs to look into it before it can be send to the store.
But maybe someone is willing to test already by installing via CLI?
branch: GitHub - harriedegroot/nl.scanno.mqtt at feature/topic-subscription
DETAILS
- Made some improvements to the topic registrations (api vs trigger)
- Fixed handling of retained messages
- Prevent losing api connections
- Addressed the bug of randomly disabled flows
- Added unsubscribe api
- Introduced a TriggerQueue to handle the rate limiting of flow card triggers and prevent disabling of flows
- Added broker online/offline flow card triggers
- Added flow card action to send an empty message to be able to remove retained messages from the broker
- Introduced TopicsRegistry to register topics by reference (e.g. app id), this reference can be used with the unsubscribe api
- Performance improvements
- General bug fixes
Commit 1
Topic registration:
- always subscribe to the broker to return retained messages on subscription
- re-subscribe to all previously registered topics on connection of the broker/mqtt-client (keep topic registry alive)
- made some calls async and actually wait for the result before continuing + fixed some async handling of promises
- extended the TopicArray with seperate methods for api & trigger topics
Commit 2
- prevent topic unsubscription if its used by another registration (api vs trigger)
Commit 3
Flow card triggers:
- prevent destruction of the full topics array, causing all api connections being lost
- subscribe to all flow card trigger topics on initialisation
- on trigger card change (flow card config): unsubscribe from removed topics & subscribe to newly added topics
- await the Homey trigger call
- check for api topic match before forwarding the message to the realtime api
Commit 4
pre-check to prevent uneccesairy calls to the Homey trigger system
prevents disabling of random flows not even listening to these topics (Homey BUG?)…
this is also a performance boost, since the Homey trigger system won’t be called for all api topics
the small downside is the duplicate topic matching, if there is a match on a trigger topic
Commit 5
(added missing file ‘Trigger.js’ for commit 3
Commit 6
On change of app settings:
- prevent destruction of the full topicsArray resulting in lost api connections
- re-connect to the broker to trigger subscription al all registered topics (instead of the trigger topics only)
- await topic unsubscription before setting up the new client
Commit 7
added unsubscribe api
Commit 8
Introduced a TriggerQueue to handle the rate limiting of flow card triggers and prevent disabling of flows
Commit 9
Added broker online/offline flow card triggers
Commit 10
Added flow card action to send an empty message to be able to remove retained messages from the broker
Commit 11
Introduced a TopicsRegistry to register topics by reference (e.g. app id). The unsubscribe api can only be called for topics with the reference provided during topic subscription. This prevents unsubscription of topics used by other apps/processes.
Also optimized the topic matching by using a topic map (instead of array lookup).
2 Likes
I’ve got this running for a few days now with several thousands of (retained) messages. Also have a test flow with a trigger on topic ‘#’, which forwards the data to another mqtt topic. Non of the flows has been disabled yet…
As soon as I can find some spare time, I will have a look at the pull request. Work and thesis consume all my time right now.
Thanks for the effort.
1 Like
