While I’m still seeing a lot of slowness and timeouts on the API, I’m also sometimes seeing errors like
{"isValid":false,"error":"missing elements -query.from,query.to,query.function,query.period","missingElements":["query.from","query.to","query.function","query.period"]}
It was ok for a while, but now I'm having regular (and at times persistent) outages on both API and MQTT. I'm also having MQTT occasionally report electricity consumption around 10k less than it is for a couple of seconds and then correcting.
We are aware that performance isn't where we want it to be yet - there are two new firewalls arriving today and once they've been configured, etc. they will be installed at the data centre, probably over the weekend.
Clive will give you all a 'chapter and verse' which is much more technical when he has time, in the meantime I appreciate you tolerating my non-technical updates and appreciating that things are not back to normal yet.
LOCAL ACCESS
I promised an answer to @sbb - there are a range of reasons why we don't currently provide local access to the data - commercially it isn't our business model. Note that we believe we are the only UK CAD manufacturer to make the data readily available via our API's or MQTT. Practically, if we didn't have internet connectivity we couldn't join the device to the meter, provide firmware updates, deliver access to historical 30 minute data from before the CAD install, provide support, or any API access for other data streams we pick up from external sources, including the DCC itself.
It is on our roadmap to support local access at some point, but as we provide the service for free to the market, the time we do give it focuses on where we think we can make the most impact in our ambition to support the UK getting to net zero - Export is next.
THANK YOU
Thank you to everyone for being supportive over the past few days, we really appreciate it.
currently Bright App functions now great !Thanking you.
However, I use Bright to directly show me my history of useage. My electric is completely working with updates happening every 30 minutes or so and all history is there.
My gas however has no history showing since the start of the outage 4 days that is. It only reports the daily standing charge everyday in pence but NO kWh of consumption.
my smets1 meters (gas & electric) adopted successfully months ago
My CAD Box (smets1 variant) from you does report useage for both gas & electric daily but only for each day ie today and then overnight it resets back to zero useage at about midnight. So there is no history for either fuel being held within the CAD device.I checked this no week/month data at all !
This is why I rely on Bright for the history from the DCC and to a lesser extent for the CAD for just ‘todays’ consumption
PS The CAD device also doesn’t record the daily standing charges for either fuel in the costs even though you can see they are correctly stored within the display settings units menu of the box. Whereas the Bright App to DCC route does include the daily standing charges with kWh units for both fuels correctly in the costs !
Hi all - as a reminder, please email us at support@glowmarkt.com for specific issues unique to your account - we do not monitor the forum all the time and do not use is as a support solution, thank you very much. Jane
RE Local Access: Wouldn't allowing the CAD to push to a local MQTT improve sales of the CAD though? It would provide a very distinct advantage over just using a free account.
Hi all - appreciate the input and thinking on local access - I think we are the only organisation that is a DCC Other User, produce CADs, sell those CADs direct to the public and supply access over MQTT and APIs - so not sure that local access would make a big impact on our sales. Do you think that we'd sell thousands of additional units if we offered local access? I'm seriously asking.
Do you feel that we'd make more impact on the drive to net zero by prioritising local access over supporting export? No-one pays for our dev time to do the work so we have to prioritise what we do and it has to fit in between commercial projects (we also have a heat network metering and billing solution and other commercial, smart meter based, offerings).
What fee would you think reasonable to pay per year for local access?
Thanks for your input on building the business case. Be warned, you'll need to work hard to persuade me to do this from an impact perspective first - before export - just warning you ;)
Happy weekend all - two new firewalls going into the CoLo tomorrow.
Happy Saturday - MQTT is currently off because IT Ops have just turned it off. We will make updates on the firewall installation etc. in the new discussion.
I would not be keen on a regular fee for local access. One of my main motivations for implementing local access is to ensure I’m not subjected to chargeable third party services. I would expect local access to be an included and basic option, especially as it would reduce the impact on your own network due to fewer people running constant MQTT connections to your cloud services. However, I would expect you to be charging for your cloud services, additional analytics in the app, etc. But, having to pay extra for me to be able to directly access a device I have already purchased totally locally on my own network does not sound reasonable. I understand there’s development effort involved to implement that functionality, and it would indeed add value to your device, but the thinking that you’re the only company offering this combined solution therefore don’t need to add that functionality would leave a gap in the market for someone else to swoop in and fill. You have the opportunity to plug that gap before it’s filled by someone else.
Well, because MQTT feed decided to stop working again, i've decided to look into what it would take to disconnect the display from the cloud (after it has been paired) and it is surprisingly little, as it turns out "latest SSL technology for secure Internet communications" as implemented by glow/hildebrand has a glaring hole in it, they do not verify SSL certificates therefore any self signed certificate (or certificate for incorrect domain) will work.
So all it took to make the display talk to my HTTP server was set up DNS entry on my DNS server to point the display at my HTTP server, add appropriate domain, configure web server to send minimum headers (one of the default headers was making the display not work correctly) and send "200" as a reply.
raw data as it comes from the display can be then sent to mqtt server.
Nice work. Was the MQTT data in the same numbering format as the feed they provide ?
Unless they roll out an update patching this flaw then this this will always be possible - wonder if a python or similar file could be written to get the local access to this data unless, of course, they provide a way for us to get to it locally?
For our customers with a CAD, Bright, our APIs and MQTT have been restored but response times may still be a bit slower than usual.
For people who don't have our CAD and are using Bright or any libraries that rely on the 30 minute updates from the DCC, API or pseudo MQTT, they are not currently enabled and are returning HTTP 503. The overnight updates should be working as normal.
AND
(5/2 )Happy Saturday - MQTT is currently off because IT Ops have just turned it off. We will make updates on the firewall installation etc. in the new discussion.
MQTT for my glowstick is still not working this morning. Has the new firewall installation run into problems or have your IT OPS forgotten to restore access ?
Bright App is working OK.
STOP PRESS
Clive posted a message to say that he'd just turned it on again a few mins before I submitted this !
All the guides for MQTT say to use TCP port 8883. I get socket errors on that. It appears that mosquitto retries on the normal MQTT port 1883, and indeed that is working. So I changed my config to avoid all the failed connection attempts for my broker bridging.
Of course port 1883 is not encrypted, I can see my data in wireshark in clear text. Is there any expectation to have port 8883 over SSL working again?
The MQTT feed started working again for me at 10:21 today Sun Feb 06
Hi @Davecl - do you have our hardware? If you do - please drop an email to support as that is an issue.
If you don't, as I said earlier "For people who don't have our CAD and are using Bright or any libraries that rely on the 30 minute updates from the DCC, API or pseudo MQTT, they are not currently enabled and are returning HTTP 503. The overnight updates should be working as normal."
For the record, we've always said that our MQTT feed is not designed for 30 minute updates.
We are reviewing when/how/whether we will restore regular DCC refresh and will update when a decision has been made.
Picking up your first point, even if data isn't being refreshed frequently at the moment, it should be correct - if you don't think it is correct then your meter is reporting incorrectly - it is our only source of truth for data. You'll need to contact your supplier.
Because some users without our CAD are calling the API much too frequently (it should be once per half hour) we have had to alter the way the call is dealt with and reject the excess traffic.
Most on this forum have our CAD so the point is moot but FYI, we've got a lot of Home Assistant users of MQTT. We tell people that MQTT isn't meant for half hourly data but they are using brokers that seem to make a lot of redundant calls.
We hope to get our usual more frequent updates of the DCC data in place soon but wanted to explain why we are currently on overnight only. Please bear with us and check back in a day or two. Thank you.
Thank you very much @si458 - I think you have just officially been designated Clive's Gold Star for the day. That would be super useful - we hadn't quite appreciated just how many people were using that community plugin and the frequency at which it was checking.
@jackyaz - the Display/CADs are due to arrive within a fortnight. I'm hoping that we'll have the firmware, etc. sorted out for the CAD without display within three weeks - and got the first pre-allocated units all shipped for the research project they are designated for.
Comments
Thanks Jane
While I’m still seeing a lot of slowness and timeouts on the API, I’m also sometimes seeing errors like
{"isValid":false,"error":"missing elements -query.from,query.to,query.function,query.period","missingElements":["query.from","query.to","query.function","query.period"]}
It was ok for a while, but now I'm having regular (and at times persistent) outages on both API and MQTT. I'm also having MQTT occasionally report electricity consumption around 10k less than it is for a couple of seconds and then correcting.
PERFORMANCE UPDATE
We are aware that performance isn't where we want it to be yet - there are two new firewalls arriving today and once they've been configured, etc. they will be installed at the data centre, probably over the weekend.
Clive will give you all a 'chapter and verse' which is much more technical when he has time, in the meantime I appreciate you tolerating my non-technical updates and appreciating that things are not back to normal yet.
LOCAL ACCESS
I promised an answer to @sbb - there are a range of reasons why we don't currently provide local access to the data - commercially it isn't our business model. Note that we believe we are the only UK CAD manufacturer to make the data readily available via our API's or MQTT. Practically, if we didn't have internet connectivity we couldn't join the device to the meter, provide firmware updates, deliver access to historical 30 minute data from before the CAD install, provide support, or any API access for other data streams we pick up from external sources, including the DCC itself.
It is on our roadmap to support local access at some point, but as we provide the service for free to the market, the time we do give it focuses on where we think we can make the most impact in our ambition to support the UK getting to net zero - Export is next.
THANK YOU
Thank you to everyone for being supportive over the past few days, we really appreciate it.
Oh! - Export would be very nice too - no complaints about that here.
Thanks for the update
Hi Jane
currently Bright App functions now great !Thanking you.
However, I use Bright to directly show me my history of useage. My electric is completely working with updates happening every 30 minutes or so and all history is there.
My gas however has no history showing since the start of the outage 4 days that is. It only reports the daily standing charge everyday in pence but NO kWh of consumption.
my smets1 meters (gas & electric) adopted successfully months ago
My CAD Box (smets1 variant) from you does report useage for both gas & electric daily but only for each day ie today and then overnight it resets back to zero useage at about midnight. So there is no history for either fuel being held within the CAD device.I checked this no week/month data at all !
This is why I rely on Bright for the history from the DCC and to a lesser extent for the CAD for just ‘todays’ consumption
PS The CAD device also doesn’t record the daily standing charges for either fuel in the costs even though you can see they are correctly stored within the display settings units menu of the box. Whereas the Bright App to DCC route does include the daily standing charges with kWh units for both fuels correctly in the costs !
any help would be appreciated
Michael B
Hi all - as a reminder, please email us at support@glowmarkt.com for specific issues unique to your account - we do not monitor the forum all the time and do not use is as a support solution, thank you very much. Jane
Hi Jane
RE Local Access: Wouldn't allowing the CAD to push to a local MQTT improve sales of the CAD though? It would provide a very distinct advantage over just using a free account.
You might even find that people would be willing to pay extra for local access.
100% would be willing to pay a small regular fee for Local Access
Hi all - appreciate the input and thinking on local access - I think we are the only organisation that is a DCC Other User, produce CADs, sell those CADs direct to the public and supply access over MQTT and APIs - so not sure that local access would make a big impact on our sales. Do you think that we'd sell thousands of additional units if we offered local access? I'm seriously asking.
Do you feel that we'd make more impact on the drive to net zero by prioritising local access over supporting export? No-one pays for our dev time to do the work so we have to prioritise what we do and it has to fit in between commercial projects (we also have a heat network metering and billing solution and other commercial, smart meter based, offerings).
What fee would you think reasonable to pay per year for local access?
Thanks for your input on building the business case. Be warned, you'll need to work hard to persuade me to do this from an impact perspective first - before export - just warning you ;)
Happy weekend all - two new firewalls going into the CoLo tomorrow.
Happy Saturday - MQTT is currently off because IT Ops have just turned it off. We will make updates on the firewall installation etc. in the new discussion.
I would not be keen on a regular fee for local access. One of my main motivations for implementing local access is to ensure I’m not subjected to chargeable third party services. I would expect local access to be an included and basic option, especially as it would reduce the impact on your own network due to fewer people running constant MQTT connections to your cloud services. However, I would expect you to be charging for your cloud services, additional analytics in the app, etc. But, having to pay extra for me to be able to directly access a device I have already purchased totally locally on my own network does not sound reasonable. I understand there’s development effort involved to implement that functionality, and it would indeed add value to your device, but the thinking that you’re the only company offering this combined solution therefore don’t need to add that functionality would leave a gap in the market for someone else to swoop in and fill. You have the opportunity to plug that gap before it’s filled by someone else.
Arguments for local MQTT access.
It would place you in a unique market position as being the first to offer this. A great marketing argument.
You would sell more CAD's. I know from the Octopus forum, that non-local access to data is the main reason people have not purchased your CAD.
You as a company, could still access the same data as you do now.
I will never pay a subscription for my data. I cannot imagine others would either.
Downtime, and lack of individual data, as experienced this month would be eliminated.
I don't export, so not something that's relevant for me.
+100 to this post ^^^
+1 vote for local access from me too =)
Well, because MQTT feed decided to stop working again, i've decided to look into what it would take to disconnect the display from the cloud (after it has been paired) and it is surprisingly little, as it turns out "latest SSL technology for secure Internet communications" as implemented by glow/hildebrand has a glaring hole in it, they do not verify SSL certificates therefore any self signed certificate (or certificate for incorrect domain) will work.
So all it took to make the display talk to my HTTP server was set up DNS entry on my DNS server to point the display at my HTTP server, add appropriate domain, configure web server to send minimum headers (one of the default headers was making the display not work correctly) and send "200" as a reply.
raw data as it comes from the display can be then sent to mqtt server.
So, local access is possible after all.
@jacekowski
Nice work. Was the MQTT data in the same numbering format as the feed they provide ?
Unless they roll out an update patching this flaw then this this will always be possible - wonder if a python or similar file could be written to get the local access to this data unless, of course, they provide a way for us to get to it locally?
Not sure why trying to quote failed !
@janeatGlow posted :-
(3/2) FURTHER UPDATE
For our customers with a CAD, Bright, our APIs and MQTT have been restored but response times may still be a bit slower than usual.
For people who don't have our CAD and are using Bright or any libraries that rely on the 30 minute updates from the DCC, API or pseudo MQTT, they are not currently enabled and are returning HTTP 503. The overnight updates should be working as normal.
AND
(5/2 )Happy Saturday - MQTT is currently off because IT Ops have just turned it off. We will make updates on the firewall installation etc. in the new discussion.
MQTT for my glowstick is still not working this morning. Has the new firewall installation run into problems or have your IT OPS forgotten to restore access ?
Bright App is working OK.
STOP PRESS
Clive posted a message to say that he'd just turned it on again a few mins before I submitted this !
It is indeed working now.
Are the DCC APIs still disabled?
All the guides for MQTT say to use TCP port 8883. I get socket errors on that. It appears that mosquitto retries on the normal MQTT port 1883, and indeed that is working. So I changed my config to avoid all the failed connection attempts for my broker bridging.
Of course port 1883 is not encrypted, I can see my data in wireshark in clear text. Is there any expectation to have port 8883 over SSL working again?
The MQTT feed started working again for me at 10:21 today Sun Feb 06
@bertrumuk
Same format, all i needed was a simple php script to send data to mqtt server https://paste.debian.net/1229776/
Update: Clive is making MQTT updates in the new discussion - https://forum.glowmarkt.com/index.php?p=/discussion/132/sunday-6th-update#latest
Still no correct figures in the app
It seemed to update at midnight but absolutely nothing since, either in the app or in my home assistant instance
Hi @Davecl - do you have our hardware? If you do - please drop an email to support as that is an issue.
If you don't, as I said earlier "For people who don't have our CAD and are using Bright or any libraries that rely on the 30 minute updates from the DCC, API or pseudo MQTT, they are not currently enabled and are returning HTTP 503. The overnight updates should be working as normal."
For the record, we've always said that our MQTT feed is not designed for 30 minute updates.
We are reviewing when/how/whether we will restore regular DCC refresh and will update when a decision has been made.
Picking up your first point, even if data isn't being refreshed frequently at the moment, it should be correct - if you don't think it is correct then your meter is reporting incorrectly - it is our only source of truth for data. You'll need to contact your supplier.
re: Local Access
Would pay a single one off fee but not an ongoing subscription - like paying to disable adverts on a Kindle.
DCC data only appears to be getting updated once a day now, is this the new normal? As if so, this service is no longer any use to me.
UPDATE from the Glow team
Because some users without our CAD are calling the API much too frequently (it should be once per half hour) we have had to alter the way the call is dealt with and reject the excess traffic.
Most on this forum have our CAD so the point is moot but FYI, we've got a lot of Home Assistant users of MQTT. We tell people that MQTT isn't meant for half hourly data but they are using brokers that seem to make a lot of redundant calls.
We hope to get our usual more frequent updates of the DCC data in place soon but wanted to explain why we are currently on overnight only. Please bear with us and check back in a day or two. Thank you.
I'd be happy to buy a CAD. When will they be back in stock?
thanks @JaneatGlow for the update
i have just suggested to our community plugin for home assistant we change the interval from 2 minutes to 30 minutes
https://github.com/HandyHat/ha-hildebrandglow-dcc/issues/126#issuecomment-1032600786
Thank you very much @si458 - I think you have just officially been designated Clive's Gold Star for the day. That would be super useful - we hadn't quite appreciated just how many people were using that community plugin and the frequency at which it was checking.
@jackyaz - the Display/CADs are due to arrive within a fortnight. I'm hoping that we'll have the firmware, etc. sorted out for the CAD without display within three weeks - and got the first pre-allocated units all shipped for the research project they are designated for.
Thanks all for your ongoing support.
Thank you @si458 very much
I did look at the plugin and whilst it is trivial to change the interval that is not what you want I think.
You need it to send once every 30 minutes, but soon after the actual :00 or :30
There should be a random delay added to that figure of up to 2 minutes so not every user of this plugin hits the api at the same time.
Sadly I don't speak python otherwise I would have coded it and sent a pull request.
Anyone want to volunteer to code it ;-)