Consistency of data across Local MQTT / API / Glow MQTT / Bright App
Hi
I've been looking at getting glow data (via the API and MQT) into influxb. In order to make sure I understood the data properly, I started to compare data received via API vs. that received by MQTT vs. data shown on the Bright App.
I thought it would be useful to summarise some of the conclusions (apologies if I'm restating parts of other posts):-
- I have found it MUCH easier to do everything in UTC, displaying in local time as required. If not, a very large rabbit hole awaits... All the comments below are based on the data being in UTC
- Read the API docs carefully three times. Then read again. Then read again.
- API data for different aggregation periods (daily, hourly, half-hourly, minute) is only self-consistent if the time stamp relates to the START of the relevant period. [ I think I noticed another post confirming this] ==> GOOD
- Glow MQTT data (electricity consumption today) is consistent with API data:- e.g. I have P1D data for a given day == 17.326kWh vs last Glow MQTT reading for the day == 17.328 kWh ==> GOOD
- I had a look at local MQ data last night - this was identical to Glow MQTT data for my sample ==> GOOD
- The local MQTT timestamps and Glow MQTTs timestamp (for the same data) are not quite the same - for me there is a difference of 8s (glow = local + ~ 8s). Looks like either DCC or Glow are not using original timestamps on the Glow MQTT flow? ==> ODD..
- The Bright App 30min / hour / day data is shown in LOCAL time (BST). The data for half hour / hour data is identical to the API data (after adjusting for timezone offset). Note that the "day" data will NOT be identical as the API summation is from midnight UTC to midnight UTC, whereas the App summation is from 23:00 UTC to 23:00 UTC . ==> GOOD - Expected Behaviour.
- "Current Day" data on the App does NOT agree with Local MQTT / Glow MQTT "Consumption Today. I was expecting App (current Day) to be [MQTT (current day) + MQTT(last reading yesterday) - MQTT (23:00 yesterday)]. Not investigated this properly, but it looks like the App totals start aggregating at around 23:30 UTC, not 23:00 UTC. ==> BAD
Has anyone found anything that conflicts with the above?
Any idea what is happening re: point (8) in particular? I'd be interested to know what is going on re: (7) also [technical interest, really]
One last point: I like the new Local MQTT feed. It removes a few dependencies, makes it a whole lot simpler, plus makes pricing data available via MQTT. I'll probably continue with a mix of local MQTT + API as (i) I want historical data also.
Pete.
Comments
...missed the caveat - the above is for electricity.
I have an issue with Gas units - but it may be a bug on my side...
Some of the issues may relate to the meter always using GMT as its base. A SMART meter that is unable to compensate for BST/GMT changes isn't that smart.
In my case the app data matches the local MQTT data but remember the data is updated on a half hourly basis. The first reading of the day would therefore be at 0030 GMT (0130 BST)
Yes, the meter always using GMT (or UTC nowadays) explains most of the artefacts that I came across. The other two important points are (i) use the start / end times that are properly aligned with period boundaries and (ii) the timestamp relates to the start of a (30 min, 1 hr...) period. If you take care of those two things then the API data is self-consistent for any period.
I solved my gas issue that I had - a bug on my side
CURRENT SETUP:-
For now, I have settled on the following:-
TODOs (still!):
NOTE:
Are you sure the meters always work in GMT/UTC? I've not checked the data returned from the API yet, but the tracking totals for both gas and electric coming via local MQTT for me are resetting at midnight BST (23:00 UTC).
So maybe it's meter/firmware specific, or Glow are adjusting something at their end? Here's data from Friday night (Sep 30th) so shows both a daily and monthly reset:
{"electricitymeter":{"timestamp":"2022-09-30T22:59:56Z","energy":{"export":{"cumulative":0.001,"units":"kWh"},"import":{"cumulative":55.519,"day":0.007,"week":0.033,"month":0.051,"units":"kWh","mpan":"not available","supplier":"P1","price":{"unitrate":0.27910,"standingcharge":0.50050}}},"power":{"value":0.126,"units":"kW"}}}
{"electricitymeter":{"timestamp":"2022-09-30T23:00:06Z","energy":{"export":{"cumulative":0.001,"units":"kWh"},"import":{"cumulative":55.519,"day":0.000,"week":0.033,"month":0.000,"units":"kWh","mpan":"not available","supplier":"P1","price":{"unitrate":0.27910,"standingcharge":0.50050}}},"power":{"value":0.126,"units":"kW"}}}
I also have the bug where my cumulative totals are 1000x less than they should be, but that's apparently a meter bug and nothing to do with Glow.
Hmm - that's odd.
My daily data is definitely reset at 00:00 UTC.
{"electricitymeter":{"timestamp":"2022-10-23T23:59:56Z","energy":{"export":{"cumulative":0.001,"units":"kWh"},"import":{"cumulative":18032.645,"day":12.367,"week":12.367,"month":297.519,"units":"kWh","mpan":"1900060123828","supplier":"EONNext","price":{"unitrate":0.35310,"standingcharge":0.42670}}},"power":{"value":0.321,"units":"kW"}}}
{"electricitymeter":{"timestamp":"2022-10-24T00:00:06Z","energy":{"export":{"cumulative":0.001,"units":"kWh"},"import":{"cumulative":18032.646,"day":0.000,"week":0.000,"month":297.520,"units":"kWh","mpan":"1900060123828","supplier":"EONNext","price":{"unitrate":0.35310,"standingcharge":0.42670}}},"power":{"value":0.326,"units":"kW"}}}
Is data from your meter with a timestamp of time 23:00Z actually produced at 23:00Z (i.e. is the clock correct?). It would be interesting to see what happens over the clock change next weekend.
Pressed "send" too soon...
Whilst my electricity daily data / weekly data is reset at 00:00 UTC, the weekly data shows an odd behaviour between 23:00 UTC and 00:00UTC:
This shows that
..net effect is that usage for the last hour of the week is not accumulated - I think someone else reported this also. I'll search the forum later.
NOTE: API data (half-hourly, hourly, daily) is self consistent, but I've not checked weekly data via the API. That should show whether the issue affects just the MQTT feed or is a more general problem
My data is definitely resetting at midnight local (both before and after BST). I've still had to do some custom coding on the gas values because they don't reset until about 10 past which screws up my grafana visualisations...a few charts show max value for the day but if "yesterday" used more than "today" then its values get plotted for both days.