Connecting using Python and the Paho MQTT client

iswoolleyiswoolley
in API Info

Here's my notes from collecting electricity data from my SMETS2 meter via a Glow IHD/CAD using Python3 and the Paho MQTT client:


Some simple Python test code looks something like:


import paho.mqtt.client as mqtt

import paho.mqtt.subscribe as subscribe


def mqtt_on_message(client, userdata, msg):

  print(msg.topic+" : "+str(msg.payload))


subscribe.callback(mqtt_on_message, "SMART/HILD/xxxxxxxxxxx", hostname="glowmqtt.energyhive.com", qos=0, userdata=None, port=8883, client_id="", keepalive=60, will=None, auth={"username":"xxxxxxxx", "password":"xxxxxxxx"}, tls={"ca_certs":"cert.pem"}, protocol=mqtt.MQTTv311)


Which will block and print out message payloads as they arrive. Note that this code is using a MQTTS connection (on port 8883), so needs a valid certificate file to establish the TLS tunnel. The cert.pem file needs to contain both the Lets Encrypt X3 intermediate certificate and the ISRG X1 root certificate with which the intermediate certificate is signed. Both are available from https://letsencrypt.org/certificates/ but are copied below. I've used this approach both on a Win10 desktop and an AWS EC2 Linux2 instance, so appears to be portable.


-----BEGIN CERTIFICATE-----

MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw

TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh

cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4

WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu

ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY

MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc

h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+

0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U

A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW

T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH

B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC

B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv

KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn

OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn

jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw

qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI

rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV

HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq

hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL

ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ

3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK

NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5

ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur

TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC

jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc

oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq

4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA

mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d

emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=

-----END CERTIFICATE-----


-----BEGIN CERTIFICATE-----

MIIFjTCCA3WgAwIBAgIRANOxciY0IzLc9AUoUSrsnGowDQYJKoZIhvcNAQELBQAw

TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh

cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTYxMDA2MTU0MzU1

WhcNMjExMDA2MTU0MzU1WjBKMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg

RW5jcnlwdDEjMCEGA1UEAxMaTGV0J3MgRW5jcnlwdCBBdXRob3JpdHkgWDMwggEi

MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCc0wzwWuUuR7dyXTeDs2hjMOrX

NSYZJeG9vjXxcJIvt7hLQQWrqZ41CFjssSrEaIcLo+N15Obzp2JxunmBYB/XkZqf

89B4Z3HIaQ6Vkc/+5pnpYDxIzH7KTXcSJJ1HG1rrueweNwAcnKx7pwXqzkrrvUHl

Npi5y/1tPJZo3yMqQpAMhnRnyH+lmrhSYRQTP2XpgofL2/oOVvaGifOFP5eGr7Dc

Gu9rDZUWfcQroGWymQQ2dYBrrErzG5BJeC+ilk8qICUpBMZ0wNAxzY8xOJUWuqgz

uEPxsR/DMH+ieTETPS02+OP88jNquTkxxa/EjQ0dZBYzqvqEKbbUC8DYfcOTAgMB

AAGjggFnMIIBYzAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADBU

BgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEBATAwMC4GCCsGAQUFBwIB

FiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQub3JnMB0GA1UdDgQWBBSo

SmpjBH3duubRObemRWXv86jsoTAzBgNVHR8ELDAqMCigJqAkhiJodHRwOi8vY3Js

LnJvb3QteDEubGV0c2VuY3J5cHQub3JnMHIGCCsGAQUFBwEBBGYwZDAwBggrBgEF

BQcwAYYkaHR0cDovL29jc3Aucm9vdC14MS5sZXRzZW5jcnlwdC5vcmcvMDAGCCsG

AQUFBzAChiRodHRwOi8vY2VydC5yb290LXgxLmxldHNlbmNyeXB0Lm9yZy8wHwYD

VR0jBBgwFoAUebRZ5nu25eQBc4AIiMgaWPbpm24wDQYJKoZIhvcNAQELBQADggIB

ABnPdSA0LTqmRf/Q1eaM2jLonG4bQdEnqOJQ8nCqxOeTRrToEKtwT++36gTSlBGx

A/5dut82jJQ2jxN8RI8L9QFXrWi4xXnA2EqA10yjHiR6H9cj6MFiOnb5In1eWsRM

UM2v3e9tNsCAgBukPHAg1lQh07rvFKm/Bz9BCjaxorALINUfZ9DD64j2igLIxle2

DPxW8dI/F2loHMjXZjqG8RkqZUdoxtID5+90FgsGIfkMpqgRS05f4zPbCEHqCXl1

eO5HyELTgcVlLXXQDgAWnRzut1hFJeczY1tjQQno6f6s+nMydLN26WuU4s3UYvOu

OsUxRlJu7TSRHqDC3lSE5XggVkzdaPkuKGQbGpny+01/47hfXXNB7HntWNZ6N2Vw

p7G6OfY+YQrZwIaQmhrIqJZuigsrbe3W+gdn5ykE9+Ky0VgVUsfxo52mwFYs1JKY

2PGDuWx8M6DlS6qQkvHaRUo0FMd8TsSlbF0/v965qGFKhSDeQoMpYnwcmQilRh/0

ayLThlHLN81gSkJjVrPI0Y8xCVPB4twb1PFUd2fPM3sA1tJ83sZ5v8vgFv2yofKR

PB0t6JzUA81mSqM3kxl5e+IZwhYAyO0OTg3/fs8HqGTNKd9BqoUwSRBzp06JMg5b

rUCGwbCUDI0mxadJ3Bz4WxR6fyNpBK2yAinWEsikxqEt

-----END CERTIFICATE-----

Share on Twitter

Comments

  • ghayneghayne

    Thanks for the information!

    Share on Twitter
  • ndfredndfred

    This is great! Based on this I was able to replicate it over the command line for easy testing (just brew install mosquitto first):

    $ mosquitto_sub -h glowmqtt.energyhive.com -u "USERNAME" -P "PASSWORD" -t "SMART/HILD/DEVICE_ID"

    I wasn't able to get the right incantation to work to use --cafile and connect over TLS but if I figure it out I'll be posting here. My best bet would have been this but it throws a "TLS error occurred" message:

    $ curl "https://letsencrypt.org/certs/isrgrootx1.pem.txt" > lets_encrypt.pem
$ curl "https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt" >> lets_encrypt.pem
$ curl "https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt" >> lets_encrypt.pem
$ mosquitto_sub -h glowmqtt.energyhive.com --cafile lets_encrypt.pem -u "USERNAME" -P "PASSWORD" -t "SMART/HILD/DEVICE_ID"
Error: A TLS error occurred.
    Share on Twitter
  • ndfredndfred

    I actually made a gist with the Paho MQTT code including parsing the values to read electricity and gas consumption. The code does not include encryption for convenience, really just meant as a way to debug stuff.

    Share on Twitter
  • KS_EricKS_Eric

    I've followed ndfred's link to his gist but have a couple of very basic queries - sorry !

    1. Just what is a "gist" : it's not a term I've ever seen before.
    2. I think I understood what was wanted for GLOW_LOGIN & "GLOW_PASSWORD" but am having doubts about "GLOW_DEVICE_ID" Is it the serial number of my Glowstick, the common 'key' for Glowmarkt queries or something else again ? When I ran the script using my serial number, it seemed to run OK except that it just sits there having displayed "Connected and subscribed". Is the script supposed to do something else or is it just a starter that will allow me to make a specific query later ? If the latter, there doesn't seem any way to enter anything else into that Python window.
    Share on Twitter
  • KS_EricKS_Eric

    I think I've (sort of) answered my two main points :-

    1. I still can't find a specific IT use of 'gist' so assume it's simply it's general meaning of 'nub' or 'salient points.
    2. From an email of Jane's, I've deduced that "GLOW_DEVICE_ID" probably refers to the Glowstick's MAC.

    However, although the script runs OK it still only gets as far as displaying "Connected and subscribed" then stops. I suspect I need an extra line (or several) to actually receive data from Hildebrand and then do something with it.

    Share on Twitter
  • KS_EricKS_Eric

    I've found my 'extra line' - I just needed to add

    print ("Payload : ", payload) 

    within the def on_message block


    script now receives a message from glowmqtt.energyhive.com and prints it out.


    I'll worry later about what to do with results 😎


    All I really wanted to do was to find the current reading of the import register without having to go out in cold or rain to read meter.

    It would have been nice also to find the current reading of the import register but that doesn't seem to be within the data set - probably because I'm not claiming export payments from Octopus so meter hasn't been programmed to send that reading.

    Share on Twitter
  • KS_EricKS_Eric

    I've pretty well accomplished what I set out to do with a suite of two python files , one php file and one batch file run from Task Scheduler.

    I can download the JSON payload with a script I call GetMeterReading.py which collects the JSON payload and dumps it into a local file. Then I run another script I call filread.py which extracts reading time & meter reading from that file. Afraid I couldn't follow the JSON 'map' that @ndfred offered but have used a Q&D method of treating payload as a string then finding the right 'slug' of data with string functions. This is followed by a call to my webpage where I run a php script called UpdateMreads.php which stores the meter data in a database table. The two python scripts are called from a batchfile by Task Scheduler every half hour.

    So at the moment I'm constructing a database table that will have half-hourly readings from my 'smart' meter. Next step will probably be to import Agile price data for each half hour so that I can calculate my own daily bills.

    There's a fuller description on the Github pages at https://github.com/bluerhinos/phpMQTT/issues/106

    Share on Twitter
Sign In or Register to comment.