LoraWAN weather station

I’ve just received a nice new ‘headless’ weather station from China, and I am engaged in connecting to its RS485 feed and understanding the protocol. They supplied a RS485-UART USB stick with it, and after playing around with some containerised NodeRED flows which couldn’t read the USB finally just used PySerial and have started dumping the data. Although it is advertised with the tag line “we will supply the communications protocol” this wasn’t forthcoming so I just dumped the bits down on the screen and have started reading them. Apparently they are a Fine Offset type of station and equivalent to the WH2950 type, without the monitor or network link. It’s all down to me!

WeatherStation

The bytes look like this:

24 B3 7A 62 61 46 00 00 00 00 00 01 00 00 00 48 A3 $.zbaF………H.
24 B3 7A 62 61 46 00 00 00 00 00 00 00 00 00 D3 2D $.zbaF……….-
24 B3 7A 62 61 46 00 00 00 00 00 02 00 00 00 D4 30 $.zbaF……….0
24 B3 7A 62 61 46 00 00 00 00 00 00 00 00 00 D3 2D $.zbaF……….-

… which is fun. I quickly worked out that the standard size was 17 bytes, now I have run it through a hex dumper and started eye-balling the bytes. I may need to spin the anemometer to generate events to get readings, but this looks good.

I’ve also now connected it via a LoRaWAN link and have the data flowing back to me via MQTT. I need to compress the data as well as recording and averaging the readings, since the standard interval time of 16 seconds would exceed the fair-use policy of The Things Network. Meteorological masts (or ‘metmasts’) conform to an IEC standard, but mine can be much less rigorous than that: I’ve decided to take ten measurements and transmit the min/avg/max of them every 160 seconds. Given enough compression I think I can get the seven measurements (wind speed and direction, temperature, humidity, UV, lux, rainfall) into less than 10 bytes.

The git repo is up on BitBucket.

Advertisements

Emergencies

I’ve just encountered one of the interesting aspects of our drive to buy everything online: emergencies. It used to be that when you lost an item, say a power cord for a computer, that you would ask a friend or colleague, or go to the shops. With the push for ‘everything online’ that last choice has disappeared.
In the 90s there were many places where you could find spare items for your travel kit. I often added to my kit bag when travelling through an airport–the electronics shops there had lots of things including mosquito zappers, bug spray, ear buds, headphones, travel pillows, clothes detergent, and many odd fitments for your average travel worrier. Failing, the absentminded would exit their warm hotel and venture into the local shops to buy things from many places which stocked bits and parts. They used to be as frequent as nail bars in my local shopping area.

Now, along with the growth of fixed-price betting shops (and those nail bars), I miss those electronics shops replaced by endless coffee bars and empty hoardings. So commerce has moved online where prices are cheaper, but the immediacy of picking things up when you wanted has gone and I’m back to ‘making do’ for two days while my latest purchase is delivered.

Feelin’ melancholic

Long way back I read a book which spoke of four types of personality. Nowadays we recognise five broad trait dimensions or domains, sometimes called the “Big Five”: Extroversion, Agreeableness, Conscientiousness, Neurotic-ism, and Openness. The 1960’s book wasn’t to blame for only choosing four since even longer back people talked of four elemental substances, such as earth, water, air, and fire. Especially the ancient Greeks, and they had some pretty good ideas.

But regardless – a personality type of ‘melancholic’ was one of the four types and these types of people were meant to be more prone to moodiness, feelings and deep thoughts.

I reject this now. My whole life experience suggests that instead of general types of people and categories into which they fall, that rather everyone has umpteen dimensions and lies in a spectrum along each of those. So I may be good at one thing and bad at another, while my neighbour is average at both. The point being that there is hundreds, if not thousands of these dimensions and everyone has them in different quantities and qualities. Understanding people through a lens of a specific type is almost as useless as handwriting analysis, which simply tells you that the writer is a person with human features – and yes, I read lot of books about that as well before realising that just speaking to someone for five minutes would reveal far more than hours of looking at their handwriting.

So when I ‘feel’ melancholy I associate it with a certain type of mood, or a combined feeling of warmth, wonder, content and amazement at things. I know that the word is meant to evoke deep feelings of gloom and pervasive sense of impending event, but I associate those more with depression and a lack of ‘agency’, or the ability to impact my own existence. I think some Heidegger, Sartre, or perhaps just plain Thoreau would be good at this part!

Building a LoRaWAN home network

I eventually went for a small SX1301 based LoRa gateway from RAK Wireless, the 831 and now have that running in my house network. Radio frequency work is different to SBC/MCU work and is a lot more ‘physical’ in the sense that you need to think about the radio frequency and antenna patterns. That’s a whole new / old world for me – and one which I thought I’d left from my childhood days.

I grew up in a country town where one of my leisure activities was being part of a ‘radio club’ for amateurs, also known as ham radio enthusiasts. These were the days of CW McCall and the song Convoy so I heard a lot of 10-codes used when travelling in cars. Our radio club toured commercial radio sites as well as taking part in radio jamborees, and constructed some monstrous antennas to pick up radio signals from all over the world – so I got familiar with terms like baluns, impedance matching, dipoles and yagis. Radio is amazing, does weird things and in my estimation just a little easier to understand than quantum mechanics. I never did get my Morse radio license though it may have been useful, and regret that now.

Setting up the RAK831 with a partner Raspberry Pi, initially using the jumper wires to connect them was easy and I soon got a signal through on LoRa. It seemed pretty simply just to follow the recipes on the internet as the device is based around a IC880a chip and there are plenty of code examples up on GitHub. My node devices are LoPy from Pycom and as they used MicroPython are pretty familiar from my work around e-ink displays. I registered the gateway with The Things Network and have it receiving stuff now, even if it is sitting on my filing cabinet and needs to be placed into a IP56 gray box and fixed to the wall.

I drilled through my house outer wall and poked the antenna out there, one of those ‘rubber duck’ type mono-pole ones, I think it is quarter wave but works quite well. Then registering on the ttn-mapper website allowed me to start mapping the reception from a smaller, hand-held LoPy transmitting a sequence of “1234” in a 30 second delayed loop so as not to abuse the duty cycle of the 868 ISM band. The results were both disappointing and amazing – amazing that such a short antenna transmitting/receiving can get anything and pass it back to the internet, and disappointing that it can’t be heard in behind some local buildings.

Screenshot from 2017-10-07 09-14-36

I’ve a couple of thoughts on this: a quality antenna such as a half- or full-wave may be better, orientation of the receiver (it is horizontal now) may be important, elevation could be critical, and not placing it on one side of my house but instead the roof so that it isn’t blocked by a close building may all improve things. But for my first simple use case that I want to merely ‘hear’ my small environmental sensors I think what I have now is probably sufficient!

Playing with LoRaWan

I’ve been interested in replacing my environmental sensor network that ran on an older communications protocol with a LoRaWan gateway and a set of button-battery sensors. Just today I started that journey using a couple of the excellent Pycom LoPy boards. This will give me the capability to measure things, and connect to a gateway using a single-channel LoRa environment.

Although I will register this with The Things Network, as it is initially not a true multi-channel gateway it probably is wise to limit it as a private gateway and not as an open public one, since the actual throughput and channels will be limited.

I’ve looked on AliExpress and there are other gateways available, although not cheap enough to consider as a impulse purchase. Some of these are based on the SX1276 and some on the SX1301. One seller even had a convoluted explanation why they still used the SX1276 rather than the multi-channel SX1301 but written in a very stilted Chinese-English hybrid and as it quickly descended into a dialogue with a block of cheese so I abandoned trying to think it through. Apparently the SX1301 needs to be front-ended by a couple of other radio-handling chips so likely is more difficult for OEM board manufacturers to build, so perhaps it may be best to avoid them until the prices drop on the multi-channel LoRaWAN gateways.

Instead I have gone with one from RAK Wireless married to a Raspberry Pi with a daughter board, and just working through the strange Reverse Polarity SMA connectors which they need to use for FCC certification so that I can glue my antenna on the outside wall.

Packets are streaming through from my PySense node so things look good for a deployment using this model.

Housing an aquarium computer

As mentioned before, I purchased a Lattepanda as a single-board computer for running Windows 10. This works great, although the Intel Atom chip runs a little hot and needs extra cooling.

The main purpose of this is to run my Seneye aquarium monitor, so it needs to fit under my aquarium along with the other items that I’ve built to automate my fish hobby. This means an IP56 custom housing which presents me with a challenge: how to get heat out of a sealed box?

IP56 LattePanda case

In the end I have decided that a completely sealed box is unreasonable, although you could imagine a small water-cooled one with the radiator external to the sealed box. Air holes drilled into each end and push/pull fans drawing the air past the computer board seemed to be an excellent idea as the Intel chip runs very hot. There were some challenges:

Getting a USB cable through a grommet

Did you spot my problem? It is in the bottom right of the photo. I was able to get a small micro-USB through a 10mm grommet by removing the sealed cable end, however I now have a bigger problem in that I need to get a USB-A type plug through to service the Seneye SUD device. For that I am using an external USB female panel mount and connect that to the USB input on the computer.

Switching it on and off

Another issue is that the LattePanda has a curious boot-up sequence. First you ‘switch it on’ by supplying power to the USB lead which goes through a sequence of blinking red and blue LEDs, then into a quiescent state of blue LED only waiting for you to trigger the Windows boot. This is done by pressing the ‘on’ micro button on the side to start the boot. But the real issue is that in a sealed box this will not be possible – I have to find a way to click that micro-button externally.

Any ideas?

Getting SMTP working on a Ricoh scanner

I have a great gel-ink Ricoh scanner, the Aficio SG 3100SNw that has a sort of industrial print quality to it. By that I mean that a whole lot of features you’d normally only get on an office printer, perhaps reflecting Ricoh’s business rather than consumer focus.

One feature on this multi-function device is a flat-bed scanner. This works great as a copier (and has a document feed as well!) however getting it to work as a scanner which would mail the scanned copies to me was difficult. I set up one of my small servers on a Beaglebone Black as an Exim4 server, and got the correct credentials from the Google SMTP servers and generated the one-time passwords required.

Having got email working using the SBC as a email relay (and woe if you put an open email relay on the internet – spam heaven!) then the next step was to get it working from the printer, to the SBC mail relay, to Google and thence to my normal email accounts. This required a few steps:

  1. Create a new account on Google Gmail to act as the ‘sender’
  2. Generate a one-time password using the Gmail features, as this email agent could not do 2-factor password authentication
  3. Store that in /etc/exim4/passwd.client
  4. Install Exim4 on the mail relay, and configure using dpkg
  5. Avoid using SSL authentication around my home network, since managing the certificates was a layer of complexity I wished to avoid

For some 1-2 years this worked wonders and I happily scanned and transferred using email, then something broke. I suspect that it was an update in the Exim software on my receiving system as I had not updated the Ricoh firmware, but endless wandering the internet via search engines took me nowhere and it lingered for about 6 months in a broken state.

Until today, when I started stepping through moving off the Beaglebone Black onto a more capable Odroid device, and started copying the Exim configurations. At some point I took the password file across and then started the command to configure Exim:

sudo dpkg-reconfigure exim4-config

I was a bit perplexed by some of the options, but as they had worked on the other box I went there and start the reconfigure utility as well, to manually mirror the same options. Some exploration and reading of Exim’s web pages helped, but a final error on connection from the printer front panel flummoxed me. This is the very helpful admin option on the Ricoh where you need to go to the following:

  • Login, using your admin PIN
  • Go to Settings, System Settings, and File Transfer
  • SMTP server, and press the ‘Com.Test’ function below which will test your SMTP server and port settings here.

I think the key this time was that I also read the Exim log file at /var/log/exim4/mainlog and saw that I got a weird message:

rejected EHLO from syntactically invalid argument(s): DIGITAL_MFP

Searching on this (and the message contains the server IP address and hostname, gets you eventually to a neat little blog post that has this same error and an easy fix. Further explanation is here but the basics are that an underscore is an invalid character in normal Exim HELO/EHLO messages. Once included as part of the template and dpkg-reconfigure re-run, all is well with the world!

Lattepanda

I purchased a LattePanda using Bitcoin.

lattepanda-logo

The Bitcoin part is unremarkable – I don’t speculate on it nor use it as a store of value, just for transactions. Buying via a maker web site was simple (their shopping cart software is Shopify and they use BitPay as a payment cartridge). But what really intrigued me was the Lattepanda itself.

It’s pretty cool.

You may know that most single-board computers (SBCs) run on ARM chips. ARM cornered the market for low-powered devices and most mobile phones use them. So too most SBCs including the well known Raspberry Pi. My collection of Beaglebones, RasPis, Odroids and others is now joined by a SBC powered by an Intel. Harking back to an earlier post, this is mostly brought on by a couple of things: a need to run some trenchant Windows software. I’d also like to play movies through it, if possible.

That second need is brought on by never really getting my Plex project working: I could get the recording part through my HD Homerun TV recorder working, and storage wasn’t a problem – it was just that the particular box I chose to put it on wasn’t strong enough to do the transcoding needed to play back the movies on different devices. Perhaps also the Lattepanda won’t be sufficient as I notice that the Atom chip runs REALLY hot and I’m just fitting it with a fan to take some heat away. Let’s see if that helps, but I may need to invest in an Odroid XU4 or something to really get the power boost my home media systems require.

For now I am gearing up the Windows platform on the Lattepanda and installing all the necessary updates, then putting it in a case and under my aquarium where it will run the Seneye Connect software and possibly a webcam on my fishes.

Reading a Seneye using a Raspberry Pi – conclusion

So it can be done, I’ve now able to read a Seneye USB device using my own server and Python code. You can read about the first couple of steps here and here. The process was difficult due to a number of quirks and barriers:

  • the Seneye code uses a C++ STRUCT for data mapping, implying byte-alignment for different data types and bit padding
  • the SUD holds local data readings until it is able to reconnect to the cloud, and will fill up to capacity if it is not connected to the cloud
  • values are decimal-shifted for display
  • if any errors occur the device seems to enter a timeout-locked state (perhaps by missing the BYESUD message?) and has to be unplugged

Firstly the official Seneye C++ code as compiled on my machine and reading the reference mug of water:

IMG_20170703_142232

Then the output from the Seneye code, without their lovely ASCII logo art:

  Device: LSDF0982LSDFOSDLJKS9E89S0D9SDMF v.2.0.16  Type:  Home
  Temperature (C)   │ 20.375                      Is Kelvin           │                   ┌────┘ └────┐
  pH                │ 7.94                        Kelvin              │                   │ ┌─┐   ┌─┐ │
  NH3 (ppm)         │ 0.02                        PAR                 │                   │ └─┘   └─┘ │
  In Water          │ True                        LUX                 │                   │           │
  Slide NOT fitted  │ False                       PUR                 │                   │   ┌───┐   │
  Slide Expired     │ False                                           │                   │   │   │   │
  Press R for reading, 1-5 to change LED, Q to quit

Taking this and observing that the pH value was 7.94, which is 031a in hex, I  scanned through the printed hex dump (and I really must write a small routine to dump binary, hex, and offset in bytes-per-line).  The latest output from my code with debugging turned on can be seen below.

('device       >>>', )
('configuration>>>', )
('interface    >>>', )
('endpoint in  >>>', )
('endpoint out >>>', )
('HELO ret code>>>', 8)
('HELO hex     >>>', (<type 'array.array'>, 64, '88:01:01:01:30:4e:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00'))
('READ ret code>>>', 7)
('sensor hex   >>>', (<type 'array.array'>, 64, '00:01:57:59:59:59:05:00:00:00:1a:03:10:00:1a:4f:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00'))
('sensor bits len>', 512)
('sensor bits  >>>', '00000000000000010101011101011001010110010101100100000101000000000000000000000000000110100000001100010000000000000001101001001111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000')
('BYE ret code >>>', 6)
{'InWater': True,
 'NH3': 0,
 'SlideExpired': False,
 'SlideNotFitted': False,
 'Temp': 20,
 'pH': 7}

This means that the pH starts at position 80, runs for two bytes, and is little endian (so x031a is the equivalent of 794 in decimal, and packed into two bytes as it is a short means it looks like ‘1a.03’. The 7.94 comes about because certain of the values returned are divided by 1000, certain ones by 100. It helps to read and understand the Seneye C++ code.

There is obviously some extra commands that are sent by the Seneye Connect software, plus I believe some cryptographic hashes in operation to ensure devices update, only upload from authorised accounts, and other things to keep the Seneye ecosystem together.

Conclusion

This means that it is unlikely that home DIYers will be able to replicate the full Seneye Connect experience. A LattePanda running Windows 10 and the Seneye Connect software or Seneye SWS probably still give the best experience, along with SMS text alerts and the Seneye dashboard. However, for those who are willing to tinker with code this project provides a reasonable solution.

Reading a Seneye using a Raspberry Pi – coding

In my previous post I discussed how to discover information about the Seneye device, here I will describe some simple code to read values from it and push these to a MQTT broker. I have this running on a small Raspberry Pi Zero W, on which I also have the Motion software and a small streaming web cam.

Having got lots of good information from the Linux commands you start by finding if the device is attached:

    dev = usb.core.find(idVendor=9463, idProduct=8708)

Next, ensure that the operating system does not have control of the device:

    interface = 0
    if dev.is_kernel_driver_active(interface) is True:
        kernel_driver_active = True
        dev.detach_kernel_driver(interface)

Then set the first configuration and claim the interface – and it needs to be done in that order, apparently!

dev.set_configuration()
usb.util.claim_interface(dev, interface)
cfg = dev.get_active_configuration()
intf = cfg[(0,0)]

Alternate settings may be ignored as most devices do not have them, so we move straight to the endpoint and search for the first in/out.

epIn = usb.util.find_descriptor(interface, custom_match= lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_IN)
epOut = usb.util.find_descriptor(interface, custom_match = lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_OUT)

Then send the READING message to the device, read the response, and read again with longer timeout so that the measurements can be read and returned.

    msg="READING"
    rc=dev.write(epOut,msg)
    ret=dev.read(epIn,epIn.wMaxPacketSize,1000)
    ret=dev.read(epIn,epIn.wMaxPacketSize,10000)

Once done the fun of picking the bit flags and integers out of the results starts and I have provided just six of them, as they were the ones I could confirm from the Seneye C++ program. At the end you need to send a closing message called “BYESUD”, I guess to tell the device to go into sleep mode or whatever.

    msg="BYESUD"
    rc=dev.write(epOut,msg)

Finished code

All this code is available in my GitHub repository and I’d encourage you to read the code, try it on one of your systems, and if you want to improve it fork the repo and submit pull requests to me. I’m not looking to make it complex with control functions and displays, just something simple and lightweight.