TP-Link HS110 Energy Monitoring API part 1

Hardware V1 and V2, and basic C# proof of concept for obtaining the Current, Voltage, and Power readings.

Useful links:

These sockets have actually proven to be fairly reliable, probably some of the most reliable WiFi sockets, but still, I’m reluctant to use them to power cycle things like computers because if they spuriously switch or lose WiFi settings or suffer interference on 2.4Ghz crowded channels or get hacked, then they could cause physical damage to my devices or possibly even cause a fire if they could switch at malicious frequencies that aggravate inrush currents synchronously for example.

I’d much rather be able to buy non-switching sockets for £10 each with well documented zigbee or similar with hardware-implemented AES encryption (at lower than 2.4Ghz frequencies too) energy monitoring channels but that’s fantasy because it’s too much like common sense so nobody seems to do that. Instead I’ve bought more of these expensive sockets and then roughly modified them, hobbling their prime function, by soldering across the switching relays – as you can see in the featured image.

Getting into them is fairly easy – you just remove the retaining screws from the bottom, and then prise the housing off using a screw driver at the top – you can see where there’s a gap in the inner lip of the housing.

I won’t show my pfSense settings for isolating these switches on one of my vLANs in case I’ve made a mistake lol. Don’t want to give away all my configuration! But obviously it makes sense to block them from potential RPC attacks on other devices should they themselves be compromised as well as controlling what can reach them in the first place.

As I’m rushed though and haven’t put these values into some kind of secret injection they’ll show up in my code for now, so keeping here for personal reference:

  • 192.168.13.31 KasaPlug01 Computer Servers (main)
  • 192.168.13.32 KasaPlug02 Computer Room Auxilliary
  • 192.168.13.40 KasaPlug04 Router Cupboard
  • 192.168.13.41 KasaPlug05 Cellar Dehumidifier

If I wanted to simply measure the total consumption used in the house (and I will do that too one day!) I’d use a CT clip on our single phase supply. Consumer devices that do this include:

https://www.amazon.co.uk/dp/B00JIMQP6Y/?coliid=I2SX20ENMMSHC4&colid=1YYM7JAHK3FJ5&psc=1&ref_=lv_ov_lig_dp_it

What I want to do though is take readings from different areas in the house. In fact I’ll probably make PRTG sensors to do this to fit into my existing methodology and that can graph the energy use for me. But the main point of doing things this way is that my new central power management Pi will be able to use these readings to confirm in some cases system operation and response to switching. It will provide more data to determine the nature of any issues.

Additionally, more specific sensor feedback should help both encourage, and, inform where I need to reduce my power consumption.


API

Investigating initially I referred to https://www.softscheck.com/en/reverse-engineering-tp-link-hs110/ and used the python software tool they made.

I’ve forked the script on GitHub so that I can use my PHP Xgithub shortcode to show it below using the Syntaxhighlighter plugin. (I don’t have any settings for python so I’m pretending it’s JavaScript so that at least some reasonable highlighting takes place).

...


2 of the devices are hardware v1, and the other two v2.

ALL I’m interested in is the energy responses. I’m locking the devices down a bit in my IoT vLAN so that hopefully it’s a bit trickier for a malicious actor to reset them and I’ll be cutting them off from direct contact with the cloud. I could just modify the python script to run on my power-management Pi, but I want a C# dotnet core 3 library because I understand C# much better and might use it in a few places including maybe my PRTG local probe.

So next, I downloaded and tried using this .Net framework based library using the supplied test WPF app:
https://www.codeproject.com/Tips/1169091/How-to-Control-TP-Link-Smart-Plug-HS-XX

BUT: I discovered it didn’t work with my V2 hardware.

The Python based tool seemed to work just fine with hardware V2. To help me understand the change for V2 in the Python (I’m not familiar with Python) I added a line:

print ("Sent:     ", encrypt(cmd))

Nb: I did this in Visual Studio Code which I normally have set to use spaces rather than tabs so first of all I converted the script to use spaces:

Ctrl+Shift+P
Convert indentation to Tabs

Oh – that’s hard to see. I’ll paste it here:

('Sent:     ', '{"emeter":{"get_realtime":{}}}')
('Sent:     ', '\x00\x00\x00\x1e\xd0\xf2\x97\xfa\x9f\xeb\x8e\xfc\xde\xe4\x9f\xbd\xda\xbf\xcb\x94\xe6\x83\xe2\x8e\xfa\x93\xfe\x9b\xb9\x83\xf8\x85\xf8\x85')
('Received: ', '{"emeter":{"get_realtime":{"current":1.366791,"voltage":243.261906,"power":275.039137,"total":184.354000,"err_code":0}}}')

Looking at the C# it becomes obvious where a change is … the 4th byte. Originally just 0x00 was fine, but now something else is needed for V2 and still works for V1. Please see https://www.codeproject.com/Tips/1169091/How-to-Control-TP-Link-Smart-Plug-HS-XX for licence details!!!

private byte[] EncryptMessage(byte[] pMessage, ProtocolType pProtocolType)
    {
      List<byte> mBuffer = new List<byte>();
      int key = 0xAB; // 171

      if ((pMessage != null) && (pMessage.Length > 0))
      {

        if (pProtocolType == ProtocolType.TCP)
        {
          mBuffer.Add(0x00);
          mBuffer.Add(0x00);
          mBuffer.Add(0x00);
          mBuffer.Add(0x1E); // I'VE CHANGED THIS TO 0x1E NOW.
        }

        for (int i = 0; i < pMessage.Length; i++)
        {
          byte b = (byte)(key ^ pMessage[i]);
          key = b;
          mBuffer.Add(b);
        }
      }

      return mBuffer.ToArray();
    }

To be sure I just wanted to see what the C# was sending in the same format as the Python print() … and that seemed to confirm the one change.

lblSentEncrypted.Text = "x" + BitConverter.ToString(mEncryptedMessage).Replace("-", "\\x").ToLower();

So I hard coded that “1e” to test, but looking closely at the Python and Googling what “pack” was all about, I realised the 4th byte was dynamically generated from the pack function acting on the length of the string being encrypted.

def encrypt(string):
    key = 171
    result = pack('>I', len(string))
    print ("temp result:", result)
    for i in string:
        a = key ^ ord(i)
        key = a
        result += chr(a)
    return result

python tplink_smartplug.py -t 192.168.13.40 -c info                                                       ✔  166  17:32:26
('temp result:', '\x00\x00\x00\x1d')
('Sent:     ', '{"system":{"get_sysinfo":{}}}')
('temp result:', '\x00\x00\x00\x1d')
('Sent:     ', '\x00\x00\x00\x1d\xd0\xf2\x81\xf8\x8b\xff\x9a\xf7\xd5\xef\x94\xb6\xd1\xb4\xc0\x9f\xec\x95\xe6\x8f\xe1\x87\xe8\xca\xf0\x8b\xf6\x8b\xf6')
('Received: ', '{"system":{"get_sysinfo":{"sw_ver":"1.5.7 Build 180806 Rel.135437","hw_ver":"2.1","type":"IOT.SMARTPLUGSWITCH","model":"HS110(UK)","mac":"68:FF:7B:1B:4D:62","dev_name":"Smart Wi-Fi Plug With Energy Monitoring","alias":"Cupboard energy meter","relay_state":1,"on_time":742,"active_mode":"none","feature":"TIM:ENE","updating":0,"icon_hash":"","rssi":-32,"led_off":0,"longitude_i":-21650,"latitude_i":534223,"hwId":"0750E2C15BB77902833ABF45366B8E9A","fwId":"00000000000000000000000000000000","deviceId":"800683CCB6399A531400B3559375AF331B3A749E","oemId":"AB8C79FE7869756511CDC455BDFE41EA","next_action":{"type":-1},"ntc_state":0,"err_code":0}}}')


For the “info” command I’m getting 0x1d for that 4th byte rather than 0x1e for the energy command.

Well – even I can see that the length of the info command is 29 decimal (0x1d) and that the length of the energy command is 30 decimal (0x1e).

I tried a longer command (e.g. the relay on/off commands are longer JSON):

python tplink_smartplug.py -t 192.168.13.32 -c on                                                         ✔  170  17:58:49
('temp result:', '\x00\x00\x00*')
('Sent:     ', '{"system":{"set_relay_state":{"state":1}}}')
('temp result:', '\x00\x00\x00*')
('Sent:     ', '\x00\x00\x00*\xd0\xf2\x81\xf8\x8b\xff\x9a\xf7\xd5\xef\x94\xb6\xc5\xa0\xd4\x8b\xf9\x9c\xf0\x91\xe8\xb7\xc4\xb0\xd1\xa5\xc0\xe2\xd8\xa3\x81\xf2\x86\xe7\x93\xf6\xd4\xee\xdf\xa2\xdf\xa2')
('Received: ', '{"system":{"set_relay_state":{"err_code":0}}}')

An asterix??? OK. Now I’m getting lost. What does that mean in bytes? What’s been actually sent over the wire? Whatever it was, the relay responded appropriately.

Not to be deterred, I just went with a bit of common sense and updated my C# encryption method simply:

    private static byte[] EncryptMessage(byte[] pMessage, ProtocolType pProtocolType)
    {
      var mBuffer = new List<byte>();

      if ((pMessage == null) || (pMessage.Length <= 0)) return mBuffer.ToArray();

      if (pProtocolType == ProtocolType.TCP)
      {
        mBuffer.Add(0x00);
        mBuffer.Add(0x00);
        mBuffer.Add(0x00);

        // For V2
        var hexLength = Convert.ToByte(pMessage.Length);
        mBuffer.Add(hexLength);
      }

      var key = 0xAB; // 171
      foreach (var t in pMessage)
      {
        var b = (byte)(key ^ t);
        key = b;
        mBuffer.Add(b);
      }

      return mBuffer.ToArray();
    }

WOOOOHOOOOOO. SEEMS TO WORK! [LITTLE DANCE OF JOY].

To help me with all this testing I’d just created a simple single form WinForms full framework application, pulling relevant bits from https://www.codeproject.com/Tips/1169091/How-to-Control-TP-Link-Smart-Plug-HS-XX and adapting them for V2 plus a little refactoring just to be as simple as possible to allow me to see what was going on. I changed the network stream reading following this example: https://docs.microsoft.com/en-us/dotnet/api/system.net.sockets.networkstream.read?view=netframework-4.8

I’m only really interested in the energy command results. The V2 hardware seem to use mv, ma and mw. I’ll assume the socket is applying some kind of power factor correction for it’s power calculations. Examples:

{"emeter":{"get_realtime":{"current":1.546371,"voltage":243.057589,"power":319.775607,"total":186.850000,"err_code":0}}}
{"emeter":{"get_realtime":{"voltage_mv":242119,"current_ma":449,"power_mw":78392,"total_wh":98,"err_code":0}}}

Obviously two sets of JSON properties complicates things a little. I’ve been wanting to switch to System.Text.Json for a while and might as well start now. Because it’ll make life simpler when using dotnet core 3 on the desktop and Pi etc. I thought about having a custom converter to map the properties. But after a bit of thought I decided it made more sense for the units just to use custom getters/setters for common “transforms”. When using JSON.Net I often turn to https://quicktype.io/csharp/ . I think I’ll still start there and then just adapt the syntax.

So this is my (very) rough ‘n’ ready WinForm just for exploring these tp-link sockets. I’m just populating a combo box items collection with the JSON commands in the designer (no enclosing quotes, escaping or spaces!).

...

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.NetworkInformation;
using System.Net.Sockets;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Windows.Forms;


namespace hs110winforms
{
  public enum ProtocolType
  {
    TCP = 0,
    UDP = 1,
  }


  public partial class EnergyMonitoring : Form
  {
    private IPAddress _mIp = null;
    public EnergyMonitoring()
    {
      InitializeComponent();
    }


    private void SendCommandAndProcessResponseForIPinThisTextBox(object sender)
    {

      txtBxDebugFeedback.Text = "";
      lblErrorMessage.Text = "";

      // no value checking at the moment
      var txtBx = sender as TextBox;
      var text = txtBx?.Text ?? "";

      var ip = IPAddress.Parse(text);
      this._mIp = ip;

      if (PingDevice(ip))
      {
        try
        {
          var responseString = this.ExecuteAndRead(this.cmbBoxCommands.Text);
          if (this.cmbBoxCommands.Text == "{\"emeter\":{\"get_realtime\":{}}}")
          {
            var jsonModel = JsonSerializer.Deserialize<EnergyReading>(responseString);
            lblmA.Text = $"{jsonModel.Emeter.GetRealtime.CurrentMa.ToString()}mA";
            lblmV.Text = $"{jsonModel.Emeter.GetRealtime.VoltageMv.ToString()}mV";
            lblmW.Text = $"{jsonModel.Emeter.GetRealtime.PowerMw.ToString()}mW";
          }
          else
          {
            lblmA.Text = @"mA";
            lblmV.Text = @"mV";
            lblmW.Text = @"mW";
          }

        }
        catch (Exception ex)
        {
          ErrorFeedback(ex.Message);
        }
      }
      else
      {
        DebugFeedback("DEVICE DISCONNECTED");
      }
    }





    private void DebugFeedback(string feedback = "") => txtBxDebugFeedback.Text += $"{feedback}\r\n";
    private void ErrorFeedback(string errorMsg = "") => lblErrorMessage.Text += $"{errorMsg}\r\n";


    // doing this a "long way round" just to make it easier for me to follow what's going on
    private string ExecuteAndRead(string pCommand)
    {
      DebugFeedback( $"Sending: {pCommand}" );
      byte[] mEncryptedMessage = EncryptMessage(Encoding.ASCII.GetBytes(pCommand), ProtocolType.TCP);
      DebugFeedback($"Sending: x{BitConverter.ToString(mEncryptedMessage).Replace("-", "\\x").ToLower()}");


      var deviceResponse = "";

      var mClient = new TcpClient {SendTimeout = 1000, ReceiveTimeout = 2000};
      DebugFeedback( "BeginConnect" );
      IAsyncResult result = mClient.BeginConnect(new IPAddress(this._mIp.GetAddressBytes()), 9999, null, null);
      bool success = result.AsyncWaitHandle.WaitOne(10000);


      if (success)
      {
        DebugFeedback( "Connection SUCCESS" );
        try
        {
          DebugFeedback( "GetStream()" );
          using (NetworkStream stream = mClient.GetStream())
          {
            DebugFeedback( "Write Message" );
            stream.Write(mEncryptedMessage, 0, mEncryptedMessage.Length);

            DebugFeedback( "Read response to buffer - check if stream.CanRead" );
            

            if (stream.CanRead)
            {
              var myReadBuffer = new byte[0];
              var myOutputArray = new byte[0];
              
              var numberOfBytesRead = 0;

              // Incoming message may be larger than the buffer size.
              do
              {
                myReadBuffer = new byte[256]; // back to 0x00's
                DebugFeedback("Can Read apparently ... attempting stream.Read");
                numberOfBytesRead = stream.Read(myReadBuffer, 0, myReadBuffer.Length);
                DebugFeedback($"numberOfBytesRead = {numberOfBytesRead}");

                Array.Resize(ref myReadBuffer, numberOfBytesRead); // get rid of trailing 0x00's

                DebugFeedback($"Before Decrypt: \nx{BitConverter.ToString(myReadBuffer).Replace("-", "\\x").ToLower()}");

                myOutputArray = myOutputArray.Length < 1 ? myReadBuffer.ToArray() : myOutputArray.Concat(myReadBuffer).ToArray();
              }
              while (stream.DataAvailable);

              DebugFeedback("End While");
              DebugFeedback($"Length of myOutputArray: {myOutputArray.Length}");
              DebugFeedback($"myOutputArray before Decrypt: ");
              DebugFeedback($"x{BitConverter.ToString(myOutputArray).Replace("-", "\\x").ToLower()}");
              deviceResponse = Encoding.ASCII.GetString((DecryptMessage(myOutputArray.ToArray(), ProtocolType.TCP)));
              DebugFeedback($"After Decrypt: \n{deviceResponse}");

            }
            else
            {
              DebugFeedback( "Sorry.  You cannot read from this NetworkStream.");
            }
            

            stream.Close();
          }
        }
        catch (Exception ex)
        {
          ErrorFeedback(ex.Message);
          DebugFeedback("ERROR - BUT EXAMINING deviceResponse ANYWAY.");
          DebugFeedback(deviceResponse);
        }
        finally
        {
          if (mClient.Connected)
            mClient.EndConnect(result);
          
        }
      }
      else
      {
        DebugFeedback( @"Connection ERROR");
      }

      return deviceResponse;
    }


    private static byte[] EncryptMessage(byte[] pMessage, ProtocolType pProtocolType)
    {
      var mBuffer = new List<byte>();

      if ((pMessage == null) || (pMessage.Length <= 0)) return mBuffer.ToArray();

      if (pProtocolType == ProtocolType.TCP)
      {
        mBuffer.Add(0x00);
        mBuffer.Add(0x00);
        mBuffer.Add(0x00);

        // For V2
        var hexLength = Convert.ToByte(pMessage.Length);
        mBuffer.Add(hexLength);
      }

      var key = 0xAB; // 171
      foreach (var t in pMessage)
      {
        var b = (byte)(key ^ t);
        key = b;
        mBuffer.Add(b);
      }

      return mBuffer.ToArray();
    }


    private static byte[] DecryptMessage(byte[] pMessage, ProtocolType pProtocolType)
    {
      var mBuffer = new List<byte>();

      //Skip the first 4 bytes in TCP communications (4 bytes header)
      var header = (pProtocolType == ProtocolType.UDP) ? (byte)0x00 : (byte)0x04;

      if ((pMessage == null) || (pMessage.Length <= 0)) return mBuffer.ToArray();

      var key = 0xAB; // 171
      for (int i = header; i < pMessage.Length; i++)
      {
        var b = (byte)(key ^ pMessage[i]);
        key = pMessage[i];
        mBuffer.Add(b);
      }

      return mBuffer.ToArray();
    }


    public static bool PingDevice(IPAddress ip)
    {
      var reply = (new Ping()).Send(ip); 
      return reply?.Status == IPStatus.Success;
    }


    private void txtBxIP1_KeyUp(object sender, KeyEventArgs e)
    {
      if (e.KeyValue == (char)Keys.Return)
        SendCommandAndProcessResponseForIPinThisTextBox(sender);
    }


    private void cmbBoxCommands_SelectedValueChanged(object sender, EventArgs e) =>
      SendCommandAndProcessResponseForIPinThisTextBox(this.txtBxIP1);


    public class EnergyReading
    {
      [JsonPropertyName("emeter")]
      public Emeter Emeter { get; set; }
    }

    public class Emeter
    {
      [JsonPropertyName("get_realtime")]
      public GetRealtime GetRealtime { get; set; }
    }


    private static long DoubleToLongMilli(double input) =>
      (long) Math.Floor(input * 1000);
    

    private static double RoundTo6dp(double input) =>
      Math.Round(input, 6, MidpointRounding.AwayFromZero);
    

    public class GetRealtime
    {
      [JsonPropertyName("voltage_mv")]
      public long VoltageMv { get; set; }

      [JsonPropertyName("voltage")]
      public double Voltage
      {
        get => RoundTo6dp( (double) VoltageMv / 1000);
        set => VoltageMv = DoubleToLongMilli(value);
      }

      [JsonPropertyName("current_ma")]
      public long CurrentMa { get; set; }

      [JsonPropertyName("current")]
      public double Current
      {
        get => RoundTo6dp((double)CurrentMa / 1000);
        set => CurrentMa = DoubleToLongMilli(value);
      }

      [JsonPropertyName("power_mw")]
      public long PowerMw { get; set; }

      [JsonPropertyName("power")]
      public double Power
      {
        get => RoundTo6dp((double)PowerMw / 1000);
        set => PowerMw = DoubleToLongMilli(value);
      }

      [JsonPropertyName("total_wh")]
      public long TotalWh { get; set; }

      [JsonPropertyName("total")]
      public double Total
      {
        get => TotalWh;
        set => TotalWh = (long) value;
      }

      [JsonPropertyName("err_code")]
      public long ErrCode { get; set; }
    }



  }
}

Just exploration/POC code. Currently this Syntax highlighting doesn't work properly for interpolated strings.

Now I’m ready to move on with a dotnet core 3 library to provide those energy reading objects. Sticking with dotnet core 3 for native System.Text.Json for simplicity rather than needing to add a package in dotnet standard – anticipating that all the things I intend to build that will use it will also be dotnet core 3.

In Part 2, I’ll demonstrate this library as – is (not Async yet and still a bit basic / not rounded – pending event hooks and more tailored exceptions etc.), used for PRTG Windows sensors.

In Part 3, I’ll use the library on the Pi to replace my current Python that updates an I2C display.

In Part 4, I’ll show the set-up of my local Nuget server in Docker, and a Web Api Docker container that uses my library as a Nuget package and using a local image registry running on my new Nuc, running on the Pi. Anticipating moving the code that updates the local (to the Pi) I2C display to another container, using the new Web Api as the interface for getting the energy readings.

I want to make use of some of the skills I learned at work and do this Web Api part using TDD.

In another article I’ll be demonstrating using CI/CD orchestrated on Gitlab on another Pi4 for the pipeline for developing the web api remotely in the container. This might not appear until February 2019 depending on priorities.