Category: Electrickery

Over the last few days, I’ve had a frank discussion in which they have explained every question I’ve posed to my satisfaction. They have even provided additional background and data to illustrate certain things.

We’ve come a long way in a short time

At one point I questioned their claim about Awesome online service, but there was a reasonable delay while they dusted off some guy in a back room who hadn’t seen daylight to take on my unusual need for precise and detailed technical answers.

Since then they’ve given him a muffin and a coke and kept him in a room with artificial light so that he was ready to come out in the day-time to deal with me very promptly.

The good news is that I now trust the Electric Kiwi process and its data.

There are a few small things I’m going to be a pain in the butt about, but the daily tracking of every minor change on their site for auditing and further processing can now cease.

If you have been following this story in sequence you will know that I believed that the first usage graph the Retailer puts up for a given day (usually two days before the current day) is made up rubbish.

It never matched real usage patterns, and the hour of power use was always too consistent.

Thanks to an email I received from the Retailer yesterday, it’s now confirmed that the first graph is fiction.

Looking at your consumption, we are generally receiving reads in the 48-72 hours delayed time range, so this will mean that most of the time your last days reading within your online account is an estimate (we are working towards noting on the account when an estimate has been used) — Retailer support person

(My emphasis).

For what it is worth, I do believe the last update for the given day, which tends to come through between 11:30 am and 12:30 pm is the final data and doesn’t get changed subsequently. I’ve not yet observed the data being updated when data for a newer day is available (or faked).

Fake it till you make it

Just as a note, I understand why the company puts up “estimated” numbers for the day. It’s not ideal in a world made up of OCD/Autistic people who crave reality and factuality at all times. But there are people who (believe it or not), are happier to be “lied to” by a computer program “guessing” their usage.

The result for the Retailer is fewer support calls about why the data isn’t “there yet”, and from what I’ve seen, the guesses are generally within 5%, so it will placate most people.

I’m going to take some time to digest this email and then I’ll report on the answers as they relate to the points I’ve raised to date.

The good news is that the email appears to be a fair attempt at being open about the process. I sense no attempts to try and hide the truth in double-speak or try to shift responsibility to other parties.

Refreshing.

Although the original aim, which is to get at my data inside my Smart Meter in a timely fashion remains a pipe dream, understanding what the Electricity Retailer receives and how they process it will repair the damage to my confidence in them.

As observed in previous posts, the Retailer throws up some kind of provisional graph at the start of the day. The first Dec 1 graph is now above.

My observations to date

The first graph does not match my usage. It has usage spikes that should not be there.

Another graph, sometimes three, will be presented over the day. This graph will have

“Unknown” usage spikes will disappear

Hour of Power half hour slots will have different usage

The original dollar charge for the day is relatively close but gets adjusted over the day

The Hour of Power is generally underestimated in use and has more electricity shifted into it

It seems to assume someone taking a quick shower at 5 am. Then it seems to assume the hour of power is maxed out on the same value for both half hours. And then it assumes we have some post-dinner extra power use. All of those do not match the way we use power on most days.

Hypothesis

So my theory is this: they won’t have the final graph computed until late morning or sometimes during the first part of the afternoon. They throw up a provisional graph which placates most people.

They assume most people won’t be looking at the detail, and who is going to complain about seeing their bill adjusted down by a few cents?

This gives them the time needed to appear to have the data for two days ago all sorted first thing in the morning.

Where do the delays come in?

The first apparent delay is that for day N to be seen, it needs to be day N+1. That way you know day N is complete.

Then, the transmission of data from day N doesn’t make it to the Meter Reading Company immediately after midnight. I have no idea at this stage what the minimum, maximum and median delays are before my Smart Meter manages to pop the data into the Meter Reader Company’s database.

Then there will be a delay as the Meter Reading Company transfers data to Retailers in batches, so if you missed a recent batch time, the data will be sitting there.

The Electricity Retailer finally gets the data. They too have processing batches. Just recently, there seems to be one that gets to my data around 5 am. We are now, on the calendar, at day N + 2.

And then there is some really fuzzy stuff that defies a theory for now

It appears my Electricity Retailer may be guessing the detail of daily usage, but they are close with the total. So it seems they are handed the totals earlier on day N + 2. This is where they run their 5 am batch with their fake half-hour graph.

During the day, and over a number of updates, the data does become more accurate. In the sense it aligns with my log.

UPDATE

I just checked out the cumulative use according to my meter and according to the Retailer over 4 days and it seems the Retailer losing a fraction of a kW every day.

This will mean that the customer doing an audit won’t come and complain about being charged an extra few cents on their bill, but I’d like to see what the difference is over a longer period. Theoretically, the difference should be +/- a few kW at most at all times because the Reading company and me reading the meter aren’t doing so at the same time.

Electricity loggers are not cheap. Certainly not cheap enough to buy one and see if you can get some data from your own power board before it hits the Smart Meter

The next option is to get an electrician to come and help you.

If you are like me, you are frustrated that data that you own is sitting meters away in a box on the wall, and you can’t get to it without breaking the law! (WHAT?)

Do you really have to invest thousands or pay “from $400” to get someone else to show you what’s going on?

Number 8 Wire solution: OCR the Smart Meter display

I’ve been toying with the idea of putting a webcam in my meter box and capturing the readings on the Smart Meter display 24/7. I have the technical know-how to automatically put that data in a spreadsheet.

It’s going to take some time to set up, test and automate, however.

As it turns out, our brothers and sisters across the pond have already done this and created a retail package that you or I can install.

An electricity data logger for a little over 100 bucks?

At AUD$99 plus shipping that is a very attractive way to get closer to what’s going on inside your Smart Meter.

But here is the sad news: there is no export function. Even though you get more detail and analysis, it is contained in yet another box.

Another box that contains the data you own but you can’t get at it.

MARTY! IT’S ALL ABOUT THE FLUX CAPACITOR!

The cheapest electricity data logger I’ve been able to find is called he Fluksometer. Made(?) and sold by a Belgian company, it clocks in at a wallet-searing €170. And then you still need the €18 current clamp. And add shipping and import duties as well as GST. Ouch.

In summary

You can build your own monitor from parts. Use a Raspberry Pi, an old PC with a webcam with some OCR, or even get the soldering iron out and create your own black box that reads the LED pulse off the Smartmeter panel.

The cheapest consumer-friendly solution appears to be from Watts Clever in Australia. They will send me one of their loggers, including shipping, for about 134 NZD. But the data is imprisoned inside the unit. There may be a way to hack the WiFi signal or the data on the unit, but that puts it beyond a consumer solution.

The Belgian Fluxometer, once shipped will set me back around 500 NZD. That’s an amount that would take a number of years to break even on. But it does allow you access to the data, and it is consumer-friendly. Fluxometer is the cheapest way to log data to a computer without having to know how to program or use a soldering iron.

Why doesn’t the Meter Reading Company help?

The amazing thing is that the Meter Reading Company gets a daily data dump from your Smartmeter. It would be so easy to be emailed a file every day with that data inside it.

The only reason we can’t do it is that the the Powers That Be have not considered it, and I suspect it will be hard-fought against. In spite of the data being ours, the company is allowed to not reveal it to you.

Stop Press

Just discovered a solution in between Whats Clever and the Fluxometer.

It comes with an Amp clamp that plugs into a WiFi transmitter. The data goes to the cloud, and you have free access to the Dashboard from any Internet-connected device in the world.

Better, it allows the download of detailed historical electricity consumption data. At 160 NZD plus shipping that would be a solution while we wait for the energy industry to sort out the simplest way: a daily email with your data. Or, web-accessible data, updated daily, allowing downloads.

The only problem with that one is it claims a +/- 10% accuracy. Which I suspect is the case with most amp-clamped readers. (sad face).

Time to be impulsive

The most accurate way to read the Smartmeter in a consumer-friendly way is to put an optical sensor on the LED. It flashes once for every 1 watt consumed. That brings us way back round to where this article started…

Sunday morning, at 4:59 am, the billing data for Friday the 30th was first updated.

Just before midday, the data changed. As shown in an earlier article, this can happen a number of times a day.

The changes are highlighted, with the more current data green, and the old data in red.

The graphs showing half-hourly usage also changed:

As you can see, some of the electricity use has shifted to different time slots.

How is this even possible? How can (theoretically the same) data produce two different results?

For most people who do not have a Free Hour of Power, it wouldn’t matter what the graph shows as long as the total kWh for the day stays the same.

But that changes too!

I know, I know. Who is worried about four cents a day?

It’s not the size of the discrepancies I’m worried about. It’s the integrity of the data.

As long as we can not get our own usage data, unaltered, from our Smart Meters, any agency that handles the data along the way is capable of changing it.

Not assuming malice, but I am assuming incompetence. Or perhaps a “close-enough-is-good-enough” attitude which should not be present.

The water companies care about every precious drop

When you have a leaky tap that drips once a second, it doesn’t seem worth the effort to go to the store, get a washer, come back, spend the time fixing it, just for one drop a second…

…except that water companies have been hammering us for decades to understand that one drop of water per second fills up a small back-yard pool in a year.

The little things do matter

…especially when they erode confidence in the integrity of the systems.

My retailer provided me different data about 6 hours later that has shifted electricity use in different time bands. It also has reduced the amount of free power it is allowing me for that day. And as a result, I am now paying more.

This article was meant to end here…

Hells Bells. The data changed AGAIN

As I was checking this article for errors, another update came in at 12:44 pm.

A THIRD go at the same data. This time, we have a really, really (really!) different looking graph.

Here are all three together, for easy comparison:

To beat this horse to death… the first was at about 5 am, the second just before midday, and the most recent at 12:44 pm. All of these are supposedly processing the same data for Friday the 30th, midnight-midnight.

As the numbers have changed again, here are all three for easy comparison, with the most recent at the bottom.

To remind you, the “Data team” of the Retailer is supposedly contacting me this coming week. I have no doubt they will have an explanation.

Every time my electricity data as reported on the Retailer’s site changes, I take a snapshot of it.

The screen you see above is where I am comparing what has changed between the latest and the latest-but-one version of things.

The RED numbers were captured at 11:44 AM Nov 30. The GREEN numbers were captured at 11:49 AM Nov 30. So the most recent numbers are on the left and are most current.

The first item to look at is the Supply charges 8 days line. On the left, it is charged at $1.9900 per day. That’s the correct rate for the account I am on. But minutes earlier, I appear to have been on a totally different tariff.

The update seems to have been to fix that the wrong tariff was erroneously used for my account.

Trust me, I know what I’m doing

I realise you may not have three+ decades of systems analysis experience (like I do), but this sends up all sorts of warning flags, sounds alarm bells, and the canary in the mine is well and truly expired.

If someone told me this is all done in a spreadsheet by Daryll at reception, I would believe it.

But as eagle-eyed readers will note, the electricity usage numbers also changed.

Question: how can my electricity usage data change in period of 5 minutes, when it theoretically arrived as a blob of data from my meter (via the Meter reading company)?

Further down my auditing report, we find the hard data.

again, the oldest data is on the right, the more up-to-date data is on the left.

The second column in each set is the total kWh for the day.

As you can see, my total kWh usage went up from 12.44 to 17.62.

The individual numbers in the specific half-hour timeslots are shown. (The data doesn’t show the time slot for each number, but it isn’t relevant to this article.) These half-hour timeslots reflect the reporting detail, and not necessarily the data as it was originally received.

In five minutes we have fixed the tariff I am on, increased my usage, and changed the electricity use report to show more electricity was used in four specific timeslots.

I believe it is entirely reasonable to ask the Retailer for an explanation.

Since the retailer contacted me earlier this afternoon, and I am waiting for their “Data team” to contact me next week, I’ll have an opportunity to do just that.

Please note that even though my electricity use reportedly went up, my bill may not have. This is because the larger number can fall in the Free Hour of power use this retailer offers.

We must return to the original objective of this series of articles: to be able to audit the electricity usage data between our Smart Meter and our electricity bill in a timely fashion.

And since then, to be able to audit the data to have confidence that every step of the data path has 100% integrity. Following from that, an increase in confidence in the various partners that make up our electricity supply.

It’s taken nearly six days to get this far, but I’m happy we’re talking.

My response

Hey [redacted]

I think my questions will be beyond the scope of the Online team.

I’m in no rush, and if I can get to ask some questions from the Data team, we might make some serious progress.

Have a good weekend.

Pete

Let’s see how far the Audit Team gets with the Data Team!

Cross fingers that I am able to get answers that explain everything I’ve been sharing with you. If we can understand the variability of this Retailer’s data, we are a small step closer to achieving end-to-end integrity.

It pays to remember that the Electricity Authority points consumers at their retailers in the first instance.

I only started this as a personal log of what I’m observing, doing and thinking because I was unable to get the retailer to engage.