It's like the arduino nano isn't sending all the data, but not sure why. I was at a baud of 115200, and changed to 128000 to see if that made any difference. It didn't decrease or increase the occurrences, that I could tell. I'm currently not counting the errors or figuring how often they occur

So what needs to be changed to make it run with less errors? I'm assuming the nano is sending "full" lines of text, as I don't see why it wouldn't. It just seems the serial read in the script isn't getting all the data for some reason.
2) In Grafana, I have some graphs for Volts/Current/Watts. What I would like to do is have a few queries that display the kWh over time. The time I would like to be static [1h, 12h, 24h, 1month, 6month, 1yr] for example, or possibly event he ability to change that on the fly using only 1 panel. And also include the costs of power over time.
My current results for the past day look like this:
I'd like to do something like this guy did: https://fablab.dyn.nerd2nerd.org/grafana...1m&orgId=1

There's nothing wrong with the emon libs or it's reporting. This is beyond that portion of it. If there was a problem with the emon lib, I "would" go ask them. But it's working as it should. This is a python/serial issue on my laptop.
I want to fix "my" code and design. That's why I'm asking here. Yes I did get some helpful info over there. Asking here for help will also yield me some good results as I know there are really good python/grafana users here. Plus I'm far more active on this forum than emons forums

Sean the error is not about Emon or influx. Its about getting the data in first place.

Korishan:
1. Dont have time to read it through now but il take a look later unless someone solved it
But it looks like you are not receiving the bginning of the data? Are you sending it to fast? Lower the speed of transmit and also the interval (I spent 10 seconds looking)

2. You need to group and calculate the value. as you can see he have set static intervals of 1day and 365 days and so forth. That is one way to do it and calculate it in grafana itself reading data from influx. This though take some cpu. (Check the raspberry solar sis thread from me with examples)
You can also have influx calulate this in the backround with set number of minutes between and just store the value that you then read ou. More examples in my thread about that.

This happens to me in many other languages where you work with serial ports, when the serial port data_received event triggers doesn't means that all the data is received, you must control by your own if all the data is received, with a parser, or a endline. In your case you don't event work with events, you only check constantly in a loop if there is data.

If I'm not wrong reading your code you do a while 200000 times checking if some data from arduino is received, trying to parse the line as 7 fields of data, these times you get errors is because not the whole line is received so it process some data in one cycle and the rest in the next 105383-105384 and 105390-105391.

I see you receive newline and return at the end of line, or at least you clean it

Code:

line = line.rstrip("\n\r")

You must cycle and store all data received in a buffer until you get \r\n and then split the data but not before you are sure the full line is received.

Are you using TTL to USB? If so you're probably picking up interference. How long is the cable and is it properly grounded? I am having the same issue with my BMS and for now I'm just ignoring some of the data. RS232 and RS485 serial are much more reliable.

Just print out the raw data to the console and you can see if the data is corrupt.

*edit* Also you get better results when you lower the baud rates. Remember the old days of modem dialups...

(09-28-2018, 08:21 PM)not2bme Wrote: Are you using TTL to USB? If so you're probably picking up interference. How long is the cable and is it properly grounded? I am having the same issue with my BMS and for now I'm just ignoring some of the data. RS232 and RS485 serial are much more reliable.

Just print out the raw data to the console and you can see if the data is corrupt.

not2bme: I'm connecting using USB cable directly between the laptop and the arduino nano. The cable is about 1 meter. So as jesusangel says, there isn't any interference. The data just gets chopped off some how.
jesusangel: yes, I'm reading each set of "lines()" that have been stored in the buffer. Each line ends with a \n\r. The data is stored in the "lines" variable as an array of strings. Then I read each "line" of "lines". Each "line" is stripped of the '\n\r' so that it'll get stored into the variables properly to be sent to the influxdb
There is supposed to be 7 entries in each line string (6 commas). In the 2 entries you quoted, there's only 5 and 3, respectively.

I hadn't thought of slowing down the baud rate. I'll give that a shot and see what happens.

(09-28-2018, 04:50 PM)daromer Wrote: Korishan:
1. Dont have time to read it through now but il take a look later unless someone solved it
But it looks like you are not receiving the bginning of the data? Are you sending it to fast? Lower the speed of transmit and also the interval (I spent 10 seconds looking)

2. You need to group and calculate the value. as you can see he have set static intervals of 1day and 365 days and so forth. That is one way to do it and calculate it in grafana itself reading data from influx. This though take some cpu. (Check the raspberry solar sis thread from me with examples)
You can also have influx calulate this in the backround with set number of minutes between and just store the value that you then read ou. More examples in my thread about that.

The totalwatts is stored in the DB, so that I don't have to worry about adding watt1 and watt2 fields in the displays. I knew I'd need to calculate the kWh over time, so that's why I did that so I didn't have to worry about it later
I looked closer at the guys json code for the panel, and it looks like he's not using a db some how. Which is partly why it's confusing. I'm not sure how he's grabbing the data. InfluxDB calls are a bit different from the SQL code I learned when dealing with MySQL and MS Access. So the tricks I'd do there don't work here.

21859/21850, 21916/21917. They are in pairs. So I took a closer look. If I take 59 & 60 'line' data, and read them together, I get:
['2781924', '118.84', '7.86', '933.66', '1'] ['2.18', '1446.90', '2380.56']

Here, we can see there are 8 entries. But are there really? If I put them together in a way I know the data is valid (but reading the data before/after the errors), I get this:
['2781924', '118.84', '7.86', '933.66', '12.18', '1446.90', '2380.56']

There's now 7 entries. The 5th entry is getting cut in half somehow. I noticed all the other errors were doing the same thing.

This is really odd. And I even slowed the baud down to 57600, and it seems there's actually more errors occurring now