After a few months working behind the scene, NodeManager got kind of a full review in terms of architecture and usability. The code should be now simpler to understand for new users, the memory utilization has been optimized and the overall user experience has been improved.

Since this change was blocking other pending changes, I hope now the development will be faster and feel free to submit your PRs now that the new architecture has been finalized.
As always, any feedback is more than welcome.
Thanks

@user2684 Thanks a lot for the architecture change, which makes a lot of sense conceptually.

Unfortunately, it appears to use more memory than before. I'm working on an air quality board with eight MQ sensors, an MH-Z19 CO2 sensor and a Plantower PMSA003 particulate matter sensor. In previous development versions (1.7-dev, before today), the eight MQ sensors and the MH-Z19 worked fine, but now after your big merge of #229 the sketch appears to run out of memory after adding the child of the sixth MQ sensor. Do you see any chance to optimize for memory?

New sketch (using the latest version of NodeManager, plus a PMSA003): https://github.com/kainhofer/NodeManager/tree/GasSensor
That sketch runs out of memory with the sixth MQ sensor (I added debug output with free memory at a lot of spots). Of course, I have commented out the PMSA003 completely to have a correct comparison.

@reinhold if you have high memory requirement, what about switching to a more modern architecture like stm32f103c8t6 on the famous "blue pill" boards ? It has 20k or RAM (10 times what you have on Arduino) and 64k of flash and it's supported by MySensors, in addition to be very cheap (2$/board). You will need voltage converter between 3.3V of the board and 5V of your sensors but that's not a big deal in exchange of the possibility to expand your board with more sensors in the future.

@nca78 Thanks for the suggestion to use the blue pill STM32 boards. I haven't thought of it (I only looked at the ESP8266, but that has only one ADC). I have a few lying around here, but never got around to trying them, as I first need to burn the bootloader. It looks quite promising, but the boards are huge, so I'll need to enlarge my PCB...
The need for a 5V->3.3V regulator is no issue, as the NRF24 already needs 3.3V. The 5V analog level of the MQ sensors will need some converter (the analog ADC inputs are marked as non-5V-tolerant...), but then in turn I can get rid of the voltage dividers for the PMSA003. The larger issue seems to be the connection of the nrf24, which takes away 4 ADC pins, so there are only 6 ADC pins left, while I have 8 analog sensors

Still, for now I have the PCBs with the ProMini, so I'll try to strip down the Sensor or SensorMQ classes for my own use.

@nca78 Two more things popped up with the latest development version of NodeManager:

In NodeManager.ino: function void before(): The sample code to set the child id of a child is sht.children.get(1)->child_id = 5, but that does not work, because at that point the children are not yet even generated. They are created by the node.before(); call at the very end of the function... So, currently this will not assign any new child ID and the default IDs will be used. I have not checked in detail whether calling node.before() at the beginning of the before() function and then assigning child_id to the children will solve the issue. From a quick look at the code, it might work...

I'm trying to implement an OLED display, where the node would wake up every 5 seconds, take a measurement and display it on the OLED. But the report interval to the MySensors gateway should only be 10 minutes. I thought thatnode.setSleepSeconds(5); node.setReportIntervalMinutes(10); would be the correct way to setup NodeManager to take measurements every 5 seconds, but only report it every 10 minutes to the network. Unfortunately, Sensor::loop(MyMessage* message) seems to immediately return without measurement if the reporting timer has not elapsed. Somehow I don't understand that logic. Why have different sleep and reporting intervals when measurements will only be taken at reporting intervals anyway? I would expect measurements to be taken with the sleep intervals (e.g. to calculate an average over the whole reporting interval) and report back to the network using report intervals.

@reinhold thanks for reporting this issue. One of the main objectives of the new architecture was to reduce the memory footprint so if this is not the case, there is definitely something to fix. I've noticed in your sketch you are using the remote configuration sensor, that one is huge in terms of memory, if not strictly required I'd comment it out (BTW very nice project!). In PR #229 I've also uploaded a spreadsheet with all tests I run for realizing what was using the memory the most. Anyway, there is still a lot of room for improvement looks like
Thanks @mfalkvidd , I'll give the directives you pointed out a try!

@reinhold you're right, assigning the child_id before calling node.before() does not work. The way to go would be to have all the methods being called before node.before() and the child_id assigned just after. Not very intuitive, I probably need to find a completely different and a better way to do so, probably by creating the Child in the constructor instead (reopening https://github.com/mysensors/NodeManager/issues/198 for this).

Regarding setSleep and setReportInterval, measurement are driven only by setReportInterval. Those are different because there are situations (e.g. relays, motion sensors) when you have nothing to report but you want the capture the heartbeat from sleep(). What you are looking for is something I never considered which is actually a pretty nice idea (taking measurements over a given time period and than reporting at the end). I've created https://github.com/mysensors/NodeManager/issues/243 for this. With the new architecture, since keeping already track of the average within Child, should not be so difficult even if would probably require an additional Timer. Thanks!

@user2684 Thanks for the spreadsheet with your memory footprints. Are these the sketch size and the free memory that the compiler reports? Or do you measure them when running the sketch (i.e. including all dynamically allocated variables in SRAM)?

Also, one problem you don't seem to have covered is heap fragmentation. Particularly your new List class seems to be prone to heap fragmentation with a large number of sensors: each time a sensor is added, the list needs to re-allocate the memory for its pointers. The old memory will be released, but it will remain as a hole in the memory, where new variables might be allocated. However, the next time you add an element to the list, the new pointer array can probably not re-use some of those freed holes.
I'll provide a PR (https://github.com/mysensors/NodeManager/pull/247) that adds an allocateBlocks(count) method to the List, which can be used in the sketch and in the sensor's onSetup methods to pre-allocate the required sized. In turn, the _preAllocBlocks member does currently not serve any purpose, and I don't think it's really required functionality. In my eyes, it's more important to ensure that at a certain moment the required memory is allocated. It is not neccessary to keep a minimum nuber of blocks allocated at all times.
For a node with 10 physical sensors (and maybe one or more built-in sensors), this will save at least 22 bytes (1,1% of total SRAM!) and probably reduce heap fragmentation by quite a bit.

Another are where >22 bytes can be saved is the power manager. None of my nodes makes use of the power manager, yet each sensor needs to keep a (null) pointer to a power manager. Providing a compiler #define NO_MODULE_POWER_MANAGER can exclude everything PowerManager-related from compilation and save one pointer per sensor and node, i.e. with 10 sensors you'll save another 22 bytes (1,1% of total SRAM!) I'll provide a PR for this, too: https://github.com/mysensors/NodeManager/pull/248

Apart from that, I simply stripped down the SensorMQ class and removed all configurability for my own use (don't worry, I won't submit this as a pull request... it is simply meant to squeeze the sketch onto the pro mini with the boards that I have already built).
BTW, the SensorConfiguration is no problem as far as SRAM is concerned (it just adds the normal SRAM footprint of a Sensor instance): It has no class members of its own. Only the flash size is considerable, but for me this is not the issue, I'm running out of SRAM.

@reinhold I measured only what the compiler reported. Main focus was to reduce the flash size, I honestly ignored any potential SRAM issues which is bad Thanks for the hit and for the great contributions and PRs so far. You're right about the List class and especially when the number of items is known, there is not reason why not allocating the blocks beforehand. I will merge your changes as they are and then take it back and adjust something (e.g. make the number of block you are passing to "node" optional, moving child creation into the constructor, always allocating the blocks for children in each sensor, generalizing a bit more the NO_MODULE thing etc.). Thanks again!

Hi, for those looking for making NodeManager time aware (with and without an attached RTC), this capability has been added to the development branch through https://github.com/mysensors/NodeManager/pull/259 and it is of course transparent to the end user. When this feature is enabled, also, the node resumes the remainder sleep time when woken up by an interrupt and allows SensorPulseMeter and subclasses to support sleep mode. Thanks

For the SensorDimmer, each sensor has only one type set so a dimmer can't receive v_status & v_percentage messages.

I've worked around this by registering two sensors at the same Analog out pin. But they don't then have access in a nice way to the status of each other as values are changed. So if I turn the led on/off via the v_status message I can't preserve the previous percentage, set by the other v_percentage sensor.

Is this normal ? I'm new to all this but at least mycontroller lets you specify for a single sensor both v_status,v_percentage so both types of messages can be sent and processed in onRecieve()...

Perhaps some enhancement on the SensorDimmer class could do this ? in ModeManager.cpp (if (message.type != _type) return;) inside Sensor::recieve.... isn't that too restrictive ?

@rfm69 yes, they are all interrupt-based. Reason is they would need to be functional either when sleeping or not so I went for this approach. If you need an example of such a sensor not interrupt-based, have a look at the SensorSonoff, it is using a different approach for handling the button. Thanks

@rfm69 you're right, this also came out a while ago. I was sure we had an associated issue on github for this but it doesn't look like. Here's the new issue for your reference https://github.com/mysensors/NodeManager/issues/296. Implementing this in v1.6 is really challenging but with the new v1.7 architecture of the development branch should be way easier since all the children share the same variables. Thanks!

@user2684 Thanks, I've implemented something myself with changes to the library, just playing around to understand it better. But will look at the SensorSonoff you mention.

Thanks again, its a great library.

Had a quick look at 1.7 dev branch is this ready for using already, testing or trying, or have you got more to do before its refactored etc ? I was interested in looking what the changes in architecture were.

@rfm69 v1.7 is still not completed but theoretically fully functional and with most of the architectural changes already done (https://github.com/mysensors/NodeManager/milestone/9 for the full list of things still to be completed and those already implemented). For sure safe to be tested out! Thanks

@user2684 in the sensebender sketch we turn on digital pin 7 to provide power to the LDR to read it.
i have many sensors in my house and would rather not re-wire all of them.
how can i add this cleanly in nodemgr to make it work as before?
pinMode(7, OUTPUT); // “power pin” for Light Sensor
digitalWrite(7, LOW); // switch power off

Hi, I am starting with MySensors and I read about the NodeManager. But I have a problem with starting with the nodemanager. I tried to make a node with a BMP280 sensor and a NRF24. Every time I tried I got the following error message
when compiling (nodemanager actual version, arduino IDE1.8.5). Even no other sensor works.

@cpl_ir super weird, looks like the linker cannot find NodeManager's object file. Are you sure all NodeManager's files are within the same directory? NodeManager cannot be installed as a standard Arduino's library but all the files need to be in the same directory of your main sketch. Thanks!

Hello, I have been experimenting with the door sensor example and have configured to wake up every hour to send a battery voltage (V_VOLTAGE)
It will also wake whenever the interrupt state on the switch pin changes. I noticed that when the node wakes up with an interrupt it seems to send three events:
I_POST_SLEEP_NOTIFICATION
V_TRIPPED (this is the important one)
I_PRE_SLEEP_NOTIFICATION
this is all good, however for my application it would be useful if it were to also send a V_VOLTAGE battery measurement along with the V_TRIPPED event.
I'm not yet completely familiar with how the nodemanager sketch links together, but is it possible for the sensor readings for all sensors to be sent on an interrupt from any other sensor so I can make use of the radio as much as possible?

@arthurdented the I_POST_SLEEP_NOTIFICATION and I_PRE_SLEEP_NOTIFICATION are due to smart sleep which is on by default. Regarding the V_VOLTAGE I think what it comes closer to your need would be a SensorBattery configured with a very short reporting interval but regardless, when an interrupt occurs, no sensors other than the one associated with the interrupt would execute their main function. For your specific use case you can try changing or tweaking the condition at this line to make it working in the way you need: https://github.com/mysensors/NodeManager/blob/master/NodeManagerLibrary.ino#L4700

@felix-haverkamp the signal node is supposed to send the RSSI when the radio supports it. What you are looking at is a v1.7 bug, fixed in the dev branch of v1.8. If you want an easy fix without the need to upgrade, just remove the uint32_t cast at line 5017 of NodeManagerLibrary.ino (https://github.com/mysensors/NodeManager/blob/master/NodeManagerLibrary.ino#L5017).
To send the values only if the value has changed, you need first of all to turn FEATURE_CONDITIONAL_REPORT on, then you can use setValueDelta() and setForceUpdateMinutes() of the Child class to achieve what you are looking for

would be nice for other users to document this function. With this i am able to only send values to the gateway when its changed. Using it for a Light Sensor and dont need the value every 10 seconds and not every 10 Minutes....
Just when it changes (1 second) and only when it changes.

@felix-haverkamp if you set setValueDelta() to e.g. 0.1 or 0.01, then the sensor will not report if the value has not changed, if I have understood correctly your point. setValueDelta() will set the minimum difference (+ or -) that the new value should have in order to be reported. Let me know I've misunderstood something. Thanks

Every new version of NodeManager, I try to review and hopefully improve its architecture. This time looked like a simple and quick task but eventually ended up in a complete and deep review of the code. Many many many things have changed with the main objective to better integrate with the MySensors library and simplify the life of users willing to contribute or to add support for new sensors.

The new file structure (with a dedicated file for each supported sensor) and the way the package is delivered (as an arduino library now) go into this direction. There are also included a good amount of new features but still in most of the cases, resulting a smaller code.

Just wanted to say that NodeManager is awesome. I can't believe I only now really had a better look at what it is. I always thought it was some kind of software you had to install, and that it was used by professionals to manage complicated sensor nets.

For the last few months I've been thinking "wouldn't it be useful if there was some kind of beginner friendly web interface where you could select which devices you want, and then it creates the code for you". This is so close to that. Thank you for this incredible work!

I am using magnetic sensor together with interrupt feature but when sensor status is change too fast before main loop is finished, sensor value remains in wrong state until magnetic sensor value is physically changed again.

So I would like to set that SensorDoor value will be set regularly like e.g. temperature sensors values. I tried to add:

door.setReportIntervalMinutes(5);

But it does not work in this case. Value is not reported based on this interval. Only way how to get SensorDoor value back to right state is physically change its value (open/close magnetic sensors).

Situation: Arduino ist connected to an raspberry gateway with openhab -> works just fine.
To the arduino I have connected a PIR on 3 and a photoresistor (LDR) with an 10k resistor on A0 -> works just fine.
On first boot (after connecting etc.) the arduino sends motion status + LDR value. But after that first msg. only the motion (tripped) is send.
What I want: I want that the arduino sleeps until there is an interrupt comming from the PIR and then sends the motion status+light value (LDR) and then sleeps again 4ever until the next interrupt.
Problem: As I said, it only sends the light value only once.. and then never again. On the arduino debug I can see, that only motion is send:
NodeManager v1.7
LIB V=2.3.0 R=N E=X T=N A=A S=S B=-
LDR I=1 P=16 T=23
MOTION I=2 P=1 T=16
RADIO...OK
PRES I=1 T=16 D=LDR
PRES I=2 T=1 D=MOTION
READY

@jiri-hron I think you can achieve what you need by using FEATURE_TIME. In this way when woken up, NodeManager will go back to sleep only for the remainder time left. Not ideal but I couldn't find a better way to understand for how long the node has been sleeping before the interrupt.

It looks like there is a limit of 1 type of each sensor. Is there a way to modify this?

My use-case is a french sliding glass door. I would like to use a single node with 2 magnetic sensors (one on each end of the node) with the node mounted in the middle of the door, so that if either side is moved the interrupt is triggered.

@iteafreely your understanding is correct. I see two options here: you can either create a new custom sensor inheriting from SensorDoor and adding an additional child and the extra logic you require or create two instances of SensorDoor and playing with hooking functions to make one checking on the other. In this latter case you probably need some additional variables to keep track of the status.

@iteafreely I don't think there is a limit of one sensor per type, not even for interrupts like door sensors. You just need to make sure you are using different pins and different child ids for the two magnetic sensors. I'm using 8 analog sensors (MQ-... gas sensors) on eight different pins on the same node and it works just fine.

Of course, if you need some interdependence on what the two magnetic sensors report, then you must follow @user2684's suggestion of creating a custom sensor (a C++ class derived from an existing sensor class).
But if you simply want two independent magnetic sensors (for each of the ends of your sliding door), then you can simply create two instances of SensorDoor on two different pins (here pins 3 and 4) and different hardcoded child_id (1 and 2 in this case):

SensorDoor doorLeft(node,3,1);
SensorDoor doorRight(node, 4,2);

Unfortunately, both of them will report to the gateway with name "DOOR" (and there's no way to change it without subclassing SensorDoor), but they will have a different child_id 1 and 2 to distinguish the two values sent by the node.

You can then use your smart home controller to handle the two door sensors like one.

@iteafreely Not sure of software issues as I don't use NodeManager, but your use-case indicates you do not need to know which door is opened, only that one or both is/are.
If by magnetic sensor you mean magnet/reed, would a single reed where both doors meet, or two reeds (one on each door end) wired in parallel, to a single interrupt pin not provide this?

@reinhold oh you're right I didn't even noticed that DOOR would show up in the debug output, thanks. Anyway with v1.8-dev this should have gone since debug output has been completely re-worked (and made compatible with https://www.mysensors.org/build/parser). Thanks

hello, I wanted to use the function for FEATURE_CONDITIONAL_REPORT for an LDR sensor but except a value when starting the arduino, no more values are transferred, must be configured for the function specifically something? Node manager 1.8dev

First a great compliment for this work! This should make things just easy, but...
It doesn't work for me. I have Ethernet Gateway and a 1wire Tempsensor node. Both flashed with the sketches from tutorials found on the mysensors.org website. Those are working without any problem.

Now I have Tsl2561 Lux sensor. This is supported by nodemanager. Just downloaded the latest dev version uncommented two lines as expected, but it the node won't appear.

Question about PowerManger:
I am a bit confused or I can't find the right information.
I have an Ultrasonic sensor (SensorHCSR04) that needs to be only powered in case of measuring, due to power consumption.
An the sensor needs to be switched on a time before the actual measuring starts.
So my first approach was to switch the
#define NODEMANAGER_POWER_MANAGER ON
Then I set

nodeManager.setPowerPins(5,6,500)

Where I a a bit surprised, why I need two pins (for vcc and ground)?
At what logic level are the when the PowerManager switches to ON or OFF?
Do I need to connect both pins, or is the vcc- connection enough?
Anyway, somehow it did not work with me. It seemed, that the pins are not switched in that way, that pin5 goes to ground and pin 6 goes to 3,3 volt.
So I found that thing :

// power all the nodes through dedicated pins
nodeManager.setPowerManager(power);

@user2684
Hello, it is me again. Going the next steps.
Now I want to operate another sensor (SensorVL53L0X).
this sensor should transmit the distance and the battery voltage every hour. in between it should sleep.
The normal reporting interval of the battery is once every hour
I set setReportIntervalHours(1); to report the sensor value.
But how do I set the node to sleep in between?setSleepOrWait(True); and sleepBetweenSend(); ?
orsetSleepOrWait(True); and setSleepMinutes(60); ?
And is the NodeManager so clever that all the transmits are at the same time,
meaning once every hour the sensor value and the battery voltage is transmitted.

@bisschopsr I now have the schematics posted for all of my boards. I also posted another board for my in-wall scene controller line. Is there any other pics or information that you think might be useful to others for any of these boards?

Advantage of this sensor is fast and easy installing by the non qualified personal. The sensor is designed for large agriculture facility with many greenhouses where is needed to be controlled temperature and humidity, the atmospheric pressure is for control the motor of air inlet and outlet. With help of the sensor installed outside the greenhouse provide information about positive/negative pressure inside the greenhouse.
Finlay the software read accurate temp/humid value of the Si7021, and only atmospheric pressure of BMP180, the temperature sensor inside the BMP180 is used only by the sensor itself to provide accurate atmospheric pressure measurements.
The sensor can be used also for home automation, and drone applications where is needed correction of the UAV barometric sensor compared to the ground barometric sensor, also humidity measurement can help the pilot to calculate more accurate flying time.