…and buffer. Improved the ability to receive a constant stream at 57600bps.
27334 bytes out of 30720.
Remember - we had to completely butcher HardwareSerial.cpp so a normal Arduino installation will not work.
C:\arduino-xxxx\hardware\arduino\cores\arduino\HardwareSerial.cpp
I removed the receive interrupt handler from HardwareSerial.cpp so that we can deal directly with USART_RX_vect in the main code. This allows us to return to the way OpenLog used to operate - by having one buffer split in half. Once one half fills, it is recorded to SD while the other half fills up. This dramatically decreases the time spent in function calls and SD recording, which leads to many fewer characters dropped.
The change log at the top of the main code got so large I've moved it to a version controlled "Changes.txt" file.
By making all these changes, I have broken many things. Ringp - I could use your help here. I apologize for stomping on so much of your work. I was not good enough to figure out how to re-link from the old function calls to the new sdfatlib setup.
Backspace is broken - ringp, I saw this fix in one of your commits, but looking at the code, I don't see how it is supposed to work. Either way, we still get 0x08 when trying to backspace.
Search for "//Error" in main pde
ls is not working fully
remove is not working
fileInfo is not working
New sdfatlib doesn't have SdFat.cpp so fileInfo doesn't work. These function calls are marked with //Error
I have chosen to dis-allow deprecated functions:
#define ALLOW_DEPRECATED_FUNCTIONS 0
This forced some trivial changes to the SD write and new file creation function calls. I believe we've successfully migrated to the new version of sdfatlib.
In the command_shell / read_line function : It may be better to pull directly from the UART buffer and use the RX interrupt. For now, we brute force it.
Because of all these changes, I need to re-test power consumption. For now, I'm confident it's low enough.
Testing with 512 buffer array size
1GB @ 57600 - dropped very little out of 3 tests
1GB @ 115200 - dropped very little out of 2 tests
8GB @ 57600 - Formatted using the sd formater (32k byte allocation size). Dropped nothing.
8GB @ 115200 - dropped very little, dropped none
16GB w/ Lots of files @ 57600 - Drops the first part of the file because of start up delay?
16GB w/ Lots of files @ 115200
1024 array size (and 800) does not run
Testing with 700 buffer array size
1GB @ 57600 - 110300 out of 111000 bytes, 110300/111000,
1GB @ 115200 - 111000/111000!, 109600/111000
8GB @ 57600 - 109000/111000, 111000/111000!,
8GB @ 115200 - 111000/111000!, 111000/111000!,
16GB w/ Lots of files @ 57600 - 85120/111000, 85120/111000
16GB w/ Lots of files @ 115200 - 56420 (but once it got going, log looks good). 56420.
I am seeing long delays on cards with lots of files. In the above tests, the 16GB test card is a good example. It has 2GB worth of random files in a sub directory. After OpenLog boots, goes to '12<'. After I send ~500 characters OpenLog freezes for a couple seconds, then returns to normal, very fast, operation. During that down time, I believe sdfatlib is searching for an open cluster. The odd thing is that after the cluster is established (after the initial down time) OpenLog performs excellently. I am hoping to create a faux file or pre-write and save to the file or some how get this allocation done before we report the '12<' indicating we are ready. That way, if a card does fill up, as long as the host system is checking for '<', it doesn't matter how long it takes sdfatlib to find the next free cluster.
You can see that we drop 700 bytes at a time. That's a bit odd - I'd expect to drop half or 350 at a time. What happens if we shrink the array size to say 256? To be expected, this is bad - way more instances of dropped characters.
Added blink for each test to the OpenLog_Test sketch so we can see as the test progresses.
http://www.sdcard.org/consumers/formatter/ is the site for SD card formatting. It looks like this program takes a guess at the correct block size. This could help a lot in the future.

…mand, new "rm" implementation. New commands: pwd (print working directory), echo <on>|<off>. Fixed bug when using "size", leaking file handle. Removed too_many_arguments_error() in order to free up memory.

…rk. Remove directory now works! Change directory up/down the tree works again.
28440 bytes used of 30720.
ringp brought back many great commands! Thanks ringp!
rm LOG*.* works
ls *.TXT works
cd .. works again
ls now correctly shows directories first and files following the directories.
To remove a directory, you have to navigate into that directory. For example:
>cd TEMP (you are now in TEMP directory)
>rm -f TEMP (you are now removing TEMP, and you will be pushed back down one level of the tree)
>ls (shows files and directories where TEMP directory used to be, but TEMP directory should be gone)
ringp added new commands:
efcount: gives the number of files in the current directory. Example: "efcount" returns "count|3". There are 3 files in the current directory.
efinfo <spot>: gives info about the file in <spot>. Example: "efinfo 2" reports "LOG00588.TXT|45". File number 2 is named LOG00588.TXT and is 45 bytes in size.
verbose <"on"|"off">: sets whether command errors are verbose (long winded) or just the "!" character. Example: "verbose off" sets verbose off. Then if a command like "blah" is received, then only "!>" is seen from the command line interface. This makes it easier for embedded systems to recognize there was an error. This setting is not recorded to EEPROM.
Updated README.

…ow work the same as in previous OpenLog versions. "cd.." now works up to a depth of 15 folders. The depth can be increase be redefining a global variable. Added new commands "efcount", "efinfo", "eem", "verbose" that are simplifying embedded usage with OpenLog. Also fixed some code indentation problems and removed old commented code not being used any more.

…ow work the same as in previous OpenLog versions. "cd.." now works up to a depth of 15 folders. The depth can be increase be redefining a global variable. Added new commands "efcount", "efinfo", "eem", "verbose" that are simplifying embedded usage with OpenLog. Also fixed some code indentation problems and removed old commented code not being used any more.

…eSerial.cpp buffer to 512 bytes.
More testing at 57600. Record times look to be 2, 5, and zero milliseconds when doing a record. This means that the split buffer doesn't really make a difference? There are some records that take 150ms, 14ms, etc. At 57600bps, that's 7200 bytes/s, 138us per byte. With a 150ms pause, that's 1,086 bytes that need to be buffered while we wait... Grrr. Too many.
I re-wrote the append_file function to use only one buffer. This allows us to more quickly pull data from the hardware serial buffer. Hardware serial buffer has to be increased manually to 512. This file (hardwareserial.cpp) is stored in the Arduino directory. With testing, it seems like recording is working more solidly at 57600bps. But now I'm seeing glitches at 19200bps so more testing is required before we make this the official OpenLog release.
Moved input_buffer into within the append function. Attempting to shave bytes of RAM.

…r consumption.
26136 bytes out of 30720
Fixed issue 30. I added printing a period ('.') for non-visible ASCII characters during a 'read' command. This cleans up the output a bit. HEX printing is still available.
Fixed issue 34. When issuing a long command such as "read log00056.txt 100 200 2" (read from spot 100 to spot 200 and print in HEX), the command shell would die at 24 spots. I increased both the general_buffer and 'buffer' in the command shell from 24 to 30. The limit is now 30 characters, so "read log00056.txt 100 20000 2" is allowed.
Works with a 16GB microSD card! High volume test: loaded 16GB card with 5GB of data. Basic serial tests work. When running at 57600, there is an odd delay. I think it has to do with the file system doing an initial scan for an available cluster. Takes 2-3 seconds before OpenLog returns to normal. This can cause significant data loss.
Fixing power management in v2. Power down after no characters for 3 seconds now works. Unit drops to 2.35mA in sleep. 7.88mA in sitting RX mode (awake but not receiving anything). There is still a weird bug where the unit comes on at 30mA. After sleep, it comes back at the correct 7-8mA. Is something not getting shut off?

…r consumption.
Fixed issue 30. I added printing a period ('.') for non-visible ASCII characters during a 'read' command. This cleans up the output a bit. HEX printing is still available.
Fixed issue 34. When issuing a long command such as "read log00056.txt 100 200 2" (read from spot 100 to spot 200 and print in HEX), the command shell would die at 24 spots. I increased both the general_buffer and 'buffer' in the command shell from 24 to 30. The limit is now 30 characters, so "read log00056.txt 100 20000 2" is allowed.
Works with a 16GB microSD card! High volume test: loaded 16GB card with 5GB of data. Basic serial tests work. When running at 57600, there is an odd delay. I think it has to do with the file system doing an initial scan for an available cluster. Takes 2-3 seconds before OpenLog returns to normal. This can cause significant data loss.
Fixing power management in v2. Power down after no characters for 3 seconds now works. Unit drops to 2.35mA in sleep. 7.88mA in sitting RX mode (awake but not receiving anything). There is still a weird bug where the unit comes on at 30mA. After sleep, it comes back at the correct 7-8mA. Is something not getting shut off?

…t working.
26058 bytes out of 30720
Fixed a bug found by Salient (Issue 35). User settings where declared at chars which allowed them to be signed. If a user went from old firmware, to v2, the safety checks would fail because the settings would be read at -1 instead of 255. Declaring user settings as byte fixed issue.
Added "a) Clear user settings" to set menu. This allows us to completely wipe an OpenLog (user settings and config file) to see how it will respond to future firmware changes.
Improved the file 'size' command.
Sequential logging is tested and works.
Receive testing: Using the Test_Sketch found on Github, I am testing the receive reliability at different UART speeds. We need to test a lot of received data. At 57600, 115200, and both from an Arduino (lots of time in between characters becuase of software overhead) and from a raw serial port (almost no time in between characters). I am hoping to make sdfatlib hiccup at 115200, full speed, across a 1MB file. If I can make it fail, then we can start to increase the buffer size and look at how much RAM sdfatlib has left open for the buffer.
9600bps from Arduino works fine
57600bps from Arduino drops characters
115200 from Arduino drops characters
It seems that sdfatlib takes longer to write to the SD card than the original file system from Robert Reigel. I'm thinking perhaps we should create a version of OpenLog firmware that is just sequantial logging, no fancy system menus... It might open up some RAM.
If only we could get control of the UART from Arduino's clutches, we could probably handle the ring buffer much better. Not sure how to handle UART interrupts without tearing out HardwareSerial.cpp.
Added README to the Test sketch. Added 115200bps to test sketch.

Welcome to version 2! We've moved from Roland Riegel's FAT library to Bill Greiman's sdfatlib. OpenLog now works with SD cards up to 16GB (that is the current largest microSD card we can get our hands on). OpenLog automatically detects and works with FAT16/FAT32 file systems. It also automatically works with normal SD cards as well as SDHC cards.
Almost all the previous commands have been ported from v1 to v2. The current commands that do not work:
cd.. - does not work. You can change to an upper directory, but you cannot navigate back down the tree.
cat - this command was deprecated. HEX printing is now done with the 'read' command. We've added a 5th argument to select between ASCII and HEX printing.
Wild cards do not yet work. So rm and ls do not have wild cards enabled - yet. Help us out!

Welcome to version 2! We've moved from Roland Riegel's FAT library to Bill Greiman's sdfatlib. OpenLog now works with SD cards up to 16GB (that is the current largest microSD card we can get our hands on). OpenLog automatically detects and works with FAT16/FAT32 file systems. It also automatically works with normal SD cards as well as SDHC cards.
Almost all the previous commands have been ported from v1 to v2. The current commands that do not work:
cd.. - does not work. You can change to an upper directory, but you cannot navigate back down the tree.
cat - this command was depricated. HEX printing is now done with the 'read' command. We've added a 5th argument to select between ASCII and HEX printing.
Wild cards do not yet work. So rm and ls do not have wild cards enabled - yet.

…his case Ctrl+Z) are no longer appended to the file. Fixed some other small terminal endline problems like for example 2 newlines displayed after each other when running terminal commands. Small indentation fixes.

… of commands -that can be used with any other micro processor- to count files in a folder (efcount), get file info (efinfo), turn off echoing (echo <on/off>), turn off extra information when something goes wrong (verbose <on/off>) and also a special command to ease the communication and separate the actual data from the result of a command (eem <on/off>). Created a "Drivers" folder to host drivers for other microcontrollers. Corrected some other minor bugs, like write command not working properly when using offset.

Fixed the firmware version in the help menu to v1.61.
Updated Eagle files to Eagle v5.9. Fixed weird airwire. Changed D1 LED from Green to Blue. Will only affect new production after 4-28-10.
Closed some tickets and wrote some more example Arduino code:
http://forum.sparkfun.com/viewtopic.php?t=21438

CONFIG.txt or through system menus.
What happens if I reset the system by pulling RX low, but the config file
has corrupt values in it? If config file has corrupt values in it, system
will default to known values 9600/ctrl+z/3/newlog. If config file is
empty, system resets to known values After some massive testing, and lots
of code to check for illegal states, it looks to be pretty stable. The
only problem is that we're running out of RAM. The buffer had to be
decreased from 900 bytes to 700 bytes to facilitate all the config file
checking. Testing at 57600bps, unit runs very well over 40kb test file on
straight RS232 connection. That's pretty good. Testing at 115200 on
straight connection, unit will drop a buffer every once and a while. Not
great, but not much we can do if the SD card times out for ~150ms while
it's writing.
8 bits to the byte plus a start/stop bit = 10 bits per byte
@ 9600bps = 960 bytes per second. Buffer will last for 729ms
@ 57600bps = 5760 bytes per second. Buffer will last for 121ms
@ 115200bps = 11520 bytes per second. Buffer will last for 60.7ms
So if the SD card pauses for more than 60ms, 115200 will have lost data,
sometimes. All other baud rates should be covered for the most part.
SD cards with larges amounts of data will have increased pause rates.
Always use a clean card where possible.

… CONFIG.txt or through system menus.
What happens if I reset the system by pulling RX low, but the config file has corrupt values in it? If config file has corrupt values in it, system will default to known values 9600/ctrl+z/3/newlog. If config file is empty, system resets to known values After some massive testing, and lots of code to check for illegal states, it looks to be pretty stable. The only problem is that we're running out of RAM. The buffer had to be decreased from 900 bytes to 700 bytes to facilitate all the config file checking. Testing at 57600bps, unit runs very well over 40kb test file on straight RS232 connection. That's pretty good. Testing at 115200 on straight connection, unit will drop a buffer every once and a while. Not great, but not much we can do if the SD card times out for ~150ms while it's writing.
8 bits to the byte plus a start/stop bit = 10 bits per byte
@ 9600bps = 960 bytes per second. Buffer will last for 729ms
@ 57600bps = 5760 bytes per second. Buffer will last for 121ms
@ 115200bps = 11520 bytes per second. Buffer will last for 60.7ms
So if the SD card pauses for more than 60ms, 115200 will have lost data, sometimes. All other baud rates should be covered for the most part.
SD cards with larges amounts of data will have increased pause rates. Always use a clean card where possible.

…s, example Arduino sketch
Added function from mungewell - check_emergency_reset. This has improved testing of the RX pin. There was potential to get a false baud reset. There is still a chance but it's now much less likely.
If OpenLog is attached to a Arduino, during bootloading of the Arduino, ctrl+z will most likely be sent to the Arduino from the computer. This character will cause OpenLog to drop to command mode - probably not what we want. So I added user selectable character (ctrl+x or '$' for example) and I added user selectable number of escape characters to look for in a row (example is 1 or 2 or 3, '$$$' is a common escape sequence). The default is now ctrl+z sent 3 times in a row.
Added an example Arduino sketch (from ScottH) to GitHub so that people can easily see how to send characters to OpenLog. Not much to it, but it does allow us to test large amounts of text thrown at OpenLog at 57600bps.

Added 4800bps and 19200bps support
Added power saving features. Current consumption at 5V is now:
In default append mode:
6.6/5.5mA while receiving characters (LED On/Off)
2.1mA during idle
In command mode: 3.2/2.1mA (LED On/Off)
So if you're actively throwing characters at the logger, it will be ~6mA. If you send the logger characters then delay 5-10 seconds, current will be ~2.5mA. (Unit records the characters in the buffer and goes into idle more if no characters are received after 5 seconds)
These power savings required some significant changes to uart.c / uart_getc()

Eliminates errors while logging at 57600bps.
Added exit options to the two menus (set and baud)
Also added display of current settin to two menus (Ex: "Baud currently: 57600bps")
Added '!' infront of 'error opening'. This pops up if there is an error while trying to append to a freshly created new log (ex: LOG00548.txt is created, then errors out because it cannot append). '!' was added so that user can parse against it.
Replicated logging errors at 57600 using 5V Arduino
Unit would systematically glitch during logging of 111054 bytes
Increasing buffer to 1000 characters caused URU error.
URU: Unit Resets Unexpectedly
To recreate URU error. Type "append ". Include the space. If you get "!error opening", then things are fine. If you get "!error opening#" where # is a weird character, then type 'ls' and the unit will unexpectedly reset (URU error). I believe this is attributed to a memory overrun somewhere in the FAT system.
Changed buffer size to 900 and declared the character buffer as volatile
#define BUFF_LEN 900
volatile char input_buffer[BUFF_LEN];
This increase to the buffer allows for clean logging of 444055 bytes with no URU errors.
Experimenting with Scott's SD cards (customer gave cards on loan for recreating logging errors):
Card with single ~740mb file produces errors when trying to open/append to new log.
Card with less stuff on it logs full 444055 bytes correctly.

Added sd_raw_sync() inside append_file. I believe this was why tz1's addition of the timeout buffer update feature was not working. Auto buffer update now working. So if you don't send anything to OpenLog for 5 seconds, the buffer will automatically record/update.

Test log roll over error 65533 to 65536: Doesn't work. Unit gets caught in endless loop.
So I added two sections to newlog() that check for max value of 65534 (65535 looks like un-initiated EEPROM spots). Works well and errors out gracefully.
Unit will throw error "!Too many logs:1!" if there are too many logs then gracefully drop to command mode.
If you are using OpenLog, parse for:
"12~<" means everything is good
"12!...." means there is an error while attempting to open a new log.
TODO: Use this style of error throwing for other types of errors (append errors, card errors maybe, etc).
New rm LOG*.* command: Tests well, I love this added feature. Works great!
LOG# increase to 16-bit (65535 max logs) looks good. The EEPROM storage locations look correctly updated. Testing works well.