Btw:
Could you please point me to the easiest way for making a full backup of stock bootloader + firmware + env (everything) and restoring this data to another GoFlexNet device before converting it? So I could revert if needed. Could I actually flash the same backup data of the GFN I'm currently using on another GoFlexNet either? Because I got to give away the one I am currently fiddling on, but I have an older one for myself, but that one already an old version modified bootloader and rescue system on it.
Thanks :)

> Could you please point me to the easiest way for
> making a full backup of stock bootloader +
> firmware + env (everything) and restoring this
> data to another GoFlexNet device before converting
> it?

How to set up U-Boot for booting in multiple drives configuration
Backup and Restore NAND mtds
UART Booting HowTo for Selected Kirkwood Devices
Migrating from Arch to Debian?
How to boot new Debian rootfs using stock u-boot tftp

BTW, the only NAND area that's going to be written over is mtd0 where u-boot is. So stock FW rootfs will not be touched.

bodhi Wrote:
-------------------------------------------------------
> bobafetthotmail,
>
> > Will have to make a chroot
>
> I'm not sure I understand why would we need to
> chroot? the tools are static, and the only thing
> need to be written to / is /etc/fw_env.config?

Yes, that config must be written to /etc/fw_env.config but I don't want have to deal with the firmware.
I don't want to have to care about specific firmware configurations in a multi-device installer system like this as I'd likely have to add hacks for many devices and any firmware update may make this a failure point.

Besides, we don't really need to change it at all as the stock firmware cannot be booted from new uboot anyway.

With a tiny chroot the tool will think that $TOOLS/etc/fw_env.config is /etc/fw_env.config, and everything will be fine regardless of how the firmware actually is. Theoretically.

The current busybox has chroot command already, I'll make some tests this evening.

It's not complicating my work nor annoying as asking people to do it by hand in a tutorial.
The script is executed by a machine after all, let's leverage this.

It's more annoying when a firmware change/difference breaks my script out of the blue and I have to troubleshoot stuff over a forum in inappropriate moments.

Btw, also the original script from jeff also had a large amount of code dealing with making a chroot to install debian from scratch, that I have removed. (for now at least)

> One more thing I forgot to mention: we need to
> check if the user is root and bail out if s/he is
> not root.

A good point. I've not seen devices that give you non-root shells but that's a possibility.

The new script attached should print the attached drive info (that is used by the script to figure out what is a disk) before quitting.
If you give me the full log of another run with that SSD I'll try to figure out why it is not detected.

It should have detected all drives that were connected at boot, hotplugging Sata isn't working well on Kirkwoods.

>I got to mention that at one time the installation pauses at
Yes, it was there for my own debugging mostly. It's now a 5 second wait.

> So do you think the sd card can't take this
> punishment?

no, my mistake again.
It's unbelievable, I have tested this on my devices and there it worked fine even with this mistake in it. :/

Yet another script. Sorry for the inconvenience.
Please run one on USB and one on Sata, if possible.

100%[======================================>] 329 --.-K/s in 0s
2000-01-01 00:01:34 (32.6 MB/s) - `/tmp/tools/etc/fw_env.config' saved [329/329]
verify file /tmp/tools/etc/fw_env.config with md5 /tmp/tools/etc/fw_env.config.md5
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 119: cat: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 119: cut: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 123: md5sum: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 123: cut: command not found
passed
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 192: rm: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 242: chmod: command not found
# Successfully installed /tmp/tools/etc/fw_env.config.
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 782: fdisk: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 786: fdisk: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 786: grep: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 786: grep: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 786: awk: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 786: awk: command not found
./file.php?3,file=953,filename=kirkwood-installer-fix5.sh: line 786: sed: command not found
no disks detected, exiting now
-sh-3.2#

Btw:
I first didn't read the title exactly and thought that the script would also install a rescue system(That's why I though of backin up all mtds). But of course that's not of priority now and an option for later. Also I would like to remind you, to maybe 'individualize' the system in the end. Honestly I don't exactly remember what this was about, but I read it here in forum and it must have been something like creating an own system certificate, SSH key or such.. :)

> It's more annoying when a firmware
> change/difference breaks my script out of the blue
> and I have to troubleshoot stuff over a forum in
> inappropriate moments.

We are not in any hurry for a deadline :)

> Btw, also the original script from jeff also had a
> large amount of code dealing with making a chroot
> to install debian from scratch, that I have
> removed. (for now at least)

One thing I hate to do is design-by-committee. So I will wait until you think it is ready, and review it.

There is one point I feel should make: the script should stay as simple as what Jeff's orginal version, as much as possible. I know we try to improve things, but I can't imagine that will be much more complicate than that script.

Stock OS does not use fw_env.config so it is OK to write it to the root. But I see your point of not touching FW. Just a reminder not to add many more moving parts that could break.

Got home and did some testing on my side too, I screwed up the setup of the toolbox and the script was using system's tools, so the behaviour was erratic.
:/

Now it properly uses the toolbox (as it can no more fallback on system tools as you see in the errors above), and I found and fixed other things too, probably also the sata issue.

I also wrote a chroot function and here works fine, it's the "02" option.
It will first do a fw_printenv using the system's tool (so will print your envs)
Then it will do a fw_printenv from inside the chroot so it will use our config file, and in your case this should error out as you don't have uboot envs in that place.

can you please test 01 with both Sata and USB and 02 (once)?

>I first didn't read the title exactly and thought that the script would also install a rescue system
The current goal is just installing Debian and uboot without fuss on the longish list of supported kirkwood boxes, as the tutorial is a bit long and scares off people.
Bodhi's modern uboot is fire-and-forget, as long as you plug a storage device with first partition labeled "rootfs" and all files inside in the right place it will boot Debian from it automatically (also works with a RAID, if you follow my tutorial from the forum's "wiki" in bodhi's signature).
As long as you have access to a Linux PC you can prepare another rootfs device with it.

To install a proper rescue system we need a rescue system for all (most) boxes first. The current one based on emdebian is relatively limited, pretty large for the embedded storage size, and works for some devices only (pogoplugs I think).

The very long-term goal (mine) is to have the script also install pre-made OpenWRT/LEDE firmware as a fallback. OpenWRT/LEDE is a third party opensource linux distro targeting embedded devices, it is mostly known for routers, it currently supports some of the kirkwood boxes, but far from all.
Being targeted at devices with 8-16 MB of storage, it's very light and allows me to fit more or less everything you ever need (also webinterface and all) in the 256MB or so that most Kirkwood boxes have.
From my limited testing with my Zyxel NSA310, it's just a matter of adding to their project the settings for making the firmware images for each box in the right way with stuff they have already, and that's what I'd like to do next.
But I'm not a hardcore developer so it's not going to be fast.

>Also I would like to remind you, to maybe 'individualize' the system in the end. Honestly I don't exactly remember what this was about, but I read it here in forum and it must have been something like creating an own system certificate, SSH key or such.. :)

Yeah I know, it's at the end of bodhi's tutorial I think. This has to be done by something else I think, even if I can chroot into the rootfs, the kernel in the stock firmware is too old to run the new programs in it.

That's a task for another simpler post-install script, also for setting buttons and lights and so on.

bodhi Wrote:
-------------------------------------------------------
> One thing I hate to do is design-by-committee. So
> I will wait until you think it is ready, and
> review it.

Well, I agree on it but I'll also say that I don't like to work on things in my free time then being told you don't want it made like that or that you use things you don't like just to not offend me or something.
So yeah, maybe not very in depth, but having some feedback from you would be nice.
It is already structured as it would be in its final form, more or less.

> There is one point I feel should make: the script
> should stay as simple as what Jeff's orginal
> version, as much as possible. I know we try to
> improve things, but I can't imagine that will be
> much more complicate than that script.

Main difference is that I'm enclosing more stuff in functions as it makes the whole thing more tidy and easier to read.
It may be a habit or personal style.

> Stock OS does not use fw_env.config so it is OK to
> write it to the root. But I see your point of not
> touching FW.

Zyxel firmwares do use that afaik, others may too.

> Just a reminder not to add many more
> moving parts that could break.

The point of using our own static toolbox was that I could take advantage of decent tools so I could do relatively advanced things without fear of issues.
That chroot trick is one of them, but I'm fine with dropping it if there are better options.
If you have a better idea to avoid writing to / while still using fw_printenv/setenv (recompile a hacked fw_printenv hardcoded to look where we need it to look without having a config anywhere?), I'm all ears.

> Well, I agree on it but I'll also say that I don't
> like to work on things in my free time then being
> told you don't want it made like that or that you
> use things you don't like just to not offend me or
> something.
> So yeah, maybe not very in depth, but having some
> feedback from you would be nice.
> It is already structured as it would be in its
> final form, more or less.

Understood :) I've been a little bit too busy so willl try to review it and and test on a couple of my spare boxes (GFNet/Home) before the weekend is over.

> Main difference is that I'm enclosing more stuff
> in functions as it makes the whole thing more tidy
> and easier to read.
> It may be a habit or personal style.

It's great.

> That chroot trick is one of them, but I'm fine
> with dropping it if there are better options.
> If you have a better idea to avoid writing to /
> while still using fw_printenv/setenv (recompile a
> hacked fw_printenv hardcoded to look where we need
> it to look without having a config anywhere?), I'm
> all ears.

I think that's the only way. If we don't chroot, we have to write to / to use fw_printenv/setenv after flashing envs image at 0xc00000. So I think chroot is cool to avoid messing with stock /.

I had both drives plugged, but couldn't get to the point for choosing one.

And for 02:

verify file /tmp/tools/etc/fw_env.config with md5 /tmp/tools/etc/fw_env.config.md5
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 122: cat: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 122: cut: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 126: cut: command not found
passed
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 195: rm: command not found
chmodding 644 -- /tmp/tools/etc/fw_env.config
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 246: chmod: command not found
# Successfully installed /tmp/tools/etc/fw_env.config.
######this is the output of normal fw_printenv
######this is the output of normal fw_printenv
######this is the output of normal fw_printenv
######this is the output of normal fw_printenv
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 1354: fw_printenv: command not found
######this is the output of chrooted fw_printenv
######this is the output of chrooted fw_printenv
######this is the output of chrooted fw_printenv
######this is the output of chrooted fw_printenv
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 529: mkdir: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 530: mount: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 532: mkdir: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 533: mount: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 535: mkdir: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 536: mount: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 540: chroot: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 543: umount: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 544: umount: command not found
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 545: umount: command not found
Installation complete
You can now reboot your device into Debian.
If your device does not start Debian after rebooting,
you may need to restart the device by disconnecting the power.
The new root password is 'root' Please change it immediately after
logging in.
Reboot now? [Y/n]
./file.php?3,file=960,filename=kirkwood-installer-6.sh: line 126: md5sum: command not found

downloading tools and making a rootfs
Connecting to pogoplug.s3.amazonaws.com (54.231.168.150:80)
wget 100% |*******************************| 2135k 00:00:00 ETA
wget: not an http or ftp url: https://dl.dropboxusercontent.com/u/47541136/linux/tools_for_Kirkwood_installer/busybox
chmod: /tmp/tools/busybox: No such file or directory
new PATH is now /tmp/tools
./file.php?3,file=965,filename=kirkwood-installer-8.sh: line 288: /tmp/tools/busybox: No such file or directory
./file.php?3,file=965,filename=kirkwood-installer-8.sh: line 290: ln: command not found

You can also see this result in 01.txt
We are still using stock wget at this point and you are trying to download through https.

Edit:

For saving us one post-cycle, I placed busybox onto my pc and changed the script to download busybox from there.
Results for both options (sata and usb) are in attachments.

ElMariachi Wrote:
-------------------------------------------------------
> Results for both options (sata and usb) are in
> attachments.

Thanks for taking the time to do the hack to test further.

It seems the device does not have the drivers to mount ext4 partitions, not even as ext2, which was the "last thing that could have gone wrong" I was talking about (I was underestimating my own ability to cause trouble).

"mount: mounting /dev/sdb1 on /tmp/rootfs failed: Invalid argument" for both, and on a system that has ext4 driver it works fine.

This means I have to switch to plan B: using dd to write a small ext4 partition image with the rootfs inside, counting on e2fsck (file system checker) to expand that into a larger partition. I'm half sure it should work.

Or Plan C and use ext3 or XFS or even ext2 depending on what drivers the device has. ext filesystems are bootable for uboot, XFS is not so I would have to make a separate /boot partition.

Or Plan D and scrap the part of the script dealing with rootfs creation and simply make a dd image of the rootfs the users can write on a drive with Rufus (windows program to make bootable usb drives and similar) or whatever other tool from their PC and distribute it in a 7z archive so it is compressed and not as large as the partition it will create.

One foolproof solution would be reversing the installation. And we only need a small toolbox. Let's try creating the basic rootfs first using ext3, and then chroot into that and do u-boot installation.

The things to watch for if we doing this: we need to gather the mtds info, ethaddr, ... things that in the rootfs chroot we can't see.

> One foolproof solution would be reversing the
> installation. And we only need a small toolbox.
> Let's try creating the basic rootfs first using
> ext3, and then chroot into that and do u-boot
> installation.

Hmm, the current toolbox is smaller than a rootfs and should be able to install uboot fine already.
Here the issue is that it would be nice to have a ext4 rootfs even if stock firmware does not have that driver.

I don't know if the Debian Jessie (or later) applications in the chroot will run at all with the kernel 2.6 of stock firmware.

We can simply try to use ext3 for the rootfs without making changes to the script, but I'm not very happy with that compromise (and I'm sure there will be devices that lack a ext3 driver too).

ElMariachi Wrote:
-------------------------------------------------------
> What about first creating mini os, that after
> creation installs uboot, then reboots and self
> destructs for making place for the final
> installation?

Yeah, this would have been the plan A if we had a recovery to flash in the internal flash. Flash uboot and recovery while into stock firmware, reboot in recovery, install Debian from recovery, reboot to Debian.

In our case we can make it anyway with a usb drive. we format a ex2 partition as rootfs, install a tweaked rootfs in it, on reboot it calls automatically a script that finishes the installation then erases that usb drive.

---
I'll have to test if Plan B (as said above) works at all. If it works it would be faster than using a recovery drive as above.

In the meantime I've attached the script, hoping it does not break again, it should now try to make a ext3 rootfs instead of a ext4 rootfs. See if it works now

> We can simply try to use ext3 for the rootfs
> without making changes to the script, but I'm not
> very happy with that compromise (and I'm sure
> there will be devices that lack a ext3 driver
> too).

On the contrary, ext3 rootfs should be used, IMO. This has ramification: backward compatibility (see my instruction for installing rootfs). Ext4 is better no doubt, but it would prevent the user to ever boot with stock u-boot again. So I always leave it to the users to upgrade it later.

Upgrading from ext3 to ext4 is quite simple to do after the installation. I think it is best to leave the rootfs as ext3. Stock uboot/OS can use ext3, if there is no ext3 driver, it wil just simply use ext2 without the journal.

Good, now we need to know what device we are working on
here is a list, please write the number of the device
00. download and unpack toolbox
01. make a Debian rootfs in a drive
02. test chroot with a printenv
1.Zyxel nsa310
2.Zyxel nsa325v1 or nsa325v2
3.Pogoplug
01
downloading tools and making a rootfs
Connecting to pogoplug.s3.amazonaws.com (54.231.168.202:80)
wget 100% |*******************************| 2135k 00:00:00 ETA
--2000-01-01 00:01:10-- https://dl.dropboxusercontent.com/u/47541136/linux/tools_for_Kirkwood_installer/busybox
Resolving dl.dropboxusercontent.com... 108.160.173.165
Connecting to dl.dropboxusercontent.com|108.160.173.165|:443... connected.
ERROR: cannot verify dl.dropboxusercontent.com's certificate, issued by `/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2':
Self-signed certificate encountered.
To connect to dl.dropboxusercontent.com insecurely, use `--no-check-certificate'.
new PATH is now /tmp/tools
./file.php?3,file=969,filename=kirkwood-installer-9.sh: line 290: ln: command not found
./file.php?3,file=969,filename=kirkwood-installer-9.sh: line 291: ln: command not found
./file.php?3,file=969,filename=kirkwood-installer-9.sh: line 292: ln: command not found

(Regarding the ext4 problem: Did you see that there's a fuse module? I edited it in, but I think this was after you had posted on already.)

> On the contrary, ext3 rootfs should be used, IMO.
> This has ramification: backward compatibility (see
> my instruction for installing rootfs). Ext4 is
> better no doubt, but it would prevent the user to
> ever boot with stock u-boot again. So I always
> leave it to the users to upgrade it later.
>
> Upgrading from ext3 to ext4 is quite simple to do
> after the installation. I think it is best to
> leave the rootfs as ext3. Stock uboot/OS can use
> ext3, if there is no ext3 driver, it wil just
> simply use ext2 without the journal.

Hm, looked up the upgrade path to ext4 https://ext4.wiki.kernel.org/index.php/UpgradeToExt4
I have some doubts that this can be performed on a live rootfs, but iit's easy enough for me to do the switch from my linux PC.
To use the older uboot again we need to have also stock uboots somewhere, or we should hope that the place the script dumps the images to does not get erased (I'd rather like to send them to a ftp server so we can offer stock uboots + envs for various devices too).

So fine for me, ext3 rootfs will be then.

ElMariachi
>(Regarding the ext4 problem: Did you see that there's a fuse module? I edited it in, but I think this was after you had posted on already.)

It's easier and less likely to blow up to just steal a ext4 kernel module from some ARM kernel 2.6 and modprobe it in with the script, but I'd rather avoid it as it is not guaranteed to work on all boxes.

---
also, added that --no-check-certificate to the script, now it should really work, hopefully.

> To use the older uboot again we need to have also
> stock uboots somewhere,
>

Jeff copied the original u-boot to stock rootfs. But we are strict about messing with stock. So I would dump it to /tmp only (1MB or less). And then at the end, copy the image to the new Debian rootfs /boot.

---everything fine
Installation complete
You can now reboot your device into Debian.
If your device does not start Debian after rebooting,
you may need to restart the device by disconnecting the power.
The new root password is 'root' Please change it immediately after
logging in.
Reboot now? [Y/n]

PS:
Again for the backup question that I had:
I would like to clone the GFN I am currently working on to another one, for making the other one stock again. The other one is completely modded including rescue system. So that's why I need to backup more than mtd0. I could use the older GFN then for more testing with stock formware and such, but I have to mod and to give away the current one.

I also tested booting with the USB drive on one of the modded GFNs, it's working fine (besides that I cannot login Debian, because of 'Permission denied (publickey,password).').

I can't test booting the SATA drive, because of that latest problem there.

> Again for the backup question that I had:
> I would like to clone the GFN I am currently
> working on to another one, for making the other
> one stock again. The other one is completely
> modded including rescue system. So that's why I
> need to backup more than mtd0. I could use the
> older GFN then for more testing with stock
> formware and such, but I have to mod and to give
> away the current one.

Please post this as a different topic in u-boot forum (how to backup/restore your NAND patitions).