A lot of good ideas have come out of this thread...To make sure they dont get missed in amongst the discussion Im going to summerize them at the top here, with the origional question below.

Conclusions

- Pre-release ticklist - Ecomoney

- Whole Forum "sandbox" section for bugs, with "bug per thread" approach, rather than one thread for all bugs -Ttuxxx

- Recruit "Package use experts" (not the compilers) and ask them to take part in testing before development begins - Patriot

- Have a "Testing Manager" to keep track of the testing process, only release when theyre happy too do so...no rush! - Ecomoney

- Make it easy to collect bug reports (default homepage) - James C

- Keep changes/package updates to only the ones needed by Users - Ecomoney

Origional Post

Ok, The puppy linux team Have just released the first official version without BarryK. Lots of new features and upgrades were added, and its received glowing praise from distro critics and even a brief #2 Ranking at Distrowatch.

Although we had a long and ardious testing process, with plenty of pre-releases, Betas and Alphas, 4.2 was released with still quite a few bugs present, and also one particularly nasty one thats been very well discussed and eventually conclusions drawn from. I dont want to go back into any of that here, this is looking forward rather than back.

What can we learn from the successes and failures of 4.2? Here are some things that have already been mentioned.

Quote:

1. Using a full release of Puppy as a test bed for individual pieces of software is a mistake.

2. Each package should be thoroughly tested using some systematic procedure before it ever gets near the overall build.

3. Introducing a new package or attempting to upgrade a package in the middle of the development process is asking for trouble.

4. Third-party criticism is an important part of the process.

from rcrsn51

There was also some discussion that a separate group/team of "test pilots" be created at the inception of each release. These should generally be separate from the developers, which would free their time for compiling etc. and need not necessarily have/need any coding skill whatsoever.

The "test should have a single person as a central co-ordinator who the developer informs of changes/upgrades/new features and packages they have implemented that need testing, and also what other areas this migh effect (say in the case of an upgraded library) That person can then assign the job of testing to an individual with experience of that package, and keep track of what has been tested. Only when that person is happy that everything is ticked off the list will the Puppy Build be released.

If we are discussing events/mistakes in the past, please try not to assign blame. Constructive comments only please._________________Puppy Linux's Mission

Sorry, my server is down atm!Last edited by ecomoney on Tue 07 Apr 2009, 09:40; edited 1 time in total

I think before all these fancy rules about how/when packages should be added, there is something much more basic: to actually see that they run.
This isn't the first release where some menu items ended up just plainly not working. I honestly don't understand what the beta-testers are doing if such a state exists -- I would expect the first thing a beta-tester would do is go over the menu and try and start every item._________________What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind

This isn't the first release where some menu items ended up just plainly not working..

Which ones, Dougal? In building each release that was one of the things I tried to do - click on each menu item, especially if it was an upgraded or new package, to see that it actually worked. Due to the time constraints, however, I didn't click on menu items where the underlying program was unchanged. I haven't yet found any menu item that doesn't work in Puppy-4.2, BUT I haven't tried them all ... so knowing which one(s) aren't working for you would help.

Cheers_________________Actions speak louder than words ... and they usually work when words don't!
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

.....Although we had a long and ardious testing process, with plenty of pre-releases, Betas and Alphas, 4.2 was released .....

IMHO, its not long enough (but thats just me) but I can understand the reasoning behind it and with puppy's recent history of changes, Whodo did what was necessary.

Quote:

1. Using a full release of Puppy as a test bed for individual pieces of software is a mistake.

Depends on the intended outcome. Some say yes, some say no.

Quote:

2. Each package should be thoroughly tested using some systematic procedure before it ever gets near the overall build.

It should be but everything's sort of informal here ... Anyway, this should have been raised much much much earlier ...

Quote:

3. Introducing a new package or attempting to upgrade a package in the middle of the development process is asking for trouble.

Huh? I must have done things wrong in my own development work then ? Again, I believe this depends on the intended outcome ...

Quote:

4. Third-party criticism is an important part of the process.

Yeah, constructive criticism that is ..... yet, I found out the hard way years ago that most have no tact when delivering their "constructive" criticism. I see most shoot first and ask questions later ... how is that being constructive ?

Quote:

from rcrsn51

There was also some discussion that a separate group/team of "test pilots" be created at the inception of each release. These should generally be separate from the developers, which would free their time for compiling etc. and need not necessarily have/need any coding skill whatsoever. .....

For one, the only real developers that I've seen here are the distro developer(s) and the puppy specific apps developers. I've yet to see the actual applications software developers for Gnumeric or Abiword commenting here. I've seen and read mtpaint's developer (wtjaguar) comments and it's good to have him here.

I don't know if its feasible to have a pool of testers making their own compiled binaries. Compiling takes time that varies between packages and even between testers machines. It's fine if one tester does one or two packages but all the packages ?

What we have here is a group of people working on packaging the various pieces of software into one distro. There's no real formal structure to it. BarryK made it into a sort of not-so-formal thing. That's one reason I was attracted to puppy. I already have enough time pressures on my back and I sure ain't looking forward to adding more from FOSS ...

I can understand the need for testing methodologies. But do we have sufficient man power to handle it ? The way I see it, I believe we need package maintainers that knows the ins-and-outs of the packages they handled. They will have to sit down and learn as much as possible about the package that they maintain. Keep track of what works and what breaks. They should handle the bug reports by looking into possible causes/fixes or to file the bug reports to the actual developer if necessary ... These package maintainers should be consulted into what should go into a distro release and what should be left as options. Being a package maintainer, they'll have to keep their guards up and be up-to-date with changes, bugfixes, tweaking, hacks, some inner-workings, whatever, etc etc ... Life can be hell for the maintainers ...

We already have a lot of users offering pet packages. Why not ask them if they're willing to become package maintainers? Or at least get them into informal groups that works towards a common goal?

"See a need, fill a need" - Bigweld (Robots)

I did CUPS package for a reason. After I got it to work, I saw the need to get fuss-free printing working in puppy. I focused on it when I can and it was a slow process. If not for wanting some functionality in newer CUPS, I wouldn't have bothered. There was no big pool of testers for CUPS. I crashed and burned with the first 1.4b2 pet ... But it was an experience that helped me understood the linux printing system better ... I would have just taken it for granted if I was on Ubuntu ...

Actually, now I'd be much much more than happy to let anyone take this off my back ... It's too much for one person to do it alone ... It would be great if there's a team in puppyland that will maintain all the different printing (not just cups) related packages.

Getting back to testing methodologies, I believe we could start some sort of an informal QA/QC team that makes a checklist of all packages that may be included into a future distro. A fresh start now will help cut down on future pre-release testing time. Having a group of selected package maintainers will also reduce the workload/time pressures/havoc. I believe that when the next distro coordinator have been appointed, he/she will find that the QA/QC checklist will be of a great help.

I know my active time here is only a few days old but again I reiterate, one of the reason I was attracted to puppy is because its a bit more relaxing here. Too much red tape hinders creativity and I've banged my head against it many times in the real world. It will be great to have some QA/QC methods but don't let it spoil the fun please ...

Just as a point of comparison, I am envolved in the Windows 7 beta testing program. Microsoft began with a limit of about 2 million downloads, then later removed that limit. I believe each download gets 3 different installation keys.....anyway,there is a large pool of testers. Crash info is automatically sent to Microsoft so it's not necessary to file bug reports.
The point is, even with all of this testing, Windows still crashes and burns. It is impossible to catch every bug, but hopefully the "showstoppers" will be caught.And I won't be purchasing 7 after the test, too big and bloated.I'm trying to get away from Windows and to Linux full time.
The puppy community is smaller, the release schedule quicker and there are way fewer developers.I don't know the solution but this thread is a beginning.

When I logged on I was really heartened to so much constructive feedback and so many good ideas have been posted on this thread. Thank you everyone

Puppy is a really important project , and its important that people take responsibility to ensure that it is as good as it can be. I agree that it does need to fun, and done in a pleasant, blame and flame free environment for our programmers/packager/compilers to produce their best work. Lets face it no one is here for the money, but even Micro$oft recognise this. Im by no means saying I havnt been guilty of this myself, but I am doing my best to learn from my mistakes.

Comparing open source with commercial development (Ive been commercial developer) there is always a pressure to get something "working" as quickly as possible and get the code (and the invoice) out. Development time is a "cost" that eats into "profit", so there is an incentive to make ugly hacks and patches that build up problems and inefficiencies for the future. This is one reason why Windows i think has become so bad. If you put programmers under pressure then they make mistakes...its just not productive to do it. Open source has no commercial pressures and doesnt need to suffer with set release deadlines....thats why its so much BETTER!

Puppy has always had a breakneck development speed, Barryk used to release versions every couple of months, and sometimes they had some quite big bugs and a new version released (wireless was broken in one version around 2.1x?). Generally speaking he did set aside a reasonable proportion of each "cycle" to testing though, and each change was small and incremental on the last.

Looking back on the events of puppy 4.2 development, Whodo chose to issue lots of small incremental versions which worked very well I thought. First released were quite a few Alphas with different packages in to try. Ok they were buggy, but they gave people the chance to try different potential packages (i.e. firepup) before a final package list was defined...great! There were then several betas to give the maintainers of those packages the time to refine their work and improve the look and the feel (pWidgets got a lot better over this time for example, and the themes were worked on). There were still a lot of issues being identified at this stage, which is why i got a bit worried when Whodo announced Release Candidate 1. I was worried there wasnt going to be enough time to finish testing, but whodo assured us it was merely a title to get more people to test it...fine. I did get the impression though whodo by this stage you were very keen to get 4.2 released and signed off? I did actually think of suggesting a pre-release checklist then and I really wish I had sorry.

@ Patriot - good points, none of them "lame". I assume because you produce the JWM window manager (which has been given the opportunity to really shine in 4.2...its so much better when its not coloured grey!) youve got a lot of experience in taking part in many different development efforts from many different open source projects. Thank you for taking the time to pass on your experience of them.

From what Ive seen of different packages, whether something is "ready" to be included is merely semantics. Im using aMSN 0.98 beta in a big cybercafe install, because Ive tested it and it seems pretty stable, as well as having the extra features. 0.97.x was stable too, 0.96 was very buggy and we had to revert back to 0.95 which we used for a long time. On the other hand you would think that Abiword 2.6.6 final would be relatively bugless, but it wasnt. The thing to learn from the future here is that we shouldnt assume/consider ANYTHING bugless or "bugless enough" until weve actually tested it ourselves. Part of the job of producing a "Distro" is to make sure we test and distribute only the best applications and the best quality version of that application? Thats the point.

Many hands make light work, and different people have different skills, and different packages they use. Assigning "Experts" to different packages (a maintainer/compiler, and a tester), who know the ins and outs of that package and can research it thoroughly is an excellent idea Splitting up the distro on the basis of its component applications is an excellent way of doing it.

The puppy community is growing as puppy becomes more popular. When I joined it, it was at number 20 in Distrowatch, now its at number 8! Ive oticed about people that when they come over to puppy at first they are very grateful they have been given something, and are more than happy to do a little bit back and be part of the project.

Perhaps a column in there as well to state if the package was an upgrade on a previous version (and therefore needed testing) would be handy...but you get the idea.

How can this be kept up to date?

@James

Theres no reason we cant learn from Windows as well...they are for some reason quite successful, at least for the moment. You are saying with the windows 7 testing program, they have the facilities for collecting bug reports actually built into the program itself? How could this concept be implemented (better) in Puppy?

How about in future, making the default home page in Seamonkey refer to a page in the Puppy Wiki. This would change depending on the release (alpha/beta/RC)...

The alpha could have a page detailing the programs being tried out

The betas could have a lits of known bugs, and what needs to be tested

The Release candidates would have the checklist above to keep track of all of the final bugs, and ensure everything was checked thoroughly before release.

As I say, this is just one general idea and open to a lot of refinement.

Thank you for your continued reading if youve made it this far..._________________Puppy Linux's Mission

Most of my experience is with XP. In XP if an application,say Firefox or Adobe reader, freezes up or crashes the user has the option to upload an error report to Microsoft. To the best of my knowledge, in Windows 7 Beta the option is taken away and the error report is automatically sent.Sort of the opposite of "automatic updates".I personally would not want that info
sent without my permission on an official release, but I don't mind on a beta release.
Anything to help get accurate bug reports back to the developers in a timely fashion would probably help.Keeping the bug reporting threads on track would probably help as well.

The Git repository has just been updated by Pizzasgood for 4.2. This is a good beginning. People can focus working on their chosen aplication/s and see improvements immediately.

Someone can start tallying people's names working on apps and probably come up with a list of individuals or groups. What this will do is help testers or developers get directed to the proper communication channels. However, the usual channel will still be the forum.

What's more, the developers can put links/directions in their forum signatures, the way MU, Zigbert et al are doing._________________Puppy user since Oct 2004. Want FreeOffice? Get the sfs (English only).

I know my active time here is only a few days old but again I reiterate, one of the reason I was attracted to puppy is because its a bit more relaxing here. Too much red tape hinders creativity and I've banged my head against it many times in the real world. It will be great to have some QA/QC methods but don't let it spoil the fun please ...

Hallelujah ... and pass the mustard!

@All - raffy is correct. Git is the opportunity to stabilise package development for Puppy. It's getting close to ready and when it is I will be using it to build Puppy-4.2.1, or perhaps 4.3, for release in a few weeks to a month.

How to proceed? Read the Git instruction manual, available online when the server is ready for general access.

Like everything else in Puppy, if you want to help in the development effort then go for it. No-one is going to tell anyone here what to do or what not to do; maybe that's a bit chaotic, sure, but highly productive in a creativity sense. OTOH I've been known to ask for things occasionally, but never with any expectation that a dev will comply. No-one can say "no" until they are asked.

Above all, HAVE FUN. Forget ego. Forget ownership. Forget personalities. Be "Nike" about the whole thing (IOW, "Just do it"!), and don't get all bent out of shape if what you do doesn't find its way into the main release. I firmly believe that no sincere effort born in generosity is ever wasted, even by exposure to ingratitude.

@Patriot - I hope for all our sakes that it just gets easier for you to stay, have fun and continue contributing great stuff the way you have so far in your relatively short time with us. If you are finding yourself under too much pressure to deliver in any area, please don't hesitate to discuss that with me via a PM before voting with your feet, ok? A big part of my job here, at least for the next release of 4.x, is to ensure that no volunteered resources - developer, packager, tester, maintainer, etc - are either abused or wasted, ok? _________________Actions speak louder than words ... and they usually work when words don't!
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

Someone can start tallying people's names working on apps and probably come up with a list of individuals or groups. What this will do is help testers or developers get directed to the proper communication channels. However, the usual channel will still be the forum.

Why not get the users to give us this information instead of going and chasing them?

I noticed a thread on here asking people how many computers they had and what OS was on them...it got very popular. What about a tempory sticky in the Users ( For the regulars ) asking "What are your favourite applications?". its been my observation that regular puppy users love posting on threads like this. It would then just be a matter of going through the thread and sending pm's to the relevant people asking them if they would be prepared to help test their favourite application in the next development cycle. This could be the job of the "testing tzar" to assemble a testing team.

It would also help us on the "Puppy should have all the applications necessary for daily use" part of the mission statement.

I think its important to have a team of people with the right skills assembled before each development cycle, with their responsibilities defined. My name was included in the release notes for 4.2 under the "testing" section ...not because I had been asked to be a tester, but because I had submitted some bug reports of things I had come across. When I pointed out about the Abiword bug I got "Well you were on the testing team...why didint you report it!". I dont even use Abiword!_________________Puppy Linux's Mission

Im not sure about this GIT repository but Im looking forward to reading about it. I hope popular apps like aMSN, Openoffice and firefox are included...look at which apps are popular in ubuntu for example. Where can i find out more?

The problem with the "nike - just do it" approach is that is someone "just doesnt do it" then faults get missed i.e. Abiword. Im not sure what background you have is software development, but testing is an important part, and should always be done methodically...not just "hope for the best". If this type of work isnt to your taste, then pass it this way.

Please dont think Im ungrateful for your efforts Whodo because I think its important to look at what didnt go so well as what did, and how we might do even better next time. Puppy 4.2 is a vast improvement on previous puppies with just a few rough edges...you should take responsibility for it all.

Not wishing to curb your enthusiasm for moving on to even bigger and better things, puppy has always had .1 and .2 releases. Currently on our main puppylinux.org download page, users are being served with a distro with a "semi-broken" wordprocessor, as well as a few other faults. Shouldnt this be put right as soon as possible before even more new features are introduced?_________________Puppy Linux's Mission

You cannot post new topics in this forumYou cannot reply to topics in this forumYou cannot edit your posts in this forumYou cannot delete your posts in this forumYou cannot vote in polls in this forumYou cannot attach files in this forumYou can download files in this forum