So I have decided to split them. If you are a skilled C programmer you might want to participate at the PPKG C-Programming Contest. There you also can read why and how the project was started.

This discussion will continue with the Puppy Linux Package Handler

PPKG Puppy Linux Package Handler
The package management in Puppy Linux is largely done by shell scipts. This concept causes performance problems for some extensive tasks. For these tasks I have designed ppkg.

At the moment ppkg is able to transform Debian/Ubuntu-Package-Info files to Puppy Linux Package-Database files with an enourmous speed win.
E.g.: The original shell script needs about one hour for the conversion of the Debian Lenny Package Info ppkg less than 400 ms.

Concept
The basic operation is to convert Debian/Ubuntu-Package-Infos like:

Additionally these packages must be assigned to the categories for the package management in Puppy Linux. This is done by puppy-category-files (extension ctg), that assign the packages to their category. They look like:

[Category-Name]
package-1
package-2
...
Usage
ppkg is intended to be inserted into the original shell scripts of Puppy Linux. But if you want to do the conversion by hand you have to proceed as follows (Example for Debian Lenny):

To do
ppkg at it's present state can only convert Debian and Ubuntu packages. Using the concepts and algorithms I used for Debian it shouldn't be a big problem to the conversion of other package formats as there are:

1. Slackware
2. Arch

If you want to realize one of them let me know and I will support you.

The appended software includes all files needed to compile the program and do some tests.

@BarryK
It should be taken care that the category system of Puppy Linux will not be garbled by the inclusion of foreign packages. At the moment the categorisation within the Pet Package Mangager mirrors the desktop menu categories. This system will be messed by the inclusion of foreign packages because until now there is no solution how to integrate them into the Puppy Menu Category System.

One solution could be to precategorize them how you do it with find_cat and to create a tool that includes them into the Puppy Menu automatically (using a default icon e.g.).

Another option could be to evaluate a distribution package for it's desktop file and to convert the distribution category to the corresponding Puppy category.

If it's for the package manager only, Debian already has a categorisation see "Section:" Enhancing the Pet-Package-Manager with this system would make it easier to distinguish the source of a package. The advantage would be that distribution categories must not be converted. The Manager's category buttons could be replaced by a list.

I think there is some reflection necessary of how the integration of foreign packages can be done user friendly without messing up the whole system.

@ Bruce_n_Duane
I have not tried it but as the Ubuntu Package Description is the same as Debian's I think it will work. The only thing that has to be done is the categorisation. Depending on your expectation you can do the conversion in 3 ways (Ubuntu-Packages: Name of bunzipped Ubuntu Package Info):

2. As Debian and Ubuntu should be about 90% identical concerning the packages, this call is worth a trial:
ppkg -c ubuntu -i lenny.ctg -x debug Ubuntu-Packages Packages-ubuntu-jaunty-main

3. To do it like Puppy Linux does it now is also no big thing. It only makes necessary a bit of manual work:
If you not have it yet, create the original Puppy Database file with 0-setup-cut (this will take about one hour and I have commented the package-name output. So wait or undo it to have some information about the progress)
0setup-cut ubuntu Ubuntu-Packages Packages-ubuntu-jaunty-main

Then you must cut the package-name together with it's category out of
Packages-ubuntu-jaunty-main. The packages must be sorted by category and to be inserted into a file jaunty.ctg following the example of lenny.ctg. Then you call:

@ BarryK
The category file structure is very simple as it only assigns package-names to their category. If all distribution categories once have been classified their source does not matter. The package-names are unique and hopefully have the same categorisation for each distribution. A universal puppy.ctg would be enough. It even could include Slackware and Arch as long as the categorisation has same structure for all distributions.

Your proposal has two disadvantages:
1. It would be necessary to include "BuildingBlock" into the category file (around 70% of all Lenny packages) to decide if the package is categorized or not, thus increasing the file and reducing the speed of the conversion.
2. find_cat must run for every conversion thus reducing speed. But the distributions are only updated once a while.

Maybe this proposal is more economic, but it needs your approval:
Using cut, your printcols function, sort and grep, the new categories easily can be extracted from the existing package-data-base files. After a run of find_cat they can be inserted into this universal space saving file. Maybe a new function in ppkg can automate as much as possible. I would like to realize it but I can't promiss when I will find time.

Once puppy.ctg is up to date this solution would be a smart:
A bzipped puppy.ctg together with it's MD5sum resides on the Puppy Mirrors. As the packages already must be downloaded a short deviation to a Puppy Mirror could do an automatic update if needed and ensure that the categories are always up to date._________________Wosh

Your proposal has two disadvantages:
1. It would be necessary to include "BuildingBlock" into the category file (around 70% of all Lenny packages) to decide if the package is categorized or not, thus increasing the file and reducing the speed of the conversion.
2. find_cat must run for every conversion thus reducing speed. But the distributions are only updated once a while.

Maybe this proposal is more economic, but it needs your approval:
Using cut, your printcols function, sort and grep, the new categories easily can be extracted from the existing package-data-base files. After a run of find_cat they can be inserted into this universal space saving file. Maybe a new function in ppkg can automate as much as possible. I would like to realize it but I can't promiss when I will find time.

I don't understand your point 2.
ppkg would lookup 'puppy.ctg', if it finds a match, fine, if not then run 'find_cat'. So, for an up-to-date 'puppy.ctg', 'find_cat' would never run even once._________________http://bkhome.org/news/

2. find_cat must run for every conversion thus reducing speed. But the distributions are only updated once a while.

This applies for the case that the info in puppy.ctg is older than the distribution converted, because ppkg has no information about the version of the package-info file. After further reflection I found a solution for this problem by an auto-update function if unknown packages are found. It could work this way:

If ppkg finds unknown packages it calls find_cat, categorizes them and stores them in an internal table. With the content of this table it automatically updates puppy.ctg.

But I am not really a friend of the current category concept for this reason:
The around 25000 packages of a Debian or Ubuntu distribution are like an elefant that stamps into a kennel called Pet Package Manager designed for a few hundred packages. Everything is smashed. The problem can be solved by these approaches. Shrink the elephant to size of a dog (that's what the current appoach does) or extend the kennel to an elephant stable. I am for the extension for these reasons:

1. At the moment find_cat only assigns a reduced number of these packages to categories. If you try to find a certain package in PPM it is probable that you won't find it even if you know it's name.
2. This could be improved by the assignment of more packages to the existing categories. But the consequence would be huge categories that are loaded slowly and are not really a great help for the user if she/he wants to find a certain package.

The current category-system is not designed for such a huge amount of packages. But it easily could be extended using this concept:
Puppy packages use the current category system, Debian packages use their system as it is represented in their package website
http://packages.debian.org/stable/
In the Debian package descriptions these categories are found behind the topic "Section:"

The representation of the categories could be realized rather simple and flexible by the exchange of the category-radio-buttons with a list that represents Debian-sections if the Debian repo is selected and Puppy-categories if the Puppy repo is selected. I do not know whether this concept collides with existing PPM concepts but the advantages are various:

1. No categorisation must be done by the conversion program.
2. The user gets the well known Debian system represented in PL.
3. All packages are categorized and selectable.
4. There is a clear separation in the PPM and in it's database concerning the categories of Debian/Ubuntu and Puppy._________________Wosh

If ppkg finds unknown packages it calls find_cat, categorizes them and stores them in an internal table. With the content of this table it automatically updates puppy.ctg.

Yes, that's what I wanted you to do.

A problem is that online repos change, new packages get added, so any 'puppy.ctg' may be out of date. There has to be this fallback that will automatically add to puppy.ctg, even if it is limited, as find_cat is.

If ppkg finds unknown packages it calls find_cat, categorizes them and stores them in an internal table. With the content of this table it automatically updates puppy.ctg.

Yes, that's what I wanted you to do.

A problem is that online repos change, new packages get added, so any 'puppy.ctg' may be out of date. There has to be this fallback that will automatically add to puppy.ctg, even if it is limited, as find_cat is.

As described two posts before the autoupdating categorisation requires to include all categories including "BuildingBlock" to the category file thus increasing it from now 86000 bytes to more than 200000 bytes. The implementation of this concept is a bit tricky and will require about two days leisure. At the moment my family, job and house make heavy demands on my time. At the next weekend I am already booked by my family.

But I found a more effective solution for this problem that could be easily implemented after work.
The current concept uses the category file as a the complete presentation of the category system. But now I want to use it as a fine tuning tool for a raw categorisation that is done this way:

1. The Debian sections are assigned to Puppy categories. Some of them will fit rather good to Puppy categories like "Editors", "Network", "Games" others won't.
2. For those that do not fit the category file is used. ppkg will check this file before it does the default categorisation.

The advantages of this concept:
1. It will work with new distributions also, because they hopefully will not contain to many new packages that must be forced into a certain category.
2. The category file becomes very small because it only contains the exceptions and not the rules.
3. The implementation will be very effective and fast because it can use conversion tables if a category is not found in the category file.
4. This concept can be implemented within a few hours._________________Wosh

I'd like to help, but the coding style of ppkg source code is like huge festering gutpile in a septic pond in hot summer. Way too much comments makes it really hard to understand and impossible to apply different style with GNU indent or any other tool of that kind. I can't stand its control flow either, FIO_FileToMemory() shows perfectly how it shoudln't have been done.

@ pstef
If it's for the coding style it shouldn't put you off participating the project. I have some good reasons to this kind of coding style which to explain would fill a book. But I also don't wan't to be forced to a certain style be it the Windows Hungarian notation or the GNU coding style and I as well do not wan't to force others to my kind of coding style. So if you really are interested in the project this won't distract you as you can realize your part of the whole (e.g.: the Slackware conversion) in a own module using your preferred style.

I think the software is hard to read for another reason:
It uses some rarley used software techniques rather from the compiler's than a human's point of view because it's focus is nothing else than speed. But the principles used are quite simple. If I will have time I'll do some explanations that make the basics more understandable.

So excuse me if my horizont is a bit resticted to the speed optimisation due to several decades micro controller programming in C and Assembler.

PS I didn't think that exactly FIO_FileToMemory of all functions would cause problems in understanding because the real tricky functions are pkg_DebianConvert and pkg_DebianConvertInfo. So here are some approaches that might help you.

Note:
FIO_FileToMemory can't be really simple because it hides a lot of demands like opening the file, checking the file size, reading the file, memory allocation and error handling from the caller. But if you produce a more understandable version that is as effective as the current one it will make it into ppkg.

You have totally misunderstood my moans. It's not the technical aspect of your program that I can't understand, it's the source code itself.

Quote:

If it's for the coding style it shouldn't put you off participating the project. I have some good reasons to this kind of coding style which to explain would fill a book.

I bet there are people with many good reasons not to wash their teeth, but even a dozen of books wouldn't convince me to follow them.

Quote:

But I also don't wan't to be forced to a certain style be it the Windows Hungarian notation or the GNU coding style and I as well do not wan't to force others to my kind of coding style.

But that's exactly what you're doing. You've made it impossible to apply different coding style. Without those stupid comments it would be as simple as typing #indent -kr ppkg.c new-ppkg.c

Quote:

I think the software is hard to read for another reason:
It uses some rarley used software techniques rather from the compiler's than a human's point of view because it's focus is nothing else than speed. But the principles used are quite simple. If I will have time I'll do some explanations that make the basics more understandable.

I didn't get to that, I'm stuck with the coding style.

Quote:

So excuse me if my horizont is a bit resticted to the speed optimisation due to several decades micro controller programming in C and Assembler.

Quote:

PS I didn't think that exactly FIO_FileToMemory of all functions would cause problems in understanding because the real tricky functions are pkg_DebianConvert and pkg_DebianConvertInfo. So here are some approaches that might help you.

No, FIO_FileToMemory() doesn't cause any problems in understanding, the coding style does. But I have mentioned FIO_FileToMemory() for another reason - control flow. It would be more obvious with tab indentation; look how deep it is at the end of the original function.

This is how I would rearrange FIO_FileToMemory(). Notice that if someone doesn't like its style, it can be easily reformatted with GNU indent. BTW, GNU indent is a program and can be found there: www.gnu.org/software/indent/

You cannot post new topics in this forumYou cannot reply to topics in this forumYou cannot edit your posts in this forumYou cannot delete your posts in this forumYou cannot vote in polls in this forumYou cannot attach files in this forumYou can download files in this forum