This document is intended to serve as a place for brainstorming ideas to improve and enhance the PBI (Push Button Installer) format for PC-BSD 9.0 and beyond.

+

<!--T:2-->

+

This document is intended to serve as a place for brainstorming ideas to improve and enhance the PBI (Push Button Installer) format for PC-BSD® 9.0 and beyond.

+

=== What is the PBI format? === <!--T:3-->

−

+

<!--T:4-->

−

=== What is the PBI format? ===

+

−

+

The idea behind creating the PBI format is that one of the things holding back mainstream adoption of open-source desktops is the package management format. With almost all open-source desktops, software is simply treated as part of your system itself. Thus when performing an update of some seemingly trivial application, you run the risk of potentially needing to upgrade other packages which in turn could break other critical parts of your desktop.

The idea behind creating the PBI format is that one of the things holding back mainstream adoption of open-source desktops is the package management format. With almost all open-source desktops, software is simply treated as part of your system itself. Thus when performing an update of some seemingly trivial application, you run the risk of potentially needing to upgrade other packages which in turn could break other critical parts of your desktop.

−

'''Example:'''

+

<!--T:5-->

−

User wants to upgrade to latest version of FireFox. However, this in turn requires new version of libFoo*,

+

:'''Example:'''

−

which is also used by Xorg and many other libs/apps. In order to perform update of libFoo, Xorg and any other

+

::User wants to upgrade to latest version of FireFox. However, this in turn requires new version of libFoo*,

−

dependent libraries must be updated as well. Newer Xorg may in turn require different version of libOtherFoo and

+

::which is also used by Xorg and many other libs/apps. In order to perform update of libFoo, Xorg and any other

−

so-on and so forth. All it takes is one failure at any point of the process, and the user may end up with a broken

+

::dependent libraries must be updated as well. Newer Xorg may in turn require different version of libOtherFoo and

−

system, no GUI and a huge loss of productive time.

+

::so-on and so forth. All it takes is one failure at any point of the process, and the user may end up with a broken

+

::system, no GUI and a huge loss of productive time.

−

While package management systems have gotten better at resolving these issues, trying to fix conflicts and prevent breakage before it occurs, it still doesn't fix the underlying problem, that every software package is simply a part of the system, and pulling one any one thread has the potential of causing a break somewhere farther down the line.

−

The PBI format tries to correct the underlying flaw in this system for desktop computing. Rather than making every application a part of your base system, PBI's are self-contained, including their own dependent library tree and related data. As a result, when you install a PBI there are no dependency issues to resolve, and applications can be added / removed freely, without fear of causing breakage to the desktop or any other installed software.

+

<!--T:6-->

+

While package management systems have gotten better at resolving these issues, trying to fix conflicts and prevent breakage before it occurs, it still does not fix the underlying problem, that every software package is simply a part of the system, and pulling one any one thread has the potential of causing a break somewhere farther down the line.

−

=== Why update the format? ===

+

<!--T:7-->

+

The PBI format tries to correct the underlying flaw in this system for desktop computing. Rather than making every application a part of your base system, PBIs are self-contained, including their own dependent library tree and related data. As a result, when you install a PBI there are no dependency issues to resolve, and applications can be added / removed freely, without fear of causing breakage to the desktop or any other installed software.

−

The current PBI format, while still very useful is beginning to show its age. The format when started was a new concept for the open-source world and now, after many years of real-world usage, it is time to take what we've learned, refine and implement it in a cleaner way, for the benefit of both PC-BSD and FreeBSD users.

+

=== Why update the format? === <!--T:8-->

+

<!--T:9-->

+

The current PBI format, while still very useful is beginning to show its age. The format when started was a new concept for the open-source world and now, after many years of real-world usage, it is time to take what we've learned, refine and implement it in a cleaner way, for the benefit of both PC-BSD® and FreeBSD users.

+

=== The current PBI format === <!--T:10-->

−

=== The current PBI format ===

+

<!--T:11-->

−

+

A .pbi file today is made up of several different components and organized as indicated below.

A .pbi file today is made up of several different components and organized as indicated below.

−

1. At the front of a PBI file is a small C binary, which makes the application executable, and starts the installation process.

+

<!--T:12-->

−

+

# At the front of a PBI file is a small C binary, which makes the application executable, and starts the installation process.<br>

−

2. Next is a small tarball which contains configuration meta-data, such as icon information, mime-data and install / uninstall scripts - all used by the GUI portion of the installer.

+

# Next is a small tarball which contains configuration meta-data, such as icon information, mime-data and install / uninstall scripts - all used by the GUI portion of the installer.<br>

−

+

# After the tarball, there is a location for an optional "icon" which can be used by thumbnailer utilities, to display the applications icon in file-managers.<br>

−

3. After the tarball, there is a location for an optional "icon" which can be used by thumbnailer utilities, to display the applications icon in file-managers.

+

# Lastly is the program archive, which is made up of a LZMA compressed tarball for both maximum compression and fast extraction.<br>

−

+

−

4. Lastly is the program archive, which is made up of a LZMA compressed tarball for both maximum compression and fast extraction.

+

−

+

+

<!--T:13-->

By using these components, the QT4 PBI installer is able to run, confirm some settings from the user, such as destination directory, and then extract the program archive to disk. The installer reads much of the included meta-data and uses it in the creation of XDG formatted desktop / menu icon creation, as well as registering mime types, etc.

By using these components, the QT4 PBI installer is able to run, confirm some settings from the user, such as destination directory, and then extract the program archive to disk. The installer reads much of the included meta-data and uses it in the creation of XDG formatted desktop / menu icon creation, as well as registering mime types, etc.

+

=== New Goals === <!--T:14-->

−

+

<!--T:15-->

−

=== New Goals ===

+

−

+

While the current format is useful and working in many ways, it is lacking some features and using some "hackish" methods to force programs (which often are not designed to be run self-contained) to work for end-users. Below are some of the ideas we want to implement in the new format to correct these issues, and make the format work on traditional FreeBSD.

While the current format is useful and working in many ways, it is lacking some features and using some "hackish" methods to force programs (which often are not designed to be run self-contained) to work for end-users. Below are some of the ideas we want to implement in the new format to correct these issues, and make the format work on traditional FreeBSD.

+

==== Managed hard-links for resources ==== <!--T:16-->

−

+

<!--T:17-->

−

==== Managed hard-links for resources ====

+

−

+

One of the drawbacks of the current implementation is its handling of libraries and resource files. With each program including its own complete set of libraries, there is often much duplication between applications, causing a waste of hard-disk space, as well as memory usage during runtime, since libraries are shared via inode lookups.

One of the drawbacks of the current implementation is its handling of libraries and resource files. With each program including its own complete set of libraries, there is often much duplication between applications, causing a waste of hard-disk space, as well as memory usage during runtime, since libraries are shared via inode lookups.

−

To improve upon this situation, a managed directory can be used for "sharing" the common files between applications, allowing less disk-space to be used and memory to be saved during runtime. This can be accomplished by creating hash-lists of files within a package and installing resources to a shared directory, creating hard-links back to the original file-names within the installed PBI's tree.

+

<!--T:18-->

+

To improve upon this situation, a managed directory can be used for "sharing" the common files between applications, allowing less disk-space to be used and memory to be saved during runtime. This can be accomplished by creating hash-lists of files within a package and installing resources to a shared directory, creating hard-links back to the original file-names within the installed PBIs tree.

+

<!--T:19-->

+

'''Pitfall:''' Different applications may include the same library with the same version (and sub-version and sub-sub...), but with ''different compile-time options'' or generated by ''different compilers''. Thus either the PBI installer must compare all candidates byte-wise (making nonsense of the choice of LZMA for fast decompression) or (better?) a strong hash is needed for an ''abstract'' of the library, which must of course include all compile-time options. For C/C++, tools to extract or generate these informations exist, but care has to be taken for other major languages.

==== Create new PBI management system for add / remove of software ====

+

<!--T:21-->

−

+

With the PBI format now, installation is performed by each application itself, using the included QT4-based binary installer. Other utilities for PBI management are also written in QT4, such as the creator / removal tool.

With the PBI format now, installation is performed by each application itself, using the included QT4-based binary installer. Other utilities for PBI management are also written in QT4, such as the creator / removal tool.

+

<!--T:22-->

This should be changed, with a new suite of tools written in shell / C, that perform the actual installation, creation and removal portions. This would allow a variety of front-ends to be developed, and help break the PBI format from being more QT/KDE specific, allowing FreeBSD users to install support for the PBI format via ports. This would also

This should be changed, with a new suite of tools written in shell / C, that perform the actual installation, creation and removal portions. This would allow a variety of front-ends to be developed, and help break the PBI format from being more QT/KDE specific, allowing FreeBSD users to install support for the PBI format via ports. This would also

removed the need for a binary loader at the front of .pbi files, instead making the file-format look like this:

removed the need for a binary loader at the front of .pbi files, instead making the file-format look like this:

+

<!--T:23-->

[meta-data]|[optional icon]|[program archive]

[meta-data]|[optional icon]|[program archive]

+

<!--T:24-->

Possible names?:

Possible names?:

+

<!--T:25-->

ports/sysutils/pbimanager?

ports/sysutils/pbimanager?

+

<!--T:26-->

pbi_add / pbi_delete / pbi_create / pbi_info

pbi_add / pbi_delete / pbi_create / pbi_info

+

<!--T:27-->

Not very original, but easy to remember for those coming from a BSD background.

Not very original, but easy to remember for those coming from a BSD background.

+

==== Explore the usage of $ORGIN as a replacement for LD_LIBRARY_PATH ==== <!--T:28-->

−

+

<!--T:29-->

−

==== Explore the usage of $ORGIN as a replacement for LD_LIBRARY_PATH ====

+

−

+

One of the ways the current version of PBI handles program execution using self-contained libraries is by creating wrapper scripts for executable, and setting some variables such as LD_LIBRARY_PATH. While this works, it has a few problems in that it is A: Hack-ish and B: Causes breakage with setuid.

One of the ways the current version of PBI handles program execution using self-contained libraries is by creating wrapper scripts for executable, and setting some variables such as LD_LIBRARY_PATH. While this works, it has a few problems in that it is A: Hack-ish and B: Causes breakage with setuid.

+

<!--T:30-->

One possible solution would be with the usage of [http://docs.sun.com/app/docs/doc/817-1984/appendixc-13?l=en&a=view $ORGIN] for the compiling of PBI applications. This feature could eliminate the idea of needing to use LD_LIBRARY_PATH and fix issues with using some setuid programs, such as VirtualBox in user-mode.

One possible solution would be with the usage of [http://docs.sun.com/app/docs/doc/817-1984/appendixc-13?l=en&a=view $ORGIN] for the compiling of PBI applications. This feature could eliminate the idea of needing to use LD_LIBRARY_PATH and fix issues with using some setuid programs, such as VirtualBox in user-mode.

+

<!--T:31-->

There is another way: Use same approach as used this util: http://www.freshports.org/devel/chrpath/

There is another way: Use same approach as used this util: http://www.freshports.org/devel/chrpath/

I tested such situation:

I tested such situation:

Line 91:

Line 103:

3. Find all text files (.sh, .xml, etc.), change there PATHes too.

3. Find all text files (.sh, .xml, etc.), change there PATHes too.

+

<!--T:32-->

With such tweak scripts, I was able to build PBI to one PREFIX, and dynamically install it to other PREFIX. Program worked just fine. However, need more testing with some complex Programs, OpenOffice probably, etc.

With such tweak scripts, I was able to build PBI to one PREFIX, and dynamically install it to other PREFIX. Program worked just fine. However, need more testing with some complex Programs, OpenOffice probably, etc.

The most common way to build a PBI currently, is by using the [http://wiki.pcbsd.org/index.php/Using_the_PBI_Builder PBI Builder Software]. This software acts as a front-end to the FreeBSD ports tree, by doing port builds with a custom PREFIX for a PBI, and using "[http://wiki.pcbsd.org/index.php/PBI_Module_Builder_Guide modules]" which contain additional meta-data used in the creation of the resulting PBI file.

The most common way to build a PBI currently, is by using the [http://wiki.pcbsd.org/index.php/Using_the_PBI_Builder PBI Builder Software]. This software acts as a front-end to the FreeBSD ports tree, by doing port builds with a custom PREFIX for a PBI, and using "[http://wiki.pcbsd.org/index.php/PBI_Module_Builder_Guide modules]" which contain additional meta-data used in the creation of the resulting PBI file.

+

<!--T:35-->

One possible solution to improve upon this method, is by creating a new Mk/ entry, that will allow the usage of "make pbi" as a build option to ports. This make would then need to perform some steps to build a workable PBI file.

One possible solution to improve upon this method, is by creating a new Mk/ entry, that will allow the usage of "make pbi" as a build option to ports. This make would then need to perform some steps to build a workable PBI file.

−

1. Run a complete 'make' on the port with PREFIX set to a target location for this PBI. (Build port + dependencies)

+

<!--T:36-->

−

+

# Run a complete 'make' on the port with PREFIX set to a target location for this PBI. (Build port + dependencies)<br>

# Run 'pbi_create' on the resulting PREFIX that the port + dependencies were installed to, resulting in a finished PBI file.<br>

−

3. Run 'pbi_create' on the resulting PREFIX that the port + dependencies were installed to, resulting in a finished PBI file.

+

+

<!--T:37-->

While this portion of the build is fairly straight forward and should be easy, the difficult part will be in the inclusion of additional meta-data to the FreeBSD ports tree. Meta-data will be necessary for most applications, since the PBI manager will need to know which binaries need to be placed in the systems $PATH for execution, what mime-entries should be created, optional plugins / ports to include, rc scripts, etc.

While this portion of the build is fairly straight forward and should be easy, the difficult part will be in the inclusion of additional meta-data to the FreeBSD ports tree. Meta-data will be necessary for most applications, since the PBI manager will need to know which binaries need to be placed in the systems $PATH for execution, what mime-entries should be created, optional plugins / ports to include, rc scripts, etc.

The PBI Format for 9.0 and Beyond

This document is intended to serve as a place for brainstorming ideas to improve and enhance the PBI (Push Button Installer) format for PC-BSD® 9.0 and beyond.

What is the PBI format?

The idea behind creating the PBI format is that one of the things holding back mainstream adoption of open-source desktops is the package management format. With almost all open-source desktops, software is simply treated as part of your system itself. Thus when performing an update of some seemingly trivial application, you run the risk of potentially needing to upgrade other packages which in turn could break other critical parts of your desktop.

Example:

User wants to upgrade to latest version of FireFox. However, this in turn requires new version of libFoo*,

which is also used by Xorg and many other libs/apps. In order to perform update of libFoo, Xorg and any other

dependent libraries must be updated as well. Newer Xorg may in turn require different version of libOtherFoo and

so-on and so forth. All it takes is one failure at any point of the process, and the user may end up with a broken

system, no GUI and a huge loss of productive time.

While package management systems have gotten better at resolving these issues, trying to fix conflicts and prevent breakage before it occurs, it still does not fix the underlying problem, that every software package is simply a part of the system, and pulling one any one thread has the potential of causing a break somewhere farther down the line.

The PBI format tries to correct the underlying flaw in this system for desktop computing. Rather than making every application a part of your base system, PBIs are self-contained, including their own dependent library tree and related data. As a result, when you install a PBI there are no dependency issues to resolve, and applications can be added / removed freely, without fear of causing breakage to the desktop or any other installed software.

Why update the format?

The current PBI format, while still very useful is beginning to show its age. The format when started was a new concept for the open-source world and now, after many years of real-world usage, it is time to take what we've learned, refine and implement it in a cleaner way, for the benefit of both PC-BSD® and FreeBSD users.

The current PBI format

A .pbi file today is made up of several different components and organized as indicated below.

At the front of a PBI file is a small C binary, which makes the application executable, and starts the installation process.

Next is a small tarball which contains configuration meta-data, such as icon information, mime-data and install / uninstall scripts - all used by the GUI portion of the installer.

After the tarball, there is a location for an optional "icon" which can be used by thumbnailer utilities, to display the applications icon in file-managers.

Lastly is the program archive, which is made up of a LZMA compressed tarball for both maximum compression and fast extraction.

By using these components, the QT4 PBI installer is able to run, confirm some settings from the user, such as destination directory, and then extract the program archive to disk. The installer reads much of the included meta-data and uses it in the creation of XDG formatted desktop / menu icon creation, as well as registering mime types, etc.

New Goals

While the current format is useful and working in many ways, it is lacking some features and using some "hackish" methods to force programs (which often are not designed to be run self-contained) to work for end-users. Below are some of the ideas we want to implement in the new format to correct these issues, and make the format work on traditional FreeBSD.

Managed hard-links for resources

One of the drawbacks of the current implementation is its handling of libraries and resource files. With each program including its own complete set of libraries, there is often much duplication between applications, causing a waste of hard-disk space, as well as memory usage during runtime, since libraries are shared via inode lookups.

To improve upon this situation, a managed directory can be used for "sharing" the common files between applications, allowing less disk-space to be used and memory to be saved during runtime. This can be accomplished by creating hash-lists of files within a package and installing resources to a shared directory, creating hard-links back to the original file-names within the installed PBIs tree.

Pitfall: Different applications may include the same library with the same version (and sub-version and sub-sub...), but with different compile-time options or generated by different compilers. Thus either the PBI installer must compare all candidates byte-wise (making nonsense of the choice of LZMA for fast decompression) or (better?) a strong hash is needed for an abstract of the library, which must of course include all compile-time options. For C/C++, tools to extract or generate these informations exist, but care has to be taken for other major languages.

Create new PBI management system for add / remove of software

With the PBI format now, installation is performed by each application itself, using the included QT4-based binary installer. Other utilities for PBI management are also written in QT4, such as the creator / removal tool.

This should be changed, with a new suite of tools written in shell / C, that perform the actual installation, creation and removal portions. This would allow a variety of front-ends to be developed, and help break the PBI format from being more QT/KDE specific, allowing FreeBSD users to install support for the PBI format via ports. This would also
removed the need for a binary loader at the front of .pbi files, instead making the file-format look like this:

[meta-data]|[optional icon]|[program archive]

Possible names?:

ports/sysutils/pbimanager?

pbi_add / pbi_delete / pbi_create / pbi_info

Not very original, but easy to remember for those coming from a BSD background.

Explore the usage of $ORGIN as a replacement for LD_LIBRARY_PATH

One of the ways the current version of PBI handles program execution using self-contained libraries is by creating wrapper scripts for executable, and setting some variables such as LD_LIBRARY_PATH. While this works, it has a few problems in that it is A: Hack-ish and B: Causes breakage with setuid.

One possible solution would be with the usage of $ORGIN for the compiling of PBI applications. This feature could eliminate the idea of needing to use LD_LIBRARY_PATH and fix issues with using some setuid programs, such as VirtualBox in user-mode.

There is another way: Use same approach as used this util: http://www.freshports.org/devel/chrpath/
I tested such situation:
1. Build PBI to directory /usr/Programs/VERY-long-name-here-very-very-long-and-unique-and-fixed-length
2. For all executable files, change DT_RPATH with chrpath;
2. For all other binary files; find all byte positions of this PATH with strings, change for example PATH/etc to NEW_PATH/etc, and write back;
dd if=/file-with-zero of=$FILE offset=$offset-from-strings-output bs=1 count=strlength(PATH/ETC)
dd if=/file-with-new-path of=$FILE offset=$offset-from-strings-output bs=1 count=strlength(NEW_PATH/ETC)
3. Find all text files (.sh, .xml, etc.), change there PATHes too.

With such tweak scripts, I was able to build PBI to one PREFIX, and dynamically install it to other PREFIX. Program worked just fine. However, need more testing with some complex Programs, OpenOffice probably, etc.

Integrate PBI building closer with FreeBSD Ports

The most common way to build a PBI currently, is by using the PBI Builder Software. This software acts as a front-end to the FreeBSD ports tree, by doing port builds with a custom PREFIX for a PBI, and using "modules" which contain additional meta-data used in the creation of the resulting PBI file.

One possible solution to improve upon this method, is by creating a new Mk/ entry, that will allow the usage of "make pbi" as a build option to ports. This make would then need to perform some steps to build a workable PBI file.

Run a complete 'make' on the port with PREFIX set to a target location for this PBI. (Build port + dependencies)

Include any optional meta-data for desktop icons / mime registration

Run 'pbi_create' on the resulting PREFIX that the port + dependencies were installed to, resulting in a finished PBI file.

While this portion of the build is fairly straight forward and should be easy, the difficult part will be in the inclusion of additional meta-data to the FreeBSD ports tree. Meta-data will be necessary for most applications, since the PBI manager will need to know which binaries need to be placed in the systems $PATH for execution, what mime-entries should be created, optional plugins / ports to include, rc scripts, etc.