''.ipk'' packages can be built using the tools contained in <tt>ipkg-utils</tt>. You may download <tt>ipkg-utils</tt> from the [[http://downloads.linkstationwiki.net downloads area]] or from the [[ftp://ftp.handhelds.org/packages/ipkg-utils/ <tt>ipkg-utils</tt> home page]]. To ensure compatibility, packages which are to be submitted to the official LinkStationWiki ipkg feed should be built using the version of <tt>ipkg-utils</tt> available from the [[http://downloads.linkstationwiki.net downloads area]].

+

''.ipk'' packages can be built using the tools contained in <tt>ipkg-utils</tt>. You may download <tt>ipkg-utils</tt> from the [http://downloads.linkstationwiki.net downloads area] or from the [ftp://ftp.handhelds.org/packages/ipkg-utils/ <tt>ipkg-utils</tt> home page]. To ensure compatibility, packages which are to be submitted to the official LinkStationWiki ipkg feed should be built using the version of <tt>ipkg-utils</tt> available from the [http://downloads.linkstationwiki.net downloads area].

Most of the tools contained in <tt>ipkg-utils</tt>, such as <tt>ipkg-build</tt>, are shell scripts. Please note, however, that some of the tools require Python. Non of them comes with any documentation. From studying the source code it has been concluded that <tt>ipkg-utils</tt> contains the following tools:

Most of the tools contained in <tt>ipkg-utils</tt>, such as <tt>ipkg-build</tt>, are shell scripts. Please note, however, that some of the tools require Python. Non of them comes with any documentation. From studying the source code it has been concluded that <tt>ipkg-utils</tt> contains the following tools:

Revision as of 20:28, 22 July 2006

This article is based on the work of nix & timtimred from www.linkstationwiki.org

Overview

This article explains how to build an ipkg package. It is intended for software developers and package maintainers.

Two methods are described

The classic way, using ipkg-build from the ipkg-utils collection, and

using a wrapper tool ipkg-mk around ipkg-build

In addition, reference information on the package format and files is provided, as well as information on setting up an own feed.

Building an .ipk package - using ipkg-utils

ipkg-utils

.ipk packages can be built using the tools contained in ipkg-utils. You may download ipkg-utils from the downloads area or from the ipkg-utils home page. To ensure compatibility, packages which are to be submitted to the official LinkStationWiki ipkg feed should be built using the version of ipkg-utils available from the downloads area.

Most of the tools contained in ipkg-utils, such as ipkg-build, are shell scripts. Please note, however, that some of the tools require Python. Non of them comes with any documentation. From studying the source code it has been concluded that ipkg-utils contains the following tools:

ipkg-accept-incoming

Used for managing the handhelds.org feed (copies uploaded packages to the actual feed and triggers the re-building of the feed's package index).

ipkg-build

Builds a package from a prepared directory structure. This is the most important tool. It can run stand-alone and is a shell script.

ipkg-buildpackage

Builds a package from a source tarball.

ipkg-compare-indexes

Compares two Package index files of two different feed directories. For each different .ipk package version the packages' contents is also compared with ipkg-diff. Requires Python.

ipkg-compare-versions

compares version numbers?

ipkg-deb-build

build a Debian file format .ipk from a directory

ipkg-deb-unbuild

Unpacks (but not installs) a Debian file format .ipk

ipkg-diff

Compares two .ipk files

ipkg-extract-file

extracts a single file from an .ipk package

ipkg-ipk

Retired. Replaced by ipkg-build

ipkg-link

Links (and unlinks) ipkg meta data from some non-volatile storrage (e.g. flash memory) to the current file system. Probably only useful on systems where packages are installed in flash ram.

ipkg-make-familiar

('familiar' is a handhelds.org distribution for the iPAQ).

ipkg-make-index

Used for generating the meta-data needed for providing a feed. The script generates the necessary index files from a list of .ipkg packages. Requires Python.

ipkg-show-deps

Shows package dependencies, requires Python

ipkg-unbuild

Unpacks (but not installs) a package back into a directory structure.

ipkg-update-index

Used for providing the index files of a feed. Requires Python.

ipkg-upload

Uploads packages to the handhelds.org feed (don't use it to supply LS packages to handhelds.org, since handhelds.org doesn't distribute LS packages)

Using ipkg-build from ipkg-utils

The normal but tedious method for building an .ipk is to use ipkg-build. You must start by compiling the software package, either directly on the LinkStation or using the Projects/CrossToolchains cross-compilation toolchain(s). ipkg-build requires several things in order to properly build an .ipk:

A special meta-data file named control (more on this in the Packge structure section)

The input files must be structured in directories exactly as they should appear when installed. Which is a tedious task. See ipkg-mk below for an alternative.

The owner, group, and access permissions of the input files must be configured as they should appear when installed

Build System | In other context a build system is the software system, e.g. a Makefile system, for building some software. Here we use the term to indentify the computer and environment where the software and packages are constructed.

We will call this directory structure the prototype structure for the purpose of this document. The system (LinkStation) on which you want to install packages will be called the target system. The system on which you build the software and the package will be called the build system.

The user and group ownership as well as the access modes of all files, directories, devices, etc. in the prototype structure need to be set up exactly as they should later appear on the target system. ipkg-build allows to force all files to have the same user and group ownership (-o and -g option), but this does not help if files in the package should have different ownership (e.g. if some should belong to root, while others should belong to bin).

This procedure of setting up a prototype directory structure is rather inconvenient and error prone. Also - this is a prinicple limitation of the used format - everything which can not be archived by GNU tar or which can not be set up on the build system can't be packed. For example, if an installed file should belong to a group abc, but if there is no group abc on the build system, then the prototype can't be set up as desired. Either the build system first has to be changed (the group added, with exactly the group number as needed on the target system), or some post-install script to fix this after installation has to be added to the package. There is little chance to set up the prototype directory structure of a non-trivial package without root access on the build system. This is a severe disadvantage of ipkg, for example in a corporate environment, where getting root access is in general verboten.

Also, files accidentally left in the prototype directory do end up in the package and are installed, even if they were not intended for deployment at all. The only exception are backup files ending with '~', since ipkg-build can clean them up before packaging.

ipkg-mk (described below) is a tool to gain more control over the building of a package, than with ipkg-build. However, ipkg-mk can't fix the problem of having to have root access to build non-trivial packages. It also does not fix the problem of missing user or group definitions on the build system.

Fixing ipkg-build v1.7

ipkg-build in the ipkg-utils-1.7 has a minor bug when a tar archive is used as the outer wrapper (-c option). In this case ipkg-build does not observer the -o and -g options on the files wraped by the outer wrapper (it does observe them for the files in the inner wrapper). This can be fixed by changing line 240 from

ipkg-build also has a bug when the -o and -g options are used in the order -g group -o owner instead of -o owner -g group. In case of -g group -o owner the group setting is not observed. The bug is in the getopts case handling, but since it can be circumvented by using the options in the right order no patch has been developed.

Building an .ipk - using ipkg-mk on top of ipkg-build

Introduction

ipkg-mk is a script which takes a prototype package description as input, spools (creates) the necessary prototype directory structure, and use ipkg-build to construct a package from the spooled prototype directory structure.

The tool's prototype description format is a variant of the Unix System V Release 4 (SVR4) package management's prototype description format. By relying on an explicite prototype description file it is possible to

gain more control over the package structure (explicite descriptions instead of implicite file attributes in some directory structure),

to have the package structure in one place (instead of having to analyse a directory structure), and to

be able to version-control the package structure (since the prototype description file can and should be placed under version control).

ipkg-mk does not fix some of the principle problems of the ipkg package format. For example, it does not fix the problem of having to have root rights. In fact, ipkg-mk requires to be executed by root.

ipkg-mk comes with documentation (man pages) describing the input format in detail. Users familiar with building SVR4 packages should feel at home.

Obtaining and installing ipkg-mk

To build a package with ipkg-mk, ipkg-mk itself and ipkg-build need to be installed. You should also be able to format man pages in order to read the documentation. Desktop Linux versions do come with man by default. If you build on the LS you might want to install a package like awf (a formater) and awman (a man clone using awf).

So, currently the installation of ipkg-mk on the LS with OpenLink requires some manual work:

Create a prototype file, which describes the desired package layout (see below for an example)

Run ipkg-mk with the prototype file as input. This should best be automated with a build rule in a Makefile.

If you build directly on the LS, specify the -c option to ipkg-mk to use tar instead of ar for the outer wrapper of the package, since the LS' busybox ar can't create archives.

The result should be the desired package.
#ProtoExampleprototype files are rather simple. As an example, the following is the prototype file which is used to pack ipkg-mk v0.1-1 itself into an .ipk package:

Package structure / prototype directory structure

Overview

The data (files etc.) which should become part of the package must be arranged in the prototype directory structure so they can be packed with ipkg-build. For the sake of this document we will call the top-level directory of the prototype directoy structure ./spool. Building such a directory structure with a tool will be called spooling (ipkg-mk is such a spooling tool). If no such a tool is available, the prototype directory structure needs to be set up manually.

CONTROL Data

Below the spool directory needs to be a CONTROL directory (can also be called DEBIAN) with a couple of optional and required files:

All files in the CONTROL directory need to be readable. Scripts also need to be executable (mode 555 or better). It is possible to place additional, non-standard files in the CONTROL directory. For example the directory is a nice place for a copyright file - which then e.g. can be displayed by a simple preinst file at the beginning of an installation.

CONTROL/control file (mandatory)

Overview

Provisioning of a control file for an ipkg package is mandatory. The control file is an ASCII text file, consisting of key/value pairs, one per line. A key and a its value are separated by a column and a space ': '.

A couple of keys within a control file are mandatory, while others are optional. In addition, it is possible to introduce own keys. Such own keys are ignored by the ipkg client. However, since it is e.g. possible to read the control file from within a pre/post install/remove script, such keys can be used for package-specific configuration. Another usage for own keys is to add packaging information, like information from a version control system. It is recommended to name all such own keys starting with 'X-' to ensure they don't clash with predefined keys.

The control file needs to be placed in the CONTROL directory of the spool directory with read permissions. A clause like

i control

or

i control=source-of-the-control-file

in a prototype file for ipkg-mk will ensure this.

Mandatory information

This section lists the mandatory information (keys), which need to be provided in a control file. However, the list, particular the key descriptions, is likely to be slightly incorrect. The information has been taken from handhelds.org, and ammanded with real-world experience. Figuring out the exact meaning and purpose of a key will require some further study of the ipkg-cl source code.

Required entries in the control file are:

Package -

Name of the package. The name must match the regular expression [a-z0-9.+-]\+. Please note that uppercase letters are not permitted. Thus, e.g. package names like myNewSoftware are not permitted.

Version -

Version number, in the form [:epoch] version[-famfrevision][-revision]]

epoch
A timestamp, seconds since the epoch (details unknown)

version
should indicate the version number of the packaged software, and must at least consist of one digit and match the regular expression [a-zA-Z0-9.+]\+. Please note that letters (uppercase and lowercase) are permitted as part of the version "number".

frevision
Optional familiar revision ? -fam[0-9]\+ handhelds.org is unclear about the exact meaning of this component of the version information. familiar is a distribution for he iPAQ.

revision
Optional, indicates the package build revision. It should be incremented each time the same software is re-packaged. If version is stepped, revision can be reset. -[0-9]\+

Architecture

Architecture name. Possible names are:

all:Indicates a processor/architecture independent package, like documentation or shell scripts.

noarch:No particular architecture specified.

mipsel:MIPSel architecture (LinkStation II with MIPSel CPU)

powerpc:PowerPC architecture (LinkStation I with PowerPC CPU)

arm:indicates software for an ARM processor\\

(There is no ARM processor in the Buffalo LS product family. Valid names for HG, TS need to be determined)

Maintainer -

Typically the name and e-mail address of the maintainer of the package, the package builder (not neccessarily the author of the software)

Section -

(exact meaning needs to be clarified)

admin:?

base:?

comm:Some communication program?

editors:Some editor?

extras:?

graphics:?

libs:Libraries?

misc:?

net:Networking tool/program?

text:Text editor / text processing program?

web: Web server, web pages, tools?

x11:X11 or X11 program?

Priority -

(meaning needs to be clarified). Normally optional should be used.

required:A requred package (what's the difference to the Essential entry?)

standard:which standard?

important:?

optional:An optional package

extra:?

Description -

A textual description of the package contents. Should be limited to 80 chars per line. Lines can be continued on the next line, which then has to start with a space. Empty lines are indicated by a single '.' after the space.

Source -

A URL or a list of filenames pointing to the source code. (mandatory, but not mentioned at handhelds.org)

Optional Information

In addition to the mandatory information, at least the following keys are recognized by ipkg.

Depends -

Dependencies on other packages. A comma separated list of package names (Can names with versions be specified here, or just names?)

lots of unexplained parameters:
Many of these are NOT supposed to be used in a package control file. They are used internally in ipkg's package database - which happens to use an enhanced version of the control file

Scripts

Things common to all CONTROL Scripts

All CONTROL scripts are optional

A script, if it exists, should return 0 on success. The only exception is preinst. A non-zero return value from preinst will abort the installation. (what happens if the other scripts return a non-zero value?)

All scripts are allowed to assume that they are connected to a terminal. Which means for example, that all scripts are allowed to interact with the user, asking for data, and expecting a response.

However, user interaction should be avoided if possible to simplify package handling and automated installation.

The environment variable PKG_ROOT is set to the root of the package installation (-d/estipkg command line argument or dest entry in /etc/ipkg.conf). This allows to access package contents in the installed, or to-be-installed, location.

All CONTROL files (scripts, data files, additional non-standard files) are extracted to the same temporary directory. Therefore it is possible to find other CONTROL files from within a script by obtainig the temporary directory as:

controlDir=`dirname "$0"`

All scripts are called with one or more arguments indicating the actual reason for why the script is called. This can be used to only provide one single script (under different symbolic names), which dispatches the actual operation according to the command line argument. This makes sense if all scripts would contain a lot of duplicated code. However prerm and postrm us the same argument and can't be distinguished just from the argument:

If the script returns a non-zero exit status then the installation of the package is aborted.

CONTROL/postinst Script (optional)

If the script exists it is called with the argument configure after the installation of the package, or when

ipkg configure packagename

was run.

CONTROL/prerm Script (optional)

If the script exists it is called with the argument remove prior to removal of the package.

Example of things which can (should) be implemented here are:

Stopping of any related processes, before the binaries are removed

Unloading modules (if relevant)

Backup of user-created data or other valuable data

CONTROL/postrm Script (optional)

If exists, called with the argument remove after the removal of the package.

Things which can/should be implemented here:

Restoring of files backed-up prior to the installation of the package.

Removal of resources allocated specifically for the package (e.g. users or groups created just for the package), or directories exclusively used by the package.

CONTROL/conffiles file (optional)

conffiles contains a list of configuration files (one per line), which are part of the package. A file mentioned in conffiles is not just removed/overriden by a new package installation. Instead the user is asked about what to do with the existing file.

Package Data Directories

The package data directories hold the actual files which should be installed on the target system:

./spool/ -+- CONTROL/ --- <control files>
|
+- <package data dirs/>

The structure of the package data directories must match the stucture of the files as intended on the target system. For example, if an application consisting of a binary, a man page, a shared library and a configuration file should be installed, the structure might look like:

The above structure is just an example. It should not be blindly copied for a real-world package. Instead the package layout, and thus the structure on the target system should be carefully planed. Things to consider include:

Should the application files be distributed into the global bin, man, lib directories (as shown in the example), or should the application files be kept in an own subdirectory structure (/opt/<app>/[bin|lib|man|etc])?

Which user and group ownerships should the files get?

Which file access modes should the files get?

Particularly, security issues should be considered here.

All this has already to be set up in the prototype directory structure. For example the ./spool/usr/bin/app application in the example should already be set to be readable and executable (e.g. mode 555) and to user:group bin:bin in the prototype directory structure.

Manually setting up the prototype directory structure is a tedious task. It should best be automated with some add-hock script or a dedicated tool like ipkg-mk, integrated into a make-based build system.

Converting the prototype structure to the package

The above described directory structure is converted into the package. You don't have to assemble the package manually (e.g. ipkg-build does it for you), but it is helpful to know the structure of a package when something doesn't work.

The structure is as it follows. The exact format is not specified anywhere, but the following is known to work.

The files in the CONTROL directory are packed into a gziped tar archive called control.tar.gz

All other files and directories are packed into a gziped tar archive called data.tar.gz

control.tar.gz and data.tar.gz, plus a file called debian-binary, are either packed in yet another tar file, or in an ar file. This is called the outer wrapper and forms the final .ipk package. The suffix .ipk is used intead of that of a gziped tar archive (.tar.gz) or an ar archive (.a).

debian-binary just contains the text 2.0, to indicate the package is also compatible with the debin package management format 2.0. The file is ignored by ipkg.

If the package contains special files like character or block special files (devices) then data.tar.gz needs to be packed with a version of tar which can pack such special files. GNU tar is such a version. Many tar's of classic Unix systems can't do this.

All in all the package structure is relative simple. It should even be possible to create such a package manually in case of emergency.. There are indications that slight variants, e.g. using uncompressed, instead of compressed tar files also work.

The package-build environment

this should probably become an own page - hey, this is a wiki :-)

Setting up your package build-environment directly on the LS

This paragraph is currently a stub.

Feel free to add content to it.

stub Hints

If you build packages on the LS be aware that the LS comes with the busybox ar archive program. That program is not capable of creating archives, it can just extract files from them. Therefore, the normal ipkg-build program will fail on the LS, unless you do one of the following:

Use the -c (lowercase 'c', not the uppercase 'C' option). This tells ipkg-build (and imkg-mk) to wrap the package up in a gziped tar file instead of an ar file. This is the easiest way to fix the problem, and the recommended way. gziped tar files are allowed as ipk files. In fact, for some time this was the default wrapper instead of ar.

Install a complete ar

Setting up your cross-package build-environment on Unix/Linux

stub - need the GNU version of tar and GNU gzip on a standard Unix system. Linux of course already comes with them

Setting up your cross-package build-environment on Windows

You need a Unix-like environment, like [| Cygwin], since the build tools are shell scripts and use other Unix tools like tar, gzip, and ar. Of course, you also need a cross-compilation environment to first build the software you want to package.

Once you have set up your cygwin environment you can follow the procedures for cross-package building under Unix/Linux.

Sample CONTROL/control files

A sample architecture-independen ipkg CONTROL/control file

Architecture independent packages are packages which e.g. control scripts which run on "any" architecture or documentation which is not architecture specific.

A sample MIPSel ipkg CONTROL/control file

A sample TeraStation ipkg CONTROL/control file

Automating the packaging with make

First of all, you should of course obtain a version of make(1). It is provided via the LinkStationWiki feed. So, if you have a working ipkg client setup, make can be installed with:

ipkg install make

The package does not contain documentation. Since the make in the package is actually a GNU make, you can find an online version of the documentation at [[1]]. If you have never worked with make it is highly recommended to study the documentation intensively (also, O'Reilly offers an nice book about GNU make).

make uses so called makefiles as input. The following is an example for a Makefile using ipkg-build to buld a package. Note the long and tedious setup of the portotype directory in the $(IPK) rule:

#Feeds Provide an anchor allowing to reference to the following section

Feeds

Once you have packaged your software, you might want to consider to make it available to others by providing it via a feed. You can either try to submit it to an existing feed (recommendet), or you could set up your own feed. Both alternatives are discussed in the following sections.

Submitting your package

There is no standard mechanism to submit a package to a particular feed. Each feed operator has set up his/her own mechnism and rules. Articles/GeneralIpkgFeeds contains a list of known feeds and pointers to the operators' submission guidelines.

Setting up your own Feed

Before seting up an own feed, consider instead to submit it to an existing feed. Also, consider that operating a useful feed requires resources and regular maintenance.

Setting up an own feed is rather simple. Actually, people often operate several feeds, e.g. a stable and an unstable feed, and feeds for different CPU architectures at the same time. Setting up a feed works as it follows:

First some ftp or web space is needed. Then:

All .ipk files for one feed should go into one directory on that ftp or web space. Typically for a LinkStation feed the directory name should be that of the particular processor architecture, e.g. mipsel, or ppc.

Further it is common to provide a stable feed for trusted and tried packages, and an unstable feed for less tested (freshly uploaded) packages.

For each feed an index file called Packages needs to be created from the .ipk packages (it can also be gziped and called Packages.gz). The file contains the control information from each package in a feed. The script ipkg-make-index from the ipkg-utils can be used to build this meta-data.

Announce the feed, so people can add the feed to their /etc/ipkg.conf configuration file.

Typically, an unstable feed is supposed to contain packages which have not been verified at all, or not extensively verified. A stable feed should contain only verified packages. Package verification should include some security check.