Autopackage Packagers Guide

Mike Hearn

Curtis Knight

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".

This documentation was built on Wed Jun 15 19:56:37 EDT 2005 for autopackage version: 1.0.3

Autopackage Developers Guide Multiple HTML Pages

Autopackage Developers Guide Single HTML Page

Autopackage API Documents


Table of Contents

1. Introduction to the framework
Basic concepts
Interface versions
Skeleton files
Names and naming
2. Creating a package
Choosing a root name
The specfile format
A quick example
Meta
Description
BuildPrepare
BuildUnprepare
Imports
Prepare
Install
Uninstall
Scripting Variables
Common Variables
Build Variables
Install Variables
Using the makeinstaller program
Sealed packages
3. Writing Skeleton Files
What are they and how do they work?
Producing skeleton files
4. Environment Variables
Build Environment Variables
Installation Environment Variables
Environment State Files
5. Prefixing
What is prefixing?
So why isn't this done already
What about the FHS?
The solution
Exception: KDE
6. Autopackage Root Names & Versioning
Dependency checking and resolution
What is a root name?
Versioning
How to integrate this with skeletons
Example
How to interface with require()
requireExact()
requireAtLeast()
More on skeletons
Automatic dependency retrieval
Summary
7. Binary portability
Symbol Versions
Symbolic collisions
Bogus dependencies
Headers
Exception handling internals
C++
Index
Glossary

List of Tables

2.1. Standard Meta Keys
2.2. APIs provided for file types
2.3. Common Variables
2.4. Build Variables
2.5. Install Variables
4.1. Build Environment Variables
4.2. Installation Environment Variables
6.1. Software & Interface Number
6.2. Software & Interface number - Complex
6.3. Software & Interface Number with Skeleton Filename
6.4. Data Definitions between Package Repository File and .apspec File

Chapter 1. Introduction to the framework

Autopackage is a framework for developers that lets them build easy to use installers for their software. It provides an API which allows delegation many things to the framework - for instance, installing menu entries, copying files, uninstalling the software, and providing feedback to the users while it is installing.

As well as providing some useful utility functions for your installers, autopackage acts as a distribution abstraction . Packages built with autopackage are designed to be distribution neutral. The underlying framework and tools take care of the differences between the systems your software is being installed on.

An autopackage is typically a self extracting shell script with a .package extension. It will first check that the autopackage tools are present. If not, then it will automatically fetch them, install them, and then proceed with the installation. It will also download the graphical frontend that best matches the users desktop.

Basic concepts

Autopackage deals with dependencies by checking the system directly for the components needed. Rather than depending on a particular version of a particular piece of software, autopackages depend on an implementation of a particular interface being present. Interfaces are abstract things - a shared library exports an interface (or more commonly, many) but an interface can also be for instance the presence of certain files in certain locations, command line arguments given, protocols between system components and so on.

The first thing to realize then is the need to tell autopackage which interfaces your package needs, and then the system tries to figure out the "best" package that can satisfy that interface, typically the most recent compatible version of that package (but an interface doesn't have to correspond one to one with a particular piece of software).

Interface versions

Autopackage abstracts interfaces behind interface versionnumbers. These take the form of two integers written as A.B, and they work similar to the libtool versions many of us are used to. The A number is the major number and is incremented when a breaking change is made to an interface (a function is removed, renamed etc). When A is incremented, B is set back to zero. B is the minor number and is incremented when an interface is added. This is in contrast to the software version, which is pretty much freeform and is usually set to the version number chosen by the software authors.

Note that interface versions don't need to bear any resemblance to the version numbers software authors assign their creations (though they can do, if that's convenient). WhizzBang 2003 may well have an interface version of 0.35, or 4.1 or whatever. On the other hand, GTK+for instance follows the libtool versioning style - every release in the 2.x series is backwards compatible. Because it also uses kernel versioning though, each release increments the minor number by two, not by one. In this case, the interface version can be equal to the software version - the meaning is still close enough that everything will work OK.

Skeleton files

The mapping from the state of the system to an interface version (or set of interface versions) is determined by the skeleton filewhich encapsulates that dependency. Skeleton files contain a small amount of metadata about a dependency, and they are used by packages to make dependency detection and resolution automatic. Typically for each dependency a package has, there must be a corresponding skeleton file - if one doesn't exist, it should be created and then let the maintainers of the software it represents know of its existence. Don't worry though, creating skeleton files is very easy.

Autopackage comes with a collection of useful skeleton files for common dependencies that can be used immediately in your own packages. If a skeleton is created, please forward the skeleton to us so that it can be in the next release.

Names and naming

Autopackage is designed to be a decentralized system. For this reason, it leverages off of the DNS network to uniquely identify things. The basic unit of currency is called a root name, and looks a bit like this: @foobar.org/frob:2000:3 . Root names have several components:

  • The path, this consists of the @foobar.org/frob part
  • The software version number, which comes after the first colon (2000 in this case)
  • The package release number, which comes after the second colon.

Warning

A root name should not be chosen based on where the package will be hosted, who built it, or anything other than whoever owns the software. Root names are designed to help reduce name conflicts without the need for yet another name "ownership" registry. Typically, if the packager is not the maintainer of the software that is being packaged, the true maintainer should be at the very least emailed to notify them that this root name has been allocated for them. Ideally of course the packager is the enlightened maintainer and wishes to build binary packages so this is not an issue.

Packages (but not skeletons) can optionally have a short name. Short names are more convenient to use than root names, and are more familiar to Linux users. They look like "frozen-bubble", "libxml" or "gnome-panel". Typically, these are the names users will use from the command line. Try and keep them unique if possible, but it's not a disaster if there are conflicts.

Finally, packages and skeletons should be given a display name. This is a human readable string, that can be optionally localized. It will be used in user interfaces where possible, and should take the form of "Product-Name Purpose", for instance "Sound Juicer CD Ripper", "Mozilla Web Browser", or "GNU Emacs Text Editor".

Chapter 2. Creating a package

Constructing an autopackage is fairly straightforward, and uses a system similar to RPM, namely the use of a specfile. This file contains most of the information needed for building the package file. The other pieces of information are pulled from skeleton files as needed.

Note

Use makeinstaller --mkspec >whatever.apspec.in as a fast way to generate a template specfile that can be customized for your own packages.

Choosing a root name

Typically, choosing a root name will be one of the first things done while building a package. The path chosen should be controlled by the maintainer of the software , even if they aren't involved in the construction of the package (though really they should be). The path doesn't even have to exist in real life, autopackage will not attempt to resolve the path given in the root name, it's there purely for namespacing reasons.

A root name should consist of an @ symbol, then a valid URI path (without the protocol) part controlled by the maintainer of the software: they should have write access to whatever server it points to, for instance. It should not be related to whoever is packaging it - always notify the maintainer of what root name was selected.

A root name must have at least one path element. The root name can not just be a DNS address on its own, for instance "@myproject.org/myproject" is valid, but "@myproject.org" is not.

Examples are: "@purity.sourceforge.net/purity", "@webspace.myisp.com/~myname/random/software/xfoo", and so on.

The specfile format

Spec files resemble the Windows INI file format of yore, now used also in .desktop files. Typically each section is denoted with a header that looks like this: [Section-Name] . Specfiles usually have a .apspec extension, and are placed in a subdirectory of the projects source tree called autopackage . This directory contains one spec file for each package, in contrast to RPM which allows creation of multiple packages from the same spec. By convention, the primary package has a filename of default.apspec , and the makeinstaller program will use this if no specfile is specified. Use the autopackage directory to store other useful things, for instance patches.

Warning

Applying patches to the source tree is an indication that something is wrong. Typically the person who should be making an autopackage should be the maintainer of the software themselves, so external patches are not relevant. This is the best arrangement because that way users are guaranteed that source code releases are accompanied by binary releases - there is no need for them to hang around waiting for somebody to rebuild the package for them.

Note

If using GNU autoconf, then call the specfile something.apspec.in and makeinstaller will automatically run config.status to update the real specfile.

A quick example

This is a specfile for the GnuTLS library. It's fairly typical. Examine it for a while, it is straight forward to understand.

                    
[Meta]
RootName: @gnutls.org/gnutls:$SOFTWAREVERSION
ShortName: gnutls
SoftwareVersion: @VERSION@
DisplayName: GNU Transport Layer Security
Summary: Provides support for the TLS/SSL protocols
Maintainer: Nikos Mavroyanopoulos <nmav@hellug.gr>
Packager: Mike Hearn <mike@theoretic.com>
InterfaceVersion: 10.3
Language: de es fr
DisplayName[de]: Gnu Transport-Schicht-Sicherheit
Summary[de]: Gibt Unterstützung für die TLS/SSL Protokolle.
DisplayName[es]: Seguridad De la Capa De Transporte del Gnu
Summary[es]: Proporciona la ayuda para los protocolos de TLS/SSL.
DisplayName[fr]: Gnu Transport-Schicht-Sicherheit
Summary[fr]: Fournit l'appui pour les protocoles de TLS/SSL.

[Description]
This is a TLS (Transport Layer Security) 1.0 and SSL (Secure Sockets Layer) 3.0
implementation for the GNU project.

[BuildPrepare]
prepareBuild --with-included-libtasn1 --disable-openpgp-authentication

[BuildUnprepare]
unprepareBuild

[Globals]
export sover="10.3.8"

[Imports]
import <<EOF
lib/libgnutls.so.$sover
lib/libgnutls-extra.so.$sover
EOF

[Prepare]
require @gnupg.org/libgcrypt 7.1
require @gnupg.org/libgpg-error 0.0

[Install]
installLib libgnutls.so.$sover libgnutls-extra.so.$sover

[Uninstall]
uninstallFromLog

Meta

This section consists of a number of keys, of the form "Key: Value". Most of these should be obvious from the discussion of basic concepts - this is where short name, software version, interface version(optional), display nameand so on are defined. Translated versions can be provided of DisplayName and Summary by appending [langcode] to the key names. The 'Scripting Key' variables are in addition to the common variables listed in the scripting variables section.

Some variables can be used as values of these keys. The $SOFTWAREVERSION is embedded into the root name - recall that root names can have the software version as a component.

Table 2.1. Standard Meta Keys

Meta Key Scripting Key Description Value Type REQ? Example
AutopackageTarget $AUTOPACKAGETARGET

Minimum targetted API version of autopackage for which the generated package requires.

numeric Yes 1.0
CPUArchitectures $CPUARCHITECTURES

The CPU architectures to be generated from the source.

string Yes x86
DisplayName[xx] $DISPLAYNAME

"Product-Name Purpose" of the package, for instance "Sound Juicer CD Ripper", "Mozilla Web Browser", or "GNU Emacs Text Editor".

Substitute a particular language code for [xx] like "DisplayName[fr]" or "DisplayName[de]". English text should not have any language declaration.

string Yes

DisplayName: Autopackage GTK+ Graphical User Interface

DisplayName[fr]: Autopackage GTK+ Frontend Graphique

InterfaceVersion $INTERFACEVERSION

Abstraction of the interface the package exposes represented as "A.B". Not useful for, say, applications or themes that do not expose an interface.

numeric No 1.0
License $LICENSE

License of the package.

string No

GNU General Public License, Version 2

GNU Lesser General Public License, Version 2.1

Apache License, Version 2.0

Public Domain

Maintainer $MAINTAINER

The name and email of the software's author or a group.

string Yes

Joe Developer <joe@developer.net>

The FooBar Development Team <development@foobar.org>

Packager $PACKAGER

Name of the person who created this package. Usually equal to the value of the Maintainer key.

string Yes Joe Packager <joe@packager.net>
PackageVersion $PACKAGEVERSION

Version of the package.

numeric No 1
RootInstallOnly $ROOTINSTALLONLY

Forces package installation to the global database which makes it available system-wide. Requires user to have super user (root) password.

boolean No Yes
RootName $ROOTNAME

Identifier for the package which is composed of package owner, software version and package release version.

string Yes @foobar.org/frob:2000:3
ShortName $SHORTNAME

Short names are more convenient to use than root names, and are more familiar to Linux users. They look like "frozen-bubble", "libxml" or "gnome-panel". Typically, these are the names users will use from the command line.

string Yes autopackage-gtk
SoftwareVersion $SOFTWAREVERSION

Version of the software usually set by the software authors.

numeric Yes 2.6
Summary[xx] $SUMMARY

A short, one-line description of the package.

Substitute a particular language code for [xx] like "Summary[fr]" or "Summary[de]". English text should not have any language declaration.

string Yes

Summary: A graphical (GTK+) frontend for installing Autopackage packages.

Summary[fr]: Un frontend (GTK+) graphique pour installer des paquets d'Autopackage.

URL $URL

Home location that relates to the package.

string No http://organization.org/application-page.html

Description

This section should contain an English description of the package (in non technical terms please if at all possible). Non-English versions should go into the Description:xx sections, where xx is the language code. A good description will provide a brief overview of the purpose of the software and it's capabilities.

BuildPrepare

This section contains a bash script that will be used to prepare (build/compile) the sources for import. Import is the process whereby the files are selected and placed into the package archive. Preparation gives the chance to do things like run configure scripts, Makefiles and other such things if the project requires them.

If your project is using GNU autoconf/automake, calling prepareBuild() function will usually do the trick nicely. It simply reconfigures the tree to a temporary prefix, then does the make, make install cycle. Please note that your software must be relocatable; refer to paragraph 3 for more information.

This section is optional but recommended. A warning will be displayed if it is missed.

BuildUnprepare

This section is run after Imports and allows for any tidying up. This section is optional. Use the unprepareBuild() function to reset the source tree back to its original configuration, assuming prepareBuild() was used.

Imports

This script is used to select files that you want to put inside the package. You use the import command to import files in your package.

The import command is invoked like this:

echo (filenames...) | import

import reads filenames from STDIN. Each file must be seperated by a newline. The command also accepts wildcards, so you can also type:

echo '*' | import

The current working directory is the build root directory ( $build_root ). If you used prepareBuild , then all built files are automatically installed to this build root (by using 'make install'), so the only thing you have to do is to call echo '*' | import .

If you're not using prepareBuild , then there are two things you can do:

  1. Invoke a build system command similar to 'make install'. Make sure the files are installed to $build_root . After that, you can simple call echo '*' | import .
  2. The files that are built during the build process are probably in the source directory. You can manually import files from that directory. The location of the source directory is specified in $source_dir .

    For example, your project is based on a simple make-based build system. Make compiles superpig.c to the binary superpig . Your source directory also has wai.png , a data file used by superpig . You write this in your specfile:

                            
    import <<EOF
    $source_dir/superpig
    $source_dir/wai.png
    EOF
    

Prepare

The prep script is the first part of any package. It is responsible for "preparing" the package, which basically means ensuring all the dependencies are present, making any decisions about what to install or not install, and so on. No installation of any files occur here.

Dependencies are best handled with the require function, although they don't have to be if your needs are quite specialist:

require @gnome.org/libxml/2.0

In this case, that's all we have to do. The correct skeleton file will automatically be selected and copied into the package by the makeinstaller script. Later, when this line of script is run, the require function will run the script associated with the skeleton file and handle the error if the dependency fails. If the check passes, the script will continue.

There are several different ways of expressing dependencies. Require is the most common but there is also recommend . Recommending a dependency means autopackage will check for it, and try to install it if it's missing, but if it can't this will not cause the function to fail. Instead, the dependency will be listed in the summary screen to inform the users that if they had this library, new functionality would be available. An easy way to make something recommended rather than required is to use dlopen, and an easy way to use dlopen is to use relaytool.

You can also use the requireAtLeast and requireExact functions if you rely on implementation details of a dependency (for instance, maybe you need a bugfix). These only work if SOFTWARE_VERSIONS is set though, and not all skeletons set this. So check before using them. Both of these functions have recommend equivalents.

Note

The prep and install scripts run in +e mode, which means that any line which returns a non-zero exit code will terminate the script . If running a command might fail but the script should carry on regardless, stick || true on the end of it.

Install

The install section is the meat of your package. It consists of a series of calls to the autopackage API, in order to affect the users system in some fashion.

In theory you can do anything you like in these scripts, including using regular shell commands like "cp" or "mv". In practice, where autopackage provides a replacement command, you should use that. For instance, use copyFiles instead of cp. The builtin versions have advantages over the raw shell commands, for instance using copyFiles will register the files in the log so they will be automatically uninstalled, and they will appear in the file list. Whilst copyFiles and copyFile are very generic, for some types of file there are more specific functions available. Here is a short list:

Table 2.2. APIs provided for file types

File type API used
Executable programs (ELF/scripts) installExe
Public ELF shared libraries installLib
Architecture-neutral data (share) installData
Man pages installMan
.desktop files for menu entries installDesktop
.desktop files for MIME type associations but not menus installMimeDesktop
Translations installLocale
MIME type sniffing/matching templates installMime
Icons installIcon
TeXInfo files installInfo
App specific config files installConfig
GConf schemas installGConfSchema

In the install section, like in the Prepare section, any command that fails will abort the install. That doesn't apply inside if statements of course.

Uninstall

This is usually easy - just a call to uninstallFromLog will suffice, as this runs the uninstall script that was generated automatically by the installer functions, and will automatically remove any files copied/any directories created etc. If any actions was done without using the autopackage APIs, now is a good time to undo them. Remember - forgive the user . They may well have screwed around with the install, so if custom actions are included, check that they make sense (IE the file exists before trying to delete it).

Scripting Variables

The following is an alphabetical list of variables that are available for scripting use.

Common Variables

The following is an alphabetical list of variables that are available for all scripting use. The variables are in addition to the 'Scripting Key' variables listed with the Standard Meta keys.

Table 2.3. Common Variables

Name Description Value Type Example
BUILDHOST

The host machine from which the package was generated.

string build-machine.organization.org
CPUARCHITECTURE

The CPU architecture of the installing or installed package.

string x86
DATASIZE

Size in bytes of the package's compressed payload.

numeric 654321
DESCRIPTION

Text that describes the package contents. Typically formatted to be a fixed column width.

string The GIMP (GNU Image Manipulation Program) is a powerful image composition and editing program, which can be extremely useful for creating logos and other graphics for Web pages. The GIMP has many of the tools and filters...

Formatted to a fixed width like 69.

DISTRIBUTIONCODENAME

The distribution code name of the target machine.

string Community
DISTRIBUTIONDESCRIPTION

The distribution description of the target machine.

string Mandrakelinux
DISTRIBUTIONID

The distribution id of the target machine.

string mandrakelinux
DISTRIBUTIONMANAGER

The distribution manager of the target machine.

string urpmi
DISTRIBUTIONMANAGERGRAPHICAL

The graphic distribution manager of the target machine.

string urpmi
DISTRIBUTIONRELEASE

The distribution release of the target machine.

string 10.1
EXPANDSIZE

The size in bytes of the expanded (uncompressed) full package.

numeric 7654321
FILETOTAL

The total file count within the package.

numeric 87
LSBVERSION

The lsb version of the target machine.

numeric 2.0
METASIZE

The size in bytes of the package's compressed meta data.

numeric 4321
PACKAGELANGUAGES

The generated list of languages that are specifically supported within the package.

string de en fr nl
PREFIX

The installation prefix where the package was installed.

string /home/user/.local
TIMESTAMPBUILD

The universal coordinated time ISO-8601 date from when the package was generated.

string 2004-10-09T01:12:59Z

Build Variables

The following is an alphabetical list of variables that are available for use in BuildPrepare, BuildUnprepare, and Imports functions along with the Common Variables.

Table 2.4. Build Variables

Name Description Value Type Example
APBUILD_BOGUS_DEPS

Specify a list of whitespace-separated bogus library dependencies. These libraries will not be linked against. This option is useful when the automatic bogus dependency detector doesn't detect bogus dependencies correctly.

string X11 ICE png
APBUILD_CC

Use this compiler instead of gcc.

string Default value: "gcc" or "g++"
APBUILD_DISABLE_BOGUS_DETECTOR

Disable the automatic bogus dependency detector. This is useful when linking to libraries don't have correct DT_NEEDED entries, like GTK 1.2.

string
APBUILD_NO_STATIC_X

Do not force static linking to certain X libraries.

All the X libraries we force static linking to are X extensions. Some distributions don't come with dynamic library versions of those libraries. The static version of all those libraries are all less than 70 KB (the biggest one is libXi, which is 61 KB; everything else is around 15 KB), so rest assured that it won't bloat your application.

string
APBUILD_PATH

Use this as the include dir for the apbuild headers.

string Default value: "/usr/local/include/apbuild"
APBUILD_STATIC

Specify a list of whitespace-separated libraries to statically link to (like: popt z). A filename can be explicitly specified to the static library.

string popt=/usr/lib/libpopt.a
APKG_BUILD_ROOT

If set, the user is forcing the root (base) directory where the package will be built.

string
AUTOPACKAGE_SKELETON_DIRS

If set, the user is forcing additional directories to be included when searching for skeleton files.

string
prefix

If set, the user is forcing the path to which the software will be installed.

string

Install Variables

The following is an alphabetical list of variables that are available for use in Globals, Prepare, Install, and Uninstall functions along with the Common Variables.

Table 2.5. Install Variables

Name Description Value Type Example
BACKUP

The directory where backup files are written relative to $PREFIX.

string @autopackage.org/tests/foobar:1.2.3/backup
DATABASE

The name of the installation database either "User" or "Global".

string User
INSTALLSIZE

The size in bytes of the package's installed files.

numeric 54321
PACKAGELOCATION

The directory location of the installation which is determined when the environment file is loaded. Used to pass base installation directory information to other functions and scripts.

string /home/user/.local/autopackage-gtk
TIMESTAMPINSTALL

The universal coordinated time ISO-8601 date from when the package was installed.

string 2004-10-09T01:12:59Z

Using the makeinstaller program

Once the specfile is written, run the makeinstaller program to generate the output .package file. Remember run this from the root of your source tree, so not in the autopackage/ directory itself.

With no parameters, makeinstaller will use the file named autopackage/default.apspec , and output the resulting package in the directory it was run from. To override this choice (because there are perhaps several possible packages to build) specify the names on the command line: for instance to build both "default" and "debug", try: makeinstaller default debug .

makeinstaller can operate in one of several modes. In the default mode it will output a self contained .package file that can be run by the user to install the software. In "split" mode it will produce two files, that end in .package.meta and .package.payload respectively, that can be placed on a web server and can be used for dependency resolution. In "both" mode it will produce a standard .package file AND the meta/payload files as well.

Sealed packages

A sealed package is one in which dependencies are placed within the package such that no other packages are required to complete the installation. Currently there is no mechanism to tell autopackage to not download anything to complete the installation. The distinction is that if a dependency is not found within the sealed package, the support code will then try to contact and resolve the dependency. For example, if only the support code is included in the sealed package, the support code would try to download and install the gtk front end automatically. So it is recommended that you include autopackage-gtk package because the support code will try to install the gtk front end during the support code installation.

The dependency packages must be located in the source tree within the `autopackage/packages' directory. The makeinstaller script will detect the packages directory and move the contents into the package.

Looking at inkscape for an example to create a sealed installer:

                
Prepare section in the apspec file states:
require @gtk.org/gtk 2.4
require @gnome.org/libxml 2.0
require @libsigc.sourceforge.net/libsigc 3
require @libpng.org/libpng 3
require @gtkmm.org/gtkmm2 3
require @zlib.org/zlib 1

Add to `autopackage/packages' directory by downloading the available autopackage of the applications for the interface required or you need to build the package and skeleton file. Do not forget to download and add the support code and autopackage-gtk package also.

                
@gtk.org/gtk 2.4 --> core component not packaged
@gnome.org/libxml 2.0 --> core component not packaged
@libsigc.sourceforge.net/libsigc 3 --> libsigc++-2.0.3.x86.package
@libpng.org/libpng 3 --> core component not packaged
@gtkmm.org/gtkmm2 3 --> glibmm-2.4.4.x86.package
@zlib.org/zlib 1 --> core component not packaged 
autopackage.tar.bz2 --> most recently released archive 
autopackage-gtk-1.0.x86.package --> most recently released package

At the moment there is no difference between a sealed installer versus including the same files in the directory where the main application package is located. In the future there will a more differences in using a sealed installer such as installing a private autopackage support code for the application which does not touch the rest of the system. The private copy of the support code would be removed along with its application.

Chapter 3. Writing Skeleton Files

What are they and how do they work?

One of the key problems with producing multi distro binary packages is determining exactly what the user has on their system at install time. The approach autopackage takes is similar to how autoconf works: it has a extendable collection of pre-written checks that inspect the state of the system and figure out which interfaces it supports. The code to do this checking and examining is contained within skeleton files, so called because they do not wrap or package actual software, but rather they just look for the "bones" around which real software is built.

A skeleton file is just a text file that looks, at first glance, a little bit like a specfile. It uses the same INI format, and they both have a [Meta] section at the top. However, they serve different purposes and therefore have different sections providing the content.

Skeletons are responsible for two different tasks. The first is to compile a list of interface versions by testing the system, usually using autopackage-provided APIs to make it easy. Optionally, skeletons can also produce a list of software versions . The difference between interface versions and software versions is described in the first chapter. The second task is to try and install an implementation satisfying the requested interface version, if it's missing. This is the dependency resolution part of autopackage. Most of the time, it's entirely automatic and you don't have to do anything in your skeleton to enable this, as the relevent section is filled in with sensible defaults for you. But, if you want to provide custom behaviour for dependency resolution you can do so.

Skeletons also provide some basic metadata. This includes the root name, a display name (which is what the user sees in the "Checking for ....." lines), and a short name. Skeletons can also be versioned themselves: this is an optional convenience and doesn't have to be used, but when it is the version number is a single incrementing integer that is unique to the skeleton file itself.

Perhaps the most important bit of metadata is the [Notes] section. This isn't parsed or used by autopackage itself, it's there purely so people using the skeleton can understand how it works. This is necessary because there is no fixed way of mapping system state to a series of interface versions. The way it's done varies between libraries, as every dependency has its own unique history of versioning. Some, like GTK+, are extremely simple to map to interface versions as they have solid and sensible backgrounds. Others are more complex: they may have forked, some users may have CVS builds, in other cases it's just plain old hard work to find out exactly what interface is being provided. The way the mapping takes place is described here, and package authors should be able to read it to find out what numbers to pass to require() or recommend().

Producing skeleton files

Generally speaking anybody can write skeleton files, but like with packages themselves the best people to do it are the developers of the library themselves. Only they know the full versioning history of their product. Even if they don't want to write it themselves, it's a good idea to check with them that you did it correctly once completed.

The first thing is to allocate the software a root name, if they don't already have one. It is of vital important that once you've done this, you send a mail to the developers list or maintainers explaining what you've done. This way, if two or more people independently try to write skeletons or packages, there is a central meeting point where duplication of effort can be avoided. It's vital that the same piece of software does not have two root names.

Next, add a [Meta] section like the following and adapt as needed:

                
[Meta]
RootName: @foobar.org/foo
DisplayName: The foobar widget is a world changing program!
ShortName: foobar
Skeleton-Author: Firstname Lastname <a@b.c>
Skeleton-Version: 1

[Notes]
We'll fill this in shortly

[Test]
Code goes here.

The metadata should be self explanatory. Like specfiles, they can be translated. There is one optional section we haven't put into the skeleton here, that's the [Retrieval] section. We can leave this for now. If it's missing, autopackage writes one for you which just calls into the retrieve API.

The [Test] section is the most important. It's expected to set the INTERFACE_VERSIONS variable to a whitespace separated list identifying which interfaces the systems implementation of the foobar widget supports. There is no need to enumerate all of them, if your tests return "2.4 1.2" as the value of INTERFACE_VERSIONS this will be expanded out to "2.4 2.3 2.2 2.1 2.0 1.2 1.1 1.0" by autopackage for you, because of the IV rules.

Writing the tests is the bulk of the work in writing a skeleton. Autopackage provides several APIs you can use to help automate this task, but there is no obligation to use them. You are free to use any techniques you wish to examine the system. One restriction is you can't assume you have root, as autopackages can be installed without superuser priviledges.

The most common case will involve checking for an ELF shared library. To help with this, we provide the testForLib() function. Using this is straightforward, as it can produce output suitable for inclusion in INTERFACE_VERSIONS directly. In the most basic form this call returns 0 if the given library soname is found on the system, and 1 if not. This involves checking the linker cache and LD_LIBRARY_PATH. The -i switch to this function will cause it to produce a list of interface versions, so:

INTERFACE_VERSIONS=$( testForLib -i libfoo.so )

... will in most cases do the right thing. There is a large collection of example skeletons shipped as part of the developer tools, so you can see the test sections there for more complex examples.

The INTERFACE_VERSIONS variable can be stacked, eg:

                
INTERFACE_VERSIONS=$( testForLib -i libfoo.so )
vers=$( testForLib -i libfoo2.so )
for v in $vers; do
    INTERFACE_VERSIONS="$INTERFACE_VERSIONS 2.`getMinor $v`"
done

The mapping between system state and interface versions is arbitrary and up to you. If the project follows best practice and has software versions equal to interface versions, a good history of stability etc then it'll be quite straightforward to do this mapping - if it's more convoluted it won't be. You have freedom to do it however you like.

One thing to be aware of is that you cannot change the mapping after the fact. This would not be backwards compatible.

Chapter 4. Environment Variables

Build Environment Variables

Various environment variables can be set to build a package in a particular way.

Table 4.1. Build Environment Variables

Name Description Value Type REQ?
APKG_BUILD_SKIP_CONFIGURE

When set to 1, prepareBuild() will not run configure. This, in combination with $APKG_BUILD_SKIP_CONFIGURE, is useful to only create a package but not recompile all the source code. If prepareBuild is not being used, your [BuildPrepare] code should respect this variable.

numeric No
APKG_BUILD_SKIP_MAKE

When set to 1, prepareBuild() will not run make. This, in combination with $APKG_BUILD_SKIP_CONFIGURE, is useful to only create a package but not recompile all the source code. If prepareBuild is not being used, your [BuildPrepare] code should respect this variable.

numeric No
APKG_BUILD_VERBOSE

When set to 0, no build output text will be echoed.

numeric No

Installation Environment Variables

Various environment variables can be set to see how a package will interact with a particular user environment.

Table 4.2. Installation Environment Variables

Name Description Value Type REQ? Example
DEBUGLEVEL

Sets the debug level to which autopackage responds. When DEBUGLEVEL is >= 3, all trace messages are echoed to ttyfe and debug logfile. When DEBUGLEVEL is == 2, only warning messages are echoed to ttyfe and debug logfile.

numeric No 3
AUTOPACKAGE_DEBUG_LOGFILE

Forces the logfile to be written to a particular filename. When DEBUGLEVEL=3, a debug (trace) logfile will be written.

string No Default value: "./autopackage.log"
AUTOPACKAGE_DISTRIBUTION_CODENAME

Forces the autopackage support code to action for a particular distribution codename.

string No

Community

Yarrow

AUTOPACKAGE_DISTRIBUTION_DESCRIPTION

Forces the autopackage support code to action for a particular distribution description.

string No

Mandrakelinux

Fedora Core

AUTOPACKAGE_DISTRIBUTION_MANAGER

Forces the autopackage support code to action for a particular distribution package manager.

string No

urpmi

apt-get

AUTOPACKAGE_DISTRIBUTION_MANAGER_GRAPHICAL

Forces the autopackage support code to action for a particular distribution graphical package manager.

string No

urpmi

up2date

AUTOPACKAGE_DISTRIBUTION_RELEASE

Forces the autopackage support code to action for a particular distribution release number.

numeric No

10.1

3

AUTOPACKAGE_DISTRIBUTION_TEXT

Forces the autopackage support code to action for a particular distribution. This will override the existence of the distribution release file and the lsb release file.

string No

Mandrakelinux release 10.1 (Community) for i586

Fedora Core release 1 (Yarrow)

AUTOPACKAGE_DUMP_VARIABLES

When displaying all environment variables, include the listed variables in the display. Use _dumpPackageEnvironment function to display environment variables.

string No _dumpPackageEnvironment $meta_dir $working_dir
AUTOPACKAGE_FRONTEND

Force the installation to use a particular front end.

string No

autopackage-frontend-gtk

apkg-ttyfe

AUTOPACKAGE_LANGUAGE

Force the installation to use a particular localized package language. This can be a list or a single entry.

string No

fr_CA fr

de fr en

AUTOPACKAGE_LSB_VERSION

Forces autopackage to action for a particular distribution lsb version.

numeric No 2.0
AUTOPACKAGE_NOCOLORS

When equalling 1, forces ttyfe output to not contain beautifying colors.

numeric No 0
AUTOPACKAGE_SKELETON_DIRS

Forces the autopackage to search in these directories in their listed order for skeleton files.

string No

Environment State Files

All environment variables for a package are written to an environment (state) file. The PACKAGELANGUAGES environment variable will define what environment files are generated for the package. The PACKAGELANGUAGES is set by makeinstaller probing the specfile and setting the variable automatically. During installation, the package loads the proper environment file of the package for the user's environment. If the particular language environment file is not found then the English (en) environment file is loaded. Additionally as the environment file was being generated, if localized information was not available then the default English information was substituted for use. In this manner, all package information will be presented in at least English.

Localized information can be generated for DisplayName and Summary from the specfile. An example is shown from the autopackage-gtk specfile.

                
[Meta]
ShortName: autopackage-gtk
DisplayName: Autopackage GTK+ Graphical User Interface
Summary: A graphical (GTK+) frontend for installing Autopackage packages.
DisplayName[de]: Autopackage GTK+ Graphisches Frontend
Summary[de]: Ein graphisches frontend (GTK+) f�r das Anbringen der Autopackage Pakete.
DisplayName[es]: Autopackage GTK+ Frontend Gr�fico
Summary[es]: Un frontend gr�fico (GTK+) para instalar los paquetes del Autopackage.
DisplayName[fr]: Autopackage GTK+ Frontend Graphique
Summary[fr]: Un frontend (GTK+) graphique pour installer des paquets d'Autopackage.
DisplayName[nl]: Autopackage GTK+ Graphische Gebruikersinterface
Summary[nl]: Een grafische frontend (GTK+) voor het installeren van Autopackage paketten.

To set a different language for use during installation, use export AUTOPACKAGE_LANGUAGE="<language list>" . For example: export AUTOPACKAGE_LANGUAGE="fr_CA fr de" . Each environment file is located in the meta directory for the package. The environment files would be of the format apkg-environment.<language> .

Variables can be added to the environment files for use by other scripting. Use the setVariables() function to set a KEY and VALUE. A Global section can also be added to the specfile to set variables.

Chapter 5. Prefixing

What is prefixing?

The term "prefixed" is made up. It's not a particularly clear word, but it adequately describes one of the problems we face in building distribution-neutral packages. In short, programs built using the autotools build system (and almost all Linux software is) often use a series of macros to hard code the prefix that the program was configured with into the source code itself. This is most often used to build in the path of data files so allowing programs to load them at runtime with the minimum of fuss, but it does mean that once a program is compiled, it can't be moved around. Here is an example of a line of C that does this:

                
MainData->xml = glade_xml_new (GNOMEICU_GLADEDIR "main.glade", "app", NULL);

GNOMEICU_GLADEDIR is defined to be the prefix specified at the time the configure script is run. Here is the relevant lines from Makefile.am :

                
#
# Makefile.am for GnomeICU
#

gladedir = $(datadir)/gnomeicu/glade/
INCLUDES = -DGNOMEICU_GLADEDIR=\"$(gladedir)\"

This sort of practice poses a problem for us, as it means that once compiled, these programs are not relocatable: they can only be installed to one prefix. As often distros require software to be installed into different prefixes, it means we can't portably package such a program until it has been deprefixed , IE all hard coded paths in the program have been replaced with soft paths (IE figured out at runtime). This has a number of other advantages, for instance a binary package can be installed to your home directory if a system accessible path is not desired.

So why isn't this done already

Mainly because although it sounds easy, in fact it's a lot less trivial than the initial thought. The simplest way of locating any data directories needed is simply to find the path of the binary and then combine that with the path "../share/program/whatever". Often, that's all it takes. It's the first bit that's the hard bit. On Linux, as long as the /proc filing system is mounted, it's very easy to locate the binary that is running. Reading /proc/pid/exe, which is a symlink to the binary location. 99% of the time, /proc will actually be mounted, but problems arise when trying to make the code portable. On other operating systems, it's not always so easy to find out this information.

If this is case, the path argv[0] must be determined to be absolute or relative. If it's absolute, that is good. Usually it isn't, and then PATH is scanned to find the binary. Needless to say, this isn't exactly fun, hence the reliance on hard coded paths.

And finally none of the above techniques works with shared libraries, which often also have data directories. Clearly then, some portable mechanism is needed for determining where the various bits of a package have been installed to.

What about the FHS?

Clearly, in the ideal world all distros would be 100% FHS compliant. Many already are, but this unfortunately does not get us very far, as the FHS is at times a rather vague document, with plenty of room for interpretation. For instance, some distros put KDE into /usr , some into /usr/local , and some into /opt. All of these are potentially correct interpretations of the standard. Working on improving the FHS is one of the priorities of this project, as often conflicting prefixes is the only problem faced in making a package cross-distro, but often there are other reasons for needing to choose the prefix as well. Sometimes it's personal preference: some people abhor putting stuff into /opt, some put everything there. Some do both. By default, the install prefix will always be as FHS compliant as possible in any autopackage, unless the prefix needs to deviate from the spec in order for the package to work properly.

The solution

Our solution: BinReloc . It provides a set of convenience macros that can easily replace build time macros in your program. Often, the changes are as simple as a global find and replace.

It works by scanning the linker maps, looking for the base address of a dummy symbol (""). Because the linker mmaps shared libraries, it's possible to find the absolute path of the calling binary. The macros provides movement backwards from there, assuming a standard FHS style hierarchy.

See http://autopackage.org/docs/binreloc/for a guide about how to use BinReloc .

Exception: KDE

One exception to the prefixing problem is KDE. As of April 21 2004, a patch which makes all KDE apps that use the KStandardDirs class automatically relocatable, has been accepted into CVS (thanks to the help of Arend van Beelen Jr). Future versions of KDE apps will be relocatable. However, this is only true for KDE applications; libraries are still not deprefixed and will have to use BinReloc.

Chapter 6. Autopackage Root Names & Versioning

Dependency checking and resolution

The approach autopackage takes to dependency checking is different to other packaging systems. Rather than maintain a database of everything on the system, it checks the system directly using a variety of scripts.

In order to make this process as simple as possible, we have some infrastructure to support the checking process. A dependency is encapsulated as a skeleton file.

Skeleton files contain all the information needed to check for a dependency, and resolve it if necessary. Typically that involves downloading a package, but it can take packages from the working directory too. Skeleton files contain some basic metadata, a script that checks the system, and a script that retrieves the dependency if it's missing. Typically these skeleton files make heavy usage of API functions that are provided to automate tasks like scanning for shared libraries.

Note

Currently, skeletons are stored in autopackage CVS in the main/share/skeletons directory. In future, they will be moved to a separate repository.

Skeleton files do something else too - they examine the state of the system and abstract it as a set of "interface versions". See the section on versioning below.

All the skeleton files used by a package are automatically included in the metadata attachment by makeinstaller . The require() API call calls the test script provided by the skeleton and interprets the results. If the results don't match what is needed, it'll invoke the retrieval script in the skeleton. That means that for a packager, expressing a dependency is as simple as a call like this:

require @gtk.org/gtk 2.4

Note that it takes 2 arguments. The first is the unversioned root name of the package to depend on. The second is the interface version needed, in this case 2.4

What is a root name?

A root name is an identifier for a particular package. Software should have a root name for each of it's packages. A root name is made of four parts:

  • The domain name. This is comparable to XML or C++ namespaces.

  • The package name.

  • The software's full version number. This is optional.

  • The package number. This indicates changes in the package , not the software. For example, there's a bug in the specfile, and it was fixed, but the software did not change, then increase this number. This is like RPM's Release number. As soon as the version of the software changes, this is reset to zero, because the package spec should be considered to be a part of the software itself.

The domain name and package name are separated by a slash. The package name, software version number and package number are separated by a colon. The package number is optional, if it isn't present, a value of zero is assumed. The software version may be optional in some contexts. If it's missing, we say it's an "unversioned rootname".

For example, root names for GTK+ 2.0.1 can look like:

@gtk.org/gtk:2.0.1

the main GTK+ package

@gtk.org/gtk/devel:2.0.1

development headers for GTK+ 2.0.1

@gtk.org/gtk/docs/devhelp:2.0.1

devhelp plugin for the GTK+ 2.0.1 documentation

or if the package was improved:

@gtk.org/gtk:2.0.1:1

Versioning

One reason why root names contain a version number is to make it possible to install two different versions of the same software in parallel.

However, version numbers are also used for dependency handling. Let's say Foobar is a GTK+ 2.0 app. Now we have a few problems:

  • We don't want to check for just "@gtk.org/gtk:2.0.1", we just want to check for *any* 2.0.x release.

  • At the time Foobar is written, GTK+ 2.2 didn't exist. GTK+ 2.2 is now released, and turns out to be binary compatible with 2.0. How can we know this?

So, we introduce a new kind of version number: interface numbers. Interface numbers are not related to the software's version number: they change when the interface has changed. Interface can mean several things, depending on your software. For shared libraries, it usually indicates binary compatibility, but it can also include things like file formats and command line arguments.

Interface numbers are used to check package compatibility.

Interface numbers are made of two numbers, each separated by a dot:

  • Major: Every time the interface has changed (binary compatibility changed), this number is increased by 1 and the minor number is reset to 0.

  • Minor: This number is increased by 1 if something has been added to the interface, like a new function in a library. Apps that depend on previous revision numbers will still work on this revision, but apps that depend on this revision will not work on previous revisions.

Interface numbers only contain numbers, not letters or anything weird like that!

Table 6.1. Software & Interface Number

Software Interface number
GTK+ 1.2.0 1.0
GTK+ 1.2.1 1.0
GTK+ 2.0.0 2.0
GTK+ 2.2.0 2.2
GTK+ 2.2.1 2.2

GTK+ 1.2.1, 2.0.1 and 2.2.1 are bugfix releases. So their major and revision numbers don't change.

GTK+ 2.0 is binary incompatible with GTK+ 1.2, so the major is increased by 1.

GTK+ 2.2 is binary compatible with GTK+ 2.0, but new functions has been added. That's why the major number doesn't change, and the revision number is increased by 1.

GTK is simple, because its software version numbers happen to reflect the interfaces. Other software is less simple, for instance:

Table 6.2. Software & Interface number - Complex

Release name Major interface number
libpng 1.0.x 1
libpng 1.2.5 2
libpng 1.2.6 3

The interface number is not part of the root name, and is stored separately. When software contains multiple interfaces, such as libraries that use symbol versions, the major number stays the same and the revision is incremented.

How to integrate this with skeletons

Skeletons represent a dependency, but not the software (or the package) itself. Skeletons are used to:

  • Check for the existence of an implementation for a certain interface.

  • Install the "best" implementation if one not found.

Skeletons root names are the same as the specfiles, but without the software version number and the package number. It's an unversioned root name Example: @gtk.org/gtk

Skeletons have a Test section, which should set SOFTWARE_VERSIONS and must set INTERFACE_VERSIONS

SOFTWARE_VERSIONS

This is a space separated list of versions of the software, if detectable.

INTERFACE_VERSIONS

This is a space separated list of the interfaces of the software

The reason there can be more than one is so skeletons can detect parallel installs. If no versions were detected, these variables should not be changed from their initial values.

Example

                    
[Meta]
RootName: @xiph.org/libvorbis
DisplayName: Vorbis audio codec
ShortName: libvorbis
Skeleton-Author: Hongli Lai <h.lai@chello.nl>
Skeleton-Version: 1

[Notes]
This skeleton does not set SOFTWARE_VERSIONS
Interface versions start at 0.

[Test]
INTERFACE_VERSIONS=`testForLib -i libvorbis.so`

The format is very similar to the specfile format. Some metadata is provided, and then a few sections most of which are optional. The Test section is required, but by relying on the API functions provided can be made extremely simple. In this case, the testForLib function is used. The -i switch to testForLib() simply makes it output interface versions as opposed to the A.B.C triples normally given.

The Notes section is a freeform text area. It's not used or parsed by autopackage, and should be used to describe quirks of how the skeleton works to the package developer. Most skeletons should follow a standard interface, however there is some leeway in how this is implemented - for instance there are no hard guarantees of how interface versions are allocated short of them being A.B pairs. It's typical for skeletons to not set SOFTWARE_VERSIONS for instance, especially in the case of simple libraries where the interface version is the software version. If that is true then it should be noted here.

How to interface with require()

Usage: require <ROOTNAME> <MAJOR>[.REVISION]

Require will check for the skeleton that has the root name ROOTNAME. As described earlier, skeletons' root names do NOT contain a software version number or a package version number. So:

                    
require "@gtk.org/gtk" 1             # valid
require "@gtk.org/gtk" 1.2           # valid
require "@gtk.org/gtk:2.0.0:1" 1     # invalid

MAJOR and REVISION are of course the required major and revision numbers. If REVISION is not specified, then it is assumed to be zero.

However, sometimes a specific version of a library is specified (for whatever reason). Let's say GTK+ 1.2.5 specifically needed.

requireExact()

For the sake of keeping require() simple, we introduce a new function:

                        
requireExact <ROOTNAME> <VERSION>

VERSION is the software's version number. It may contain letters to a certain degree, the exact capabilities of autopackages version number comparison routines will be documented later.

And sometimes a version that's higher than a specific number is needed. Because of a bug in GTK+ 1.2, AbiWord will only work on GTK+ 1.2.5 or higher, even though the interface hasn't changed. So we introduce another function:

requireAtLeast()

                        
requireAtLeast <ROOTNAME> <VERSION>

This function is like require() , but also checks whether the software version of the software installed on the computer equals VERSION or is higher than VERSION.

It is the job of the skeleton file to accurately report what the version number of the installed software is, the test code should set the SOFTWARE_VERSIONS variable to the detected value, or empty if the version could not be located.

More on skeletons

Skeletons contain code to check whether certain software is present on the system, but it is impractical if one single skeleton must be able to detect all versions of the software it represents.

Sometimes, a change the skeleton file is needed. Because of that, skeleton files have a version number as well. This is a single integer value that increments in steps of one. The number has no related to any other version number, it is purely there to allow distinction between different sets of metadata for the software.

Table 6.3. Software & Interface Number with Skeleton Filename

Software Interface number Skeleton filename
GTK+ 1.2.0 0.0 skeleton.0
GTK+ 2.0.0 1.0 skeleton.1 (gtk has a new soname)
GTK+ 2.2.0 1.1 skeleton.2 (oops, bug in skeleton)

Be aware that skeleton.2 must be able to detect GTK+ 2.2.0 and 1.2.0 too. A skeleton must always be able to detect all known versions of the software, simply to keep things sane.

Automatic dependency retrieval

Autopackage is capable of downloading packages from the network in order to satisfy dependencies. This functionality is provided by luau, and for it to work correctly a few pieces of information are required:

  • A package repository XML file
  • A URL to that XML file in the skeleton file of the dependency

The URL is stored in the Repository key of the skeleton file. This isn't the only way register a repository for a particular package - the registerRepository function is available as well. The XML file pointed to by the URL should be on an HTTP or FTP server.

A sample repository XML looks like this: below:

                
<?xml version="1.0" ?>
<!DOCTYPE luau-repository SYSTEM
        "http://luau.sourceforge.net/luau-repository-1.1.dtd">

<luau-repository interface="1.1">

        <program-info id="@foo.org/libbar"> 

                <shortname>libbar</shortname>
                <fullname>Bar widget library</fullname>
                <desc>A library for barmen</desc> 

        </program-info>

        <software version="1.2">
                <date>2005-06-11</date>
                <interface version="1.2" />
                <keyword>UNSTABLE</keyword> 

                <short>Short declarative statement(s) about release for language en.</short>
                <long>
                        Paragraphs describing the release for language en.
                </long> 

                <package type="autopackage"
                         size="8746"
                         md5="e5ece8caf2b00d37d30cab0def847dd6">
                        http://foo.org/bar/libbar-1.2.x86.package
                </package>
        </software>

</luau-repository>

Most of the information in the repository XML file should match the package .apspec file, but particular details of this release need to be added to the software section.

Adding parameters b and x to makeinstaller when the package is generated will automatically create a new repository XML file for the package. makeinstaller will generate a single full package and two split packages into a meta package and a software payload package. In the above instance, the automatically created repository XML file would be named libbar.xml or libbar.xml.new to not overwrite a previous repository XML file.

                
$ makeinstaller -bx default

generates files:
    libbar-1.2.x86.package
    libbar-1.2.x86.package.meta
    libbar-1.2.x86.package.payload
    libbar.xml or libbar.xml.new [if .xml file already exists]

The table shows how the .apspec file relates to the repository XML file with some comments.

Table 6.4. Data Definitions between Package Repository File and .apspec File

XML Tag/Argument .apspec Meta Key Comments REQ?
program-info id Unversioned RootName Yes
shortname ShortName Yes
fullname DisplayName Yes
desc Summary No
url Matches Repository key in .apspec specfile No
date Matches the build date from the package environment file. No
short Short declarative statements about this release. No
long Paragraphs to describe this release. No
type Use 'autopackage' to describe package as an autopackage. Yes
size File size in bytes of the meta package. No
md5 md5 of the meta package. No
software version SoftwareVersion Yes
interface version InterfaceVersion Yes
version PackageVersion No
weight Proportional amount of time that the following mirror uri should be used. No
mirror Uri to be used to retrieve the package. http, https or ftp are available through curl. Yes

Note

The md5 and size are of the meta package. autopackage initially downloads the meta package to determine dependencies. The luau downloader uses the md5 to validate what to download and size gives some additional status information to the user.

All of the package files should all be in the same directory so that autopackage can find what it needs. If a user wants to start installing this package, the user would download and execute the full package ( .package ) . If another package has a dependency for this package, then autopackage would download the package repository XML file (.skeleton key) to determine where to find the packages. autopackage proceeds to download the meta package ( .package.meta ) to validate the package and determine what dependencies exist. This process is repeated until all the dependencies are fulfilled.

So how to tie this new XML file into peoples packages? The easiest way is to put a new Repository key into the skeleton files Meta section. It should contain the URL of the XML file. In this way, when people use the skeleton for libbar to depend on it, the URL of the XML file is compiled into the package. If the URL changes the dep resolution will break, so make sure it's stable!

Most of the time that's all you'll have to do. Autopackage provides more flexibility though for if you need it. The first mechanism is the [Retrieval] section. This goes into skeleton files and is invoked when a dependency is missing in order to try and retrieve it. Most of the time this section will be missing so autopackage will use the default instead, which just calls the "retrieve" API. If you want to, you can provide any arbitrary script here.

The second mechanism is the repository registration mechanism. Repositories are XML files and you can associate root names with URLs to XML files using the registerRepository() API. If you add a Repository key to the skeleton file this call is done for you, but any package or skeleton can register a repository for any root name. This is most often useful when a packager wishes to also package some dependencies, but because they aren't the maintainer of the dependency they shouldn't modify the skeleton file. Instead, in the [Prepare] section of the package, they can call registerRepository for their own private dependency repository and this will override any repositories registered later.

Luau supports mirroring. To use mirrors, define a set of them at the top of the XML file like so:

                
<mirror-list id="sourceforge">
    <mirror-def id="osdn">http://osdn.dl.sourceforge.net/sourceforge/libbar/</mirror-def>
    <mirror-def id="internap">http://internap.dl.sourceforge.net/sourceforge/libbar/</mirror-def>
    ... etc ...
</mirror-list>

Then refer to the mirrors in your package tags, like this:

                
<package type="autopackage" mirror-id="osdn" filename="libbar-1.2.x86.package" />

Summary

This is a summary of the different version numbers. We have:

Software version

The version number of the software, as defined by the maintainer. Tends to be put on websites etc

Package version

The number of the package. Similar to RPM's Release number. Used only in the specfile. Used for "bugfix releases" of a package itself.

Interface version

Version number used to determine the interface compatibility of the software. + Major + Revision

Chapter 7. Binary portability

It turns out that it's quite hard to make a binary which will work reliably on different distros and even versions of the same distro. However, in order for your package to work correctly on all Linux distros this is a subject that must be tackled. Ignore this section at your peril!

Symbol Versions

The first problem is glibc symbol versions. The GNU C library is a rather special package. All programs use it, either directly or indirectly, as it interfaces programs to the kernel and provides functions for services like file system access, DNS lookup, arithmetic and so on. It also deals with shared library support. Normally, when a library breaks binary compatability, the maintainer increases the major number in the soname, effectively renaming the library (but in a standard way). This means several versions can be present on the system at once. glibc does not do this, for a variety of reasons I won't go into here. Instead, it renames each individual function, meaning there can be several different versions of the same function. The ELF interpreter glibc provides is capable of dealing with this transparently. To see them, run nm /lib/libc.so.6 | grep chown .

When a program is compiled, it's linked against the latest versions of the symbols. If those symbol versions are not present on a system when the program is run, the link process will fail and the program won't start. This problem, and variants of it, have plagued UNIX-like systems for a very long time now. Because users cannot usefully upgrade glibc, the application must be compiled to use a reasonably old set of symbol versions - the versions used bracket the range of distros your binary can run on. Luckily, there is at least a partial solution in the form of apbuild . It is a drop-in replacement for gcc that controls the symbol versions used by the program. If any version that are too new, the build will fail. It will automatically select old versions of the symbols - by default, the binary will be portable (at least for glibc) back to glibc 2.2 - that's before Debian Stable and RH 7.2!

Symbolic collisions

Another problem is that of symbol collisions. The semantics of ELF are unfortunately based on the old static linking days. When a program is executed, the dependency tree of the binary is walked by /lib/ld-linux.so (which is the ELF dynamic linker). If a program, "foo" depends on libbar.so, which in turn is linked against libpng.so, then foo, libbar.so and libpng.so will all be mapped into memory. Semantically, all the symbols from these objects are dumped into one big pot, and this is the crux of the problem. When performing symbol fixup, the glibc ELF interpreter will always choose the first symbol that matches, regardless of what the object being processed is linked against.

For example, let's take our foo binary, and link it against two libraries, libA and libB. libA is in turn linked against libA1, and libB is linked against libB1. Now libA1 and libB1 are different libraries, BUT they both define a symbol called someFunction() . They have the same name, but do completely different things. The expectation is to expect libA to be linked to the definition in libA1, and libB to be linked to the definition in libB1, that is what makes intuitive sense (and is what happens on Windows). But that's not what happens on Linux. They will BOTH be linked to the symbol in libA1, because that's the one that came first. D'oh. This usually results in a nearly instant segfault on startup.

OK, so why does this cause problems with binary portability? Well, although having two libraries that declare a function with the same name is unusual, having two different versions of the same library in use at once is a lot more common. Libpng has 2 major versions in wide usage, libpng.so.2 and libpng.so.3 - they are source compatible (but not binary compatible). If I compile on a Linux distro that uses libpng.so.3, then my program will also be linked against libpng.so.3. If a user then wishes to run it on an older distro, say one which was compiled against libpng.so.2, they'll need to install the newer version for my app to work. Normally we say, so what? Unfortunately, my app (let's pretend it's a game) doesn't just link against libpng.so.3, it also links against libSDL.

Now libSDL links against libSDL_image, which in turn links against libpng.so.2 because it was compiled by the distro vendor. So, now when my app is loaded, 2 different versions of libpng, both libpng.so.2 and libpng.so.3 will be linked in together, and things go boom. Not good.

Note that the two versions are source, but not ABI compatible. That means the user can fix the problem by recompiling my app against libpng.so.2 - this time. It's not always that easy.

As a result, binaries can occasionally end up tied, often unknowingly, to the set of libraries the developer used when compiling. Running it on another distro might work, but there are no guarantees.

Luckily, there is a solution to this problem in the form of an extension to the ELF symbol fixup rules, originally implemented by Sun in Solaris. Direct and grouped fixup allows scope restriction of the symbols, preventing such collisions. Unluckily, it's not implemented by glibc. Volunteers? The problem is big enough that at some point, we (the autopackage hackers) may have to down tools and go work on glibc for a few months.

Bogus dependencies

Again, linking and symbol related. It's typical for libraries to provide foo-config scripts, or more recently pkg-config files. These are handy ways to configure a build system. Unfortunately, they have a nasty side effect - these programs produce libraries link commands like the following -L/opt/foolib/lib -lfoo -lxml2 -lglib-2.0 . The first two options are straightforward enough, but what about the last two? They are there because foolib, internally, uses libxml2 and glib2. Unfortunately it appears that some older versions of ld attempted to close the symbol set (IE ensure that every symbol involved in the program could be linked). It means that in order to link against a library, every library it used also had to linked against, and so on all the way down the dependency tree.

This is problematic. If glib or libxml was not used in your own program, your binary now has a DT_NEEDED entry for every lib involved in the link tree. If a compatible version of foolib (with the same SONAME) is available on the end users system but it was linked against a different version of a library it uses internally your binary will now break, because it has a bogus/unneeded link against it.

This issue is fixed by the latest version of apbuild, which scans binaries as they are linked for bogus dependencies, and then relinks them without the unnecessary -l options.

It is also fixed by binutils 2.15.90.0.2 and upwards if the switch --as-needed linker is specified. In the future apbuild will hopefully detect the presence of this option and use it instead of its built in dependency stripping.

Headers

Some libraries, like GTK+ and glibc, assume the use of headers for the version targetted. In other words, even if the "Since 2.4" functions are avoided, if it is compiled against the GTK+ 2.4 headers there is a risk of gaining an implicit dependency on GTK+ 2.4.

The only current solution to this is to get a copy of the older headers for the version targetted and drop them into the apbuild directory so they are searched before your system headers are.

In particular, this affects the g_return_if_fail macro in glib, the ctype functions in glibc, the GDK_THREADS_ENTER/EXIT macros in GDK, and pthread_cleanup_push/pop.

Exception handling internals

On modern Linux systems an optimization was made to the GNU binary toolchain (the gcc, binutils, glibc triple) to eliminate the relocations the dynamic linker needed to make when loading C libraries which exceptions could propagate through but that did not actually throw and catch exceptions themselves.

The code to perform exception propagation is unfortunately a bit of a mess and parts are duplicated between glibc and gcc3. The key functions are __register_frame_info and __register_frame_info_bases . These functions deal with the internals of throwing exceptions across shared library boundaries.

By default if your systems copy of "ld", the compile time linker, supports the mostly undocumented --eh-frame-hdr option, gcc will optimize away the link in order to improve startup time. Unfortunately while doing so it delegates the exception handling code to glibc rather than libgcc and as such your code gains a dependency on glibc which is much harder to upgrade than libgcc. So, apbuild gives --shared-libgcc to gcc which means it will always be a separate library. When nobody uses pre gcc3 systems anymore we can remove this, but for now it is required.

C++

C++ is what people most often think of when people say Linux binary incompatibility. This section looks at the problems.

The "ABI" is the set of rules used by compilers as they generate binary code from source code. The rules dictate things like how C++ classes are laid out in memory, and how C++ type signatures are converted into unique symbol names (mangling). There are 3 ABI versions in circulation on Linux. The first is the one generated by the GCC 2.95 series of compilers, this is sufficiently far in the past that we no longer need to care about it. The second is the GCC 3.2/3.3 ABI, and the third is the GCC 3.4/4.0 ABI.

The latter two ABIs are virtually identical: only bugfixes distinguish them. They both attempt to follow the Itanium C++ ABI specification, which is fully documented. Unfortunately the bug fixes are severe enough that the two ABIs cannot be mixed.

To "mix" ABIs means to take a binary that was imports one ABI, and link it against a library that exports another. Doing this will usually cause a crash or link failure. Theoretically, linking two C++ binaries indirectly (eg via a C library) is safe, but in practice due to an apparent bug in GCC this is not the case.

To tell a binary compiled with the GCC 3.2/3.3 ABI apart from a 3.4/4.0 ABI binary, look at which version of libstdc++.so it links against. Older binaries use .so.5, newer binaries use .so.6. According to the GCC documentation these two standard libraries can be mixed in the same ELF image, but unfortunately template inlines are given GLOBAL WEAK scope by GCC which can cause symbolic collisions (see earlier) so this is not in fact safe. You should only have one copy of libstdc++.so linked into a process at any one time, for maximum safety.

Index

Glossary

short name

An ANSI, typically lower case name that uses a dash (-) instead of spaces. These names are often used by packaging systems like RPM, DPKG, emerge and so on. Short names do not have to be globally unique, they exist for convenience only.

interface version

An expression of the form A.B, where A and B are integers. Interfaces are assigned a version, and all interfaces where A is the same are backwards compatible. Therefore if an interface of for instance 2.5 also effectively provides 2.4, 2.3, 2.1 and 2.0, but not 1.x or 3.x

software version

The software version is a semi-opaque string that might (or might not) be meaningful to humans. It is generally assigned to a particular release of some software by the maintainers, example software version strings are "4.2", "1.4b", "7.4-pre3", "2000", "XP" and so on.

skeleton file

A skeleton file contains enough information to allow autopackage to detect the presence of an interfaces implementation, and fetch and install one if none is found. It typically comprises of a test script, a notes section, some small amount of metadata and optionally a retrieval script.

root name

A root name is a globally unique identifier based on domain names. It takes the form of "@something.tld/somepath"

display name

Package name that will be shown to the user in their native language, if possible. Should take the form "Product-Name Purpose", IE "Mozilla Web Browser".

binreloc

Binary relocatability technology - it provides a drop in replacement for macros traditionally used in autotools based projects.