The Syllable Build System, Manual
*********************************

Copyright (c) 2003,2006-2008,2010 Kaj de Vos

2 May 2010

http://syllable.org


Contents
========

1 Introduction
2 How to install the build system on Syllable
    2.1 
    2.2 Configuring the build system
    2.3 Uninstalling & upgrading the build system
3 Getting started with the build system
    3.1 Command line syntax
    3.2 Modes of operation
    3.3 Logging
    3.4 Building a single native application
    3.5 Building a third-party package
        3.5.1 About build recipes
        3.5.2 Installing the source
            3.5.2.1 
            3.5.2.2 Patching the source
        3.5.3 Building it
    3.6 Building a single system module
        3.6.1 
        3.6.2 Installing the source
        3.6.3 Building it
    3.7 Building multiple modules
        3.7.1 
        3.7.2 Writing a profile
        3.7.3 Predefined profiles
    3.8 Building a compound package
    3.9 Building the Syllable base system
4 Developing your own packages
5 Known problems and limitations
6 Contact information

1 Introduction
==============

This documentation applies to version 0.11.x of the build system for
the Syllable operating systems. If you encounter problems or have any
other interesting observations, I would be interested to hear about
them (see contact information).

2 How to install the build system on Syllable
=============================================

2.1 
----

Please note that the build system is written in the Ruby language,
hence you need to have the Ruby interpreter installed. Refer to the
Ruby package for Syllable for instructions on installing Ruby.

Now for the installation of the build system. You need to install a
version that matches your system. There are different download packages
for Syllable Desktop and Syllable Server. In the case of Builder, there
is not a big difference between them, but they are configured differently.

If you downloaded this software as a single Syllable resource package, you
should have unpacked this file with

    unzip Builder-0.7.63.i586.resource

or something equivalent. This package contains an anonymous CVS
checkout, so you can update it over the Internet. This will ensure
that you have the latest version, with the latest package definitions.
It is also possible to get the build system from the Syllable CVS
repository manually. The location of the build system in Syllable CVS is

    /syllable/system/apps/utils/Builder/

The software is contained in the subdirectory "Builder". This can be
installed anywhere. In Syllable, it is usually installed under /resources/.
If you unpacked or copied it somewhere else, do:

    mv Builder /resources/

You may have to become the super user first:

    su

Then run:

    package register Builder

This creates links in Syllable to enable it to find the build system's
files.

2.2 Configuring the build system
--------------------------------

If you installed manually from the Syllable CVS repository, you should be
aware that the default configuration is for Syllable Desktop. If you
install Builder on Syllable Server from CVS, you must adapt the
configuration accordingly in the file

    Builder/settings

There are also other settings options in there that you may want to
change. This file is a plain text file with a structured content. You
can edit it with any text editor, but you should be careful to adhere
to the structure. All files that Builder uses to store its own Builder
specific data have this same basic structure.

You can also use Builder on Linux systems other than Syllable Server (and
probably on other POSIX-like systems, but this has not been tested yet).
An easy way to do so is to install it in your home directory and add the
Builder/programs directory to your $PATH environment variable. This way
you don't need super-user access. You should also adapt the settings file
to your situation.

You should now be able to use the build system.

2.3 Uninstalling & upgrading the build system
---------------------------------------------

If you ever want to remove the build system, follow these steps. Again,
you may have to become the super user first:

    su

Then you should remove the links to the build system from Syllable.

    package unregister Builder

To delete the files belonging to the build system, run:

    rm -r /resources/Builder

Keeping the build system up to date is easy if you have an Internet
connection. Just do

    build update
    build log

This requires that you have the CVS program installed. If you have
made your own modifications to files in the build system, you should
always check the log for conflicts and resolve them if there are any.

3 Getting started with the build system
=======================================

3.1 Command line syntax
-----------------------

The build system is called with one command: build. To see all
possible uses of this command, run

    build help

The various options to the build command provide a toolbox for
building several kinds of software packages. Some of the options will
be used more often than others. Some of them are usually not
specified directly, but used internally by the build system.

3.2 Modes of operation
----------------------

The build system can work in several different modes of operation. It
can build a single software module or multiple modules at the same
time, and it differentiates between building the Syllable system and
building external applications. Which mode is used is determined by
how and where the build command is called.

A single module is built by passing the path to where the module
resides to the build command. Multiple modules are built by passing
the name of a profile, which is a list of software modules. If there
is a name conflict between a single module and a profile, you have to
make sure to include a slash in the directory name of the module to
distinguish it. You can do this by making sure there is a slash at the
end:

    build example/

or by including the current directory in the path:

    build ./example

When the build command is called from a working directory named
"system", it assumes it is building the Syllable base system. This is
the root directory of the Syllable directory tree in its CVS
repository. In this mode, several functions are performed
differently, so if you are building a separate application, make sure
it is not from a directory named "system".

3.3 Logging
-----------

Not all variations of the build command perform actual build
operations, but those that do are logged in several forms. During
building, only progress information and error messages are sent to
the screen, in order to provide a better overview of the process
(that is, stderr is sent to the screen as normal, but stdout is not).
The rest of the output (stdout) is captured in a log file. To see this
log of the last build operation that was performed, do

    build log | less

(It's usually quite long, so you will probably want to view it in a
pager such as "less".) If you are building multiple modules, you may
want to have a summary of the results, without all the process output:

    build log summary

When building multiple modules, the build system doesn't stop at a
module that won't build successfully. To see a list of only those
modules that didn't build, do:

    build log failures

3.4 Building a single native application
----------------------------------------

The simplest case of building a software module, as far as the build
system is concerned, is that of a native Syllable application. The
source code of the application should be self-contained, and should
conform to the specifications that the build system expects from a
native application. In this situation, the build system doesn't add
much value to building the module "by hand" by calling "make"
commands directly, so you may prefer that method. However, it forms
the basis for more useful operations.

You should place the source code in a subdirectory that will
typically have the same name as the application. Suppose the root
directory of the application is called "example". The first step in
building it would then be

    build example

Or if you placed the application directory in a subdirectory, for
example "applications":

    build applications/example

Watch the output to see exactly what operations are performed by the
build system. It will clean the source code, configure it, and then
try to compile it. If you want to see the standard process output as
well, do

    build log | less

If the build procedure was successful, you can optionally run a suite
of tests, if the application has one:

    build test example

If all is well, you can install the application. You may have to
become the super user to do that:

    su

Then run

    build install example

3.5 Building a third-party package
----------------------------------

3.5.1 About build recipes
-------------------------

Building a third-party software package, i.e. software that is not
native to Syllable but was ported to it from other systems, is much
like building a native module. Here, the build system is starting to
become useful. It shields you from many differences between the build
procedures of third-party packages. In order to do that, it needs
descriptions of these differences, that are written in the form of
something called a recipe.

The build system contains a number of recipes for packages that have
been ported to Syllable, so if you are building one of these
predefined packages, you don't have to worry about recipes. If you
are trying to build a package that is not known by the build system,
it is still possible that the default build procedure is able to
build it. The default procedure is oriented towards commonly used
build steps, and in particular towards the customary build procedures
of packages from the GNU project (http://www.gnu.org). If the default
procedure does not succeed in building a package, you will have to
find a recipe for it, or write one yourself. A common case would be
trying to build a newer version of a package that was ported before.
In that situation, chances are that the recipe for the older version
will work. In any case, refer to the chapter "Developing your own
packages".

3.5.2 Installing the source
---------------------------

3.5.2.1 
--------

You could install the source code for a third-party package manually,
in a similar way to how you would put the sources for a native module
into place. You would probably receive the official release of a
source package in the form of a compressed tarball, which you would
have to unpack in the place where you want to build it. However, the
build system can do this job for you. Just drop the source package
into the build system in the directory

    /resources/Builder/sources/

When you ask the build system to build a package for which the source
is not installed, it will look for it in this location and unpack it
automatically. This is most useful when you regularly have to remove
sources to clean up. As long as the source package is in the build
system, the sources will be restored when needed.

Even better, if the package has a build recipe that defines a download
location, and you have the cURL program installed, the build system
will download the source package and place it in Builder/sources/
automatically. cURL is included in the Syllable operating systems, so
you don't have to do anything extra unless you're using Builder on
another system.

3.5.2.2 Patching the source
---------------------------

In addition to the build recipe described earlier, the description of
a third-party package may also contain patches to the software that
are required for Syllable. These patches will automatically be
applied to the sources when you allow the build system to unpack the
official source package. If you install the sources yourself,
patching is also your own responsibility. However, you can still
explicitly tell the build system to apply any patches it has, like
this:

    build patch example

3.5.3 Building it
-----------------

As mentioned before, building a third-party software package is much
like building a native module. The name of the directory that
contains the source code will typically include the version number of
the package. The first step in building it would for example be

    build example-1.2

If you want to see the standard process output, do

    build log | less

If the application has a test suite, you can run it:

    build test example-1.2

To install the package, you may have to become the super user:

    su

The installion is done with

    build install example-1.2

Third-party packages will be installed into their own subdirectories
in the /resources/ directory. The Syllable package manager will be run
to register the package with Syllable.

The installation and registration can be undone with

    build uninstall example-1.2

If you want to distribute the package to others, you can build a
distribution package while the package is installed with

    build package example-1.2

The distribution package will be generated in your working directory.
Take care that you are not distributing any sensitive data that the
application may have written to its own directories while it was
running. If you want to be certain about this, you should build the
distribution package immediately after installing the program, before
running it.

3.6 Building a single system module
-----------------------------------

3.6.1 
------

For the purpose of this documentation, a system module is a Syllable
module that is built in the context of the source tree for the
Syllable base system. The build system assumes this context when it
is called from a working directory named "system". Consequently, the
path names that you use to designate the modules you want to build
must be relative to this "system" directory.

The build system will set up a number of environment variables that
native system modules typically use. It assumes that the source tree
contains the complete Syllable source code, in particular the system
headers. It will also create a new directory tree structure in which
the module can be installed. This directory tree ultimately serves to
create an image for a new Syllable distribution.

Although the build system will make a number of preparations for
building a system module, it is still your job to ensure the right
environment. System modules often rely on particular system headers
and libraries. Those tend to change when Syllable evolves, which can
break the build if different versions are not compatible. By default,
when compiling software in the "system" directory, headers from the
Syllable source tree are used. These things can be tricky, and there
is no easy instruction for them.

3.6.2 Installing the source
---------------------------

Typically, the sources for the Syllable tree and the module to build
would be installed from CVS. If the module is a third-party package,
the same considerations apply that were described in the section for
building a third-party package. The build system will unpack and
patch the sources if it can. You are free to modify the tree, as long
as you know what you are doing.

3.6.3 Building it
-----------------

The commands for building a system module are no different than those
for stand-alone applications:

    build example

Installing the module into the running system is done with

    build install example

However, modifying your installed system can be dangerous. Modules can
be built and installed in one go into a staging area.

    image example

will install the module in the staging area where a partial directory
tree is built for a Syllable distribution. The staging area is located
in the subdirectory "stage/image/" within the "system" directory. From
there, you can inspect the results of the build and move files into your
running system selectively.

3.7 Building multiple modules
-----------------------------

3.7.1 
------

More than one module can be handled in one go by the operations of
the build system. This is done by passing a profile name to the build
command, instead of a module path. For example, to build a number of
modules, enter:

    build example-profile

The build system recognises a profile name because it doesn't contain
a slash ('/'), as opposed to a module path. A profile contains a
list of modules. All the modules in the profile will be acted on
sequentially. The logs will contain all the actions on all modules,
even if one or more of the modules failed.

You can get a list of the modules in a profile by entering

    build modules example-profile

You can test all the modules in a profile in one go:

    build test example-profile

And you can install them all at once:

    build install example-profile

A profile can contain both native and third-party modules. The way
they are handled corresponds to what was described earlier in their
respective sections. Because of that, the

    build uninstall example-profile

command and the

    build package example-profile

command will have different effects, depending on what modules are in
the profile and whether you are building stand-alone applications or
part of the Syllable base system.

3.7.2 Writing a profile
-----------------------

Profiles are stored in the build system at this location:

    /resources/Builder/profiles/

A profile is simply a text file that you can write yourself. Please
refer to the existing profiles as examples. Comment lines are allowed;
they should start with a semicolon (';') in the first position.

3.7.3 Predefined profiles
-------------------------

The build system contains some predefined profiles that can be
useful. Some profiles are generated automatically by the build
system itself. After a build operation, the profile named "failures"
will contain a list of the modules that failed. This is very handy if
you are building a long list of modules where some of them have
errors. You can use

    build log failures

to be shown the log of failing modules from your last operation.
After making fixes, using

    build failures

will rerun the build procedure on only those packages that weren't
built successfully yet. This will even work if you invoked the build
command on single modules inbetween while making fixes, because the
profile "failures" is only updated when working with a profile. The
procedure can be repeated until all modules are repaired.

Another auto-generated profile that you can use is named "last". This
profile always contains the last profile that the build command was
used with.

3.8 Building a collection pack
------------------------------

The build system is capable of producing distribution packages
consisting of multiple third-party packages. For example, the build
system contains a profile named "dev-pack", for the Developer's
Delight pack. This is a collection of third-party packages commonly
used by developers. You can build all those packages with:

    build dev-pack

To install such a list of packages, you would normally do:

    build install dev-pack

(In this case, it's more complicated than that, because the developer
pack contains several packages that need themselves to build and
sometimes even to install themselves, such as Make, BinUtils and GCC.
You would have to replace these packages carefully and individually.)

When you are satisfied with single distribution packages. you should
move them to

    /resources/Builder/distributions/

Then,

    build pack dev-pack

will create one distribution pack in your working directory that
contains all the packages from the profile, and optionally extra
files such as documentation.

3.9 Building the Syllable base system
-------------------------------------

At the time of writing, it is not possible to give a definitive
instruction for building the entire Syllable base system. Although
the build system is capable of doing this and evolves over time with
the development of Syllable, the Syllable system is a work in progress.
Furthermore, if you try to build from CVS sources, you will most likely
encounter build failures that exist along the way to the next Syllable
release.

That said, here is a basic description of how the system is built. For
starters, there is a large number of prerequisites that you have to
fulfill before this makes any sense. Here is a - subject to change,
and probably incomplete - list of packages you need to install. Most
of them are available as binary packages, but some you may have to build
from source with the build system.

Source packs
  - For Syllable Desktop, the Syllable source code
Collection packs
  - Shell Essentials
  - Network Necessities
  - Developer's Delight
  - The Perl Pit
Single packages
  - CMake

There are several profiles that define a list of all the modules of
the Syllable Desktop base system. The process goes like this:

    cd system

    image syllable
    build log failures

    image base
    build log failures

    image gui-base
    build log failures

    image gui
    build log failures

    image desktop
    build log failures

    su
    image finish-su
    build log failures

    image finish
    build log failures

    cd stage/image
    build scrub

    cd ../..
    construct distro SyllableDesktop 1.0 i586

The last command, "construct", is an extra tool included with Builder
that will split the system into the base distribution and the special
development files pack, and create distribution packages of them.

It is basically similar for Syllable Server:

    cd system

    image linux
    build log failures

    image base
    build log failures

    image gui-linux
    build log failures

    image gui-base
    build log failures

    image server
    build log failures

    image linux-finish
    build log failures

    cd stage/image
    sudo build scrub

    cd ../..
    sudo construct distro SyllableServer 1.0 i486

Along the way you will have to fix build failures, using techniques
from this manual. Please have patience, and use at your own risk. Or,
in the words of our enlightened project leader:

"Attempts to build Syllable from source may cause temporary blindness,
paranoia, madness, rudeness and athlete's foot. Do not use Builder if
pregnant or stupid. If in doubt, consult a trained arborist."

Especially for Syllable Desktop, more than the above procedure is needed
to arrive at a complete system. We use extra scripts to set up a build
environment and drive Builder. They are here in Syllable's CVS repository:

    /syllable/system/build-tools/scripts/auto-build/

Although these scripts automate the process, they are also a work in
progress, developing together with Syllable. At any point in time, they
are aimed at helping us produce the next Syllable development build.

4 Developing your own packages
==============================

To port a third-party package to Syllable and integrate it with the
build system, you have to write a Builder recipe for it. Full
documentation for this needs to be written, but there is an up to date
specification for the syntax of recipes, in the file

    Builder/packages/skeleton.recipe

For examples of recipes, please study the many existing ports in
Builder/packages/.

5 Known problems and limitations
================================

The Syllable build system is a work in progress. It develops with
Syllable, so changes and improvements are made continuously. Therefore,
it is best to use a recent version, or at least a version that matches
the software you are building.

6 Contact information
=====================

The Syllable build system was created by Kaj de Vos, and started from
the work of Kristian Van Der Vliet. You can send problem reports,
fixes and the like on our forum:

    http://forum.syllable.org

The latest version of this software can be found through the download
section of the Syllable web site, or in the CVS repository:

    http://syllable.org
