Syllable Desktop Change Log

History

Syllable Desktop 0.6.7
Documentation
Installation
Security
Desktop Environment
Command Line Environment
Graphical Applications
Resource Packages
Multimedia
Server Functionality
LibSyllable
Kernel
Drivers
System Structure
Programming

Syllable Desktop 0.6.7

Kaj de Vos, 12 April 2012

This release is dedicated to the memory of Jack Tramiel, who passed away when I was making the finishing touches to this document. He was dealt hard blows in life and always recovered. I will never forget how he saved Atari from bankruptcy and orchestrated their comeback.

We are happy and proud to present this new Syllable version. We thank all of the volunteers who contributed to this release for their generous efforts. Most of the work that went into this version was done by Kristian Van Der Vliet, Anthony Morphett and Kaj de Vos. Other contributors are mentioned with their contributions.

The highlights of this release include:

- Improved documentation.

- Improved installation, with support for specific configurations such as the Asus EeePC.

- The system is much more usable with a keyboard. Installation and log-in can now be completed without a mouse.

- Auto-login: you can set a user account to log in automatically without entering a password.

- POSIX interfaces for memory mapped and async I/O implemented in the kernel.

- Improved support for Intel eepro100 network driver and Intellipoint PS/2 mice.

- Improved native applications. The web browser was fixed to remember cookies again. An FTP client was added.

- Improved support for SDL programs. Almost hundred SDL programs that “Bogomips” ported can now be used without extending the system:

sites.google.com/site/syllablesoftware/

- Upgraded multimedia support. Flash videos can now be played.

- First 3D support implemented, through porting of Mesa3D and PicoGL.

- QEmu now included for running Virtual Machines, with improved performance due to memory mapped and async I/O.

- SaMBa now included for Windows compatible networking.

- The server part of OpenSSH now works for remote access to Syllable.

- SQLite now included and Fossil version control system ported.

- System restructuring to make it more user friendly.

- REBOL 3 and Boron now included for programming.

This release has been a long time in the making. Of course it is good to have regular releases, as Syllable had in the past, but we are a volunteer organisation, so we need to make do with limited means. A lot of work is needed to make a release, and we need to fit that into our lifes. Along the way, we lost our release engineer and build infrastructure. This is an intermediate release in the process of replacing them.

To make it easier for us to make this release, its scope is limited. It is not based on the latest source code in CVS, but on the previous development build, which had been tested for a long time, with many fixes and additions made to it. Building from the latest source would have meant a series of extra issues to fix and test. Due to this decision, no separate source package is available for this release. It can be constructed from Syllable CVS, but we currently don't have the resources to do it. The next release will have a separate source package again. At the time of writing, there is no package to upgrade from Syllable 0.6.6. The Virtual Machine image, the live CD and the API documentation have not been updated, but will probably be released later, as time permits. Sadly, Lulu.com has stopped producing CD's, so there is no Premium CD anymore, either. We are looking into replacing it.

Syllable 0.6.6 was a good release, so there wasn't an urgent need to replace it. We are making this release now to prepare for several future developments. The system is in a transition state in several ways. Syllable 0.6.6 was a stable base to work on, so we need to move to a new stable state, with as little disruption as possible. In the past, new Syllable releases suffered from too many regressions. We want to minimise those. We made a new development release two months before this release and tested it quite heavily. No problems were reported and we fixed all the regressions compared to Syllable 0.6.6 that we could find.

The structure of the system and software packages are being reorganised, to make them easier to understand. During this migration, both the old and new package structures need to be supported. Resource packages built according to the new structure on Syllable 0.6.7 will not be compatible with older releases. Further, the system now supports the POSIX mmap interface for memory mapped I/O, which is used by a number of third-party packages, including GCC. To port newer versions of GCC, we need to move to this new Syllable release, and to complete these various migrations, software packages need to be rebuilt on this new Syllable on a new toolchain. This release constitutes the migration path between these configurations, including several one-off measures that will only be needed during the migration.

We also need this release to prepare for the upcoming high-level Red programming language. Its memory allocator uses mmap on POSIX systems. In the past year we made Red/System fully support Syllable, and now Syllable should be prepared to support Red.

Documentation

The base documentation for the system was overhauled, updated and cleaned up. Several documents were converted from plain text to HTML format, including these release notes and the installation manual. This fixes Unicode encoding issues when displaying the Russian and German versions, particularly from our web sites. A new style was applied to them that is easier to read, and being HTML, they now have clickable tables of contents and other links. These release notes are now included in the system.

Some documents are still in transition to this new format, particularly the translated versions.

Installation

Anthony has implemented keyboard navigation in Disk Manager. This means you will be able to install Syllable without using a mouse, in case the installation CD has problems with it. To do this, Anthony had to make fixes to keyboard handling in LibSyllable Views. This will also improve keyboard use in other applications.

The start-up menu was further extended to help with specific installation environments and incompatible hardware.

The options for IDE and USB CD players were merged, so that the troubleshooting options can now also be tried with a USB CD player. A specific installation option was added for ASUS EeePC netbooks. The EeePC requires compensation for its shifting of drive positions, which is now performed by the installer. This was tested for us by Hans Rood on the Summer SylCon. There are now more safe mode options for troubleshooting, such as an option to fully remove the USB 2 driver, which is buggy on some systems. The original Acer Aspire One netbook was tested to work for us by Ruud Kuin, when disabling ACPI.

Added extra boot menu option to remove the buggy EHCI USB 2 driver. The installer also removes it for EeePC.

Added extra boot menu option and boot parameter for SATA configurations that shift drive numbers.
Extended installer to compensate for drive number swapping and shifting on EeePC and some SATA configurations.

Experimental installation options were added for installing from and to USB memory sticks. These don't work yet, because Syllable doesn't start from USB devices other than CD players yet, but this is being worked on. Apart from Syllable's own AFS format, there's an option for starting from USB memory formatted with Linux's Ext2FS file system. Installing to USB is still unreliable on Syllable, so the Ext2FS option allows to experiment with creating a USB installation medium from Linux.

Work on the boot menu:
- Added option without AGP.
- Added USB drivers to most options, so they can be used with both ATAPI and USB CD drives.
Left a failsafe option without early loading of USB drivers.
- Split off Virtual Machine options and failsafe options into separate submenus.
- Added eeepc=true kernel parameter to EeePC option, so the installer can act on it.
- Added experimental option to boot from Ext2FS formatted USB media.
- Clarified descriptions of several options.

Graphical Applications

Applications were moved to more functional names. While this is a tad boring, it reduces confusion for new users trying to learn the system and the function of applications. Only the launch names were changed, not the display names of applications once they are started.

The Webster web browser that was new in Syllable 0.6.6 fails to remember cookies, because it tries to store them in the location for the old ABrowse browser that it succeeded. This release has a newer version that fixes this, which was already on the Syllable 0.6.6 live CD. You need it for example to make web sites remember your log-in status.

The collection of default bookmarks for the web browser was updated.

File Transferrer was added, our FTP client application. Anthony has made a number of fixes to the functionality and the appearance, and has integrated work from contributors. This is originally our community project. The application is not done yet, but it is becoming usable for your downloading pleasure, and your uploads if you are so inclined.

Several applications from the sIDE development collection that are useful outside of programming were added to the base system. Code Editor (Sourcery) as an advanced text editor, Translation Editor for editing program translations and Event Viewer for showing available system events.

Several classic Syllable demos were added: Double Buffering (moving objects without flickering), Eyes (eyes following the mouse cursor) and Stars (scrolling star field).

The new PicoGL package includes several classic 3D animation demos: Cube (spinning solid cube with texture), Gears (spinning gears), Mech (walking mechanical robot), Spin (spinning wire-frame box) and Texture (spinning textures).

Made the disk partitioner fully keyboard navigable.

Patches for Settings Editor from John Aspras.

Resource Packages

Most packages in the system were upgraded, including key components such as Packager, FFMPEG, FreeType, BASh and NCurses. Other important packages such as OpenSSH and Transmission were also updated.

OpenSSL, ReadLine and LibJPEG are now shared libraries included in separate packages, instead of statically compiled into the programs that use them.

The collection of well-known root certificates from Certification Authorities was upgraded from the Debian CA Certificates collection to the LFS Mozilla CA Bundle collection. CA-Certificates has too much Debian specific machinery to get the certificates in the needed form.

XZ-Utils was ported to Syllable, providing the same LZMA compression as in 7-Zip, but in a different format that is becoming popular, and is better integrated with POSIX systems. TAr was upgraded and this version has support for XZ-Utils. Compression of the development files pack was changed from 7-Zip to XZ format.

QEmu was added, so you can run other operating systems or test versions of Syllable as Virtual Machines out of the box.

SaMBa was added, so you can connect to Windows machines over the network, or to other machines running a Windows network. You can use the smbclient program from the command line to access files on other machines, or you can activate the SaMBa server to make files of your choice available to the network.

Fossil and the latest version of SQLite were ported to Syllable. Recent SQLite versions work again thanks to file locking fixes in the Syllable kernel. Some stability problems were observed in the Fossil port, so it is still experimental.

Multimedia

The FFMPEG codecs collection was updated to version 0.5.0. The Syllable media framework needed to be adapted to match. As a result, the Media Player can now play Flash videos.

“Bogomips” found that the SDL subsystem didn't support PNG images yet. We tracked the problem to the SDL-Image component and found TIFF image support missing, as well. Oddly, even though JPEG (and GIF) images already worked, the fix required making LibJPEG a shared library. Fixed versions of SDL-Image and LibJPEG are included.

SDL-Image configure didn't detect LibPNG and LibTIFF, so it didn't have support for PNG and TIFF images. The configuration has been extended to fix this. The only way I got the new combination to work was by making the LibJPEG library dynamic. There are some combinations left that could be tested, for example in case the problem is with LibTIFF, that is now being used and uses LibJPEG itself. However, this debugging takes a lot of time, and I have always thought that forcing the popular image libraries in the system to be static libraries may be a bridge too far. LibJPEG is used in several places in the base system, so making it dynamic is the most straigthforward solution, and also makes the system more efficient.

Kaj ported the latest version of Mesa3D. Kaj and “Bogomips” ported PicoGL. This means the first 3D functionality is now available in Syllable. Previous attempts at porting Mesa3D long ago were never completed. It has a new graphics architecture now, so a Syllable driver for these old versions does not work anymore. Therefore, Mesa3D currently only provides off-screen rendering. However, PicoGL is built on SDL, so programs written for PicoGL can render directly to the screen. Several demos are included.

Server Functionality

Configurations, including initialisation scripts, were added for the OpenSSH remote access server, the SaMBa Windows-compatible file server and the Fossil version-control file server. To activate one of the servers, edit its start-up script in /resources/index/tasks/start/ and uncomment the line that activates the server, according to the instructions there. After that, the server will be started on each system start-up.

After many years of trying by several people, Kaj collected their hints and got the server part of OpenSSH to work. The integration into Syllable Desktop is based on earlier integration into Syllable Server of the system configuration needed for the OpenSSH server. Kristian implemented the socketpair function in Syllable 0.6.6 to support OpenSSH, although this currently still needs to be disabled. Michael Pavone and Adam Kirchhoff supplied and tested several other options that need to be disabled to get the server running and stable.

Unique encryption keys for OpenSSH that identify your system were already generated at the first start-up of your Syllable installation.

LibSyllable

IconView key handling bugfix.

Keyboard navigation improvements for compound views such as os::Spinner and DropdownMenu.

Kernel

POSIX interfaces for memory mapped and asynchronous input/output were implemented. Bugs in file locking and random number generation were fixed.

Fixed /dev/random. It was throwing away 24 bits of each call to rand() (rand() % 256) which seriously compromised the result to the caller.

flock.c/lock_inode_record(): Do not return EWOULDBLOCK if the caller is replacing a lock it already owns, which is obviously a non-blocking call.

Kristian has implemented asynchronous input/output. This has been tested with QEmu, which shows increased performance both due to this and the also new implementation of memory-mapped files.

Async I/O is implemented as a native POSIX interface. The available functions are aio_read() and aio_write(). In addition, the asynchronous case for msync() in memory-mapped files is now supported. In Syllable 0.6.6, async I/O was simulated in the GNU C library and executed synchronously. This was enough to make QEmu work, but now async I/O is actually implemented through a syscall kernel interface.

Originally, Syllable relied solely on pervasive multithreading for its exceptional responsiveness. Several years ago, a new scheduler was introduced that is more suitable to desktop use than the original round-robin scheduler. Async I/O further enhances responsiveness in ported applications that use it.

Implemented async I/O support:
- aio_read() & aio_write() are supported via a generic interface (sys_aio_request()).
- msync() MS_ASYNC is now supported via the async I/O backend.

Kristian has implemented the POSIX interface for memory-mapped input/output. Ever more third-party software requires this functionality. Both the interfaces for memory-backed files and anonymous memory maps are supported. Syllable has always used memory-mapped files for its in-kernel ELF loader, so the POSIX interface is implemented on top of this. Several components in the system make use of mmap now.

The mmap(), munmap(), mprotect() and msync() functions are implemented. In the process, Kristian fixed some bugs in the underlying memory management implementation in the kernel. One of the next planned implementations of standard interfaces is async I/O, among other things for completing the mmap support. Further, it was determined that the time is nearing that the syscall interface needs to change from int 0x80 to SYSENTER/SYSCALL. mmap() is the kernel's first syscall with six arguments, requiring extensions to the syscall table and strace(). The newer SYSENTER interface is needed to fully support this and 64-bits arguments, also for a future 64-bits port of Syllable.

There is a tendency in much open source software in recent years to require advanced features of the host platform, that are only available on the major systems; sometimes only on Linux. This is problematic for alternative operating systems and undermines the portability of open source and the diversity and choice of platforms. Through the addition of the mmap interface, we hope to gain access to such software as GCC beyond the 4.1 versions, Gnash, Cherokee and maybe PostgreSQL. Software that will perform better includes FreeType, SDL and DOSBox.

Implemented mmap(), mprotect(), msync() & munmap():
- Both anonymous and file-backed maps are supported by mmap().
- MAP_PRIVATE is unsupported.
- mmap() on anything other than “normal” files are not yet supported.
- MS_ASYNC (async msync()) is not yet supported and the sync is performed synchronously.

areas.c/modify_area_pages(): Modify all PTEs for a given area with new PTE flags derived from the area protection flags.

pagefault.c/load_area_page(): Fixed to check the area protection flags for the faulting page before it either creates a COW page or loading a new page from disc: attempts to write an area which has been created without the AREA_WRITE flag will now generate a SIGSEGV; attempts to read or write a page which has neither AREA_READ & AREA_WRITE flags will also generate a SIGSEGV.

Drivers

Fixed support for Intellipoint mice by completing the switching to IMPS2 protocol. Patch by Evgeniy Philippov for Defender PS/2 mouse. This was probably a regression because IMPS2 support was added long ago.

Added another OEM PCI ID to the eepro100 network driver. From Andy Shissler.
Added an extra PCI ID to the eepro100 network driver for HP laptops. Patch from Carlos Pizarro.

Added chip-specific controller drivers for the AGP bus driver, for Nvidia, Ali and Sis chips, from Dee Sharpe. The Via driver is waiting for a fix to memory management in the AGP bus manager.

Implemented O_DIRECT support in AFS:
- Added maybe_cached_read() and maybe_cached_write() to the block cache.
- Added the MNTF_DIRECT flag, which tells the filesystem to open the device (or file) with the O_DIRECT flag.

Removed the old MNTF_SLOW_DEVICE flag.

System Structure

Syllable is an operating system with a high degree of compatibility with the POSIX standards and de facto conventions in Unix systems and Linux distributions. This is a double edged sword. It makes it much easier to get existing third-party software to run on Syllable, but it also burdens us with ancient methods from the beginning of computing that have partly lost their meaning. One example is the layout and naming of files in the system and in additional software packages. We have always tried to strike a balance between following conventions and inventing more modern and user friendly methods. With a decade of experience, we have gathered the confidence to make a number of extra steps towards more user friendly structures, with as little disruption to compatibility as possible.

Instead of trying to shield people from the workings of their system, we want them to be able to go into the directories of the system or a separate package just with a filer, and be able to make some sense out of the files there and find what they are looking for. To that end, we are introducing a series of new measures to rename and move subdirectories that often form the starting point for searches for information.

The system tools Packager and Builder were further developed to support these restructurings, among other things to improve the use of symlinking for package registration. Packager is included in the system, but Builder needs to be installed as part of the Developer's Delight package, if you want to compile software yourself. When Builder is installed on Syllable 0.6.6, an older version is used that is still compatible with the older system structures. The Syllable branch in Builder/packages/ is also still largely compatible with Syllable 0.6.6, so that some packages can still be built on older systems. After the release of Syllable 0.6.7, Builder and its recipe tree will be updated to the latest versions, using the new structures.

Graphical programs that, besides the desktop menu, also need to be available from the command line are now kept in the search path locations and symlinked from /Applications/. Previously, this was usually done the other way around, but that meant that the program in /Applications/ couldn't be moved.

Packages are in the process of moving to clearer names for their main package directory. Instead of the traditional, short, lowercase names that are confusing to common people, common typographical standards are now being used. Resource packages have project names, which are proper names, so they are capitalised. Generic terms remain lowercased. When names consist of multiple words, CamelCase is used to avoid spaces. When uppercase abbreviations would lead to words of only capitals being concatenated, hyphens are used for more clarity. Abbreviations are expanded when they don't take too much space; roughly twenty characters maximum. Exceptions can be made to keep names recognisable to people who already know the short names, and to keep them roughly in the same place in alphabetical order. Around half the resource packages in the system are currently renamed.

The internal structure of resource packages is in the process of being heavily reorganised, doing away with the traditional Unix naming that is confusing and has lost its meaning over time. The structure of the symlink pools has changed accordingly. Subdirectories are being moved and renamed thus:

etc             -> settings
bin             -> programs
sbin            -> system-programs
libexec         -> framework/executables
lib             -> framework/libraries
include         -> framework/headers
share/aclocal   -> framework/AutoConfigure
share/pkgconfig -> framework/PackageConfigure
lib/pkgconfig   -> framework/PackageConfigure
share           -> data
share/doc       -> documentation
man             -> manuals
share/man       -> manuals
info            -> manuals/info
share/info      -> manuals/info

Subdirectories that are not immediately meaningful to common users are tucked away in the framework subdirectory, a name which should at least signal to users that it should not be tampered with without appropriate knowledge. The documentation directories are split between an unmanaged and a managed (structured manuals) part, to make a distinction in how they should be used and the degree to which a user could tamper with them.

Since there is no standard for the location of REBOL and Boron libraries, they can go into framework/REBOL.

The Syllable-specific initialisation scripts were consolidated into one subdirectory. This leaves room for adding the common Unix script classes for stop, reload, restart and status commands:

early-init      -> tasks/setup
init            -> tasks/start

An extra subdirectory applications was introduced for resource packages that contain graphical programs that should appear in an applications menu. This was prompted by the port of the VICE emulator. This and other packages need to be in /resources/, but want to make entry points in the Syllable menu like /Applications/ do.
Support in the menu needs to be implemented separately.

A few packages need extra, internal compatibility symlinks because they don't fully support this reorganisation. On the other hand, such links can also be used to solve interdependencies between packages.

The symlink pool /resources/index/ and Packager have compatibility facilities to support most older binary packages.

Moved bin/ to programs/.
Most common people would probably associate “bin” with a dustbin, and wonder why every package has its own trashcan. We'd have to explain that it means “binaries”, then explain what binaries are and what a binary number system is, and then explain that bin/ directories don't just contain binaries, and that binaries can also be found elsewhere... bin/ has only been a fitting name for a short period in the earliest Unix history, before the advent of scripts. Since the invention of libraries, there are also binary files elsewhere.

Moved sbin/ to system-programs/.
The “root” super user in Syllable Server has already been renamed to “system”. This is also planned for Desktop. system-programs/ will then be a matching name for programs that need to be executed by the super user.

Moved share/ to data/.
share/ originally meant files that can be shared over a network between multiple machines, because they're data instead of architecture dependent binaries. This name is not very meaningful on Syllable, because disk space is not an issue anymore for these files and we build self-contained systems that can be operated off-line from a network. share/ directories are symlinked to make them available as an aggregate within the system, but most resource subdirectories are shared that way.

init/ and early-init/ were already consolidated into tasks/. The remaining subdirectories are not meaningful to common users, because they contain support files for programs/ and applications/, so I folded them into a new subdirectory framework/ to signify their function and move them out of sight.

Moved lib/ to framework/libraries/.
Libel? Liberation? Makes it clear what we're talking about. It's still not the same as a public library, but at least it's clear that it belongs to the framework of a resource package, and libraries is the term used everywhere in system and programming documentation, so it would be confusing for techies to change it more than this.

Moved libexec/ to framework/executables/.
libexec/ clearly has a confused personality, so this makes that explicit. It can contain both support programs that don't need to be in the search path and libraries, which are both executables, so it clearly belongs to the framework.

Moved include/ to framework/headers/.
Include which in what? We're talking about compiled language programming headers for the libraries, and that's still a very technical term, so we tuck it away in framework/.

Moved share/aclocal/ to framework/AutoConfigure/.
This contains M4 macros for the GNU AutoTools, AutoConfigure and AutoMake, that many packages use for source code configuration. Very technical and applies to the libraries, so it's part of the framework. Besides, M4 macros are scripts, so it's not even proper data/. Not even techies can know what aclocal/ means until they find the explanation. Having a directory named local inside the directory share/ is also pretty disingenious. This is not a universally used configuration system, so we name it after the clearest component, that also has a relation to the original name.

Consolidated lib/pkgconfig/ and share/pkgconfig/ into framework/PackageConfigure/.
This is for another software configuration system named pkg-config, and contains data files describing properties of the package. Again, this is mostly meant for other packages to use the libraries in this resource package, so it's part of the framework. We use the clear CamelCase version of the name of the configuration system. Packages in the wild are confused about whether this is data or belongs with the libraries. Consolidating those two locations into one simplifies our search path for PackageConfigure files.

Moved lib/REBOL/ to framework/REBOL/.
Programming languages tend to have their own hierarchy under lib/ for their support files.
There is no standard place for REBOL and Boron libraries, so we are free to change this.
Having it immediately under framework/ instead of under framework/libraries/ shortens paths.

Moved etc/ to settings/.
etc/ must be the dumbest computer term ever introduced. It seems the designers were completely out of inspiration. I imagine it was scribbled on a napkin over lunch at the end of the other brilliant ideas for the Unix filesystem layout, and later implemented without further thought. While it's understandable that these people didn't count on common people needing to understand their system in the early 1970's, the naming is even of exceptionally poor quality in a technical context. What's really stupefying, though, is that these design choices have been defended as high culture by Unix types ever since.
Note that eventually, we shouldn't have changing data in program directories, so in most cases we direct this settings directory to the global and unfortunately standard /etc/.

Unlike earlier restructurings, the locations of these directories are critical to the operation of the software packages. Many components have accompanying search paths, that I have adapted in different places in the system. However, packages don't always use these and may try to find components in fixed locations or relative to themselves. Many packages configured with the GNU AutoTools support build options to change these locations. As much as possible, Builder now uses those options to perform these moves, so the packages are aware of them. However, many package configurations are partly broken, especially regarding this because few systems are interested in changing these locations. Such packages may need to be repaired by adding compatibility symlinks from the traditional subdirectories to the new ones.

This can actually be turned into an advantage. Quite a few resource packages act as platforms for other packages. Those other packages contain extensions in the same location as in the platform package. In other systems, these files would end up in the very same directories. In Syllable, the platform packages can't find those modules because packages are built into their own, separate directories. However, they are registered in our symlink pools to create the same view as in other systems. By using the traditional subdirectories in platform packages to symlink into the symlink pools, they can be pointed to the extensions in other packages.

Moved ^/lib/ to ^/libraries/.
This is the subdirectory relative to a program executable, that can contain private dynamic libaries for the program. Changed it to match the other renaming.

Some of these changes still have the old counterparts in the system, so that most old resource packages and applications will still work on systems with the new structure, until they are obsolete.

Reorganised the init scripts.

We've been porting more Unix server packages over time. They may need setup and teardown procedures that are usually categorised in five types: start, stop, reload, restart and status. We have init/ and early-init/ subdirectories within ported packages. init/ corresponds to the start script in Unix systems, which is actually a better name, because common users have no idea what “init” means. There's no real equivalent to early-init/, and we use that mainly to set variables in users' shell environments. Eventually, we would have to support the other script types, as well. Having them each in a different subdirectory within a package would be a mess.

Created a new (optional) subdirectory tasks/ within resource packages. Builder and Packager work together to symlink the content into /resources/index/ and other symlink pools.
Moved the init/ subdirectory to tasks/start/.
Moved early-init/, having no naming identification with init/ anymore, to tasks/setup/.
This name also better reflects the fact that setup scripts are called at every shell start, not just at system start-up or user log-in.
Future scripts can be added under tasks/ as tasks/stop/, tasks/reload/, tasks/restart/ and tasks/status/.

Reorganised, unified and integrated the documentation in ported packages.

Third-party packages have at least six locations for documentation. We collect loose documentation files in a documentation/ subdirectory. Many packages also have freeform documentation in a share/doc/ subdirectory. Traditionally, “man” pages are in man/ but GNU has been moving them to share/man/ lately. They prefer “info” files, anyway, and have first defined those in info/ but are now moving them to share/info/.

This is a mess for users. We want them to be able to go into a package directory just with a filer, and find documentation they are looking for. We were already moving share/man/ and share/info/ back to man/ and info/ because the share/ subdirectory is not very meaningful in Syllable. We build a self-contained system that typically doesn't share system files over networks, and we symlink several subdirectories to make them globally available within the system, not just the share/ hierarchy.

Any files in share/doc/ are now merged into documentation/. The man and info pages should not go in there, because they're structured databases that are meant to be read in their respective viewers. man/ and share/man/ are now moved to manuals/, a much clearer name. Because they have a structured content, it's possible to also move info/ and share/info/ to manuals/info/. The latter was pioneered by GoboLinux. Builder and Packager work together to accomplish this. As much as possible, Builder uses packages' configure options so they are aware of the moves.
Remaining files are moved afterwards.

The GCC libraries and headers installed in the system now have priority over a separately installed full GCC package. This protects system integrity while allowing to install versions of GCC older than the system libraries. This is a cleaner way than manipulating the GCC package, which was done before.

There are several legitimate reasons for installing a GCC package that is not in sync with the system libraries. The GCC package may be newer than the system release. It may be older, or an old GCC may be required to compile some program. If the versions diverge too much, this can disrupt the working of the system.
Originally, I removed the shared libraries from the GCC packages, but that may not be enough. Then I removed the complete lib/ subdirectory, but GCC does not appreciate that in all cases. Currently, the GCC packages have their include/ and lib/ subdirectories linked to the versions in the system. However, that may not work when the GCC version is too different from the system. Also, it makes the complete /system/index/include/ and /system/index/lib/ structures appear in /resources/index/, where they may conflict with other versions of the same packages.

What's really needed is to give the system libraries priority over GCC, while letting GCC keep its own versions for when it needs those internally. I chose to do that through the system paths. Other options are installing GCC in a separate package pool, or making an exception for it when symlinking in Packager.

Note that other packages in /resources/ still have priority over /system/resources/, so apart from GCC, it's still possible to overrule system components.

The symlink pools were moved from /system/indexes/ and /resources/indexes/ to /system/index/ and /resources/index/ to shorten paths throughout the system. The old locations are still available as compatibility symlinks to the new locations, to support packages that were built before Syllable 0.6.7.

/usr/ is now a symbolic link to /resources/ instead of the other way around. In the next step of this migration, they will be separated into two directories.

Implemented, in Packager, extra symlinking in the symlink pools to point to the main version of installed resource packages. This allows to point to packages in versioned pools without mentioning the version. The package symlink points to the main/currently selected version.

The links are made in the symlink pools, not in the installed packages. This allows to have multiple symlink pools making different selections of the same packages, for example for different user accounts or software build environments.

Symlinks that are made from the system into resource packages (usually for compatibility reasons) can be completed by Builder. So far, Builder added the complete path of the specific resource package, including the version for packages in the system area. This is not robust against upgrades of a versioned package or a change of the package name, because these symlinks are not included in single binary packages.

Now, Builder completes incomplete paths in a recipe based on the symlink pool of the package. Thus, the symlink does not contain the package name and version anymore and remains valid when those change.

Programming

The situation of the REBOL family of programming languages is currently in a transition phase. We try to select only one of each class of software sub-systems in Syllable, but we make an exception for REBOL because of its innovative qualities. As we predicted many years ago, there are now several open source alternatives, but they haven't yet reached the level where they can fully replace REBOL. In particular, we need REBOL because our in-house content management system still runs on it. We won't make the Syllable base platform dependent on closed source components, so we used ORCA to implement some system tools. Boron is now available as its successor, but we expect that both REBOL and Boron will eventually be replaced by the new Red language.

Boron, the successor to the ORCA programming language, was added. Syllable's system tools (such as Packager) haven't been migrated yet, so ORCA is still included as well.

Here is a Boron presentation we did for Software Freedom Day. We also made a web site where you can try out and compare REBOL and Boron on-line.

REBOL 3 was added, and a cURL binding for it, written by us.

REBOL 3 has been split into a core interpreter library and an open source host environment for integration with as many systems as possible. Kaj has got this version to work on Syllable Desktop. This is the first time that a version of REBOL can run there (besides emulating a complete operating system), so this is a major step towards our REBOL cross-platform strategy.

Interestingly, this R3 version is not fully native to Syllable yet and is not fully ported. The open host environment has been compiled on Syllable, but is using the binary Linux version of the core interpreter library. Normally, this is impossible, but it works in this case through a trick. Both Linux and Syllable use the GNU GLibC C library, but the porting layer and the internal library versions are different. Providing symlinks from the Linux library versions to the Syllable versions makes this work. This is exceptional and is a testament both to Syllable's compatibility with Linux and to the excellent isolation the host environment achieves between REBOL 3 and the host platform.

These compatibility symlinks could potentially be used to run other binary Linux libraries.

The current version of REBOL 3 for Syllable interfaces with the system at the POSIX level. Beyond that, you can write your own bindings with system and user libraries. Bindings are written as REBOL 3 extensions. We have created a binding with the popular cURL library, which is included in Syllable as its networking library.

Just like the core REBOL 3 interpreter library, Syllable Desktop can run the Linux binary of the binding unchanged, so that the same binary can be shared between Syllable Desktop and Syllable Server. This is possible because the interfaces between REBOL 3 and its extensions are designed for maximal portability, to avoid dependency hell. Documentation for the cURL binding is available here.

A massive overhaul of the Syllable system headers for C and C++ development. The intention is to stop kernel only types and structures “leaking” into user-space, where they can cause problems.
The changes should not impact user-space code except in cases where the code is not explicitly including headers it should be and relying on the side-effect of other system headers including them (e.g. using snooze() without including syllable/time.h) or using headers which they should not or do not use (e.g. kernel/udelay.h).

Another important change has been to deprecate the atheos/kernel.h header, which was pretty much used as a dumping ground for “everything else”. The contents of this header has been split into separate headers, and a new header has been provided for compatibility. atheos/kernel.h should not be used at all for kernel or driver code.

The changes have added a new kernel directory, which contains headers which are used by the kernel and device drivers, and the syllable symlink: this is currently a symbolic link to the atheos directory but the intention is to remove the atheos directory totally at some point in the future.

From this point forward new code should include headers from the syllable directory rather than atheos. Old code will be changed over time.

There is still work to be done to clean up the headers to ensure the licence and copyright information are correct for all headers, move anything which is in the wrong place (a lot of stuff in the kernel/ headers is internal to the kernel so should be moved to the kernels own inc/ directory, for example).
These changes can and will be made gradually over time.