diff --git a/doc/sdk-01-overview.md b/doc/sdk-01-overview.md new file mode 100644 index 0000000..b36dbb9 --- /dev/null +++ b/doc/sdk-01-overview.md @@ -0,0 +1,287 @@ +The Devuan SDK +============== + +The Devuan SDK is a unique build framework written to ease maintenance +and production of various types of the Devuan distribution images, such +as: live ISOs, virtual machine images, and images targeted at embedded +ARM boards. This paper explains how to use the SDK, gives and inside +look at its various parts and documents the workflow to be used when +modifying its code. + +The SDK is designed in such a way that there are levels of priority +within the scripts. First there is libdevuansdk, which holds the vanilla +configuration, then come the various wrappers targeted around specific +targets (live, virtual, embedded), and afterwards we optionally add more +on top of it if we need to customize or override specific functions. +This is for example the case with DECODE OS, where we have to add +additional software and extra components on top of the base Devuan +system. + + +libdevuansdk +------------ + +libdevuansdk is the core of any part of the Devuan SDK. It holds the +common knowledge between all of the upper wrappers such as live-sdk, +vm-sdk, and arm-sdk. Simply put, it is a shell script library to unify +the use and creation of various functions spread throughout the complete +Devuan SDK. + +The wrappers are designed to be used interactively from a terminal, as +well as automated from shell scripts. libdevuansdk uses an additional +zsh library called [zuper](https://github.com/dyne/zuper) to ease the +variable declaration and scoping, as well as error checking and +debugging. However, zuper is not included in libdevuansdk itself - one +is required to include it in its respective wrapper. live-sdk, vm-sdk, +and arm-sdk can be taken as example. libdevuansdk itself has some +software dependencies though: + +``` +zsh +debootstrap +sudo +kpartx +cgpt +xz-utils +``` + + +### Workflow + +Working with libdevuansdk splits into categories of what you want to do. +_zlibs_ are files separated into these categories: + +* ***bootstrap*** Contains the functions for the bootstrap process. + Creating a minimal debootstrap base, and making it into a tarball for + later use so one does not have to wait for the lengthy bootstrap + process on each consequent build. + +* ***helpers*** Contains the helper functions for libdevuansdk that make + the workflow a bit easier to use and handle. + +* ***imaging*** Contains the functions necessary for creating raw + dd-able images. + +* ***rsync*** Contains rsync and copying functions. + +* ***sysconf*** Contains the default system configuration. + + +### Usage + +As libdevuansdk is not very helpful when being used on its own, its +usage will be explained at later parts, for each specific wrapper. The +Technical documentation of libdevuansdk will follow in its appropriate +section. + + +The wrappers +------------ + +As mentioned, libdevuansdk is the core library we wrap around. The +currently existing wrappers are called _live-sdk_, _vm-sdk_, and +_arm-sdk_. These facilitate the builds of liveCDs, virtual machines, and +images for embedded ARM devices, respectively. Each of them have their +own section in this paper. + +Since all of these wrappers, along with libdevuansdk, hold a vanilla +Devuan configuration, you might prefer not to change their code. Due to +this, a concept called *blends* was introduced. Blends are a simple way +to customize the base image before building it, allowing you to very +easily add packages, kernels, and virtually anything one might want to +do in the image. This exactly is the case with DECODE OS. + + +arm-sdk +------- + +The _arm-sdk_ is our way of facilitating builds for embedded ARM boards +such as Allwinner-based CPUs, Raspberry Pis, Chromebooks, etc. It holds +a knowledgebase for a number of embedded devices, and how to build +according kernels and bootloaders. + + +### Directory structure + +arm-sdk's directory structure is separated into places where we hold our +boards and their kernel configurations, device-specific directories with +firmware and/or configuration, and a lib directory (where we keep +libdevuansdk and the like). + + +### Obtaining arm-sdk + +The SDK, like any other, should be obtained via git. The repositories +are hosted on Devuan's Gitlab. To grab it, we simply issue a _git clone_ +command, an since it contains git submodules - we append _--recursive_ +to it: + +``` +$ git clone https://git.devuan.org/sdk/arm-sdk --recursive +``` + +Consult the README.md file found in this repository to see what are the +required dependencies to use arm-sdk. + + +### Using arm-sdk + +Once the build system is obtained, it can now be used interactively. The +process is very simple, and to build an image one can actually use a +single shell command. However, we shall first show how it works. + +In arm-sdk, every board has its own script located in the _boards_ +directory. In most cases, these scripts contain functions to build the +Linux kernel, and a bootloader needed for the board to boot. This is the +only difference between all the boards, which requires every board to +have their own script. We are able to reuse the rootfs that is +bootstrapped before. For our example, let's take the _Nokia N900_ build +script. To build a vanilla image for it, we simply issue: + + +``` +$ zsh -f -c 'source sdk && load devuan n900 && build_image_dist' + +``` + +This will fire up the build process, and after a certain amount of time +we will have our compressed image ready and checksummed inside the +_dist_ directory. + +The oneliner above is self-explanatory: We first start a new untainted +shell, source the sdk file to get an interactive SDK shell, then we +initialize the operating system along with the board we are building, +and finally we issue a helper command that calls all the necessary +functions to build our image. The _load_ command takes an optional third +argument which is the name of our blend (the way to customize our +vanilla image) which will be explained later. So in this case, our +oneliner would look like: + +``` +$ zsh -f -c 'source sdk && load devuan n900 decode && build_image_dist' +``` + +This would create an image with the _"decode"_ blend, which is available +by cloning the DECODE OS git repository. The *build_image_dist* command +is a helper function located in libdevuansdk that wraps around the 8 +functions needed to build our image. They are all explained in the +technical part of this paper. + + +live-sdk +-------- + +The _live-sdk_ is used to build bootable images, better known as Live +CDs. Its structure is very similar to _vm-sdk_ and is a lot smaller than +_arm-sdk_. + + +### Directory structure + +Unlike arm-sdk, in live-sdk we have no need for specific boards or +setups, so in this case we only host the interactive shell init, and +libraries. + + +### Obtaining live-sdk + +The SDK, like any other, should be obtained via git. The repositories +are hosted on Devuan's Gitlab. To grab it, we simply issue a _git clone_ +command, an since it contains git submodules - we append _--recursive_ +to it: + +``` +$ git clone https://git.devuan.org/sdk/live-sdk --recursive +``` + +Consult the README.md file found in this repository to see what are the +required dependencies to use live-sdk. + + +### Using live-sdk + +Much like _arm-sdk_, the _live-sdk_ is used the same way. With two +specific differences. Since we don't have any need for specific boards, +with loading we don't specify a board, but rather the CPU architecture +we are building for. Currently supported are *i386* and *amd64* which +represent 32bit and 64bit respectively. To build a vanilla live ISO, we +issue: + +``` +$ zsh -f -c 'source sdk && load devuan amd64 && build_iso_dist' +``` + +This will start the build process, and after a certain amount of time we +will have our ISO ready and inside the _dist_ directory. + +Just like in arm-sdk, we can use a _blend_ and customize our OS: + +``` +$ zsh -f -c 'source sdk && load devuan amd64 decode && build_iso_dist' +``` + +So this would create a live ISO of DECODE OS. Again as noted, this can +be obtained by recursively cloning the decode-os git repository. + +The *build_iso_dist* command is a helper function located in +libdevuansdk that wraps around the 9 functions needed to build our +image. They are all explained in the technical part of this manual. + + +vm-sdk +------ + +The _vm-sdk_ is used to build VirtualBox/Vagrant boxes, and virtual +images for emulation, in QCOW2 format, which is a nifty byproduct of +building a Vagrant box. Its structure is very similar to _live-sdk_ and +is the smallest of the three wrappers currently found in the Devuan SDK. + + +### Directory structure + +Like with live-sdk, in vm-sdk we have no need for specific boards or +setups, so in this case we only host the interactive shell init, and +libraries. + + +### Obtaining vm-sdk + +The SDK, like any other, should be obtained via git. The repositories +are hosted on Devuan's Gitlab. To grab it, we simply issue a _git clone_ +command, an since it contains git submodules - we append _--recursive_ +to it: + +``` +$ git clone https://git.devuan.org/sdk/vm-sdk --recursive +``` + +Consult the README.md file found in this repository to see what are the +required dependencies to use vm-sdk. + + +### Using vm-sdk + +Once obtained, we can use it interactively. The process is very simple, +and to build an image we use the oneliner we've already seen above. + +Also like with live-sdk, we don't need specific boards, however we also +do not create any non-amd64 images, so we don't have to pass an +architecture to the load command either. To build a vanilla Vagrant Box, +VirtualBox image, qcow2 image, and a cloud-based qcow2 image, we issue: + +``` +$ zsh -f -c 'source sdk && load devuan && build_vagrant_dist' +``` + +This line would create al the four types of the VM image. + +As shown with the previous two, the _blend_ concept works as advertised +here as well: + +``` +$ zsh -f -c 'source sdk && load deuvan decode && build_vagrant_dist' +``` + +The *build_vagrant_dist* command is a helper function located in +libdevuansdk that wraps around the 11 functions needed to build our +image. They are all explained in the technical part of this manual. diff --git a/doc/sdk-02-blends.md b/doc/sdk-02-blends.md new file mode 100644 index 0000000..f8ef73c --- /dev/null +++ b/doc/sdk-02-blends.md @@ -0,0 +1,317 @@ +Blends +====== + + +Introduction +------------ + +In the Devuan SDK, a _blend_ is the preferred way we use to make +customizations to the vanilla image. Using blends we can very easily +create different flavors of our image, by easily including/excluding +certain software packages, files, or anything we wish to do as a matter +of fact. Blends can become a very quick way of creating entire new +derivatives of the vanilla distribution we are building. + +This time, we will take the DECODE OS as a blend example. In DECODE OS +we provide a blend called _decode_ which is the blend we use to create +a production release of DECODE OS. The blend's files are contained +within their own directory in the decode-os git repository. + + +Configuration +------------- + +Any SDK requires a single file to act as a blend. This file is also a +zsh script, and, at the very least, it must contain two functions +called: + +``` +blend_preinst() +blend_postinst() +``` + +These functions are your pathway to expanding your blend into whatever +you would like to do. The _preinst_ function is usually called right +after bootstrapping the vanilla root filesystem, and the _postinst_ +function is called near the very end, just before packing or compressing +the image. These two strategic places should be enough to do changes +within the image. If this is not enough, blends also allow you to simply +**override any variable or function** contained within libdevuansdk or +the sdk you are using. + +Our _decode_ blend is such an example. It is a somewhat expanded blend, +not contained within a single file, but rather a directory. This allows +easier maintenance and makes the scripts clearer and cleaner. + + +### Adding and removing packages + +When we want to add or remove specific packages to our build, we have to +override or append to libdevuansdk's arrays. The array for packages we +want installed is called *extra_packages*, and the array for packages we +want purged is called *purge_packages*. In the decode blend, these can +be found in the _config_ file located inside the decode-os blend +directory. Keep in mind that these arrays could already contain +specific packages, so you are advised to rather append to them, than +overriding them. + +If the packages you want to install are not available in the repos, you +still have a way of automatically installing them. All you need to do to +take care of it is at some point in your blend - copy your .deb files to +the following directory: + +``` +$R/extra/custom-packages/ +``` + +And when that is done, just call the function *install-custdebs* + + +Creating a blend +---------------- + +Rather than explaining theory, you are best off viewing the blend files +that are provided with _decode-os_. It is a fairly simple blend and +should give you enough insight on creating your own blend. Here are some +important guidelines for creating a blend: + + +* The blend should always contain at least two functions + +This means you must provide *blend_preinst* and *blend_postinst* in your +blend. They don't even have to do anything, but they should be there. +These two functions open the path for you to call any other functions +you created for your blend. + + +* When overriding functions, make sure they provide a result that + doesn't break the API + +Breaking the API may result in unwanted behavior. You should always +study well the functions you are planning to override and figure out if +it is safe to override them in the way you want. The same goes for any +variables as well. + + +* Any arguments used after the blend name when loading from the SDK are + free for you to use in the blend. + +This means you can use anything **after $4** inside your blend if you +require passing arguments to it. + +These are some of the more important guidelines. There is plenty more +tricks and quirks, but it's easy to find out once you read a blend or +two on your own... + + +### Enable the blend + +To use your blend in the first place, you need to make the sdk know +about it. To make this work, you need to append the path to your new +blend inside the **blend_map** of the _sdk_ file: + +``` +blend_map=( + "devuan-live" "$R/blends/devuan-live/devuan-live.blend" + "decode" "$R/../decode.blend" + "heads" "$R/../heads.blend" + "ournewblend" "$R/blends/newblend/new-blend.blend" +) +``` + +As you can see, the map is a key-value storage. So you can have an alias +(name) for your blend, and just use that to point to the path of the +blend. The blend file will be sourced by the sdk once it is told to do +so. + + +### A configuration file + +For having a finer-grained control of what goes into our build, we can +create a config file for our blend. From here we can easily control any +configurable aspect of our blend, such as packages that go in or out, +the blend name, and much more. **Make sure you source this file from +your blend.** + +Adding and removing packages was abstractly mentioned earlier: it goes +into two separate arrays holding package names. To add packages, we +append to the **extra_packages** array, which would look like this: + +``` +extra_packages+=( + my_new_package + foo + bar + baz +) +``` + +This would install these four packages, along with the ones predefined +in either libdevuansdk or the sdk you are using. You may also want to +see which those are in case you wish to exclude them, but they are sane +and useful utilities which should be included in your build if possible. +Overriding all those packages, you would need to reset the whole array, +so you would simply issue this: + +``` +extra_packages=( + my_new_package + foo + bar + baz +) +``` + +As you can see, we no longer have the _+=_, but rather only _=_, which +means we are not appending to the array, but rather redefining it. + +All of the above applies as well for removing packages, but in this case +the array is called **purge_packages**. + + +#### Custom packages + +If you want to install deb packages that aren't in any repositories, put +them in the blend directory and simply add them to another array in the +configuration file. The contents of the arrays are the paths to the +debs, relative to this configuration file: + +``` +custom_deb_packages=( + yad_0.27.0-1_amd64.deb + palemoon_27.2.0~repack-1_amd64.deb +) +``` + +To trigger installation of these packages, you will need to copy them to +`$R/extra/custom_packages`, and then call the **install_custdebs** +function somewhere from your blend. + + +### Custom files + +Any files you want to add to the system to override what's there by +default you can add using a *rootfs overlay*. Create a directory inside +your blend directory called *rootfs-overlay* and simply put files inside +it. The directory structure is absolute to the image we are building. +For example what's in "rootfs-overlay/etc/" would end up in the "/etc" +of our final image. See _hier(7)_ from the Linux manpages for more +explanation on this directory hierarchy. + +If you end up with any files here, to actually copy them, you will need +to `cp -f` it, or `rsync` it if you prefer. + + +### The .blend file + +We listed a path to the .blend file in our first step. We need to create +this file now. + +Start your blend file with the following, so the sdk is aware of the +environment: + +``` +BLENDPATH="${BLENDPATH:-$(dirname $0)}" +source $BLENDPATH/config +``` + +The minimum blend should contain two functions: **blend_preinst** and +**blend_postinst**. These functions are called at specific points in the +build, where they give the most power: just after bootstrapping the +vanilla system, and just before packaging the final build, respectively. + + +#### blend_preinst + +A preinst function can look like this: + +``` +blend_preinst() { + fn blend_preinst + req=(BLENDPATH R) + ckreq || return 1 + + notice "executing blend preinst" + + add-user "user" "pass" + cp -fv "$BLENDPATH"/*.deb "$R/extra/custom-packages" || zerr + install-custdebs || zerr +} +``` + +So as you can see, the preinst function will add a new user with the +credentials `user:pass`, it will copy our custom debs where they can be +used, and finally it will trigger their installation. + +The `fn, req, ckreq` part on the top of the function is a safety check +for the function that is enabled by zuper. It allows us to check if +variables are defined when the function is called and fail if it is +wrong. You should utilize this as much as possible. The `zerr` calls are +used to exit if the function fails. + + +#### blend_postinst + +A postinst function can look like the following: + +``` +blend_postinst() { + fn blend_postinst + req=(BLENDPATH strapdir) + ckreq || return 1 + + notice "executing blend postinst" + + sudo cp -vf "$BLENDPATH"/rootfs-overlay/* $strapdir || zerr + + blend_finalize || zerr +} +``` + +This function would copy the `rootfs-overlay` to the `strapdir` (which +holds our image's filesystem) and it would call the `blend_finalize` +function. By default this function doesn't exist, but it's an example so +you can see you can call your own functions as well. You can define them +within the blend file. + + +Using a blend +------------- + +As explained in previous chapters, you can use your blends through the +interactive SDK shell. In decode-os the blend is placed in the root of +the git repository, and the sdk wrappers are located within. Therefore +an sdk would have to source it with such a path: + +``` +$R/../decode.blend +``` + +If you take a look at vm-sdk's *sdk* file, you can see it in the +*blend_map*. Using a new blend requires you to add it to this map in +the same manner. The map is key-value formatted, and on the left you +have an alias of your blend, and on the right you have a script you have +to write. It can either be the blend itself or any helper file you might +need to initialize your blend. + +After you've added it to the blend map, you simply initialize the sdk, +and use the same *load* command we learned earlier, while appending the +blend alias and any optional argument. + +``` +$ zsh -f +$ source sdk +$ load devuan decode +``` + +And we've initialized our *decode* blend. It's always good to add a +*notice()* call to your blend to signal it's been loaded successfully. + +After this is done, we simply build the image the same way we learned +before: + +``` +$ build_vagrant_dist +``` + +Consult the vm-sdk chapter for this. diff --git a/doc/sdk-03-technical.md b/doc/sdk-03-technical.md new file mode 100644 index 0000000..f9d9e3c --- /dev/null +++ b/doc/sdk-03-technical.md @@ -0,0 +1,130 @@ +The Devuan SDK more in-depth +============================ + +The following parts will explain the Devuan SDK more technically. It +will show its configuration, important functions, and show how it all +glues together. + + +Configuration +------------- + +Much of the libdevuansdk configuration is done in `libdevuansdk/config`. +Here you can edit the defaults if you wish to do something your needs +are expressing. However, overriding these through upper levels is +recommended. + + +### `config` file + +`vars` and `arrs` are global arrays for holding other global variables +and arrays, respectively. This is required for `zuper` and helps a lot +with debugging. If you declare new variables or arrays, add them to the +aforementioned variables. + + +* `os` holds the name of the distribution being worked on. + +* `release` holds the release codename of the distribution. Used for apt + repositories mostly. + +* `version` is the version of the distribution being worked on. + +* `mirror` is a mirror holding the required packages for `debootstrap`. + +* `section` are the sections of the repository. For adding in + `/etc/apt/sources.list`. Separate them with whitespaces. + +* `image_name` is the output name of the raw image. If you declare a + blend or a device name (arm-sdk), they will be appended to this name. + +* `rootcredentials` and `usercredentials` are currently placeholders. + +* `core_packages` is an array holding the core packages that will be + installed in the bootstrap process. + +* `base_packages` is an array holding the base packages that will be + installed at a later point in the bootstrap process. + +* `purge_packages` is an array of packages that will get purged at the + end of the bootstrap process. + + +Helper functions +---------------- + +You can find useful helper functions in `libdevuansdk/zlibs/helpers`. +They are intended to help when it comes to writing wrappers, as well as +making the developers' jobs easier for developing libdevuansdk. Some of +these functions are required for libdevuansdk to work properly as well. + + +### `build_image_dist()` + +This function is a kind of a wrapper function. It's used in arm-sdk to +build a complete dd-able image from start to end. To run, it requires +`$arch`, `$size`, `$parted_type`, `$workdir`, `$strapdir`, and +`$image_name` to be declared. See the part of "Creating wrappers" for +insight on these variables. + +The workflow of this function is bootstrapping a complete rootfs, +creating a raw image, installing/compiling a kernel, rsyncing everything +to the raw image, and finally compressing the raw image. + +This same workflow is applied in the next two functions in this file, +which are `build_iso_dist` and `build_vagrant_dist`. To get a better +understanding of libdevuansdk, it's recommended to go through one of +these functions and following it deeper to find and figure out the other +functions and how they work together. + + +### `devprocsys()` + +This function is a simple helper function that takes two arguments. It +mounts or unmounts `/dev`, `/proc`, and `/sys` filesystems to or from +wherever you tell it to. For example: + +``` +$ devprocsys mount $strapdir +$ devprocsys umount $strapdir + +``` + +It is very necessary to use this if one wants to do anything requiring +access to hardware or the system's resources, i.e. cryptography. + + +### `dpkgdivert()` + +This function, like `devprocsys` takes two arguments and will create or +remove a dpkg diversion in the place you tell it to and remove +`invoke-rc.d` so that apt does not autostart daemons when they are +installed. + + +### `chroot-script()` + +This very useful functions allows you to chroot into `$strapdir` and +execute the script/binary that's passed as a parameter to this function. +It also takes an optional argument `-d` that will call dpkgdivert on and +off before and after execution. + +The `chroot-script` is also an example on its own that shows how to use +the `chroot-script` function. + + +Mandatory variables +------------------- + +* `$R` is the root directory of a wrapper. It's defined already in all + the existing ones. In almost evert situation it can be `$PWD`. + +* `$workdir` is the working directory of the current build. A sane + default is `$R/tmp/workdir` + +* `$strapdir` is the bootstrap directory of the build. It holds the + rootfs when you debootstrap it, and customize it further on. Default + is `$workdir/rootfs`. + +* `$arch` is the CPU architecture of the build. I.e. `amd64`, `armhf`, + etc. diff --git a/doc/toaster-01-overview.md b/doc/toaster-01-overview.md new file mode 100644 index 0000000..936b84a --- /dev/null +++ b/doc/toaster-01-overview.md @@ -0,0 +1,33 @@ +toaster.do +========== + +The **toaster.do** setup is an ecosystem of modular parts of software +used to facilitate builds of customized Devuan images using Dockerfiles +and a web interface. It allows us to have a seamless way of using the +Dockerfiles that are used in testing to make production images using the +same Dockerfile. This brings a deterministic approach to debugging and +allows centralization of resources, while avoiding extra work needed to +write a Devuan blend. + +The setup is comprised of a web interface written in Clojure, a backend +glue written in Python, the Devuan SDK, and the Jenkins CI system. + + +Clojure frontend +---------------- + +The Clojure frontend is an embedded web server with its own database, +which allows for managing of users. A user registered within this part +is then allowed to upload Dockerfiles and manage their image builds. + +The frontend talks to the Python backend through SSH, and runs a +specific command to enable or disable a build job. + + +Jenkins backend +--------------- + +The backend glue is a Python tool which talks to Jenkins itself and +does all the managing and configuration of build jobs. It serves as the +backend to the Devuan SDK's web interface and is executed by the web CGI +when a build function is requested.