Configuration

When building packages, Bob executes the instructions defined by the recipes. All recipes are located relative to the project root directory in the recipes subdirectory. Recipes are YAML files with a defined structure. The name of the recipe and the resulting package(s) is derived from the file name by removing the trailing ‘.yaml’. To aid further organization of the recipes, they may be put into subdirectories under recipes. The directory name becomes part of the package name with a ::-separator.

To minimize repetition of common functionality, there is also an optional classes subdirectory. Classes have the same structure as recipes and can be included from recipes and other classes to factor out common stuff. Files that do not have the ‘.yaml’ extension are ignored when parsing the recipes and classes directories.

There are two additional configuration files: config.yaml and default.yaml. The former contains static configuration options while the latter holds some options that can have a default value and might be overridden on the command line. Putting that all together, a typical recipe tree looks like the following:

.
├── classes
│   └── make.yaml
├── config.yaml
├── default.yaml
└── recipes
    ├── busybox.yaml
    ├── initramfs
    │   ├── inittab
    │   └── rcS
    ├── initramfs.yaml
    ├── linux.yaml
    ├── toolchain
    │   ├── arm-linux-gnueabihf.yaml
    │   ├── make.yaml
    │   └── x86.yaml
    └── vexpress.yaml

Such a recipe and configuration tree is meant to be handled by an SCM (Source Code Manager). As you can see in the above tree there is a toolchain subdirectory in the recipes. The packages in this directory will be named toolchain::arm-linux-gnueabihf, toolchain::make and toolchain::x86. You can also see that there are other files (initramfs/…) that are included by recipes but are otherwise ignored by Bob.

In addition to the single project tree configuration, Bob supports layers. A layer has the same structure as shown above but is merged with the other layers during parsing. This structure is recursive (a layer may contain another layer) but is flattened during parsing. A typical structure might look like the following:

.
├── classes
│   └── ...
├── config.yaml
├── default.yaml
├── layers
│   ├── bsp
│   │   ├── default.yaml
│   │   └── recipes
│   │       ├── linux.yaml
│   │       └── toolchain
│   │           └── arm-linux-gnueabihf.yaml
│   └── myapp
│       └── recipes
│           └── fancy-app.yaml
└── recipes
    ├── busybox.yaml
    ├── initramfs
    │   ├── inittab
    │   └── rcS
    ├── initramfs.yaml
    ├── toolchain
    │   ├── make.yaml
    │   └── x86.yaml
    └── vexpress.yaml

Conceptually the recipes (and classes) from the bsp and myapp layers are parsed together at the same level with the recipes of the project root. The recipes may reference each other freely. The only restriction is that a particular recipe or class must only be defined by one layer. It is not allowed to “overwrite” a recipe in a layer above.

Layers must be configured in config.yaml to be picked up. This is just a simple list of layer names that are then expected in the layers directory:

layers:
    - myapp
    - bsp

The order of layers is important with respect to settings made by default.yaml in the various layers. The project root has the highest precedence. The layers in config.yaml are named from highest to lowest precedence. A layer with a higher precedence can override settings from layers of lower precedence.

Layers allow you to structure your projects into larger entities that can be reused in other projects. This modularity helps to separate different aspects of bigger projects like the used toolchain, the board support package and the applications integration.

Principle operation

All packages are built by traversing the recipe tree starting from one or more root recipes. These are recipes that have the root attribute set to True. There must be at least one root recipe in a project. The tree of recipes is traversed depth first. While following the dependencies, Bob keeps a local state that consists of the following information:

Environment
Bob always keeps the full set of variables but only a subset is visible when executing the scripts. Initially, only the variables defined in default.yaml in the environment section are available. Environment variables can be set at various points that are described below in more detail.
Tools
Tools are aliases for paths to executables. Initially there are no tools. They are defined by provideTools and must be explicitly imported by upstream recipes by listing tools in the use attribute. Like environment variables, the tools are kept as key value pairs where the key is a string and the value is the executable and library paths that are imported when using a tool.
Sandbox
This defines the root file system and paths that are used to build the package. Unless a sandbox is consumed by listing sandbox in the use attribute of a dependency, the normal host executables are used. Sandboxed builds are described in a separate section below.

All of this information is carried as local state when traversing the dependency tree. Each recipe gets a local copy that is propagated downstream. Any updates to upstream recipes must be done by explicitly offering the information with one of the provide* keywords and the upstream recipe must consume it by adding the relevant item to the use attribute of the dependency.

Step execution

The actual work when building a package is done in the following three steps. They are scripts that are executed with (and only with) the declared environment and tools.

Checkout
The checkout step is there to fetch the source code or any external input of the package. Despite the script defined by checkoutScript, Bob supports a number of source code management systems natively. They can be listed in checkoutSCM and are fetched/updated before the checkoutScript is run.
Build
This is the step where most of the work should be done to build the package. The buildScript receives the result of the checkout step as argument $1, and any further dependency whose result is consumed is passed in order starting with $2. If no checkout step was provided, $1 will point to some invalid path.
Package
Typically, the build step will produce a lot of intermediate files (e.g. object files). The package step has the responsibility to distill a clean result of the package. The packageScript will receive a single argument with the path to the build step.

Each step of a recipe is executed separately and always in the above order. The scripts’ working directory is already where the result is expected. The scripts should make no assumption about the absolute path or the relative path to other steps. Only the working directory might be modified.

Script languages

Bob itself is written in the Python scripting language but actually independent of the scripting language that is used during step execution (see above). Currently Bob supports two scripting languages: bash and PowerShell. Classes and recipes may define their scripts in one or both scripting languages. The actually used language at build time is determined by the scriptLanguage key or, if nothing was specified, by the project scriptLanguage setting. The other language scripts are ignored.

Environment handling

The variables listed in environment of default.yaml with their configured value are mangled through String substitution by the current OS environment and are then taken over into the initial environment. The user might additionally override or set certain variables from the command line. Such variables are always taken over verbatim. The so calculated set of variables is the starting point for each root recipe.

Note

Depending on the cleanEnvironment policy the initial environment may first be populated with the whitelisted variables named by whitelist from the current OS environment. The new behaviour (i.e. enabled policy) is to start with a clean environment.

The next steps are repeated for each recipe as the dependency tree is traversed. A copy of the environment is inherited from the upstream recipe.

  1. Any variable defined in environment is set to the given value.
  2. Make a copy of the local environment that is subsequently passed to each dependency (named “forwarded environment” thereafter).
  3. For each dependency do the following:
    1. Make a dedicated copy of the environment for the dependency.
    2. Set variables given in the environment attribute of the dependency in this copy.
    3. Descend to the dependency recipe with that environment.
    4. Merge all variables of the provideVars section of the dependency into the local environment if environment is listed in the use attribute of the dependency.
    5. If the forward attribute of the dependency is True then any merged variable of the previous step is updated in the forwarded environment too.

After all dependencies have been processed, the environment variables of tools (see provideTools) that are used in the recipe are merged into the local environment. Finally, variables defined in privateEnvironment and metaEnvironment are merged too.

A subset of the resulting local environment can be passed to the three execution steps. The variables available to the scripts are defined by {checkout,build,package}Vars and {checkout,build,package}VarsWeak. The former property defines variables that are considered to influence the build while the latter names variables that are expected to not influence the outcome of the build.

A variable that is consumed in one step is also set in the following. This means a variable consumed through checkoutVars is also set during the build and package steps. Likewise, a variable consumed by buildVars is set in the package step too. The rationale is that all three steps form a small pipeline. If a step depends on a certain variable then the result of the following step is already indirectly dependent on this variable. Thus it can be set during the following step anyway.

A recipe might optionally offer some variables to the upstream recipe with a provideVars section. The values of these variables might use variable substitution where the substituted values are coming from the local environment. The upstream recipe must explicitly consume these provided variables by adding environment to the use attribute of the dependency.

Tool handling

Tools are handled very similar to environment variables when being passed in the recipe dependency tree. Tools are aliases for a package together with a relative path to the executable(s) and optionally some library paths for shared libraries. Another recipe using a tool gets the path to the executable(s) added to its $PATH.

Starting at the root recipe, there are no tools. The next steps are repeated for each recipe as the dependency tree is traversed. A copy of the tool aliases is inherited from the upstream recipe.

  1. Make a copy of the local tool aliases that is subsequently passed to each dependency (named “forwarded tools” thereafter).
  2. For each dependency do the following:
    1. Descend to the dependency recipe with the forwarded tools
    2. Merge all tools of the provideTools section of the dependency into the local tools if tools is listed in the use attribute of the dependency.
    3. If the forward attribute of the dependency is True then any merged tools of the previous step are updated in the forwarded tools too.

While the full set of tools is carried through the dependency tree, only a specified subset of these tools is available when executing the steps of a recipe. The available tools are defined by {checkout,build,package}Tools. A tool that is consumed in one step is also set in the following. This means a tool consumed through checkoutTools is also available during the build and package steps. Likewise, a tool consumed by buildTools is available in the package step too.

To define one or more tools, a recipe must include a provideTools section that defines the relative execution path and library paths of one or more tool aliases. These aliases may be picked up by the upstream recipe by having tools in the use attribute of the dependency.

Sandbox operation

Unless a sandbox is configured for a recipe, the steps are executed directly on the host. Bob adds any consumed tools to the front of $PATH and controls the available environment variables. Apart from this, the build result is pretty much dependent on the installed applications of the host.

By utilizing user namespaces on Linux, Bob is able to execute the package steps in a tightly controlled and reproducible environment. This is key to enable binary reproducible builds. The sandbox image itself is also represented by a recipe in the project.

Initially no sandbox is defined. A downstream recipe might offer its built package as sandbox through provideSandbox. The upstream recipe must define sandbox in the use attribute of this dependency to pick it up as sandbox. This sandbox is effective only for the current recipe. If forward is additionally set to True the following dependencies will inherit this sandbox for their execution.

Inside the sandbox, the result of the consumed or inherited sandbox image is used as root file system. Only direct inputs of the executed step are visible. Everything except the working directory and /tmp is mounted read only to restrict side effects. The only component used from the host is the Linux kernel and indirectly Python because Bob is written in this language. The sandbox image must provide everything to execute the steps. In particular, the following things must be provided by the sandbox image:

  • There must be an etc/passwd file containing the “nobody” user with uid 65534.
  • There must not be a home directory. Bob creates this directory on demand and will fail if it already exists.
  • There must not be a tmp directory for the same reason.
  • The interpreter of the used script language must be available (bash or pwsh) and it must be in $PATH. When using bash (the default) at least version 4 must be installed. Bob uses associative arrays that are not available in earlier versions.

String substitution

At most places where strings are handled in keywords, it is possible to use variable substitution. These substitutions might be simple variables, but a variety of string processing functions is also available that can optionally be extended by plugins. The following syntax is supported:

  • Variable substitution
    • ${var}: The value of var is substituted. The variable has to be defined or an error will be raised. The braces can be omitted if the variable name consists only of letters, numbers and _ and when the name is followed by a character that is not interpreted as part of its name.
    • ${var:-default}: If variable var is unset or null, the expansion of default is substituted. Otherwise the value of var is substituted. Omitting the colon results in a test only for var being unset.
    • ${var:+alternate}: If variable var is unset or null, nothing is substituted. Otherwise the expansion of alternate is substituted. Omitting the colon results in a test only for var being unset.
  • $(fun,arg1,...): Substitutes the result of calling fun with the given arguments. Unlike unix shells, which employ word splitting at whitespaces, the function arguments are separated by commas. Any white spaces are kept and belong to the arguments. To put a comma or closing parenthesis into an argument it has to be escaped by a backslash or double/single quotes.
  • Quoting
    • "...": Double quotes begin a new substitution context that runs until the matching closing double quote. All substituions are still recognized.
    • '...': Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.
    • \.: A backslash preserves the literal meaning of the following character. The only exception is within single quotes where backslash is not recognized as meta character.

The following built in string functions are supported:

  • $(eq,left,right): Returns true if the expansions of left and right are equal, false otherwise.
  • $(match,string,pattern[,flags]): Returns true if pattern is found in string, false otherwise. Quoting the pattern is recommended. Flags are optional. The only currently supported flag is i to ignore case while searching.
  • $(if-then-else,condition,then,else): The expansion of condition is interpreted as a boolean value. If the condition is true the expansion of then is returned. Otherwise else is returned.
  • $(is-sandbox-enabled): Return true if a sandbox is enabled in the current context, false otherwise.
  • $(is-tool-defined,name): If name is a defined tool in the current context the function will return true. Otherwise false is returned.
  • $(ne,left,right): Returns true if the expansions of left and right differ, otherwise false is returned.
  • $(not,condition): Interpret the expansion of condition as boolean value and return the opposite.
  • $(or,condition1,condition2,...): Expand each condition and then interpret each condition as boolean. Return false when all conditions are false, otherwise true.
  • $(and,condition1,condition2,...): Expand each condition and the interpret each condition as booelan. Return true when all conditions are true, otherwise false.
  • $(strip,text): Remove leading and trailing whitespaces from the expansion of text.
  • $(subst,from,to,text): Replace every occurence of from with to in text.

The following built in string functions are additionally supported in package path queries. They cannot be used in recipes as they work on packages:

  • $(matchScm,property,pattern): Return true if there is at least one checkoutSCM in the package that has a property that matches the pattern. Otherwise returns false. Shell globbing patterns may be used as pattern.

Plugins may provide additional functions as described in String functions. If a string is interpreted as a boolean then the empty string, “0” (zero) and “false” (case insensitive) are considered as logical “false”. Any other value is considered as “true”.

Host dependency fingerprinting

Bob closely tracks the input of all packages. This includes all checked out sources and the dependencies to other packages. If something is changed, Bob can accurately determine which packages have to be rebuilt. This information is also used to find matching binary artifacts. If a recipe depends on resources that are outside of the declared recipes, the situation changes though. Bob cannot infer what external resources are actually used and how these influence the build result.

A common host dependency that “taints” the build result is the host compiler. While the host compiler typically does not change, it limits the portability across machines in the form of binary artifacts. The dependency on the host architecture is obvious, but also the libc has to be considered. This can be extended to other libraries that might be used by the recipe.

To let Bob know about the usage and state of an external host resource, a fingerprint script can be used in the recipe. The output of the fingerprint script is used to “tag” the created package. If the fingerprint changes, the package is rebuilt. The fingerprint is also attached to the binary artifact. To download a binary artifact of a package, the fingerprint has to match.

The fingerprint does not apply to the checkoutScript, though. If the result of your checkoutScript depends on the host that it runs on, you have to set checkoutDeterministic to False. The fingerprint serves only as a virtual input to the build and package steps to declare to Bob what part of the host is used by the recipe.

The impact of the host that is declared by a fingerprint script applies only to the result of a recipe. Specifically, it does not apply to the implied behaviour of any provided tools. This means that when using a tool from another recipe that is directly or indirectly affected by a fingerprint, the using recipe is not affected. The rationale for this exception of transitivity is that it typically does not matter where a tool is built but how it behaves.

See fingerprintScript[{Bash,Pwsh}] and provideTools for information where fingerprint scripts can be configured.

Recipe and class keywords

{checkout,build,package}Script[{Bash,Pwsh}]

Type: String

This is the script that is executed by Bob at the respective stage when building the Packet. It is strongly recommended to write the script as a newline preserving block literal. See the following example (note the pipe symbol on the end of the first line):

buildScript: |
    $1/configure
    make

The suffix of the keyword determines the language of the script. Using the Bash suffix (e.g. buildScriptBash) defines a script that is interpreted with bash. Likewise, the Pwsh suffix (e.g. buildScriptPwsh) defines a PowerShell script. Which language is used at build time is determined by the scriptLanguage key or, if nothing was specified, by the project scriptLanguage setting. A keyword without a suffix (e.g. buildScript) is interpreted in whatever language is finally used at build time. If both the keyword with the build time language suffix and without a suffix are present then the keyword with the build language suffix takes precedence.

The script is subject to file inclusion with the $<<path>> and $<'path'> syntax. The files are included relative to the current recipe. The given path might be a shell globbing pattern. If multiple files are matched by path the files are sorted by name and then concatenated. The $<<path>> syntax imports the file(s) as is and replaces the escape pattern with a (possibly temporary) file name which has the same content. Similar to that, the $<'path'> syntax includes the file(s) inline as a quoted string. In any case, the strings are fully quoted and not subject to any parameter substitution.

Note

When including files as quoted strings ($<'path'> syntax), they have to be UTF-8 encoded.

The scripts of any classes that are inherited which define a script for the same step are joined in front of this script in the order the inheritance is specified. The inheritance graph is traversed depth first and every class is included exactly once.

During execution of the script only the environment variables SHELL, USER, TERM, HOME and anything that was declared via {checkout,build,package}Vars are set. The PATH is reset to “/usr/local/bin:/bin:/usr/bin” or whatever was declared in config.yaml. Any tools that are consumed by a {checkout,build,package}Tools declaration are added to the front of PATH. The same holds for $LD_LIBRARY_PATH with the difference of starting completely empty.

Additionally, the following (environment) variables are populated automatically:

  • BOB_CWD: Environment variable holding the working directory of the current script as absolute path.
  • BOB_ALL_PATHS: An associative array that holds the paths to the results of all dependencies indexed by the package name. This also includes indirect dependencies such as consumed tools or the sandbox.
  • BOB_DEP_PATHS: An associative array of all direct dependencies. This array comes in handy if you want to refer to a dependency by name (e.g. ${BOB_DEP_PATHS[libfoo-dev]}) instead of the position (e.g. $2).
  • BOB_TOOL_PATHS: An associative array that holds the execution paths to consumed tools indexed by the package name. All these paths are in $PATH resp. %PATH%.

The associative arrays are no regular environment variables. Hence they are not inherited by other processes that are invoked by the executed scripts. In bash scripts they are associative arrays. See Bash Arrays for more information. In PowerShell scripts they are defined as Hash Tables.

For PowerShell scripts a utility function called Check-Command is available. It has two arguments: the first one (ScriptBlock) expects a script block that is executed. The optional second argument (ErrorAction) lets you override the error action. After the script block was executed the Check-Command function will check the last exit status and invoke the error action if it is not zero. Example:

Check-Command { cmake --build . }

By default it will halt the script execution. This helper is needed because there is no possibility to configure PowerShell to stop execution when an external command fails. Make sure to wrap calls to external tools with Check-Command or check $lastexitcode yourself. Otherwise the build will not detect errors involving external commands!

{checkout,build,package}Tools

Type: List of strings

This is a list of tools that should be added to $PATH during the execution of the respective checkout/build/package script. A tool denotes a folder in an (indirect) dependency. A tool might declare some library paths that are then added to $LD_LIBRARY_PATH. The order of tools in $PATH and $LD_LIBRARY_PATH is unspecified. It is assumed that each tool provides a separate set of executables so that the order of their inclusion does not matter.

A tool that is consumed in one step is also set in the following. This means a tool consumed through checkoutTools is also available during the build and package steps. Likewise a tool consumed by buildTools is available in the package step too. The rationale is that all three steps form a small pipeline. If a step depends on a certain tool then the result of the following step is already indirectly dependent on this tool. Thus it can be available during the following step anyway.

{checkout,build,package}Vars

Type: List of strings

This is a list of environment variables that should be set during the execution of the checkout/build/package script. This declares the dependency of the respective step to the named variables.

It is not an error if a variable listed here is unset. This is especially useful for classes or to implement default behaviour that can be overridden by the user from the command line. If you expect a variable to be unset, it is your responsibility to handle that case in the script. Every reference to such a variable should be guarded with ${VAR-somthing} or ${VAR+something}.

A variable that is consumed in one step is also set in the following. This means a variable consumed through checkoutVars is also set during the build and package steps. Likewise, a variable consumed by buildVars is set in the package step too. The rationale is that all three steps form a small pipeline. If a step depends on a certain variable then the result of the following step is already indirectly dependent on this variable. Thus it can be set during the following step anyway.

The following variables are populated internally by Bob and might be added to the variable list:

  • BOB_HOST_PLATFORM - the platform identifier where Bob is running on. The following values are defined:
    • linux: Linux
    • msys: Windows/MSYS2
    • win32: Windows
    • darwin: Mac OS X
  • BOB_RECIPE_NAME - name of the recipe that defined the package
  • BOB_PACKAGE_NAME - name of the actual package. Might be different from the recipe name if multiPackage is used.

Note that you should keep the usage of these variables to a minimum because they may force separate builds of packages that are otherwise identical. For example using BOB_PACKAGE_NAME in buildVars will force separate builds of all involved multiPackage keys even if they have a common buildScript because BOB_PACKAGE_NAME will be unique for each multiPackage entry.

{checkout,build,package}VarsWeak

Type: List of strings

This is a list of environment variables that should be set during the execution of the checkout/build/package script. These variables are not considered to influence the result, very much like the variables listed in whitelist.

Warning

Bob expects that the content of these variables is irrelevant for the actual build result. They neither contribute to variant management nor will they trigger a rebuild of a package if they change.

For example, a typical usage of buildVarsWeak is to specify the number of parallel make jobs. While it changes the behaviour of the job (the number of parallel compiler processes) it will not change the actual build result. The weak inclusion of a variable has no effect if it is also referenced by {checkout,build,package}Vars. In this case the variable will always be considered significant for the build result.

It is not an error that a variable listed here is unset. This is especially useful for classes or to implement default behaviour that can be overridden by the user from the command line. If you expect a variable to be unset it is your responsibility to handle that case in the script. Every reference to such a variable should be guarded with ${VAR-somthing} or ${VAR+something}.

A variable that is consumed in one step is also set in the following. This means a variable consumed through checkoutVarsWeak is also set during the build and package steps. Likewise, a variable consumed by buildVarsWeak is set in the package step too. The rationale is that all three steps form a small pipeline. If a step depends on a certain variable then the result of the following step is already indirectly dependent on this variable. Thus it can be set during the following step anyway.

{build,package}NetAccess

Type: Boolean

By default the external network is not accessible during build or package steps when building inside a sandbox. Checkout steps always have network access. If such access is still needed a recipe may set the buildNetAccess or the packageNetAccess to True.

Warning

Bob assumes that build and package steps are deterministic. Do not rely on external state that changes the behavior of the build. Unless the input of a package changes (sources, dependencies) Bob will not re-build a package.

Note

Before Bob 0.14 (see offlineBuild policy) the network access was always possible. The policy will determine the default value of this property.

To configure the network access based on the actually used tools by a recipe you can set the netAccess property in provideTools. The {build,package}NetAccess should only be set if the script in the recipe itself requires the network access during build or package steps.

checkoutAssert

Type: List of checkout assertions

Using checkoutAssert you can make a build fail if a file content has been changed. This is especially useful to detect modifications in license files and copyright notices in source files.

The following properties are supported:

Property Description
file The file in the workspace to check. Must be a relative path.
digestSHA1 Digest of the file / part (lower case). Either pre calculate it using sha1sum command or take the output of the first (failing) run.
start First line of the file that is checked. Optional integer number. Defaults to 1 (first line of file).
end Last line of file that is checked. Optional integer number. Defaults to last line of file.

Line numbers start at 1 and are inclusive. The start line is always taken into account even if the end line is equal or smaller. The line terminator is always \n (ASCII “LF”, 0x0a) regardless of the host operating system.

Example:

checkoutAssert:
    - file: LICENSE
      digestSHA1: "2f7285314f4c057c75dbc0e5fad403b2d0691628"
    - file: src/namespace-sandbox/namespace-sandbox.c
      digestSHA1: "5ee22fb054c92560ec17202dec67202563e0d145"
      start: 3
      end: 13

checkoutDeterministic

Type: Boolean

By default any checkoutScript is considered indeterministic. The rationale is that extra care must be taken for a script to fetch always the same sources. If you are sure that the result of the checkout script is always the same you may set this to True. All checkoutSCMs on the other hand are capable of determining automatically whether they are determinstic.

If the checkout is deemed deterministic it enables Bob to apply various optimizations. It is also the basis for binary artifacts.

checkoutSCM

Type: SCM-Dictionary or List of SCM-Dictionaries

Bob understands several source code management systems natively. On one hand it enables the usage of dedicated plugins on a Jenkins server. On the other hand Bob can manage the checkout step workspace much better in the development build mode.

All SCMs are fetched/updated before the checkoutScript of the package are run. The checkoutScript should not move or modify the checkoutSCM directories, though.

If the package consists of a single git module you can specify the SCM directly:

checkoutSCM:
    scm: git
    url: git://git.kernel.org/pub/scm/network/ethtool/ethtool.git

If the package is built from multiple modules you can give a list of SCMs:

checkoutSCM:
    -
        scm: git
        url: git://...
        dir: src/foo
    -
        scm: svn
        url: https://...
        dir: src/bar

There are three common (string) attributes in all SCM specifications: scm, dir and if. By default the SCMs check out to the root of the workspace. You may specify any relative path in dir to checkout to this directory.

By using if you can selectively enable or disable a particular SCM using either a string or a expression. In case a string is given to the if-keyword it is substituted according to String substitution and the final string is interpreted as a boolean value (everything except the empty string, 0 and false is considered true). In case you’re using the expression syntax you can use String literals and String function calls to express a condition. The SCM will only be considered if the condition passes.

Currently the following scm values are supported:

scm Description Additional attributes
cvs CVS repository
cvsroot: repository location (“:ext:...”, path name, etc.)
module: module name
rev: revision, branch, or tag name (optional)
git Git project
url: URL of remote repository
branch: Branch to check out (optional, default: master)
tag: Checkout this tag (optional, overrides branch attribute)
commit: SHA1 commit Id to check out (optional, overrides branch or tag attribute)
rev: Canonical git-rev-parse revision specification (optional, see below)
remote-*: additional remote repositories (optional, see below)
sslVerify: Whether to verify the SSL certificate when fetching (optional)
shallow: Number of commits or cutoff date that should be fetched (optional)
singleBranch: Fetch only single branch instead of all (optional)
import Import directory from project
url: Directory path relative to project root.
prune: Delete destination directory before importing files.
svn Svn repository
url: URL of SVN module
revision: Optional revision number (optional)
sslVerify: Whether to verify the SSL certificate when fetching (optional)
url While not a real SCM it allows to download (and extract) files/archives.
url: File that should be downloaded
digestSHA1: Expected SHA1 digest of the file (optional)
digestSHA256: Expected SHA256 digest of the file (optional)
extract: Extract directive (optional, default: auto)
fileName: Local file name (optional, default: url file name)
sslVerify: Whether to verify the SSL certificate when fetching (optional)
stripComponents: Number of leading components stripped from file name (optional, tar files only)

Most SCMs support the sslVerify attribute. This is a boolean that controls whether to verify the SSL certificate when fetching. It defaults to True with the notable exception of git before Bob 0.15 which was rectified by the introduction of the secureSSL policy. If at all possible, fixing a certificate problem is preferable to using this option.

cvs
The CVS SCM requires a cvsroot, which is what you would normally put in your CVSROOT environment variable or pass to CVS using -d. If you specify a revision, branch, or tag name, Bob will check out that instead of the HEAD. Unfortunately, because Bob cannot know beforehand whether the rev you gave it points to a branch or tag, it must consider this SCM nondeterministic. To check out using ssh, you can use the syntax :ssh:user@host:/path, which will be translated into an appropriate CVS_RSH assignment by Bob. Alternatively, you can use a normal :ext: CVSROOT and manually pass the CVS_RSH value into the recipe using checkoutVars.
git

The git SCM requires at least an url attribute. The URL might be any valid Git URL. To checkout a branch other than master add a branch attribute with the branch name. To checkout a tag instead of a branch specify it with tag. You may specify the commit id directly with a commit attribute too.

Note

The default branch of the remote repository is not used. Bob will always checkout “master” unless branch, tag or commit is given.

The rev property of the git SCM unifies the specification of the desired branch/tag/commit into one single property. If present it will be evaluated first. Any other branch, tag or commit property is evalued after it and may override a precious setting made by rev. The branch/tag/commit precedence is still respected, though. Following the patterns described in git-rev-parse(1) the following formats are currently supported:

  • <sha1>, e.g. dae86e1950b1277e545cee180551750029cfe735. The full SHA-1 object name (40-byte hexadecimal string).
  • refs/tags/<tagname>, e.g. refs/tags/v1.0. The symbolic name of a tag.
  • refs/heads/<branchname>, e.g. refs/heads/master. The name of a branch.

The remote-* property allows adding extra remotes whereas the part after remote- corresponds to the remote name and the value given corresponds to the remote URL. For example remote-my_name set to some/url.git will result in an additional remote named my_name and the URL set to some/url.git.

To reduce the amount of data that is fetched from the remote repository the optional shallow attribute can be set. If it is an integer then only this number of commits are fetched from the tip of the remote branches (--depth clone parameter). It can also be a string that should be a date understood by git (passed as --shallow-since= to git). Either option will imply singleBranch to be true. This further restricts the fetching of remote branches to the configured branch only. Set singleBranch either to False to explicitly fetch all remote branches or to True to fetch only the current branch, regardless of the shallow setting.

Tip

You can set the shallow and singleBranch properties with scmOverrides too. This can be used to improve the build times of existing projects or to fetch the whole history if shallow is used in the recipes.

import

The import SCM copies the directory specified in url to the workspace. By default the destination is not overwritten unless the source file was changed more recently than the exiting destination in the workspace. Set prune to True to always delete the destination directory before importing the files.

Attention

Do not import large source trees when working with Jenkins builds. The content is included in the job configuration that will get too large otherwise.

svn
The Svn SCM, like git, requires the url attribute too. If you specify a numeric revision Bob considers the SCM as deterministic.
url

The url SCM naturally needs an url attribute. If a SHA digest is given with digestSHA1 and/or digestSHA256 the downloaded file will be checked for a matching hash sum. This also makes the URL deterministic for Bob. Otherwise the URL will be checked in each build for updates. Based on the file name ending Bob will try to extract the downloaded file. You may prevent this by setting the extract attribute to no or False. If the heuristic fails the extraction tool may be specified as tar, gzip, xz, 7z or zip directly. For tar files it is possible to strip a configurable number of leading components from file names on extraction by the stripComponents attribute.

Note

Starting with Bob 0.14 (see tidyUrlScm policy) the whole directory where the file is downloaded is claimed by the SCM. It is not possible to fetch multiple files in the same directory. This is done to separate possibly extracted files safely from other checkouts.

depends

Type: List of Strings or Dependency-Dictionaries

Declares a list of other recipes that this recipe depends on. Each list entry might either be a single string with the recipe name or a dictionary with more fine grained settings. Such entries might either name another recipe directly (name) or a list of further dependencies (depends) that inherit the settings from the current entry. See the following example for both formats:

depends:
    - foo
    - bar
    -
        name: toolchain
        use: [tools, environment]
        forward: True
    -
        if: "${FOOBAR}"
        depends:
            - baz
            - qux

In the first and second case only the package is named, meaning the build result of recipe foo resp. bar is fed as $2 and $3 to the build script. Any provided dependencies of these packages (provideDeps) will be implicitly added to the dependency list too.

In the third case a recipe named toolchain is required but instead of using its result the recipe imports any declared tools and environment variables from toolchain. Additionally, because of the forward attribute, these imported tools and variables are not only imported into the current recipe but also forwarded to the following recipes (baz and qux).

The 4th case is a recursive definition where the simple dependencies baz and qux are guarded by a common condition. These dependencies will only be considered if the variable FOOBAR expands to a value that is evaluated as boolean true. If the condition passes these dependencies will be available as $4 and $5 to the build script. Recursive definitions might be nested freely and they might override any setting mentioned in the table below. All if properties on each nesting level must evaluate to true for an entry to take effect.

Detailed entries must either contain a name property or a depends list. The following settings are supported:

Name Type Description
name String The name of the required recipe.
depends List of Dependencies A list of dependencies inheriting the settings of this entry.
use List of strings

List of the results that are used from the package. The following values are allowed:

  • deps: provided dependencies of the recipe. These dependencies will be added at the end of the dependency list unless the dependency is already on the list.
  • environment: exported environment variables of the recipe.
  • result: build result of the recipe.
  • tools: declared build tools of the recipe.
  • sandbox: declared sandbox of the recipe.

Default: Use the result and dependencies ([deps, result]).

forward Boolean

If true, the imported environment, tools and sandbox will be forwarded to the dependencies following this one. Otherwise these variables, tools and/or sandbox will only be accessible in the current recipe.

Default: False.

environment Dictionary (String -> String)

This clause allows to define or override environment variables for the dependencies. Example:

environment:
    FOO: value
    BAR: baz
if String

The string (subject to substitution) is interpreted as boolean value. The dependency is only considered if the string is considered as true. See String substitution.

Default: true

if: !expr String

The string is parsed as expression using the same String literals and String function calls as available for bobpaths. The dependency is only considered if the expression is considered as true.

Default: true Example:

if: !expr |
      "${FOO}" == "bar" || "${BAZ}"

environment

Type: Dictionary (String -> String)

Defines environment variables in the scope of the current recipe. Any inherited variables of the upstream recipe with the same name are overwritten. All variables are passed to downstream recipes.

Example:

environment:
   PKG_VERSION: "1.2.3"

The environment of the recipe and inherited classes are merged together. The exact way of merging is subject to the mergeEnvironment policy.

See also privateEnvironment.

filter

Type: Dictionary ( “environment” | “sandbox” | “tools” -> List of Strings)

The filter keyword allows to restrict the environment variables, tools and sandboxes inherited from upstream recipes. This way a recipe can effectively restrict the number of package variants.

The filters specifications may use shell globbing patterns. As a special extension there is also a negative match if the pattern starts with a “!”. Such patterns will filter out entries that have been otherwise included by previous patterns in the list (e.g. by inherited classes).

Example:

filter:
    environment: [ "*_MIRROR" ]
    tools: [ "*toolchain*", "!host-toolchain" ]
    sandbox: [ "*" ]

In the above example the recipe would inherit only environment variables that end with “_MIRROR”. All other variables are unset. Likewise all tools that have “toolchain” in their name are inherited, except the “host-toolchain”. Anything is accepted as sandbox which would also be the default if left out.

Warning

The filter keyword is still experimental and may change in the future or might be removed completely.

fingerprintScript[{Bash,Pwsh}]

Type: String

The fingerprint script is executed before a package is built or downloaded. The script is supposed to gather information about whatever external resource is used in the recipe and output that in a stable format. The actual output is irrelevant to Bob as long as it detects all relevant external influences of the build result and that subsequent executions of the script generate the same output if the external components have not changed.

Note

Defining a fingerprintScript does not enable fingerprinting yet. At least one inherited class, used tool or the recipe itself must enable it by setting fingerprintIf accordingly.

Bob will incrementally rebuild the package whenever the fingerprint script output changes. The output of the script is also used to tag binary artifacts. An artifacts will only be downloaded if the fingerprint script generated the same output. This enables Bob to prevent false sharing of binary artifacts across otherwise incompatible machines.

The fingerprint script is executed in an empty temporary directory. It does not have access to any dependencies of the recipe nor to the checked out sources. A subset of environment variables of the package (see {checkout,build,package}Vars) as defined by fingerprintVars is set. The usual bash options are applied (nounset, errexit, pipefail) too. If the script returns with a non-zero exit status it will fail the build. The output on stderr is ignored but will be displayed in the error message if the script fails. The scripts of inherited classes are concatenated (but only if their fingerprintIf condition did not evaluate to false). Any fingerprint scripts that are defined by used tools (see provideTools) are concatenated too.

The suffix of the keyword determines the language of the script. Using the Bash suffix (fingerprintScriptBash) defines a script that is interpreted with bash. Likewise, the Pwsh suffix (fingerprintScriptPwsh) defines a PowerShell script. Which language is used at build time is determined by the scriptLanguage key or, if nothing was specified, by the project scriptLanguage setting. The keyword without a suffix (fingerprintScript) is interpreted in whatever language is finally used at build time. If both the keyword with the build time language suffix and without a suffix are present then the keyword with the build language suffix takes precedence.

For common fingerprint tasks the following built-in functions are provided by Bob:

bob-libc-version

Checks the host architecture together with the type and version of the libc library. The C-compiler that is used can be configured either with the first parameter of the function or it will use the CC environment variable. If both are not set the cc command is used.

This helper should typically be used with the host compiler recipe.

bob-libstdc++-version

Checks the host architecture together with the type and version of the C++ standard library. The C++-compiler that is used can be configured either with the first parameter of the function or it will use the CXX environment variable. If both are not set the c++ command is used.

This helper should typically be used with the host compiler recipe.

bob-hash-libraries

Takes a list of libraries as arguments that should be hashed. This will link an executable that links with the given libraries, call ldd and hash all used libraries.

Use this helper if no other information is available about a library / libraries except the name.

These helpers can be used in the fingerprint script. Their actual implementation and output may change in the future as more systems are supported by Bob.

fingerprintIf

Type: String | Boolean | null | IfExpression

By default no fingerprinting is done unless at least one inherited class, used tool or the recipe explicitly enables it. This is done by either setting fingerprintIf to True or by a boolean expression string that is evaluated to True. This can be used e.g. to apply a fingerprint only if the package is built for the host and not cross-compiled. The fingerprintScript[{Bash,Pwsh}] of the recipe is only evaluated if fingerprintIf is true. Otherwise the fingerprint script is ignored even if some other class enables fingerprinting. Setting fingerprintIf to False will unconditionally disable the associated fingerprintScript.

A null value has a special semantic. It does not enable fingerprinting for a package but retains the associated fingerprint script. If some inherited class, the recipe or a used tool does enable fingerprinting then the fingerprint script will still be evaluated. This is useful for classes to provide some fingerprinting functions but, unless an inheriting recipe defines a fingerprint script, does not enable fingerprinting of the recipe by itself.

Examples:

fingerprintIf: True                       # unconditionally enable fingerprinting
fingerprintIf: "$(eq,${TOOLCHAIN},host)"  # boolean experession
fingerprintIf: null                       # same as if unset
fingerprintIf: !expr |                    # IfExpression
                  "${TOOLCHAIN}" == "host"

If not given it defaults to null.

fingerprintVars

Type: List of strings

This declares the subset of the environment variables of the affected package that should be set during the execution of the fingerprintScript. Only variables that are selected by {checkout,build,package}Vars can be used. It is not an error that a variable listed here is unset. The variables will only be set if the corresponding fingerprintScript is enabled too.

Note

Before Bob 0.16 (see fingerprintVars policy) all environment variables of the affected package were set during the execution of the fingerprintScript. If the policy is set to the old behaviour then this key will be ignored and has no effect.

inherit

Type: List of Strings

Include classes with the given name into the current recipe. Example:

inherit: [cmake]

Classes are searched in the classes/ directory with the given name. The syntax of classes is the same as the recipes. In particular classes can inherit other classes too. The inheritance graph is traversed depth first and every class is included exactly once.

All attributes of the class are merged with the attributes of the current recipe. If the order is important the attributes of the class are put in front of the respective attributes of the recipe. For example the scripts of the inherited class of all steps are inserted in front of the scripts of the current recipe.

metaEnvironment

Type: Dictionary (String -> String)

metaEnvironment variables behave like privateEnvironment variables. They overrule other environment variables and can be used in all steps, but substitution is not available. In addition all metaEnvironment variables are added to the audit no matter they are used in a step or not. This predestines metaEnvironment variables to add the license type or version of a package.

The bob-query-meta command can be used to retrieve metaEnvironment variables.

multiPackage

Type: Dictionary (String -> Recipe)

By utilizing the multiPackage keyword it is possible to unify multiple recipes into one. The final package name is derived from the current recipe name by appending the key under multiPackage separated by a “-“. If an empty string is given as key the separator is not inserted. Nested multiPackages are also supported. Every level of multiPackages appends another suffix to the package name. The following example recipe foo.yaml declares four packages: foo, foo-bar-x, foo-bar-y and foo-baz:

multiPackage:
   "":
      ...
   bar:
      buildScript: ...
      multiPackage:
         x:
            packageScript: ...
         y:
            packageScript: ...
   baz:
      ...

All other keywords on the same level are treated as an anonymous base class that is inherited by the defined multiPackage’s. That way you can have common parts to all multiPackage entries and keep just the distinct parts separately.

A typical use case for this feature are recipes for libraries. There are two packages that are built from a library: a -target packet that has the shared libraries needed during runtime and a -dev packet that has the header files and other needed files to link with this library.

privateEnvironment

Type: Dictionary (String -> String)

Defines environment variables just for the current recipe. Any inherited variables with the same name of the upstream recipe or others that were consumed from the dependencies are overwritten. All variables defined or replaced by this keyword are private to the current recipe.

Example:

privateEnvironment:
   APPLY_FOO_PATCH: "no"

The privateEnvironment of the recipe and inherited classes are merged together. The exact way of merging is subject to the mergeEnvironment policy.

See also environment.

provideDeps

Type: List of Patterns

The provideDeps keyword receives a list of dependency names. These must be dependencies of the current recipe, i.e. they must appear in the depends section. It is no error if the condition of such a dependency evaluates to false. In this case the entry is silently dropped. To specify multiple dependencies with a single entry shell globbing patterns may be used.

Provided dependencies are subsequently injected into the dependency list of the upstream recipe that has a dependency to this one (if deps is included in the use attribute of the dependency, which is the default). This works in a transitive fashion too, that is provided dependencies of a downstream recipe are forwarded to the upstream recipe too.

Example:

depends:
    - common-dev
    - communication-dev
    - config

...

provideDeps: [ "*-dev" ]

Bob will make sure that the forwarded dependencies are compatible in the injected recipe. That is, any duplicates through injected dependencies must result in the same package being used.

provideTools

Type: Dictionary (String -> Path | Tool-Dictionary)

The provideTools keyword defines an arbitrary number of build tools that may be used by other steps during the build process. In essence the definition declares a path (and optionally several library paths) under a certain name that, if consumed, are added to $PATH (and $LD_LIBRARY_PATH) of consuming recipes. Example:

provideTools:
   host-toolchain:
      path: bin
      libs: [ "sysroot/lib/i386-linux-gnu", "sysroot/usr/lib", "sysroot/usr/lib/i386-linux-gnu" ]
      netAccess: True
      environment:
         CC: gcc
         LD: ld
      fingerprintIf: True
      fingerprintScript: |
         bob-libc-version gcc

The path attribute is always needed. The libs attribute, if present, must be a list of paths to needed shared libraries. Any path that is specified must be relative. If the recipe makes use of existing host binaries and wants to provide them as tool you should create symlinks to the host paths.

The netAccess attribute allows the tool to request network access during build/package step execution even if the recipe has not requested it (see {build,package}NetAccess). The network access is only granted if the tool is used. This attribute might be needed if the recipe cannot know if a particular tool actually requires network access. A prominent example are proprietary compilers that need to talk to a license server. Unless a package is built with such a compiler the network access is not needed.

The environment attribute provides the ability to define environment variables that are automatically picked up by the recipe where the tool is used. This allows for much more fine-grained variable provisioning than provideVars. If multiple tools are used in a recipe they must define distinct variables because no particular order between tools is defined. The values defined in this attribute are subject to variable substitution.

The fingerprintScript attribute defines a fingerprint script like in a normal recipe by fingerprintScript[{Bash,Pwsh}]. A fingerprint script defined by a tool is implicitly added to the fingerprint scripts of all recipes that use the particular tool. Use it to automatically apply a fingerprint to all recipes whose result will depend on the host environment by using the tool. The fingerprintIf and fingerprintVars attributes are handled the in the same way.

If no attributes except path are present the declaration may be abbreviated by giving the relative path directly:

provideTools:
   host-toolchain: bin

provideVars

Type: Dictionary (String -> String)

Declares arbitrary environment variables with values that should be passed to the upstream recipe. The values of the declared variables are subject to variable substitution. The substituted values are taken from the current package environment. Example:

provideVars:
    ARCH: "arm"
    CROSS_COMPILE: "arm-linux-${ABI}-"

By default these provided variables are not picked up by upstream recipes. This must be declared explicitly by a use: [environment] attribute in the dependency section of the upstream recipe. Only then are the provided variables merged into the upstream recipes environment.

provideSandbox

Type: Sandbox-Dictionary

The provideSandbox keyword offers the current recipe as sandbox for the upstream recipe. Any consuming upstream recipe (via use: [sandbox]) will be built in a sandbox where the root file system is the result of the current recipe. The initial $PATH is defined with the required paths keyword that should hold a list of paths. This will completely replace $PATH of the host for consuming recipes.

Attention

The build result is considered to be an invariant of such a sandbox (see sandboxInvariant policy). This implies that recipes shall produce the same result whether the sandbox is used or not.

Optionally there can be a mount keyword. With mount it is possible to specify additional paths of the host that are mounted read only in the sandbox. The paths are specified as a list of either strings or lists of two or three elements. Use a simple string when host and sandbox path are the same without any special options. To specify distinct paths use a list with two entries where the host path is the first element and the second element is the path in the sandbox.

The long format with three items additionally allows to specify a list of mount flags. The shorter formats described above have no flags set. The following flags are available:

  • nofail: Don’t fail the build if the host path is not available. Instead drop the mount silently.
  • nolocal: Do not use this mount in local builds.
  • nojenkins: Do not use this mount in Jenkins builds.
  • rw: Mount as read-writable instead of read-only.

Additionally there can be an optional environment keyword. This works like the provideVars keyword and defines environment variables that are picked up by the depending recipe. In contrast to provideVars the variables defined here are only consumed if the sandbox is actually used (i.e. the parent recipe defined sandbox in the use section and the user builds with --sandbox). In this case the variables defined here have a higher precedence that the ones defined in provideVars.

Variable substitution is possible for the mount paths and environment variables. See String substitution for the available substations. The mount paths are also subject to an additional variable expansion when a step using the sandbox is actually executed. This can be useful e.g. to expand variables that are only available on the build server. Example:

provideSandbox:
    paths: ["/bin", "/usr/bin"]
    mount:
        - "/etc/resolv.conf"
        - "${MYREPO}"
        - "\\$HOME/.ssh"
        - ["\\$SSH_AUTH_SOCK", "\\SSH_AUTH_SOCK", [nofail, nojenkins]]
    environment:
        AUTOCONF_BUILD: "x86_64-linux-gnu"

The example assumes that the variable MYREPO was set somewhere in the recipes. On the other hand $HOME is expanded later at build time. This is quite useful on Jenkins because the home directory there is certainly different from the one where Bob runs. The last entry shows two mount option being used. This line mounts the ssh-agent socket into the sandbox if available. This won’t be done on Jenkins at all and the build will proceed even if $SSH_AUTH_SOCK is unset or invalid. Note that such variables have to be in the whitelist to be available to the shell.

Note

The mount paths are considered invariants of the build. That is changing the mounts will neither automatically cause a rebuild of the sandbox (and affected packages) nor will binary artifacts be re-fetched.

The user might amend the mount and search paths in default.yaml by a sandbox entry.

relocatable

Type: Boolean

If True Bob can assume that the package result is independent of the actual location in the file system. Usually all packages should be relocatable as this is a fundamental assumption of Bob’s working model. There might be particular tools, though, that depend on their installed location. For such tools the property should be set to False.

If the property is not set the default will be True unless the recipe defines at least one tool. In this case the default value is False if the allRelocatable policy is unset or disabled. If the policy is set the default value is always True. Inherited values from a class will be overwritten by the recipe or inheriting class.

root

Type: Boolean

Recipe attribute which defaults to False. If set to True the recipe is declared a root recipe and becomes a top level package. There must be at least one root package in a project.

scriptLanguage

Type: Enumeration: bash, PowerShell.

Defines the scripting language which is used to run the {checkout,build,package,fingerprint}Script scripts when building the package. If nothing is specified the scriptLanguage setting from config.yaml is used. Depending on the chosen language Bob will either invoke bash or pwsh/powershell as script interpreter. In either case the command must be present in $PATH/%PATH%.

shared

Type: Boolean

Marking a recipe as shared implies that the result may be shared between different projects or workspaces. Only completely deterministic packages may be marked as such. Typically large static packages (such as toolchains) are enabled as shared packages. By reusing the result the hard disk usage can be sometimes reduced drastically.

The exact behaviour depends on the build backend. Currently the setting has no influence on local builds. On Jenkins the result will be copied to a separate directory in the Jenkins installation and will be used from there. This reduces the job workspace size considerably at the expense of having artifacts outside of Jenkins’s regular control.

Project configuration (config.yaml)

The file config.yaml holds all static configuration options that are not subject to be changed when building packages. The following sections describe the top level keys that are currently understood. The file is optional or could be empty.

bobMinimumVersion

Type: String

Defines the minimum required version of Bob that is needed to build this project. Any older version will refuse to build the project. The version number given here might be any prefix of the actual version number, e.g. “0.1” instead of the actual version number (e.g. “0.1.42”). Bob’s version number is specified according to Semantic Versioning. Therefore it is usually only needed to specify the major and minor version.

The version string has to be compliant to Python PEP 440. It is allowed to specify pre-release versions (e.g. 0.16.0rc1) and even development versions (e.g. 0.15.1.dev42). A version without pre-release suffix is considered more recent than a version with a pre-release suffix. The development release number is only relevant if the main version and the pre-release versions are equal.

layers

Type: List of strings

The layers section consists of a list of layer names that are then expected in the layers directory relative to the conig.yaml referencing them:

layers:
    - myapp
    - bsp

Layers that are not named in this section but that are present in the layers directory are ignored. Layers that are named but that do not exist lead to a parse error. Layers can be nested, that is, a layer can itself have layers below it.

The order of layers is important with respect to settings made by default.yaml in the various layers. The project root has the highest precedence. The layers in config.yaml are named from highest to lowest precedence. A layer with a higher precedence can override settings from layers of lower precedence.

See Configuration for more information.

plugins

Type: List of strings

Plugins are loaded in the same order as listed here. For each name in this section there must be a .py-file in the plugins directory next to the recipes. For a detailed description of plugins see Plugins.

policies

Type: Dictionaly (Policy name -> Bool)

The policies section allows to individually set policies to their old (disabled) or new (enabled) behaviour. See Defined policies for a list of all policies and their rationale.

Example:

policies:
    relativeIncludes: False

This will explicitly request old behaviour for the relativeIncludes policy.

scriptLanguage

Type: Enumeration: bash, PowerShell.

Defines the scripting language which is used to run the {checkout,build,package,fingerprint}Script scripts. Defaults to bash. Might be overrided on a case-by-case basis in a class or recipe with scriptLanguage. Depending on the chosen language Bob will either invoke bash or pwsh/powershell as script interpreter. In either case the command must be present in $PATH/%PATH%.

User configuration (default.yaml)

The default.yaml file holds configuration options that may be overridden by the user. Most commands will also take an ‘-c’ option where any number of additional configuration files with the same syntax can be specified.

Like git there are three locations where bob is looking for a configuration file. They are parsed in descending order making it possible to locally override global settings.:

/etc/bobdefault.yaml:
    System-wide configuration file.

$XDG_CONFIG_HOME/bob/default.yaml resp. ~/.config/bob/default.yaml:
    User-specific configuration File. If XDG_CONFIG_HOME is not set
    ~/.config/bob/default.yaml is used.

./default.yaml.
    Workspace-specific configuration file.

User configuration files may optionally include other configuration files. These includes are parsed after the current file, meaning that options of included configuration files take precedence over the current one. Included files do not need to exist and are silently ignored if missing. Includes are specified without the .yaml extension:

include:
    - overrides

Note

Depending on the relativeIncludes policy the base directory from where includes are resolved is different. Normally files are included relative to the currently processed file unless the relativeIncludes policy is disabled. In this case files included by default.yaml and by the command line use the project root directory as base directory.

It is possible for plugins to define additional settings. See Plugin settings for more information. Their meaning and typing is completely controlled by the respective plugin and Bob will just pass the data as-is without further interpretation.

User configuration files may also require specific files to be included. The require keyword behaves just like the include keyword with the exception that Bob raises a parsing error if the file to be included cannot be found:

require:
   - overrides
   - /path/to/some/file

Required include files have a lower precedence that optional include files.

environment

Type: Dictionary (String -> String)

Specifies default environment variables. Example:

environment:
   # Number of make jobs is determined by the number of available processors
   # (nproc).  If desired it can be set to a specific number, e.g. "2". See
   # classes/make.yaml for details.
   MAKE_JOBS: "nproc"

If the cleanEnvironment policy is enabled then these variables are subject to String substitution with the current OS environment. This allows to take over certain variables from the OS environment in a controlled fashion.

whitelist

Type: List of Strings

Specifies a list of environment variable keys that should be passed unchanged to all scripts during execution. The content of these variables are considered invariants of the build. It is no error if any variable specified in this list is not set. By default the following environment variables are passed to all scripts: TERM, SHELL, USER and HOME. The names given with whitelist are added to the list and does not replace the default list.

Example:

# Keep ssh-agent working
whitelist: ["SSH_AGENT_PID", "SSH_AUTH_SOCK"]

archive

Type: Dictionary or list of dictionaries

The archive key configures the default binary artifact server(s) that should be used. It is either directly an archive backend entry or a list of archive backends. For each entry at least the backend key must be specified. Optionally there can be a flags key that receives a list of various flags, in particular for what operations the backend might be used. See the following list for possible flags. The default is [download, upload].

download
Use this archive to download artifacts. Note that you still have to explicitly enable downloads on Jenkins servers. For local builds the exact download behaviour depends on the build mode (release vs. develop).
upload
Use this archive to upload artifacts. To actually upload to the archive the build must be performed with uploads enabled (--upload).
nofail
Don’t fail the build if the upload or download from this archive fails. In any case it is never an error if a download does not find the requested archive on the backend. This option additionally suppresses other errors such as unknown hosts or interrupted transfers.
nolocal
Do not use this archive in local builds.
nojenkins
Do not use this archive in Jenkins builds.

Depending on the backend further specific keys are available or required. See the following table for supported backends and their configuration.

Backend Description
none Do not use a binary repository (default).
azure Microsoft Azure Blob storage backend. The account must be specified in the account key. Either a key or a sasToken may be set to authenticate, otherwise an anonymous access is used. Finally the container must be given in container. Requires the azure-storage-blob Python3 library to be installed.
file Use a local directory as binary artifact repository. The directory is specified in the path key as absolute path. The optional fileMode and directoryMode keys take the desired access modes as numeric value to override the default umask derived modes.
http Uses a HTTP server as binary artifact repository. The server has to support the HEAD, PUT and GET methods. The base URL is given in the url key. The optional sslVerify boolean key controls whether to verify the SSL certificate.
shell This backend can be used to execute commands that do the actual up- or download. A download and/or upload key provides the commands that are executed for the respective operation. The configured commands are executed by bash and are expected to copy between the local archive (given as $BOB_LOCAL_ARTIFACT) and the remote one (available as $BOB_REMOTE_ARTIFACT). See the example below for a possible use with scp.

The directory layouts of the azure, file, http and shell ($BOB_REMOTE_ARTIFACT) backends are compatible. If multiple download backends are available they will be tried in order until a matching artifact is found. All available upload backends are used for uploading artifacts. Any failing upload will fail the whole build.

Note

The uploaded artifacts can be managed by bob-archive. It might be wise to use different repositories for release builds and for continous builds to keep them separated.

Example:

archive:
   backend: http
   url: "http://localhost:8001/upload"

It is also possible to use separate methods for upload and download:

archive:
    -
        backend: http
        url: "http://localhost:8001/archive"
        flags: [download]
    -
        backend: shell
        upload: "scp -q ${BOB_LOCAL_ARTIFACT} localhost:archive/${BOB_REMOTE_ARTIFACT}"
        download: "scp -q localhost:archive/${BOB_REMOTE_ARTIFACT} ${BOB_LOCAL_ARTIFACT}"
        flags: [upload]

The azure backend can also be used in conjunction with the http backend in case of publicly readable containers. Given a typical configuration like this:

archive:
    backend: azure
    account: <account>
    container: <container name>
    key: <access key>

the anonymous access to the container can be used like this:

archive:
    backend: http
    url: https://<account>.blob.core.windows.net/<container name>
    flags: [download]

The flags: [download] makes sure that Bob does not try to upload artifacts in case other backends are configured too.

scmOverrides

Type: List of override specifications

SCM overrides allow the user to alter any attribute of SCMs (checkoutSCM) without touching the recipes. They are quite useful to change e.g. the server url or to override the branch of some SCMs. Overrides are applied after string substitution. The general syntax looks like the following:

scmOverrides:
  -
    match:
      url: "[email protected]:/foo/repo.git"
    del: [commit, tag]
    set:
      branch: develop
    replace:
      url:
        pattern: "foo"
        replacement: "bar"
    if: !expr |
      "${BOB_RECIPE_NAME}" == "foo"

The scmOverrides key takes a list of one or more override specifications. You can select overrides using a if expression. If if condition evaluates to true the override is first matched via pattens that are in the match section. All entries under match must be matching for the override to apply. The right side of a match entry can use shell globbing patterns.

If an override is matching the actions are then applied in the following order:

  • del: The list of attributes that are removed.
  • set: The attributes and their values are taken, overwriting previous values.
  • replace: Performs a substitution based on regular expressions. This section can hold any number of attributes with a pattern and a replacement. Each occurrence of pattern is replaced by replacement.

All overrides values are mangled through String substitution. Mangling is performed during calculation of the checkoutStep so that the full environment for this step is available for substitution.

alias

Type: Dictionary (String -> String)

Aliases allow a string to be substituted for the first step of a relative location path:

alias:
   myApp: "host/files/group/app42"
   allTests: "//*-unittest"

See Alias substitution for the rules that apply to aliases.

command

Type: Dict of command dicts

Override default command settings:

command:
    dev:
        [..]
    build:
        [..]
    graph:
        [..]

build / dev

Set default build arguments here. See bob-dev or bob-build for details.:

command:
    dev:
        no_logfile: True
        build_mode: "build-only"
    build:
        verbosity: 3
        download: No

The following table lists possible arguments and their type:

Key Command line switch Type
always_checkout --always-checkout List of strings (regular expression patterns)
audit --[no]-audit Boolean
build_mode -b | -B | --normal String (normal, build-only or checkout-only)
clean --clean | --incremental Boolean
clean_checkout --clean-checkout Boolean
destination --destination String (Path)
download --download String (yes, no, deps, forced, forced-deps, forced-fallback or packages=<packages>)
force -f Boolean
link_deps --[no-]link-deps Boolean
no_deps -n Boolean
no_logfiles --no-logfiles Boolean
sandbox --[no-]sandbox Boolean
upload --upload Boolean
verbosity -q | -v Integer (-2[quiet] .. 3[verbose], default 0)

graph

Set default graph arguments here. See bob-graph for details.:

command:
    graph:
        options:
            d3.dragNodes: True
        type: "d3"
        max_depth: 2

Supported arguments and their type:

Key Type
options Dictonary of String key value pairs
type “d3” or “dot”
max_depth Integer

hooks

Hooks are other programs or scripts that can be executed by Bob at certain points, e.g. before or after a build. Unless otherwise noted they are executed with the project root directory as working directory. Example:

hooks:
    postBuildHook: ./contrib/notify.sh

where contrib/notify.sh is:

#!/bin/bash
HEADLINE="Bob build finished"
BODY="The build in $PWD has finished: $1"
if [[ ${XDG_CURRENT_DESKTOP:-unknown} == KDE ]] ; then
    kdialog --passivepopup "$BODY" 10 --title "$HEADLINE"
else
    notify-send -u normal -t 10000 "$HEADLINE" "$BODY"
fi

The currently supported hooks are described below.

preBuildHook

The pre-build hook is run directly before a local build (bob dev / bob build). It receives the paths of all packages that are built as arguments.

If the hook returns with a non-zero status the build will be interrupted.

postBuildHook

The post-build hook is run after a local build finished, regardless if the build succeeded or failed. It receives the status as first argument (success or fail) and the relative paths to the workspaces of the results as further arguments.

The return status of the hook is ignored.

rootFilter

Type: List of Strings

Filter root recipes. The effect of this is a faster package parsing due to the fact, that the tree is not build for filtered roots.

Works like the filter keyword.

sandbox

Type: Sandbox-Dictionary

The default paths and mounts of a sandbox are defined by the provideSandbox keyword. The sandbox section in the user configuration allows to specify additional mounts and additional search paths. The format of the settings is the same as in the provideSandbox keyword.

Example:

sandbox:
    mount:
        - [ "$HOME/bin", "/mnt" ]
    paths:
        - /mnt

The search paths from paths are added to $PATH in reverse order so that later entries have a higher precedence. In contrast to provideSandbox no variable substitution is possible for the mounts. The mount paths are still subject to shell variable expansion when a step using the sandbox is actually executed, though.

The example above will mount the bin directory of the users home directory as /mnt inside the sandbox. The /mnt directory will be in $PATH before any other search directory of the sandbox but still after any used tool (if any).

ui

Type: Dictionary

Specifies options of user interface.

color

Color mode of console output. Can be also overridden by command line option --color.

never
No colors in output
always
Use colors in output
auto
Use colors only when TTY console detected (default)