Bob 0.14 Release Notes

Changes made since Bob 0.13.0 include the following.

New features

  • Add rootFilter setting to default.yaml

    This setting allows to filter root recipes by matching then with a globbing pattern. The effect of this is a faster package parsing due to the fact, that the tree is not build for filtered roots.

  • Extended colored output options

    Bob gained a --color= switch that can be set to auto (default), always or never to control the colorized output of all commands.

  • Add Microsoft Azure Blob storage backend

    The azure backend uses the Microsoft cloud block blob storage. It will use the same layout as all other backends so that the container can be made public readable and be fetched by the http backend without giving out the credentials.

    See archive for all options. Requires the azure-storage-blob Python3 library to be installed.

  • The -D switch gained the ability to take values that contain =.

    The value of an environment variable can have any character in the recipes. When passed on the command line this is now possible too. Previously values that had a = (e.g. -DFOO=A=B were rejected or lead to crashes.

  • Allow user to amend sandbox mounts/paths

    The default sandbox mount and search paths are defined in the sandbox recipe(s). If the user wanted to mount additional directories inside the sandbox (e.g. a local source code mirror in conjunction with a scmOverride) he previously had to edit the recipes.

    Now it is possible to add such paths to default.yaml. See sandbox.

  • Allow substitution on scmOverrides

    The values of scmOverrides are mangled through String substitution.

Recipes

  • Add checkoutAssert keyword

    With checkoutAssert you can define a checksum for some files or parts of files. This can be useful to detect license changes.

    Example:

    checkoutAssert:
    -
        file: 'LICENSE'
        digestSHA1: be520183980a2e06b5272f3669395782f186d6d0
    -
        file: 'main.c'
        start: 1
        end: 20
        digestSHA1: 949d02bed248e000d23bb81dfcce55e5603e8789

  • Add stripComponents attribute for URL SCM

    With stripComponents it is possible to strip a configurable number of leading components from file names on extraction. This works only for tar files, though.

  • Define relocatable property

    Historically any recipe that did define at least one tool was deemed to be non-relocatable. This implied that it cannot be stored as binary artifact unless we’re in a sandbox.

    This is overly pessimistic. By introducing a relocatable property the user is able to express if a package is really relocatable or not. For backwards compatibility the default value is False if the recipe defines a tool, otherwise it is True.

    See relocatable.

  • Implement proper inheritance for shared keyword

    The shared keyword was always ignored in classes. Now we deduct the final value by the usual inheritance rules just like all other values.

  • Add environment property to provideSandbox

    A sandbox brings its own host environment that overrides the host environment when used. This can be quite different and be even a different architecture (think of an i386 sandbox image that is used on an x86_64 host).

    Bob regards the sandbox as a build invariant, that is it must have no impact on the results of the recipes that are built inside the sandbox. On the other hand there can be situations where this still the case. To properly handle this a sandbox recipe can now define environment variables that are picked up when this sandbox is actually used. By consuming this variable in other recipes Bob can determine the impact of the sandbox, if any.

    See provideSandbox.

Bob build / bob dev

  • Allow build workspace to symlink checkout result

    This adds support for having symlinks to files of the checkout workspace in the build workspace (e.g. linking big archives). It has always been guaranteed to have the checkout and build workspace of the package available in the package step. Thus the package step can access this link instead of having to copy the archive in the build step.

Bob clean

  • Added -c option

  • Added -D option

Bob graph

  • Added --(no-)sandbox option

Bob jenkins

  • Prevent running in stale workspaces

    Jenkins does not delete workspaces automatically. He also has no clue if the job that is running in an old workspace is compatible or not. Hence we have to make sure ourself that we do not collide with old data in case of incremental builds.

    To pro-actively prevent such clashes every Jenkins server in Bob now get’s a 32-bit random UUID which is appended to every path. This should minimize the chance to collide with another or old instance.

    Because of the low entropy the UUID does not provide a guarantee that no collision occurs. It also does not help on existing instances where no UUID was assigned. Therefore, as a 2nd line of defence, every workspace stores a canary with the variant-id. If the canary exists it has to match. Otherwise the build is aborted because we don’t build the same thing in this existing workspace. The user has to clean up the workspace manually then.

  • Add jobs.policy job trigger policy extended option

    This extended option that makes the actual trigger threshold between jobs configurable. It defaults to “stable” which was the default before. The “unstable” setting triggers on unstable builds and takes artifacts from stable and unstable builds. The “always” setting does what it says but might not be that useful.

Bob ls

  • Add --unsorted option

    If given, show the packages in the order they were named in the recipe. By default they are sorted by name for better readability.

Plugins

  • Add plugin settings support

    Sometimes plugin behaviour needs to be configurable by the user. On the other hand Bob expects plugins to be deterministic. To have a common interface for such settings it is possible for a plugin to define additional keywords in default.yaml. This provides Bob with the information to validate the settings and detect changes in a reliable manner.

    See Plugin settings and bob.input.PluginSetting for more details.

Changed behaviour

Backwards compatible policies

Bob will retain the old behavior unless instructed otherwise. See Policies for more information.

  • url SCM: track checkout directory instead of file

    Historically the URL SCM was not tracking the checkout directory but the individual files that are downloaded by the SCM. This had the advantage that it is possible to download more than one file into the same directory. There are a couple of major disadvantages, though, that are now solved by “owning” the whole directory by the SCM where the file is downloaded.

    See tidyUrlScm policy for more details.

  • Define allRelocatable policy

    The allRelocatable policy changes the default of the relocatable property to True regardless of any defined tools. This get’s rid of the old heuristic which was too pessimistic in most cases.

  • Define offline build properties and policy

    Bob will prevent network access by default during build and package steps when using a sandbox. The {build,package}NetAccess properties can override this behavior and the offlineBuild policy controls the default setting.

  • Define sandboxInvariant policy

    Traditionally the impact of a sandbox to the build has not been handled consistently. On one hand the actual usage of a sandbox was not relevant for binary artifacts. As such, an artifact that was built inside a sandbox was also used when building without the sandbox (and vice versa). On the other hand Bob did rebuild everything from scratch when switching between sandbox/non-sandbox builds. This inconsistent behavior is rectified by the sandboxInvariant policy that consistently declares builds as invariant of the sandbox.

  • Warn/Fail on duplicate dependencies (uniqueDependency policy)

    Naming the same dependency multiple times in a recipe is deprecated. If encountered a warning is shown unless the uniqueDependency policy is set to the new behavior. In this case the parsing is stopped with an error.

    See uniqueDependency policy for more details.

Dev/build behavioural changes

  • Clean workspace on manual invocation of build/package.sh too

    For package steps the workspace is always cleaned. This was not the case if package.sh was invoked manually, though. Fix this and also adapt the behaviour of build.sh to the last Bob invocation (clean vs. incremental).

  • Check all dependencies for input changes

    Changing the source code of a tool did not trigger an incremental rebuild of packages that use this tool. Such changes are now taken into account an will trigger an incremental build (if possible) of affected packages.

Other behavioural changes

  • Handle dependency correctly if it is named multiple times

    It is possible (but not useful) to name the same dependency multiple times in a recipe. If only the tools are used it was not detected when multiple variants of the same package were specified. We now do the duplicate check independent of what is actually used.

    If a package is named multiple times in the depends section of a recipe and they are the same variant then we will issue a warning. It is (and was already in the past) defined that only the first result in the list is taken. But it is most likely an error of the user if there is more than one reference to the same package. It is also possible that, even if the packages themself are of the same variant, they might provide different dependencies or variables upwards. This is handled but not easily detectable by the user.

    Setting the uniqueDependency policy to the new behavior will halt the parsing with an error when a dependency is detected multiple times.

  • In rare circumstances it was possible that a dependency was considered more than once in a package. This bug lead to wrongly calculated variant- and build-IDs. Unfortunately fixing this bug can lead to incompatible binary artifacts with previous versions of Bob.

    There is no impact on the build result by this fix because the bug only affected the internal calculation. Bob already passed the dependency only once to the build script of such a package.

Performance improvements

  • Optimize internal data structures to lower memory footprint

    Depending on the recipes the memory consumption is roughly halved. This also improves package calculation time to some extent.

  • Bob dev: improve startup time

    The directory layout is now cached across invocations. This saves a couple of seconds until a build starts. It will also keep the directory assignment more stable in case of recipe changes.

  • Bob dev: optimize audit trail generation time

  • Bob project: improve project generation speed on large projects

  • Jenkins: projects with many jobs are calculated an order of magnitude faster

  • archive: improve local upload speed

    The compression level was reduced to 6 which produces only marginally bigger archives but is dramatically faster on compression. Some performance numbers with a reasonably (~600MB) sized workspace of text and binary files:

    Bob 0.13 (level 9): 220MiB, 2m10s
Bob 0.14 (level 6): 222MiB, 0m25s

  • Plugins: fix PluginState class comparison.

    Any plugin that used the bob.input.PluginState class caused a massive performance drop on huge projects. This is fixed for most plugins but it might still be necessary to update some plugins. See bob.input.PluginState for the details.