Extending Bob

Bob may be extended through plugins. Right now the functionality that can be tweaked through plugins is intentionally limited. If you can make a case what should be added to the plugin interface, please open an issue at GitHub or write to the mailing list.

See contrib/plugins in the Bob repository for some examples.

Plugins

Plugins can be put into a plugins directory as .py files. A plugin is only loaded when it is listed in config.yaml in the plugins section. Each plugin must provide a ‘manifest’ dict that must have at least an ‘apiVersion’ entry. The apiVersion is compared to the Bob version and must not be greater for the plugin to load. At minimum this looks like this:

manifest = {
    'apiVersion' : "0.2"
}

Class documentation

Plugins might only access the following classes with the members documented in this manual. All other parts of the bob Python package namespace are considered internal and might change without notice.

class bob.input.PluginProperty(present, value)

Base class for plugin property handlers.

A plugin should sub-class this class to parse custom properties in a recipe. For each recipe an object of that class is created then. The default constructor just stores the present and value parameters as attributes in the object.

Parameters:

  • present (bool) – True if property is present in recipe

  • value – Unmodified value of property from recipe or None if not present.

getValue()

Get (parsed) value of the property.

inherit(cls)

Inherit from a class.

The default implementation will use the value from the class if the property was not present. Otherwise the class value will be ignored.

Parameters:

cls (PluginProperty) – The property instance of the class

isPresent()

Return True if the property was present in the recipe.

static validate(data)

Validate type of property.

Ususally the plugin will reimplement this static method and return True only if data has the expected type. The default implementation will always return True.

Parameters:

data – Parsed property data from the recipe

Returns:

True if data has expected type, otherwise False.

class bob.input.PluginSetting(settings)
getSettings()

Getter for settings data.

merge(other)

Merge other settings into current ones.

This method is called when other configuration files with a higher precedence have been parsed. The settings in these files are first validated by invoking the validate static method. Then this method is called that should update the current object with the value of other.

The default implementation implements the following policy:

  • Dictionaries are merged recursively on a key-by-key basis

  • Lists are appended to each other

  • Everything else in other replaces the current settings

It is assumed that the actual settings are stored in the settings member variable.

Parameters:

other – Other settings with higher precedence

priority = 50

Base class for plugin settings.

Plugins can be configured in the user configuration of a project. The plugin must derive from this class, create an object with the default value and assign it to ‘settings’ in the plugin manifest. The default constructor will just store the passed value in the settings member.

Parameters:

settings – The default settings

static validate(data)

Validate type of settings.

Ususally the plugin will reimplement this method and return True only if data has the expected type. The default implementation will always return True.

Parameters:

data – Parsed settings data from user configuration

Returns:

True if data has expected type, otherwise False.

class bob.input.PluginState

Base class for plugin state trackers.

State trackers are used by plugins to compute the value of one or more properties as the dependency tree of all recipes is traversed.

Attention

Objects of this class are tested for equivalence. The default implementation compares all members of the involved objects. If custom types are stored in the object you have to provide a suitable __eq__ and __ne__ implementation because Python falls back to object identity which might not be correct. If these operators are not working correctly then Bob may slow down considerably.

copy()

Return a copy of the object.

The default implementation uses copy.deepcopy() which should usually be enough. If the plugin uses a sophisticated state tracker, especially when holding references to created packages, it might be usefull to provide a specialized implementation.

onEnter(env, properties)

Begin creation of a package.

The state tracker is about to witness the creation of a package. The passed environment, tools and (custom) properties are in their initial state that was inherited from the parent recipe.

Parameters:

  • env (Mapping[str, str]) – Complete environment

  • properties (Mapping[str, bob.input.PluginProperty]) – All custom properties

onFinish(env, properties)

Finish creation of a package.

The package was computed. The passed env and properties have their final state after all downstream dependencies have been resolved.

Parameters:

  • env (Mapping[str, str]) – Complete environment

  • properties (Mapping[str, bob.input.PluginProperty]) – All custom properties

onUse(downstream)

Use provided state of downstream package.

This method is called if the user added the name of the state tracker to the use clause in the recipe. A state tracker supporting this notion should somehow pick up and merge the state of the downstream package.

The default implementation does nothing.

Parameters:

downstream (bob.input.PluginState) – State of downstream package

class bob.input.RecipeSet

The RecipeSet corresponds to the project root directory.

It holds global information about the project.

defaultEnv()

The default environment that each root recipe inherits

Return type:

Mapping[str, str]

envWhiteList()

The set of all white listed environment variables

Return type:

Set[str]

generatePackages(nameFormatter, sandboxEnabled=False, stablePaths=None)

Generate package set.

Parameters:

  • nameFormatter (Callable[[Step, Mapping[str, PluginState]], str]) – Function returning path for a given step.

  • sandboxEnabled (bool) – Enable sandbox image dependencies.

  • stablePaths (None | bool) – Configure usage of stable execution paths (/bob/…). * None: Use stable path in sandbox image, otherwise workspace path. * False: Always use workspace path. * True: Always use stable path (/bob/…).

getProjectRoot()

Get project root directory.

The project root is where the recipes, classes and layers are located. In case of out-of-tree builds it will be distinct from the build directory.

class bob.input.Recipe

Representation of a single recipe

Multiple instaces of this class will be created if the recipe used the multiPackage keyword. In this case the getName() method will return the name of the original recipe but the getPackageName() method will return it with some addition suffix. Without a multiPackage keyword there will only be one Recipe instance.

getLayer()

Get layer to which this recipe belongs.

Returns a list of the layer hierarchy. The root layer is represented by an empty list. If the recipe belongs to a nested layer the layers are named from top to bottom. Example: layers/foo/layers/bar/recipes/baz.yaml -> ['foo', 'bar'].

If the managedLayers policy is set to the new behaviour, nested layers are flattened. This means that layers are always returnd as single-item lists.

Return type:

List[str]

getName()

Get plain recipe name.

In case of a multiPackage multiple packages may be derived from the same recipe. This method returns the plain recipe name.

getPackageName()

Get the name of the package that is drived from this recipe.

Usually the package name is the same as the recipe name. But in case of a multiPackage the package name has an additional suffix.

getPluginProperties()

Get all plugin defined properties of recipe.

The values of all properties have their final value, i.e. after all classes have been resolved.

Returns:

Plugin defined properties of recipe

Return type:

Mapping[str, bob.input.PluginProperty]

getRecipeSet()

Get the RecipeSet to which the recipe belongs

isRoot()

Returns True if this is a root recipe.

class bob.input.Package

Representation of a package that was created from a recipe.

Usually multiple packages will be created from a single recipe. This is either due to multiple upstream recipes or different variants of the same package. This does not preclude the possibility that multiple Package objects describe exactly the same package (read: same Variant-Id). It is the responsibility of the build backend to detect this and build only one package.

getAllDepSteps()

Return list of all dependencies of the package.

This list includes all direct and indirect dependencies. Additionally the used sandbox and tools are included too.

getBuildStep()

Return the build step of this package.

getCheckoutStep()

Return the checkout step of this package.

getDirectDepSteps()

Return list of the package steps of the direct dependencies.

Direct dependencies are the ones that are named explicitly in the depends section of the recipe. The order of the items is preserved from the recipe.

getIndirectDepSteps()

Return list of indirect dependencies of the package.

Indirect dependencies are dependencies that were provided by downstream recipes. They are not directly named in the recipe.

getMetaEnv()

meta variables of package

getName()

Name of the package

getPackageStep()

Return the package step of this package.

getPluginStates()

Return state trackers of this package.

Returns:

All plugin defined state trackers of the package

Return type:

Mapping[str, bob.input.PluginState]

getRecipe()

Return Recipe object that was the template for this package.

getStack()

Returns the recipe processing stack leading to this package.

The method returns a list of package names. The first entry is a root recipe and the last entry is this package.

isRelocatable()

Returns True if the packages is relocatable.

class bob.input.Step

Represents the smallest unit of execution of a package.

A step is what gets actually executed when building packages.

Steps can be compared and sorted. This is done based on the Variant-Id of the step. See bob.input.Step.getVariantId() for details.

doesProvideTools()

Return True if this step provides at least one tool.

getAllDepSteps()

Get all dependent steps of this Step.

This includes the direct input to the Step as well as indirect inputs such as the used tools or the sandbox.

getArguments()

Get list of all inputs for this Step.

The arguments are passed as absolute paths to the script starting from $1.

getDigestScript()

Return a long term stable script.

The digest script will not be executed but is the basis to calculate if the step has changed. In case of the checkout step the involved SCMs will return a stable representation of what is checked out and not the real script of how this is done.

getEnv()

Return dict of environment variables.

getLabel()

Return path label for step.

This is currently defined as “src”, “build” and “dist” for the respective steps.

getPackage()

Get Package object that is the parent of this Step.

getSandbox()

Return Sandbox used in this Step.

Returns a Sandbox object or None if this Step is built without one.

getScript()

Return a single big script of the whole step.

Besides considerations of special backends (such as Jenkins) this script is what should be executed to build this step.

getTools()

Get dictionary of tools.

The dict maps the tool name to a bob.input.Tool.

getVariantId()

Return Variant-Id of this Step.

The Variant-Id is used to distinguish different packages or multiple variants of a package. Each Variant-Id need only be built once but subsequent builds might yield different results (e.g. when building from branches).

getWorkspacePath()

Return the workspace path of the step.

The workspace path represents the location of the step in the user’s workspace. When building in a sandbox this path is not passed to the script but the one from getExecPath() instead.

isBuildStep()

Return True if this is a build step.

isCheckoutStep()

Return True if this is a checkout step.

isDeterministic()

Return whether the step is deterministic.

Checkout steps that have a script are considered indeterministic unless the recipe declares it otherwise (checkoutDeterministic). Then the SCMs are checked if they all consider themselves deterministic. Build and package steps are always deterministic.

The determinism is defined recursively for all arguments, tools and the sandbox of the step too. That is, the step is only deterministic if all its dependencies and this step itself is deterministic.

isPackageStep()

Return True if this is a package step.

isRelocatable()

Returns True if the step is relocatable.

isShared()

Returns True if the result of the Step should be shared globally.

The exact behaviour of a shared step/package depends on the build backend. In general a shared package means that the result is put into some shared location where it is likely that the same result is needed again.

isValid()

Returns True if this step is valid, False otherwise.

jobServer()

Returns True if the jobserver should be used to schedule builds for this step.

class bob.input.Sandbox

Represents a sandbox that is used when executing a step.

getEnvironment()

Get environment variables.

Returns the dictionary of environment variables that are defined by the sandbox.

getMounts()

Get custom mounts.

This returns a list of tuples where each tuple has the format (hostPath, sandboxPath, options).

getPaths()

Return list of global search paths.

This is the base $PATH in the sandbox.

getStep()

Get the package step that yields the content of the sandbox image.

getUser()

Get user identity in sandbox.

Returns one of ‘nobody’, ‘root’ or ‘$USER’.

isEnabled()

Return True if the sandbox is used in the current build configuration.

class bob.input.Tool

Representation of a tool.

A tool is made of the result of a package, a relative path into this result and some optional relative library paths.

getEnvironment()

Get environment variables.

Returns the dictionary of environment variables that are defined by the tool.

getLibs()

Get list of relative library paths into the result.

Returns:

List[str]

getNetAccess()

Does tool require network access?

This reflects the netAccess tool property.

Returns:

bool

getPath()

Get relative path into the result.

getStep()

Return package step that produces the result holding the tool binaries/scripts.

Returns:

bob.input.Step

Hooks

Path formatters

Path formatters are responsible for calculating the workspace path of a package. There are three plugin hooks that override the path name calculation:

  • releaseNameFormatter: local build in release mode

  • developNameFormatter: local build in development mode

  • jenkinsNameFormatter: Jenkins builds

All hooks must be a function that return the relative path for the workspace. The function gets two parameters: the step for which the path should be returned and a dictionary to the state trackers of the package that is processed. The default implementation in Bob looks like this:

def releaseNameFormatter(step, properties):
    return os.path.join("work", step.getPackage().getName().replace('::', os.sep),
                        step.getLabel())

def developNameFormatter(step, properties):
    return os.path.join("dev", step.getLabel(),
                        step.getPackage().getName().replace('::', os.sep))

def jenkinsNameFormatter(step, props):
    return step.getPackage().getName().replace('::', "/") + "/" + step.getLabel()

manifest = {
    'apiVersion' : "0.1",
    'hooks' : {
        'releaseNameFormatter' : releaseFormatter,
        'developNameFormatter' : developFormatter,
        'jenkinsNameFormatter' : jenkinsNameFormatter
    }
}

Additionally there is a special hook (developNamePersister) that is responsible to create a surjective mapping between steps and workspace paths with the restriction that different variant ids must not be mapped to the same directory. The hook function is taking the configured develop name formatter (see above) and is expected to return a callable name formatter too. The developNamePersister must handle two cases in the following way:

  • The passed name formatter returns different paths for steps that have the same variant id. In this case the developNamePersister should only return one such path for the same variant id.

  • The name formatter returns the same path for different variant ids. In this case the developNamePersister must disambiguate the path (e.g. by adding a unique suffix) to return different paths for the different variants of the step(s).

Even though it is not strictly required by Bob it is highly recommended to map all steps with the same variant id to a single directory. The hook is currently only available for the develop mode. The default implementation in Bob is to append an incrementing number starting by one for each variant to the path returned by the configured name formatter:

def developNamePersister(nameFormatter):
    dirs = {}

    def fmt(step, props):
        baseDir = nameFormatter(step, props)
        digest = step.getVariantId()
        if digest in dirs:
            res = dirs[digest]
        else:
            num = dirs.setdefault(baseDir, 0) + 1
            res = os.path.join(baseDir, str(num))
            dirs[baseDir] = num
            dirs[digest] = res
        return res

    return fmt

manifest = {
    'apiVersion' : "0.1",
    'hooks' : {
        'developNamePersister' : developNamePersister
    }
}

If your name formatter generates unique names for each variant of the steps you may want to override the persister to change this behavior, e.g. to not add a number for the first variant.

String functions

String functions can be invoked from any place where string substitution as described in String substitution is allowed. These functions are called with at least one positional parameter for the arguments that were specified when invoking the string function. They are expected to return a string and shall have no side effects. The function has to accept any number of additional keyword arguments. Currently the following additional kwargs are passed:

  • env: dict of all available environment variables at the current context

  • recipe: the current bob.input.Recipe

  • sandbox: True if a sandbox image is used. False if no sandbox image was configured or if it is disabled (e.g. --no-sandbox option was specified).

In the future additional keyword args may be added without notice. Such string functions should therefore have a catch-all **kwargs parameter. A sample implementation could look like this:

def echo(args, **kwargs):
    return " ".join(args)

manifest = {
    'apiVersion' : "0.2",
    'stringFunctions' : {
        "echo" : echo
    }
}

Jenkins job mangling

Jenkins jobs that are created by Bob are very simple and contain only information that was taken from the recipes. It might be necessary to enable additional plugins, add build steps or alter the job configuration in special ways. For such use cases the following hooks are available:

  • jenkinsJobCreate: initial creation of a job

  • jenkinsJobPreUpdate: called before updating a job config

  • jenkinsJobPostUpdate: called after updating a job config

All hooks take a single mandatory positional parameter: the job config XML as string. The hook is expected to return the altered config XML as string too. The function has to accept any number of additional keyword arguments. Currently the following additional kwargs are passed:

  • alias: alias name used for jenkins

  • buildSteps: list of all build steps (bob.input.Step) used in the job

  • checkoutSteps: list of all checkout steps (bob.input.Step) used in the job

  • hostPlatform: Jenkins host platform type (linux, msys or win32)

  • name: name of Jenkins job

  • nodes: The nodes where the job should run

  • packageSteps: list of all package steps (bob.input.Step) used in the job

  • prefix: Prefix of all job names

  • sandbox: Boolean whether sandbox should be used. Plugins starting with API version 0.25 are passed the sandbox mode as string (no, yes, slim, dev or strict).

  • url: URL of Jenkins instance

  • windows: True if Jenkins runs on Windows

See the jenkins-cobertura plugin in the contrib directory for an example. The default implementation in Bob looks like this:

def jenkinsJobCreate(config, **info):
    return config

def jenkinsJobPreUpdate(config, **info):
    return config

def jenkinsJobPostUpdate(config, **info):
    return config

manifest = {
    'apiVersion' : "0.4",
    'hooks' : {
        'jenkinsJobCreate' : jenkinsJobCreate,
        'jenkinsJobPreUpdate' : jenkinsJobPreUpdate,
        'jenkinsJobPostUpdate' : jenkinsJobPostUpdate
    }
}

Generators

The main purpose of a generator is to generate project files for one or more IDEs. There are several built-in generators, e.g. for QtCreator project files.

A generator is called with at least 3 arguments:

  • package: the bob.input.Package to build the project for.

  • argv: Arguments not consumed by bob project.

  • extra: Extra arguments to be passed back to bob dev when called from the IDE. These are the generic arguments that bob project parses for all generators.

Starting with Bob 0.17 an additional 4th argument is passed to the generator function:

  • bob: The fully qualified path name to the Bob executable that runs the generator. This may be used to generate project files that work even if Bob is not in $PATH.

The presence of the 4th parameter is determined by the apiVersion of the manifest.

A simple generator may look like:

def nullGenerator(package, argv, extra, bob):
    return 0

manifest = {
    'apiVersion' : "0.17",
    'projectGenerators' : {
        'nullGenerator' : nullGenerator,
    }
}

Traditionally a generator handles only one package. When running the generator this package needs to be provided using the complete path. Starting with Bob 0.23 a generator can specify that he can handle multiple packages as a result of a package query. This is done by setting the optional query property to True. In this case the first argument of the generator are the package objects returned by bob.pathspec.PackageSet.queryPackagePath().:

def nullQueryGenerator(packages, argv, extra, bob):
    for p in packages:
        print(p.getName())
    return 0

manifest = {
     'apiVersion' : "0.23",
     'projectGenerators' : {
         'nullQueryGenerator' : {
            'func' : nullQueryGenerator,
            'query' : True,
     }
 }

Plugin settings

Sometimes plugin behaviour needs to be configurable by the user. On the other hand Bob expects plugins to be deterministic. To have a common interface for such settings it is possible for a plugin to define additional keywords in the User configuration (default.yaml). This provides Bob with the information to validate the settings and detect changes in a reliable manner.

To define such settings the plugin must derive from bob.input.PluginSetting, create an instance of that class and store it in the manifest under settings. A minimal example looks like the following:

from bob.input import PluginSetting

class MySettings(PluginSetting):
    @staticmethod
    def validate(data):
        return isinstance(data, str)

mySettings = MySettings("")

manifest = {
    'apiVersion' : "0.14",
    'settings' : {
        'MySettings' : mySettings
    }
}

This will define a new, optional “MySettings” keyword for the user configuration that will accept any string. The default value, if nothing is configured in default.yaml, is specified when constructing MySettings. In the above example it is an empty string.

Attention

Do not configure your plugins by any other means. Bob will not detect changes and, due to aggressive caching, might not call the plugin again to process the new settings. So reading external files or using environment variables results in undefined behavior.

It is not possible to re-define already existing setting keywords. This applies both to Bob built-in settings as well as settings defined by other plugins. Because Bob is expected to define new settings in the future a plugin defined setting must not start with a lower case letter. These names are reserved for Bob.

Custom recipe properties

A plugin may define any number of additional recipe properties. A property describes a key in a recipe that is parsed. The class handling the property is responsible to validate the data in the recipe and store the value. It must be derived from bob.input.PluginProperty. The property class handles the inheritance between recipes and classes too.

The following example shows two trivial properties:

class StringProperty(PluginProperty):
    @staticmethod
    def validate(data):
        return isinstance(data, str)

manifest = {
    'apiVersion' : "0.21",
    'properties' : {
        "checkoutDir" : StringProperty,
        "platform" : StringProperty
    },
}

The above example defines two new keywords in recipes: checkoutDir and platform. As verified by the validate method they need to be strings. Because bob.input.PluginProperty.inherit() was not overridden, the recipe and higher priority classes will simply replace the value of lower priority classes. Other plugin extensions can query the value of a property by calling bob.input.Recipe.getPluginProperties() to fetch the instances of a particular recipe. For example this might be used by project generators to supply recipe specific data to the generator.

If custom properties need to be propagated in the recipe dependency hierarchy, a property state tracker is required. A state tracker is a class that is invoked on every step when walking the dependency tree to instantiate the packages. The state tracker thus has the responsibility to calculate the final values associated with the properties for every package. Like properties there can be more than one state tracker. Any state tracker provided by a plugin must be derived from bob.input.PluginState.