Gaffer for pipeline – part 1 – App structure and startup

It can be hard to know where to start when you first start integrating a new application into a pipeline. This post is the first part in a series that takes a bottom-up approach and brings together a few helpful links for getting started with Gaffer configuration, scripting and development.

In this part, we take a look at how the gaffer application is structured, and what happens at startup.

Gaffer Overview

Gaffer itself is a series of c++ libraries and Python modules (Gaffer, GafferScene, etc.). UI code is kept distinctly separate – in libraries with the UI suffix. These libraries are then assembled into several applications to perform specific tasks. Apps are loaded from $GAFFER_APP_PATHS and can be run using the gaffer command. The Gaffer distribution provides several standard apps in its ./apps folder.

These are assembled into several applications to perform specific tasks. Apps are loaded from $GAFFER_APP_PATHS and can be run using the gaffer command. The Gaffer distribution provides several standard apps in its ./apps folder.

Running gaffer by itself defaults to the gui app, gaffer dispatch gaffer browser and gaffer execute for example, run the dispatch, browser and execute apps respectively. A full list of registered applications can found by running gaffer --help.

Common apps you’ll use in production:

gui The default when no app is specified, forms the main artist facing UI.
execute All farm tasks use this headless app for rendering/etc…
dispatch Submit jobs using one of the gaffer dispatchers.
env Run arbitrary commands in the gaffer environment.

Having separate apps for different tasks avoids loading code you don’t need (eg: no GUI on the farm), or allows you develop independent tools (eg: gaffer browser). Understanding this separation of libraries and apps is critical to understanding how to configure ‘gaffer’.

Configuration

Most of gaffer can be customised by modifying the configuration via python. You can (amongst other things) add new menus, tools and editors, adjust defaults for nodes or even change the appearance of the app. We’ll talk more about specific tasks in subsequent posts. A common entry point for adjusting config is in a startup script.

Startup scripts

When gaffer starts up, it loads all files in the directories specified in the $GAFFER_STARTUP_PATHS environment variable. This is a : separated path-style list of directories. This makes it a natural fit for configuration or loading custom code (ie: ‘plug-ins’).

Starting on the right, and working to the left, gaffer looks in each entry for a subdirectory whose name matches the app or library being loaded, and runs any python files it finds. For example:

<entry>/gui/*.py will be run when the gui app loads
<entry>/GafferScene/*.py will be run whenever the GafferScene module is imported in python (ie: import GafferScene)

The general convention is:

lowercase for apps
UpperCamelCase for modules

It’s important to understand this as choosing where you put your script impacts what it can affect. Anything in the gui folder for example wouldn’t be loaded by execute, and so unavailable on the farm. Something in the Gaffer folder would be available to anyone that made use of the core Gaffer library – even someone typing python in a shell and then import Gaffer – choose wisely!

info

At startup, all the scripts found on the path are run. Gaffer doesn’t check to see if it has already sourced a script with any given name or similar. As such, there is no hard and fast rule as to what ‘wins’. Simply that code in a more leftward directory will run after that in any directories to the right of it on the path. Depending on what your script does, this may overwrite previous actions, or simply add to them!

Headless .vs. UI

It’s critical that no UI code is loaded for batch/headless applications (such as execute). As such, you should never import a Gaffer*UI module in your scripts unless you’re in an interactive app or UI module startup. This may mean that you need to split your startup scripts and custom code up such that headless code is available on the farm, and UI code in an interactive session.

The startup-script interpreter

Startup scripts are run using gaffer’s bundled python interpreter using execfile. They are run with an empty context aside from the addition of:

__file__ as per python’s built-in.
application (only for app startup, not libraries) the Gaffer.ApplicationRoot object that is loading the script.

As such, any variables you define in a startup script will go out-of-scope at the end of your file.

Useful Links

The apps that form the gaffer distribution and their docs.
Gaffer’s own startup scripts, generally these assemble the component libraries and provide compatibility shims. They can be a useful reference.