Traitlets - an introduction & use in Jupyter configuration management
Published August 30, 2020
You have probably seen Traitlets in applications, you likely even use it. The package has nearly 5 million downloads on conda-forge alone.
But, what is Traitlets ?
In this post we'll answer this question along with where Traitlets came from, its applications, and a bit of history.
traitlets 5.0 has recently been released; 5 years after the earliest versions of
traitlets 4.0. The latest and greatest 5.0 release brings
new features and a cleaner codebase while maintaining backward compatibility with 4.0.
This is a big upgrade to our interactive computing tools because Traitlets is used everywhere in Jupyter/IPython - for configuration, runtime type checking, widgets, and CLI parsing.
Traitlets is a library that provides objects that can expose individual configuration options in an intelligent way. They are used in almost all the Jupyter Projects.
Traitlets began as a pure Python implementation of the Enthought Traits library.
These libraries implement the object-oriented trait pattern. Prior to 2015,
traitlets was a part of the IPython (pre-jupyter) code base; then during "The Big
Split" it was moved to their own reusable package.
traits addressed the challenge of using typed code in interactive Python REPLs. They offer type checking, coercion and validation at run time.
Traitlets for development
The general idea when developing with Traitlets is:
The developer defines a class-level attribute,
This class level attribute will automatically be converted into a property-like element with runtime type and value checking, configurability and observable events and changes.
Traitlets minimizes boilerplate for Python applications. Traitlets maintains a uniform naming convention and helps your users configure their applications.
A Traitlets usage example
We'll demonstrate the flexibility of Traitlets by configuring the
traitlet/option of IPython from the command line interface, configuration
file, as well as observe effect of dynamically changing values.
Below is an excerpt of the
IPython main class that defines
from traitlets import SingletonConfigurable, Enumclass InteractiveShell(SingletonConfigurable): ... autocall = Enum((0,1,2), default_value=0, help= """ Make IPython automatically call any callable object even if you didn't type explicit parentheses. For example, 'str 43' becomes 'str(43)' automatically. The value can be '0' to disable the feature, '1' for 'smart' autocall, where it is not applied if there are no more arguments on the line, and '2' for 'full' autocall, where all callable objects are automatically called (even if no arguments are present). """ ).tag(config=True) ...
autocall class attribute will be converted at instantiation to an
property, in particular an
Enum whose values are ensured to be
2. Traitlets provides a number of utilities to decide
whether assigning incorrect values should raise an exception; or coerce to one
of the valid ones. Here a wrong assignment will fail:
$ ipython --no-bannerIn : ip = get_ipython()In : ip.autocallOut: 0In : ip.autocall = 5...TraitError: The 'autocall' trait of a TerminalInteractiveShell instance expected any of [0, 1, 2], not the int 5.
While type – and value – checking at runtime is a nice feature, most of these
options are usually user preferences. Traitlets provides a way to automatically
create configuration files with help, as well as command line arguments
autocall traitlet has
strings that are tagged with
config=True. This informs the current
application on how to automatically generate configuration files, decide on
the option name and document it. No need for the developer to spend time on
deciding on a configuration parameter name.
A generated configuration file.
On a brand new machine with IPython installed you will find the following in
your default configuration file
referring to our
autocall traitlet of previous section:
...## Make IPython automatically call any callable object even if you didn't type# explicit parentheses. For example, 'str 43' becomes 'str(43)' automatically.# The value can be '0' to disable the feature, '1' for 'smart' autocall, where# it is not applied if there are no more arguments on the line, and '2' for# 'full' autocall, where all callable objects are automatically called (even if# no arguments are present).#c.InteractiveShell.autocall = 0...
You will recognize there the help string provided before, as well as the default value.
If you update this file by uncommenting the last line and changing the value,
as you would expect, new instances of
InteractiveShell will be instantiated
with this new default value.
Alternatively, Traitlets can also generate and parse command line arguments, so
ipython --InteractiveShell.autocall=3 will take precedence over
configuration files and start IPython with this new option:
$ ipython --no-banner --InteractiveShell.autocall=2In : get_ipython().autocallOut: 2
It will display the same help if you try
$ ipython --help-all...--InteractiveShell.autocall=<Enum> Make IPython automatically call any callable object even if you didn't type explicit parentheses. For example, 'str 43' becomes 'str(43)' automatically. The value can be '0' to disable the feature, '1' for 'smart' autocall, where it is not applied if there are no more arguments on the line, and '2' for 'full' autocall, where all callable objects are automatically called (even if no arguments are present). Choices: any of [0, 1, 2] Default: 0
Adding a configuration option is thus a breeze, for example IPython can reformat your code with Black since this pull request. Beyond the logic to actually do the reformatting, the complete diff to add the options to the CLI, configuration file, and automatic generation of the option documentation in Sphinx documentation is as follows:
@IPython/terminal/interactiveshell.py:98Class TerminalInteractiveShell(InteractiveShell).snip ...+ autoformatter = Unicode(None,+ help="Autoformatter to reformat code.",+ allow_none=True+ ).tag(config=True)
If you have an application or library which potentially has a really large number of configuration knobs and you want to isolate changes: Traitlets can help you expose all of those cleanly to a user, without having to think about the option name or writing the logic to set a default value.
Unlike many other libraries likes argparse or click, the listing of configuration options is decentralized and lives on the object being configured. And there is no differences between adding a configuration option in the configuration file or at the command line interface.
Configure what you do not yet know about
In an application with few parameters and only a couple of plugins it might be relatively straightforward to provide options and CLI arguments; this becomes harder when arbitrary plugins are involved and those plugins have arbitrary configuration options you may or may not know about at startup time.
One good example is JupyterHub. JupyterHub has various plugins, one category of
which is Spawners. Spawners decide how notebook servers are started. You can
use a custom spawner,
and many institutions employ only minimal changes to the default Spawner to
accommodate their use case. A few of the common Spawners are
KubeSpawner, each with their own parameters.
It is critical to make it as simple as possible to provide configuration options and make them available from Jupyter Configuration files and from the command line.
Using a custom Spawner is simple:
c.JupyterHub.spawner_class = 'mypackage:MySpawner'
and Traitlets allows you to also arbitrarily configure
c.MySpawner.mem_limit = '200G'
Traitlets is aware of class hierarchy, thus when
MySpawner inherits from the
default Spawner, all
c.Spawner... options will affect
c.MySpawner... options will of course only affect
It is thus also easy to configure siblings differently; a good example is Terminal
IPython vs IPykernel (used in notebooks and lab). Both applications are subclasses
I can configure both with
c.InteractiveShell.attribute=, or decide that only
terminal IPython will be affected via
c.TerminalInteractiveShell.attribute=, I can also
target only notebook-like interfaces with
On my own machine for example,
%matplotlib is setup to be
inline only for
kernels and not terminal.
Thus if you have an application or library with a number of plugins, and for which configurability can be thought of as tree-like similar to a class hierarchy, Traitlets can help you.
Beyond the configurability part is observability; as we already had a great type system with hooks, and sometimes you may want to mutate configuration of a running application, traitlets allow you to observe values and propagate them to other places.
To look at the above example with code reformatting, the reformatter can be changed dynamically to a different one with the following:
Class TerminalInteractiveShell(InteractiveShell)... @observe('autoformatter') def _autoformatter_changed(self, change): formatter = change.new if formatter is None: self.reformat_handler = lambda x:x elif formatter == 'black': self.reformat_handler = black_reformat_handler else: raise ValueError
Observability is also what powers ipywidgets; it allows dynamic binding of sliders, buttons and other controls to visualisations.
This can be useful to glue together models, with parameters needing to be synchronized and models needing to react to changes.
Traitlets supports dynamic defaults, i.e. your default values may depend on some context (OS, Python version).
Configurations are by default Python scripts (but can be JSON), so user config files can be shared across machines and have dynamic values.
Context base configuration
Object configuration can also look at the creator of the object at instantiation time.
c.Foo.Bar.attr = 1c.Qux.Bar.attr = 2
With the above, the
Bar object created by
Foo or by
Qux will get different
default values. This is for example used by nbconvert
to decide which options to activate depending on whether you do
Flags and aliases
Long flag names like
--InteractiveShell.debug=True can be cumbersome to type.
That is why traitlets provide aliases and flags, so that
ipython --debug will
debug=True to enable logging on all classes supporting it while still
allowing you to tweak the login level used on a per-class basis.
Of course all this power doesn't come completely for free:
- Your class and attribute names become part of your API
- As configuration is loaded/read before any potential plugins are loaded, it is impossible to definitively detect typos or invalid configuration options.
- Traitlets relies heavily on metaclasses, so it can add a construction cost to your objects and can be hard to debug.
This was a short introduction to Traitlets. I hope it made it a little clearer how the Jupyter configuration system works, and we are looking forward to see how this can be used to adapt Jupyter to your work flow.
Are you struggling with a system that has too many configuration options? Do you have a use case where you believe Traitlets can be useful? We'll be happy to hear from you.
Try the new 5.x version and let us know if you have questions.