configmanager¶

configmanager on github: https://github.com/jbasko/configmanager

Contents

Introduction ¶

Objectives ¶

Provide a clean, object-oriented interface to:

declare configuration items; organise them into a configuration tree of arbitrary depth
parse various configuration sources and initialise configuration items accordingly
read values of configuration items
change values of configuration items
write configuration to various destinations
enrich configuration items with additional meta information and more advanced methods of calculating values at run-time.
inspect meta information of configuration items

Why not just use `ConfigParser`?¶

ConfigParser does what its name suggests – it parses configuration files (of one certain format).

It only starts to appear quite limited if you misuse it as a configuration manager. By configuration manager we mean a party responsible for things like managing a list of allowed configuration items and their defaults, loading custom configuration values from different sources (files, command-line arguments, environment variables etc.), casting values to their right types, providing interface to retrieve these values consistently. Just because an instance of ConfigParser holds the data it has parsed doesn’t mean it should be used in application code to look up configuration values.

If ConfigParser is used as a configuration manager some of its limitations are:

a configuration value is just a string which is cast to the right type only if you specifically ask to every time when you are looking it up.
a configuration item (called option in ConfigParser) must belong to a section.
you can’t have configuration trees deeper than one section + one option.
it only supports INI-format files.

Key Terms and Principles ¶

Configuration of a system is a hierarchical tree of arbitrary depth in which leaf nodes are configuration items, and non-leaf nodes are configuration sections.

Each sub-tree (section) of the tree can be regarded as a configuration of a sub-system.

Configuration can also be seen as a collection of key-value pairs in which value is a configuration item (including its name and state), and key is the path of the item in the tree.

All referenced configuration sections and items have to be declared before they are used to stop user from referring to non-existent, mistyped configuration paths.

Configuration Item ¶

has a name
has a type
has a value which is determined dynamically when requested
may have a default value which is returned as its value when no custom value is set
may have a custom value which overrides the default value
may be marked as required in which case either a custom value or default value has to be available when item’s value is requested.
may have its value requested using Item.get() which accepts a fallback which is used when neither a default value, nor a custom value is available.
does NOT know its path in a configuration tree
knows the section it has been added to
can be extended with other attributes and to consult other sources to calculate its value

Configuration Section ¶

is an ordered collection of named items and named sections
has a name which is assigned to it when it is added to a parent section. Root section does not have a name

Quick Start ¶

Install the latest version from pypi.python.org
```
pip install configmanager
```
Import Config.
```
from configmanager import Config
```

Create your main config section and declare defaults.

config = Config([
    ('greeting', 'Hello, world!'),
])

Inspect config values.

>>> config.greeting.value
'Hello, world!'

>>> config.dump_values()
{'greeting': 'Hello, world!'}

Change config values.

>>> config.greeting.value = 'Hey!'
>>> config.greeting.value
'Hey!'
>>> config.greeting.default
'Hello, world!'

>>> config.load_values({'greeting': 'Good evening!'})
>>> config.greeting.value
'Good evening!'

>>> config.dump_values()
{'greeting': 'Good evening!'}

Persist the configuration.
```
config.configparser.dump('config.ini')
```

User Guide ¶

Schemas ¶

By default, configmanager requires user to declare all configuration items that their application code depends on in schemas. Our view is that if the set of available configuration items varies then it’s not really a configuration anymore.

A schema is basically a listing of what sections and items are there. The simplest way to declare a section is to pass a dictionary to Config initialiser:

from configmanager import Config

config = Config({
    'user': 'admin',
    'password': 'secret',
})

The code above creates a configuration section that will manage two configuration items, user and password, with default values set to admin and secret respectively, and both declared as having type string (guessed from the default value provided).

If all your configuration values are strings with no default values (an anti-pattern in our view), you can declare them by passing a list of strings:

# anti-pattern: lots of configuration items with no defaults
db_config = Config(['host', 'user', 'password', 'name'])

The first example was better by providing some sensible defaults, but it is not how you want to declare your configuration. We believe that the order of items and sections in configuration matters, so a better schema would either pass an OrderedDict, or a list of (name, default_value) pairs:

config = Config([
    ('user', 'admin'),
    ('password', 'secret'),
])

Default value is not the only thing that can be specified in schema. If we pass a list of Item instances then we allow ourselves to be more specific:

from configmanager import Config, Item

config = Config([
    ('hostname', 'localhost'),  # it's ok
    ('port', Item(type=int, required=True)),  # better
    Item('enabled', default=False),  # name can be set directly on Item
])

Sections can also contain other sections. In the example below, we use dictionary notation just for clarity – you are advised to use the list of tuples notation to preserve order of sections and items.

import logging_config  # module in which you configure logging

config = Config({
    'uploads': {  # this will become a Config too
        'enabled': True,
        'tmp_dir': '/tmp',
        'db': {  # and this will become a Config
            'user': 'root',
            'password': 'secret',
            'host': 'localhost',
            'name': 'exampledb',
        },
    },
    'logging': logging_config,  # a Python module is also a valid part of schema
})

This allows to maintain multiple Config instances, one for each component of your code, which are then combined into a containing instance only when you have a component that relies on configuration of multiple components.

Names, Aliases, and Paths ¶

The examples that follow will rely on this instance of Config:

config = Config([
    ('uploads', Config([
        ('enabled', True)
    ])),
    ('greeting', 'Hello'),
    ('tmp_dir', '/tmp'),
])

Name of a configuration item or alias of a configuration section is a string which has to be unique in the section to which the item or the section is added. In the example above, 'uploads' is an alias for a section, 'enabled' is a name, and so are 'greeting', and 'tmp_dir'. Note that the root section – the configuration tree – does not have an alias.

Path of an item or section is a tuple of names and aliases describing the item’s or section’s place in a configuration tree. In the example above, ('uploads',), ('uploads', 'enabled'), ('greeting',), and ('tmp_dir',) are all the existing paths.

Note that the path of an item is relative to where you observe it from. If you were iterating over all paths of 'uploads' section, the path of its only item would be ('enabled',), not ('uploads', 'enabled').

Names of items and aliases of sections have no meaning when traversing the configuration tree recursively, so iterators that do that yield paths instead.

If you have a name of an item or an alias of a section, for example, greeting, you can retrieve it from its parent section in two ways:

>>> config.greeting
<Item greeting 'Hello'>

>>> config['greeting']
<Item greeting 'Hello'>

To retrieve a section or an item using its path, you have to know the root section relative to which the path was generated and use the [] notation:

>>> config[('greeting',)]
<Item greeting 'Hello'>

>>> config[('uploads',)]
<Config uploads at 4436269600>

>>> config[('uploads', 'enabled')]
<Item enabled True>

Iterators ¶

configmanager provides several handy iterators to walk through items and sections of a configuration tree (which itself is a section).

The examples that follow will rely on this instance of Config:

config = Config([
    ('uploads', Config([
        ('enabled', True)
    ])),
    ('greeting', 'Hello'),
    ('tmp_dir', '/tmp'),
])

Iterable Sections ¶

An instance of Config (which is used to represent both sections and the whole configuration tree) is an iterable, and iterating over it will yield names of sections and items contained directly in it. Note that sub-sections aren’t inspected.

>>> for name in config:
...     print(name)
...
uploads
greeting
tmp_dir

>>> for name in config.uploads:
...     print(name)
enabled

len() of a section will return number of items and sections it contains.

>>> len(config)
3

>>> len(config.uploads)
1

`iter_all` and `iter_paths`¶

Config.iter_all() is the main iterator that is used by all others. It yields key-value pairs where keys are paths and values are sections or items. It accepts an optional recursive= kwarg which if set to True will make the iterator yield contents of sub-sections too.

>>> for path, obj in config.iter_all(recursive=True):
...     print(path, obj)
...
('uploads',) <Config uploads at 4436269600>
('uploads', 'enabled') <Item enabled True>
('greeting',) <Item greeting 'Hello'>
('tmp_dir',) <Item tmp_dir '/tmp'>

If you wish to just iterate over all available paths, you can do so with Config.iter_paths():

>>> list(config.iter_paths(recursive=True))
[('uploads',), ('uploads', 'enabled'), ('greeting',), ('tmp_dir',)]

`.is_section` and `.is_item` helpers ¶

When traversing a configuration tree with Config.iter_all(), you may want to easily detect whether you are looking at a section or an item. You can do that by inspecting .is_section and .is_item which are defined on Config as well as on Item.

>>> config.is_section
True
>>> config.is_item
False

>>> config.greeting.is_section
False
>>> config.greeting.is_item
True

`iter_items` and `iter_sections`¶

If you know that you only care about items or only about sections, you can use Config.iter_items() and Config.iter_sections() which accept not only recursive= kwarg, but also key= which determines what is returned as key in key-value pairs.

By default, path of the returned item or section is used as the key:

>>> for path, item in config.iter_items(recursive=True):
...  print(path, item)
...
('uploads', 'enabled') <Item enabled True>
('greeting',) <Item greeting 'Hello'>
('tmp_dir',) <Item tmp_dir '/tmp'>

To get item names or section aliases (plain strings) as keys, use key='alias' for sections and key='name' for items.

>>> for name, item in config.iter_items(key='name'):
...  print(name, item)
...
greeting <Item greeting 'Hello'>
tmp_dir <Item tmp_dir '/tmp'>

>>> for alias, section in config.iter_sections(key='alias'):
...     print(alias, section)
...
uploads <Config uploads at 4436269600>

Note that using alias or name as key doesn’t make much sense in recursive context (recursive=True) as keys yielded from one section may clash with those from another. For example, ('uploads', 'tmp_dir') would have the same key as ('downloads', 'tmp_dir').

Exceptions ¶

All exception types raised by configmanager that program can recover from inherit ConfigError.

`NotFound`¶

When an unknown configuration item or section is requested, a NotFound exception is raised.

`RequiredValueMissing`¶

When an item with no default value and no custom value set is marked required and has its value requested, a RequiredValueMissing exception is raised.

Hooks ¶

`Config.hooks.not_found`¶

name, the name that was requested and was not found
section, the section in which the name was requested

`Config.hooks.item_added_to_section`¶

subject - item which was added
section - section to which the subject item was added
alias - name under which the subject item was added

`Config.hooks.section_added_to_section`¶

subject - subject which was added
section - parent section to which the subject section was added
alias - name under which the subject section was added

`Config.hooks.item_value_changed`¶

item
old_value
new_value

How to disable hooks?¶

Hooks are enabled by default whenever a first hook is registered, but can be manually disabled by passing hooks_enabled=False when initialising Config.

click Integration ¶

click is the best framework out there to create usable command-line interfaces with readable code. If you are a click user, you will find it easy to make your options and arguments fall back to configuration items when user does not specify values for them in the command line.

configmanager dependencies don’t include click package as it is an optional feature. To install configmanger with click:

pip install configmanager[click]

In your click command definition, instead of using click.option and click.argument, use <config>.click.option and <config>.click.argument where <config> is your instance of Config.

To specify which configuration item is to be used as a fallback, pass it as the last positional argument to <config>.click.option or <config>.click.argument.

import click
from configmanager import Config


config = Config({
    'greeting': 'Hello!',
})


@click.command()
@config.click.option('--greeting', config.greeting)
def say_hello(greeting):
    click.echo(greeting)


if __name__ == '__main__':
    say_hello()

Note that if you are using PlainConfig, you will have to pass config.get_item('greeting') to @config.click.option because config.greeting would be just the primitive value.

If you now run this simple program it will say Hello! by default and whatever you supply with --greeting otherwise.

Note that in click, arguments are not meant to be optional (but they can if they are marked with required=False). Since a fallback makes sense only for optional input, you will have to mark your arguments with required=False if you want them to fall back to configuration items:

@click.command()
@config.click.argument('greeting', config.greeting, required=False)
def say_hello(greeting):
    click.echo(greeting)

API Reference ¶

Public Interface ¶

`Config`¶

class configmanager.Config(schema=None, **configmanager_settings)¶

Represents a configuration tree.

Config(schema=None, **kwargs)¶

Creates a configuration tree from a schema.

Args:: schema: can be a dictionary, a list, a simple class, a module, another Config instance, and a combination of these.

Keyword Args:

config_parser_factory:

Examples:

config = Config([
    ('greeting', 'Hello!'),
    ('uploads', Config({
        'enabled': True,
        'tmp_dir': '/tmp',
    })),
    ('db', {
        'host': 'localhost',
        'user': 'root',
        'password': 'secret',
        'name': 'test',
    }),
    ('api', Config([
        'host',
        'port',
        'default_user',
        ('enabled', Item(type=bool)),
    ])),
])

<config>[<name_or_path>]

Access item by its name, section by its alias, or either by its path.

Args:: name (str): name of an item or alias of a section
Args:: path (tuple): path of an item or a section
Returns:: Item or Config

Examples:

>>> config['greeting']
<Item greeting 'Hello!'>

>>> config['uploads']
<Config uploads at 4436269600>

>>> config['uploads', 'enabled'].value
True

<config>.<name>

Access an item by its name or a section by its alias.

For names and aliases that break Python grammar rules, use config[name] notation instead.

Returns:: Item or Config

<name_or_path> in <Config>: Returns True if an item or section with the specified name or path is to be found in this section.

len(<Config>)¶: Returns the number of items and sections in this section (does not include sections and items in sub-sections).

__iter__¶: Returns an iterator over all item names and section aliases in this section.

configparser¶

Adapter to dump/load INI format strings and files using standard library’s ConfigParser (or the backported configparser module in Python 2).

Returns: ConfigPersistenceAdapter

json¶

Adapter to dump/load JSON format strings and files.

Returns: ConfigPersistenceAdapter

yaml¶

Adapter to dump/load YAML format strings and files.

Returns: ConfigPersistenceAdapter

alias¶

Returns alias with which this section was added to another or None if it hasn’t been added to any.

Returns: (str)

dump_values(with_defaults=True, dict_cls=<class 'dict'>, flat=False)¶

Export values of all items contained in this section to a dictionary.

Items with no values set (and no defaults set if with_defaults=True) will be excluded.

Returns: A dictionary of key-value pairs, where for sections values are dictionaries of their contents.
Return type: dict

is_default¶: True if values of all config items in this section and its subsections have their values equal to defaults or have no value set.

iter_all(recursive=False, path=None, key='path')¶

Parameters

recursive – if True, recurse into sub-sections
path (tuple or string) – optional path to limit iteration over.
key – path (default), str_path, name, None, or a function to calculate the key from (k, v) tuple.

Returns

iterator over (path, obj) pairs of all items and sections contained in this section.

Return type

iterator

iter_items(recursive=False, path=None, key='path')¶

See iter_all() for standard iterator argument descriptions.

Returns

iterator over (key, item) pairs of all items: in this section (and sub-sections if recursive=True).

Return type

iterator

iter_paths(recursive=False, path=None, key='path')¶

See iter_all() for standard iterator argument descriptions.

Returns: iterator over paths of all items and sections contained in this section.
Return type: iterator

iter_sections(recursive=False, path=None, key='path')¶

See iter_all() for standard iterator argument descriptions.

Returns

iterator over (key, section) pairs of all sections: in this section (and sub-sections if recursive=True).

Return type

iterator

load_values(dictionary, as_defaults=False, flat=False)¶

Import config values from a dictionary.

When as_defaults is set to True, the values imported will be set as defaults. This can be used to declare the sections and items of configuration. Values of sections and items in dictionary can be dictionaries as well as instances of Item and Config.

Parameters

dictionary –
as_defaults – if True, the imported values will be set as defaults.

reset()¶: Recursively resets values of all items contained in this section and its subsections to their default values.

section¶: Returns: (Config): section to which this section belongs or None if this hasn’t been added to any section.

`Item`¶

class configmanager.Item(name=<NotSet>, **kwargs)¶

Represents a configuration item – something that has a name, a type, a default value, a user- or environment-specific (custom) value, and other attributes.

Item attribute name should start with a letter.

When instantiating an item, you can pass any attributes, even ones not declared in configmanager code:

>>> threads = Item(default=5, comment='I was here')
>>> threads.comment
'I was here'

If you pass attributes as kwargs, names prefixed with @ symbol will have @ removed and will be treated like normal attributes:

>>> t = Item(**{'@name': 'threads', '@default': 5})
>>> t.name
'threads'
>>> t.default
5
>>> t.type
int

required = False¶: True if config item requires a value. Note that if an item has a default value, marking it as required will have no effect.

envvar = None¶: If set to a string, will use that as the name of the environment variable and will not consult envvar_name. If set to True, will use value of envvar_name as the name of environment variable to check for value override. If set to True and envvar_name is not set, will use auto-generated name based on item’s path in the configuration tree: SECTION1_SECTION2_ITEMNAME.

envvar_name¶: See envvar. Note that you can override this so you don’t have to specify name for each enabled envvar individually.

name = <NotSet>¶: Name of the config item.

type = <_StrType ('str', 'string', 'unicode')>¶: Type of the config item’s value, a callable. Defaults to string.

value¶: The property through which to read and set value of config item.

get(fallback=<NotSet>)¶: Returns config value.

See also

set() and value

set(value)¶: Sets config value.

reset()¶: Resets the value of config item to its default value.

is_default¶

True if the item’s value is its default value or if no value and no default value are set.

If the item is backed by an environment variable, this will be True only if the environment variable is set and is different to the default value of the item.

has_value¶: True if item has a default value or custom value set.

section¶: Config section (an instance of Config) to which the item has been added or None if it hasn’t been added to a section yet.

get_path()¶

Calculate item’s path in configuration tree. Use this sparingly – path is calculated by going up the configuration tree. For a large number of items, it is more efficient to use iterators that return paths as keys.

Path value is stable only once the configuration tree is completely initialised.

validate()¶: Validate item.

`ConfigPersistenceAdapter`¶

class configmanager.ConfigPersistenceAdapter(config, reader_writer)¶

load(source, as_defaults=False)¶

Load configuration values from the specified source.

Parameters

source –
as_defaults (bool) – if True, contents of source will be treated as schema of configuration items.

loads(config_str, as_defaults=False)¶

Load configuration values from the specified source string.

Parameters

config_str –
as_defaults (bool) – if True, contents of source will be treated as schema of configuration items.

dump(destination, with_defaults=False)¶

Write configuration values to the specified destination.

Parameters

destination –
with_defaults (bool) – if True, values of items with no custom values will be included in the output if they have a default value set.

dumps(with_defaults=False)¶

Generate a string representing all the configuration values.

Parameters: with_defaults (bool) – if True, values of items with no custom values will be included in the output if they have a default value set.

store_exists(store)¶: Returns True if configuration can be loaded from the store.

Exceptions ¶

`ConfigError`¶

class configmanager.ConfigError¶: Base class for all exceptions raised by configmanager that user may be able to recover from.

`NotFound`¶

class configmanager.NotFound(name, section=None)¶

Section or item with the requested name or path is not found in the section it is being requested from.

name¶: Name of the section or item which was not found

section¶: Config instance which does not contain the sought name

`RequiredValueMissing`¶

class configmanager.RequiredValueMissing(name, item=None)¶

Value was requested from an item which requires a value, but had no default or custom value set.

name¶: Name of the item

item¶: Item instance

Design ¶

These are internals. Do not use in your application code.

Key, Section, Item, and Item Value Access ¶

get an item or a section: config._get_item(*key), config._get_section(*key), or config._get_item_or_section(key).
get a key (the meaning of this depends on user settings): config._get_by_key(key).