configuration.options README

Introduction

The configuration.options system provides

• data structures and functions for hierarchical configuration schemata and options
• sources of option values (builtin sources are configuration files, environment variables and commandline options)
• and handling of changes of option values

All of these aspects are extensible via protocols.

STARTED Tutorial

Implementing configuration processing using the configuration.options system involves at least three steps:

1. *Specifying a Schema
2. *Constructing and Populating a Configuration based on the schema
3. *Querying a Configuration

Names

Since options (and the corresponding schema items) are organized into a hierarchy, option names are a sequence of multiple components. The notation COMPONENT₁.COMPONENT₂.… is used when representing names as strings.

“Wildcard names” are names in which one or more components is :wild or :wild-inferiors.

The following functions deal with names:

Form	Result
(parse-name “a.b."c.d"”)	(“a” “b” “c.d”)
(make-name “a.b.c”)	(“a” “b” “c”)
(make-name “a.*.c”)	#<WILDCARD-NAME a.*.c {100D70EF23}>
(make-name “a.**.c”)	#<WILDCARD-NAME a.**.c {100D712923}>
(make-name (“a” “b” “c”))	(“a” “b” “c”)
(name-components #<WILDCARD-NAME a.**.c {100D6E5FE3}>)	(“a” :WILD-INFERIORS “c”)
(name-equal (“a” “b” “c”) (“d” “e” “f”))	NIL
(name-matches #<WILDCARD-NAME d.**.g {100D6EFFC3}> (“d” “e” “f” “g”))	T
(name-equal (“a” “b” “c”) (“d” “e” “f”))	NIL
(merge-names (“a” “b” “c”) (“d” “e” “f”))	(“a” “b” “c” “d” “e” “f”)

Specifying a Schema

A schema can be defined in multiple ways:

• “Manually” via multiple function and method calls
• Declaratively using configuration.options:eval-schema-spec
• Declaratively using configuration.options:define-schema

Since the third method is likely the most commonly used (and uses the same syntax as the second method), it is probably sufficient to only discuss configuration.options:define-schema. Here is an example:

(configuration.options:define-schema *my-schema*
  "Configuration schema for my program."
  ("logging"
   ("appender"                :type    '(member :file :standard-output)
                              :default :standard-output
                              :documentation
                              "Appender to use.")
   ((:wild-inferiors "level") :type    '(member :info :warning :error)
                              :documentation
                              "Package/module/component log level.")))

The above code creates a schema object and stores it in the parameter *my-schema*. The schema consists of two items:

1. logging.appender with allowed values :file and :standard-output and default value :standard-output
2. a “template” option named logging.**.level with allowed values :info, :warning and :error and without a default value

(describe *my-schema*)

#<STANDARD-SCHEMA  (2) (C 0) {1002EE8E03}>

Tree:
  <root>
  │ Configuration schema for my program.
  └─logging
    ├─appender
    │   Type    (MEMBER FILE STANDARD-OUTPUT)
    │   Default :STANDARD-OUTPUT
    │   Appender to use.
    └─**
      └─level
          Type    (MEMBER INFO WARNING ERROR)
          Default <no default>
          Package/module/component log level.

STARTED Types

As demonstrated above, each schema item has an associated type that describes the allowed values of associated options, as types tend to do. In addition to that, types are used to control the parsing and unparsing of option values. For better or worse, schema item types are specified using Common Lisp type specifiers such as (member :info :warning :error) in the above example. The validation, parsing and unparsing behavior for types is implemented using an extensible protocol. This protocol is used by, for example, the configuration.options-and-puri system to add support for additional types.

The builtin types are:

AND	«standard»
BOOLEAN	«standard»
CONFIGURATION.OPTIONS:DIRECTORY-PATHNAME	A pathname syntactically suitable for designating a directory.
CONFIGURATION.OPTIONS:FILE-PATHNAME	A pathname syntactically suitable for designating a file.
INTEGER	«standard»
LIST	«standard»
MEMBER	«standard»
NULL	«standard»
OR	«standard»
PATHNAME	«standard»
STRING	«standard»

TODO Sub-schemata

Constructing and Populating a Configuration

Configurations are created from schemata by first creating an empty configuration object and then populating it with option objects corresponding to schema item objects in the schema:

(defparameter *my-configuration* (configuration.options:make-configuration *my-schema*))

The created configuration is empty:

(describe *my-configuration*)

#<STANDARD-CONFIGURATION  (0) {10076052B3}>

Tree:
  <empty>

There are several ways to create option objects from schema item objects:

1. “Manually”, options can be created using the make-option generic function (this also works if the corresponding to schema items have wild names):

   (let* ((name        "logging.mypackage.myparser.level")
          (schema-item (configuration.options:find-option
                        name *my-schema*
                        :interpret-wildcards? :container)))
     (setf (configuration.options:find-option name *my-configuration*)
           (configuration.options:make-option schema-item name)))
       

   #<STANDARD-OPTION  logging.mypackage.myparser.level: (MEMBER INFO WARNING ERROR) <no value> {100B8A4FE3}>
       

   Note that the schema item named logging.**.level matches the requested name because of its :wild-inferiors name component. Also note that creating an option object does not automatically assign a value to it (even if the schema item specifies a default value).

   The schema item lookup and make-option call in the above code can be done automatically, shortening the example to:

   (configuration.options:find-option
    "logging.mypackage.mylexer.level" *my-configuration*
    :if-does-not-exist :create)
       

   #<STANDARD-OPTION  logging.mypackage.mylexer.level: (MEMBER INFO WARNING ERROR) <no value> {100B8DD5C3}>
       

2. Using a “synchronizer” which integrates data from sources such as configuration files into configuration objects:

   (defun populate-configuration (schema configuration)
     (let ((synchronizer (make-instance 'configuration.options:standard-synchronizer
                                        :target configuration))
           (source       (configuration.options.sources:make-source :defaults)))
       (configuration.options.sources:initialize source schema)
       (configuration.options.sources:process source synchronizer)))
   
   (populate-configuration *my-schema* *my-configuration*)
       

   The above example uses the simple “default values” source which instantiates option objects for all schema items with non-wild names and sets their values to the respective default values (if any) stored in corresponding schema items.

After creating these option objects, the configuration looks like this:

(describe *my-configuration*)

#<STANDARD-CONFIGURATION  (3) {1002F54013}>

Tree:
  <root>
  └─logging
    ├─appender
    │   Type    (MEMBER FILE STANDARD-OUTPUT)
    │   Default :STANDARD-OUTPUT
    │   Value   :STANDARD-OUTPUT
    │   Sources DEFAULT:
    │             :STANDARD-OUTPUT
    │   Appender to use.
    └─mypackage
      ├─mylexer
      │ └─level
      │     Type    (MEMBER INFO WARNING ERROR)
      │     Default <no default>
      │     Value   <no value>
      │     Package/module/component log level.
      └─myparser
        └─level
            Type    (MEMBER INFO WARNING ERROR)
            Default <no default>
            Value   <no value>
            Package/module/component log level.

In a more realistic setting, populating the configuration would be done exclusively using a synchronizer but with a “cascade” of sources ¹ instead of just the “default values” source.

TODO Querying a Configuration

TODO Tracking Changes of Option Values

More on Sources

*Constructing and Populating a Configuration introduced the “source” and “synchronizer” concepts by demonstrating the default values source.

In more realistic settings, a combination of multiple sources like (from highest to lowest priority)

1. Commandline options
2. Environment variables
3. Configuration file(s) and directories
4. Default values

will be used. Cascades of this kind can be constructed by instantiating the :cascade source with appropriate subordinate sources:

(configuration.options.sources:make-source
 :cascade
 :sources '((:commandline)
            (:environment-variables)
            (:config-file-cascade :config-file "my-program.conf"
                                  :syntax      :ini)
            (:defaults)))

#<CASCADE-SOURCE  (4) {100B9D0953}>

A similar cascade of sources is constructed by the :common-cascade source without the need for manually specifying the involved sources.

(configuration.options.sources:make-source
 :common-cascade :basename "my-program" :syntax :ini)

#<COMMON-CASCADE-SOURCE  (4) {100B962693}>

Currently available sources are:

Name	Documentation
:CASCADE	This source organizes a set of sources into a prioritized cascade.
:COMMANDLINE	This source obtains option values from commandline arguments.
:COMMON-CASCADE	This source implements a typical cascade for commandline programs.
:CONFIG-FILE-CASCADE	This source implements a cascade of file-based sources.
:DEFAULTS	This source assigns default values to options.
:DIRECTORY	Collects config files and creates corresponding subordinate sources.
:ENVIRONMENT-VARIABLES	This source reads values of environment variables.
:FILE	This source reads configuration data from files.
:STREAM	This source reads and configuration data from streams.

The :stream (and therefore :file, :config-file-cascade and :common-cascade) source supports the following syntaxes:

Name	Documentation
:INI	Parse textual configuration information in “ini” syntax.
:XML	This syntax allows using some kinds of XML documents as

STARTED Configuration Debugging

With multiple configuration sources such as environment variables and various configuration files, it can sometimes be hard to understand how a particular option got its value (or did not get an expected value). This is true in particular for users who cannot poke around inside the program.

To alleviate this problem, the configuration.options system provides a simple configuration debugging facility aimed at users. This facility can be enabled by calling

(configuration.options.debug:enable-debugging STREAM)

To enable debug output to STREAM unconditionally

(configuration.options.debug:maybe-enable-debugging PREFIX :stream STREAM)

To enable debug output to STREAM if the environment variable PREFIXCONFIG_DEBUG is set

The intention is that a program using this system calls one of these functions before configuration processing starts.

For example, using the schema defined above:

(setf (uiop:getenv "MY_PROGRAM_LOGGING_APPENDER") "file")

(configuration.options.debug:enable-debugging *standard-output*)

(let* ((schema        *my-schema*)
       (configuration (configuration.options:make-configuration schema))
       (synchronizer  (make-instance 'configuration.options:standard-synchronizer
                                     :target configuration))
       (source        (configuration.options.sources:make-source
                       :common-cascade :basename "my-program" :syntax :ini)))
  (configuration.options.sources:initialize source schema)
  (configuration.options.sources:process source synchronizer))

Configuring COMMON-CASCADE-SOURCE with child sources (highest priority first)

  1. Environment variables with prefix mapping
     MY_PROGRAM_LOGGING_APPENDER=file (mapped to logging.appender) -> "file"

  2. Configuring CONFIG-FILE-CASCADE-SOURCE with child sources (highest priority first)

     1. Current directory file "my-program.conf" does not exist

     2. User config file "/home/jmoringe/.config/my-program.conf" does not exist

     3. System-wide config file "/etc/my-program.conf" does not exist

STARTED Integration with the architecture.service-provider System

The architecture.service-provider system allows defining services and providers of these services. The integration described here adds the ability to automatically define a configuration schema for a given service and use a configuration object to choose, instantiate and configure a provider:

This functionionality is provided in the separate configuration.options-and-service-provider system:

(asdf:load-system :configuration.options-and-service-provider)

STARTED Deriving a Schema for A Schema

(service-provider:define-service my-service)

(defclass my-provider () ((a :initarg :a :type string)))
(service-provider:register-provider/class
 'my-service :my-provider :class 'my-provider)

(describe
 (configuration.options.service-provider:service-schema
  (service-provider:find-service 'my-service)))

#<STANDARD-SCHEMA  (1) (C 1) {10083C2913}>

Tree:
  <root>
  │ Configuration options of the MY-SERVICE service.
  ├─provider
  │   Type    (PROVIDER-DESIGNATOR-MEMBER MY-PROVIDER)
  │   Default <no default>
  │   Selects one of the providers of the MY-SERVICE service for
  │   instantiation.
  └─my-provider
    │ Configuration of the MY-PROVIDER provider.
    └─a
        Type    STRING
        Default <no default>

STARTED Creating a Configured Provider

(let* ((schema        (configuration.options.service-provider:service-schema
                       'my-service))
       (configuration (configuration.options:make-configuration schema)))

  (populate-configuration schema configuration)
  (setf (configuration.options:option-value
         (configuration.options:find-option "provider" configuration))
        :my-provider
        (configuration.options:option-value
         (configuration.options:find-option "my-provider.a" configuration))
        "foo")

  (describe (service-provider:make-provider 'my-service configuration)))

#<MY-PROVIDER {1007F19023}>
  [standard-object]

Slots with :INSTANCE allocation:
  A  = "foo"

TODO Tracking Service Changes

TODO Reference

TODO Related Work

• https://github.com/Shinmera/universal-config/
• https://github.com/Shinmera/ubiquitous
• https://docs.python.org/3/library/configparser.html
• cl-config

Settings

Footnotes

¹ See *More on Sources