Proposal for the introduction of extensions

.github/ISSUE_TEMPLATE/petab-extensions.md

@@ -0,0 +1,26 @@
+---
+name: PEtab Extension
+about: Suggest a new extension for PEtab core
+title: ''
+labels: file format
+assignees: ''
+
+---
+
+**Name of the Extension**
+Please make sure that the extension name matches the regular expression `^[a-zA-Z_]\w*$`.
+
+**Which problem would you like to address?**
+A clear and concise description of which use case you want to address and, if applicable, why the current specifications do not fulfill your requirements.
+
+**Describe the solution you would like**
+A clear and concise description of the changes you want to propose. Please describe any additional fields / files you would want to add, including allowed inputs and implications.
+
+**Describe why this should not be implemented by changes to PEtab core**
+A clear and concise description in what way the proposed changes introduce features that are orthogonal to the PEtab core specification.
+
+**List the extension library that implements validation checks**
+A link to the website or github repository that accompanies the proposed extension.
+
+**List two toolboxes that support the proposed standard**
+A link to the website or github repository that contains the software that implements support for the standard.

.rtd_pip_reqs.txt

@@ -1,6 +1,37 @@
-sphinx
-sphinxcontrib-napoleon
-sphinx-markdown-tables
-sphinx-rtd-theme
-recommonmark
-m2r
+# Direct dependencies:
+# Sphinx
+# sphinx-markdown-tables
+# sphinx-rtd-theme
+# sphinxcontrib-napoleon
+# m2r2
+alabaster==0.7.12
+Babel==2.9.1
+certifi==2021.10.8
+charset-normalizer==2.0.7
+docutils==0.17.1
+idna==3.3
+imagesize==1.2.0
+Jinja2==3.0.2
+m2r2==0.3.1
+Markdown==3.3.4
+MarkupSafe==2.0.1
+mistune==0.8.4
+packaging==21.0
+pockets==0.9.1
+Pygments==2.10.0
+pyparsing==3.0.3
+pytz==2021.3
+requests==2.26.0
+six==1.16.0
+snowballstemmer==2.1.0
+Sphinx==4.2.0
+sphinx-markdown-tables==0.0.15
+sphinx-rtd-theme==1.0.0
+sphinxcontrib-applehelp==1.0.2
+sphinxcontrib-devhelp==1.0.2
+sphinxcontrib-htmlhelp==2.0.0
+sphinxcontrib-jsmath==1.0.1
+sphinxcontrib-napoleon==0.7
+sphinxcontrib-qthelp==1.0.3
+sphinxcontrib-serializinghtml==1.1.5
+urllib3==1.26.7

CHANGELOG.md

@@ -14,6 +14,13 @@ available at https://github.com/PEtab-dev/libpetab-python/.
 * Update tutorial.rst (#512)
 * Update how-to-cite (Closes #432) (#509)
 
+## 0.2 series
+
+### 0.2.0
+
+* Specify how PEtab functionality can be expanded through extensions.
+* YAML files are now required for the specification of PEtab problems
+
 ## 0.1 series
 
 ### 0.1.14

README.md

@@ -125,6 +125,8 @@ will have to:
 
 1. Create a parameter table.
 
+1. Create a yaml file that lists the model and all of the tables above.
+
 If you are using Python, some handy functions of the
 [PEtab library](https://github.com/PEtab-dev/libpetab-python/) can help
 you with that. This includes also a PEtab validator called `petablint` which

doc/_static/petab_schema.yaml

@@ -0,0 +1,111 @@
+# For syntax see: https://json-schema.org/understanding-json-schema/index.html
+#$schema: "https://json-schema.org/draft/2019-09/meta/core"
+$schema: "http://json-schema.org/draft-06/schema"
+description: PEtab parameter estimation problem config file schema
+
+properties:
+
+  format_version:
+    anyof:
+      - type: string
+        #  (corresponding to PEP 440).
+        pattern: ^([1-9][0-9]*!)?(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*))*((a|b|rc)(0|[1-9][0-9]*))?(\.post(0|[1-9][0-9]*))?(\.dev(0|[1-9][0-9]*))?$
+      - type: integer
+
+    description: Version of the PEtab format
+
+  parameter_file:
+    oneOf:
+    - type: string
+    - type: array
+    description: |
+      File name (absolute or relative) or URL to PEtab parameter table
+      containing parameters of all models listed in `problems`. A single
+      table may be split into multiple files and described as an array here.
+  problems:
+    type: array
+    description: |
+      One or multiple PEtab problems (sets of model, condition, observable
+      and measurement files). If different model and data files are
+      independent, they can be specified as separate PEtab problems, which
+      may allow more efficient handling. Files in one problem cannot refer
+      to models entities or data specified inside another problem.
+    items:
+
+      type: object
+      description: |
+        A set of PEtab model, condition, observable and measurement
+        files and optional visualization files.
+      properties:
+
+        sbml_files:
+          type: array
+          description: List of PEtab SBML files.
+
+          items:
+            type: string
+            description: PEtab SBML file name or URL.
+
+        measurement_files:
+          type: array
+          description: List of PEtab measurement files.
+
+          items:
+            type: string
+            description: PEtab measurement file name or URL.
+
+        condition_files:
+          type: array
+          description: List of PEtab condition files.
+
+          items:
+            type: string
+            description: PEtab condition file name or URL.
+
+        observable_files:
+          type: array
+          description: List of PEtab observable files.
+
+          items:
+            type: string
+            description: PEtab observable file name or URL.
+
+        visualization_files:
+          type: array
+          description: List of PEtab visualization files.
+
+          items:
+            type: string
+            description: PEtab visualization file name or URL.
+
+      required:
+        - sbml_files
+        - observable_files
+        - measurement_files
+        - condition_files
+
+  extensions:
+    type: object
+    description: |
+      PEtab extensions being used.
+    patternProperties:
+      "^[a-zA-Z_]\\w*$":
+
+        type: object
+        description: |
+          Information on a specific extension
+        properties:
+          version:
+            type: string
+            pattern: ^([1-9][0-9]*!)?(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*))*((a|b|rc)(0|[1-9][0-9]*))?(\.post(0|[1-9][0-9]*))?(\.dev(0|[1-9][0-9]*))?$
+
+        required:
+          - version
+      additionalProperties: true
+
+    additionalProperties: false
+
+required:
+  - format_version
+  - parameter_file
+  - problems

doc/conf.py

@@ -37,7 +37,7 @@
 extensions = [
     'sphinx.ext.napoleon',
     'sphinx.ext.autodoc',
-    'recommonmark',
+    'm2r2',
     'sphinx.ext.autosummary',
     'sphinx_markdown_tables',
 ]
@@ -78,7 +78,7 @@
 #
 html_theme = 'sphinx_rtd_theme'
 def setup(app):
-    app.add_stylesheet('custom.css')
+    app.add_css_file('custom.css')
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".

doc/development.rst

@@ -192,6 +192,59 @@ Upon a new release, the PEtab editors ensure that
 * the new version of the specifications is deposited at Zenodo
 * the new release is announced on the PEtab mailing list
 
+PEtab Extensions
+----------------
+
+An elaborate, monolithic format would make it difficult to understand and
+implement support for PEtab, leading to a steep learning curve and discouraging
+support in new toolboxes. To address this issue, the PEtab format is modular and
+permits modifications through extensions that complement the core standard.
+This modular specification evens the learning curve and provides toolbox
+developers with more guidance on which features to implement to maximize
+support for real world applications. Moreover, such modular extensions
+facilitate and promote the use of specialized tools for specific, non-parameter
+estimation tasks such as visualization.
+
+Requirements for new extensions:
+
+* Specifications in PEtab extensions take precedence over PEtab core, i.e., they
+can ease or refine format restrictions imposed by PEtab core.
+* PEtab extensions should extend PEtab core with new orthogonal features or
+tasks, i.e., they should not make trivial changes to PEtab core.
+* PEtab extensions must be named according to ^[a-zA-Z_]\w*$
+* PEtab extensions must be versioned using semantic versioning.
+* PEtab extensions required for interpretation of a problem specification must
+be specified in the PEtab-YAML files
+* There is at least one tool that supports the proposed extension
+* The authors provide a library that implements validation checks for the
+proposed format.
+
+Developers are free to develop any PEtab extension. To become an official
+PEtab extension, it needs to go through the following process.
+
+1. The developers write a proposal describing the motivation and specification
+of the extension, following the respective issue template provided in this
+repository.
+1. The proposal is submitted as an issue in this repository.
+1. The technical specification and documentation of the extension is submitted
+as a pull request in this repository that references the respective issue.
+1. The developers submit a pull request that adds test cases to validate support
+for the extension in the https://github.com/PEtab-dev/petab_test_suite
+repository. The external pull request must also reference the issue containing
+the proposal.
+
+The PEtab editors jointly decide whether an extension meets the requirements
+described here. In case of a positive evaluation, they announce a poll for the
+acceptance as official extension to the PEtab forum. All members of the PEtab
+community are eligible to vote. If at least 50% of the votes are in favor,
+the extension is accepted and the respective pull requests with specifications,
+documentation and test cases are merged.
+
+It is encouraged that extensions are informally discussed with the community
+before initiating the process of becoming an official extension. Such
+discussions can be conducted through the communication channels mentioned
+above.
+
 Versioning of the PEtab format
 ------------------------------
 

doc/documentation_data_format.rst

@@ -58,6 +58,9 @@ and
 - A parameter file specifying optimization parameters and related information
   [TSV]
 
+- A grouping file that lists all of the files and provides additional information
+  including employed extensions [YAML]
+
 - (optional) A simulation file, which has the same format as the measurement
   file, but contains model simulations [TSV]
 
@@ -697,7 +700,7 @@ To link the SBML model, measurement table, condition table, etc. in an
 unambiguous way, we use a `YAML <https://yaml.org/>`_ file.
 
 This file also allows specifying a PEtab version (as the format is not unlikely
-to change in the future).
+to change in the future) and employed PEtab extensions.
 
 Furthermore, this can be used to describe parameter estimation problems
 comprising multiple models (more details below).

doc/md2rst.py

@@ -1,4 +1,4 @@
-import m2r
+import m2r2
 import re
 
 
@@ -25,7 +25,7 @@ def absolute_links(txt):
 
 def md2rst(source: str, target: str):
     txt = absolute_links(read(source))
-    txt = m2r.convert(txt)
+    txt = m2r2.convert(txt, anonymous_references=True)
     with open(target, 'w') as f:
         f.write(txt)
 

doc/tutorial.rst

@@ -19,9 +19,9 @@ details of the model are not relevant for the purpose of this tutorial.
 For more details, we refer to the original publication.
 
 A PEtab problem consists of 1) an SBML model of a biological system, 2)
-condition, observable and measurement definitions, and 3) the
-specification of the parameters. We will show how to generate the
-respective files in the following.
+condition, observable and measurement definitions, 3) the
+specification of the parameters and 4) a configuration file that lists all of
+these files. We will show how to generate the respective files in the following.
 
 1. The model
 ++++++++++++
@@ -324,11 +324,10 @@ https://petab.readthedocs.io/en/latest/documentation_data_format.html#visualizat
 5. YAML file
 ++++++++++++
 
-To group the previously mentioned PEtab files, a YAML file can be used,
-defining which files constitute a PEtab problem. While being optional,
-this makes it easier to import a PEtab problem into tools, and allows
-reusing files for different PEtab problems. This file has the following
-format (``Boehm_JProteomeRes2014.yaml``):
+To group the previously mentioned PEtab files, a YAML file must be used,
+defining which files constitute a PEtab problem. This makes it easier to import
+a PEtab problem into tools, and allows reusing files for different PEtab
+problems. This file has the following format (``Boehm_JProteomeRes2014.yaml``):
 
 .. code-block:: yaml