diff --git a/README.md b/README.md index 1743e39..73983e4 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # vo-models -`vo-models` an open-source project to provide Python models for [IVOA](https://www.ivoa.net/) service protocols. +`vo-models` is an open-source project to provide Python models for [IVOA](https://www.ivoa.net/) service protocols. The project is designed to be used by IVOA members, service implementors, and developers to help facilitate the development of IVOA-compliant services and clients. @@ -8,7 +8,7 @@ The project is designed to be used by IVOA members, service implementors, and de - **Pydantic-xml Models:** The project includes Python models for IVOA protocols, using [pydantic-xml](https://github.com/dapper91/pydantic-xml). Based on [Pydantic](https://docs.pydantic.dev/latest/), these models describe transactions for an IVOA protocol, such as UWS, and feature automatic validation, parsing and serialization of XML data for use with Python clients and web frameworks. -- **Expandability:** The project is designed with future expansion in mind. Plans include extending the schema and models to cover other IVOA standards. +- **Expandability:** The project is designed with future expansion in mind. Plans include extending the schema and models to cover other IVOA standards and future versions of existing standards where possible. ## Protocols @@ -17,6 +17,15 @@ The following IVOA protocols are currently supported: - **UWS (Universal Worker Service) version 1.1** - **VOSI (IVOA Support Interfaces) version 1.1** - VOSI Availability + - VOSI Tables +- **VODataService version 1.2 (limited)** + - DataType + - FKColumn + - ForeignKey + - Table + - TableParam + - TableSchema + - TableSet You can read more about using these models in our documentation: https://vo-models.readthedocs.io/ @@ -48,12 +57,6 @@ For active development, install the project in development mode: pip install -e .[dev,test] ``` -### Pydantic-XML Models - -Python models using [pydantic-xml](https://github.com/dapper91/pydantic-xml), a library based on [Pydantic](https://docs.pydantic.dev/latest/), are provided in the `vo/models/xml` directory. - -These models can be used to parse and validate XML data into Python objects, as well as serialize Python objects into XML data. These models can be used with any Python web framework, but are particularly useful when used libraries that leverage the power of Pydantic, such as [FastAPI](https://fastapi.tiangolo.com/). - ### Contributing Contributions to the project are more than welcome. Collaboration and discussion with other IVOA members, service implementors, and developers is what started this project, and is what makes the IVOA great. diff --git a/docs/source/conf.py b/docs/source/conf.py index 0e5acb8..32e895b 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -33,6 +33,7 @@ "sphinx.ext.viewcode", "sphinx.ext.autodoc", "sphinx.ext.napoleon", + "sphinx_autodoc_typehints", ] autodoc_default_options = { diff --git a/docs/source/pages/api/index.rst b/docs/source/pages/api/index.rst index e1a782a..1d8493b 100644 --- a/docs/source/pages/api/index.rst +++ b/docs/source/pages/api/index.rst @@ -10,4 +10,5 @@ This section contains documentation on the package's modules and classes. uws_api vosi_api - voresource_api \ No newline at end of file + voresource_api + vodataservice_api \ No newline at end of file diff --git a/docs/source/pages/api/vodataservice_api.rst b/docs/source/pages/api/vodataservice_api.rst new file mode 100644 index 0000000..87adfdd --- /dev/null +++ b/docs/source/pages/api/vodataservice_api.rst @@ -0,0 +1,12 @@ +.. _vodataservice_api: + +VODataservice API +----------------- + +Models +^^^^^^ + +.. automodule:: vo_models.vodataservice.models + :members: + :no-inherited-members: + :exclude-members: model_config, model_fields, Job \ No newline at end of file diff --git a/docs/source/pages/api/vosi_api.rst b/docs/source/pages/api/vosi_api.rst index 19c52b5..2b8d574 100644 --- a/docs/source/pages/api/vosi_api.rst +++ b/docs/source/pages/api/vosi_api.rst @@ -7,6 +7,14 @@ Availability ^^^^^^^^^^^^ .. automodule:: vo_models.vosi.availability.models + :members: + :no-inherited-members: + :exclude-members: model_config, model_fields, + +Tables +^^^^^^ + +.. automodule:: vo_models.vosi.tables.models :members: :no-inherited-members: :exclude-members: model_config, model_fields, \ No newline at end of file diff --git a/docs/source/pages/protocols/index.rst b/docs/source/pages/protocols/index.rst index 2cbe31e..5811eea 100644 --- a/docs/source/pages/protocols/index.rst +++ b/docs/source/pages/protocols/index.rst @@ -9,4 +9,5 @@ The following IVOA protocols are currently supported: :maxdepth: 3 uws - vosi \ No newline at end of file + vosi + vodataservice \ No newline at end of file diff --git a/docs/source/pages/protocols/vodataservice.rst b/docs/source/pages/protocols/vodataservice.rst new file mode 100644 index 0000000..3e36523 --- /dev/null +++ b/docs/source/pages/protocols/vodataservice.rst @@ -0,0 +1,173 @@ +.. _vodataservice: + +VODataService +------------- + +VODataService is an IVOA XML encoding standard for data collections and services that access them. It is an extension of the VOResource standard. + +`vo-models` currently supports the following VODataService v1.2 elements: + +Models +^^^^^^ + +FKColumn +******** + +Represents a single foreign key column. + +.. grid:: 2 + :gutter: 2 + + .. grid-item-card:: Model + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: python + :start-after: FKColumn-model-start + :end-before: FKColumn-model-end + + .. grid-item-card:: XML Output + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: xml + :lines: 2- + :start-after: FKColumn-xml-start + :end-before: FKColumn-xml-end + +ForeignKey +********** + +Represents one or more foreign key columns. + +.. grid:: 2 + :gutter: 2 + + .. grid-item-card:: Model + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: python + :start-after: ForeignKey-model-start + :end-before: ForeignKey-model-end + + .. grid-item-card:: XML Output + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: xml + :lines: 2- + :start-after: ForeignKey-xml-start + :end-before: ForeignKey-xml-end + +DataType +******** + +A simple element containing a column's datatype. + +.. grid:: 2 + :gutter: 2 + + .. grid-item-card:: Model + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: python + :start-after: DataType-model-start + :end-before: DataType-model-end + + .. grid-item-card:: XML Output + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: xml + :lines: 2- + :start-after: DataType-xml-start + :end-before: DataType-xml-end + +TableParam +********** + +A description of a table parameter (a column within the table) with a fixed datatype. + +.. grid:: 2 + :gutter: 2 + + .. grid-item-card:: Model + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: python + :start-after: TableParam-model-start + :end-before: TableParam-model-end + + .. grid-item-card:: XML Output + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: xml + :lines: 2- + :start-after: TableParam-xml-start + :end-before: TableParam-xml-end + +Table +***** + +A single table element. + +.. grid:: 2 + :gutter: 2 + + .. grid-item-card:: Model + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: python + :start-after: Table-model-start + :end-before: Table-model-end + + .. grid-item-card:: XML Output + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: xml + :lines: 2- + :start-after: Table-xml-start + :end-before: Table-xml-end + +TableSchema +*********** + +Represents a description of a logically related group of tables. + +.. grid:: 2 + :gutter: 2 + + .. grid-item-card:: Model + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: python + :start-after: TableSchema-model-start + :end-before: TableSchema-model-end + + .. grid-item-card:: XML Output + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: xml + :lines: 2- + :start-after: TableSchema-xml-start + :end-before: TableSchema-xml-end + + +TableSet +******** + +Represents a collection of tables that are part of a single resource. + +.. grid:: 2 + :gutter: 2 + + .. grid-item-card:: Model + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: python + :start-after: TableSet-model-start + :end-before: TableSet-model-end + + .. grid-item-card:: XML Output + + .. literalinclude:: ../../../../examples/snippets/vodataservice/vodataservice.py + :language: xml + :lines: 2- + :start-after: TableSet-xml-start + :end-before: TableSet-xml-end \ No newline at end of file diff --git a/docs/source/pages/protocols/vosi.rst b/docs/source/pages/protocols/vosi.rst index 0e0d402..abe58c8 100644 --- a/docs/source/pages/protocols/vosi.rst +++ b/docs/source/pages/protocols/vosi.rst @@ -6,7 +6,7 @@ VOSI (VO Support Interface) ``vo-models`` supports the following VOSI v1.0 protocols: Availability -***************** +^^^^^^^^^^^^ The Availability model is used to represent the response given by a UWS service to a ``GET /availability`` request. @@ -27,4 +27,58 @@ The Availability model is used to represent the response given by a UWS service :language: xml :lines: 2- :start-after: xml-start - :end-before: xml-end \ No newline at end of file + :end-before: xml-end + +Tables +^^^^^^ + +VOSITable +********* + +For requests for a single table from the ``GET /tables/{table_name}`` endpoint, you can use the ``Table`` model. + +.. note:: This model is functionally identical to the :ref:`pages/protocols/vodataservice:table` element, specifically namespaced under VOSI. + +.. grid:: 2 + :gutter: 2 + + .. grid-item-card:: Model + + .. literalinclude:: ../../../../examples/snippets/vosi/tables.py + :language: python + :start-after: table-model-start + :end-before: table-model-end + + .. grid-item-card:: XML Output + + .. literalinclude:: ../../../../examples/snippets/vosi/tables.py + :language: xml + :lines: 2- + :start-after: table-xml-start + :end-before: table-xml-end + + +VOSITableSet +************ + +For requests to the ``GET /tables`` endpoint, you can use the ``TableSet`` model to represent table schemas, their child tables, and columns. + +.. note:: This model is functionally identical to the :ref:`pages/protocols/vodataservice:tableset` element, specifically namespaced under VOSI. + +.. grid:: 2 + :gutter: 2 + + .. grid-item-card:: Model + + .. literalinclude:: ../../../../examples/snippets/vosi/tables.py + :language: python + :start-after: tableset-model-start + :end-before: tableset-model-end + + .. grid-item-card:: XML Output + + .. literalinclude:: ../../../../examples/snippets/vosi/tables.py + :language: xml + :lines: 2- + :start-after: tableset-xml-start + :end-before: tableset-xml-end \ No newline at end of file diff --git a/docs/source/pages/quickstart.rst b/docs/source/pages/quickstart.rst index 4263a71..926016a 100644 --- a/docs/source/pages/quickstart.rst +++ b/docs/source/pages/quickstart.rst @@ -6,7 +6,7 @@ Quickstart Basic Usage ----------- -Working with ``vo-models`` classes is easy. They can be created and modified like any Pydantic model. +``vo-models`` objects can be created and modified like any Python class or Pydantic model. The following example creates a UWS :ref:`pages/protocols/uws:shortjobdescription` model using keyword arguments and updates the phase: diff --git a/examples/snippets/vodataservice/vodataservice.py b/examples/snippets/vodataservice/vodataservice.py new file mode 100644 index 0000000..46872b7 --- /dev/null +++ b/examples/snippets/vodataservice/vodataservice.py @@ -0,0 +1,209 @@ +from vo_models.vodataservice.models import ( + DataType, + FKColumn, + ForeignKey, + Table, + TableParam, + TableSchema, + TableSet, +) + +# [FKColumn-model-start] +fk_column = FKColumn( + from_column="from_column", + target_column="target_column", +) +# [FKColumn-model-end] + +# [FKColumn-xml-start] +fk_column_xml = """ + + from_column + target_column + +""" # [FKColumn-xml-end] + +# [ForeignKey-model-start] +foreign_key = ForeignKey( + target_table="target_table", + fk_column=fk_column, + description="description", + utype="utype", +) +# [ForeignKey-model-end] + +# [ForeignKey-xml-start] +foreign_key_xml = """ + + target_table + + from_column + target_column + + description + utype + +""" # [ForeignKey-xml-end] + +# [DataType-model-start] +data_type = DataType( + type="string", + arraysize="32*", + value="value", +) +# [DataType-model-end] + +# [DataType-xml-start] +data_type_xml = """ +value +""" +# [DataType-xml-end] + +# [TableParam-model-start] +table_param = TableParam( + column_name="example_column", + description="Example column description", + datatype=data_type, + flag=["std", "indexed"], +) +# [TableParam-model-end] + +# [TableParam-xml-start] +table_param_xml = """ + + example_column + Example column description + + + + + + value + + std + indexed + +""" # [TableParam-xml-end] + +# [Table-model-start] +table = Table( + table_type="table", + table_name="example_table", + description="Example table description", + column=table_param, + foreign_key=foreign_key, +) +# [Table-model-end] + +# [Table-xml-start] +table_xml = """ + + example_table + Example table description + + example_column + Example column description + + value + + std + indexed + + + target_table + + from_column + target_column + + description + utype + +
+""" # [Table-xml-end] + +# [TableSchema-model-start] +table_schema = TableSchema( + name="example_schema", + title="Example schema title", + description="Example schema description", + table=table, +) +# [TableSchema-model-end] + +# [TableSchema-xml-start] +table_schema_xml = """ + + default + Example schema title + Example schema description + + example_table + Example table description + + example_column + Example column description + value + std + indexed + + + target_table + + from_column + target_column + + description + utype + +
+ +""" # [TableSchema-xml-end] + +# [TableSet-model-start] +table_set = TableSet( + tableset_schema=table_schema, +) +# [TableSet-model-end] + +# [TableSet-xml-start] +table_set_xml = """ + + + default + Example schema title + Example schema description + + example_table + Example table description + + example_column + Example column description + value + std + indexed + + + target_table + + from_column + target_column + + description + utype + +
+ + +""" # [TableSet-xml-end] diff --git a/examples/snippets/vosi/tables.py b/examples/snippets/vosi/tables.py new file mode 100644 index 0000000..372fdeb --- /dev/null +++ b/examples/snippets/vosi/tables.py @@ -0,0 +1,104 @@ +from vo_models.vodataservice import DataType, Table, TableParam, TableSchema +from vo_models.vosi.tables import VOSITable, VOSITableSet + +# [table-model-start] +test_element = VOSITable( + table_type="table", + table_name="tap_schema.schemas", + title=None, + description="description of schemas in this dataset", + utype=None, + nrows=None, + column=[ + TableParam( + column_name="schema_name", + description="Fully qualified schema name", + unit=None, + ucd=None, + datatype=DataType(type_attr="vs:VOTableType", arraysize="*", value="char"), + flag=["std"], + ), + TableParam( + column_name="description", + description="Brief description of the schema", + unit=None, + ucd=None, + datatype=DataType(type_attr="vs:VOTableType", arraysize="*", value="char"), + flag=["std"], + ), + ], + foreign_key=None, +) +# [table-model-end] + +# [table-xml-start] +table_xml = """ + + tap_schema.schemas + description of schemas in this dataset + + schema_name + Fully qualified schema name + char + std + + + description + Brief description of the schema + char + std + + +""" # [table-xml-end] + +# [tableset-model-start] +test_element = VOSITableSet( + tableset_schema=[ + TableSchema( + schema_name="tap_schema", + title=None, + description="schema information for TAP services", + table=[ + Table( + table_type="table", + table_name="tap_schema.schemas", + title=None, + description="description of schemas in this dataset", + utype=None, + nrows=None, + column=None, + foreign_key=None, + ), + Table( + table_type="table", + table_name="tap_schema.tables", + title=None, + description="description of tables in this dataset", + utype=None, + nrows=None, + column=None, + foreign_key=None, + ), + ], + ), + ] +) +# [tableset-model-end] + +# [tableset-xml-start] +tableset_xml = """ + + + tap_schema + schema information for TAP services + + tap_schema.schemas + description of schemas in this dataset +
+ + tap_schema.tables + description of tables in this dataset +
+ + +""" # [tableset-xml-end] diff --git a/pyproject.toml b/pyproject.toml index 42a1236..46a80d9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "vo-models" -version = "0.2.2" +version = "0.3.0" authors = [ {name = "Joshua Fraustro", email="jfraustro@stsci.edu"}, {name = "MAST Archive Developers", email="archive@stsci.edu"} @@ -26,10 +26,8 @@ classifiers = [ ] keywords = [ "ivoa", - "uws", "pydantic", "pydantic-xml", - "openapi", ] [project.optional-dependencies] diff --git a/requirements.txt b/requirements.txt index c0fd4cf..600e5e1 100644 --- a/requirements.txt +++ b/requirements.txt @@ -213,9 +213,9 @@ pydantic-core==2.14.5 \ --hash=sha256:fb2ed8b3fe4bf4506d6dab3b93b83bbc22237e230cba03866d561c3577517d18 \ --hash=sha256:fe0a5a1025eb797752136ac8b4fa21aa891e3d74fd340f864ff982d649691867 # via pydantic -pydantic-xml[lxml]==2.6.0 \ - --hash=sha256:2a7019dcfcf68b0136378e72efbe0747ec21f7ba16bfcaf6ab5a47dd1f2d68e9 \ - --hash=sha256:8a88d0e8af20406eca06af20e57b1d2ccb7d1630c00f927e1f839629c0decb1e +pydantic-xml[lxml]==2.7.0 \ + --hash=sha256:6bb16cee3166482365b7def95416285d56b325041d995fbfa58979a06ff1d51c \ + --hash=sha256:e336bca6e01984f5995ea197394bedff415a7d8449d0910a9647a1216990069c # via # pydantic-xml # vo-models (pyproject.toml) diff --git a/tests/vodataservice/__init__.py b/tests/vodataservice/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/vodataservice/vodataservice_models_test.py b/tests/vodataservice/vodataservice_models_test.py new file mode 100644 index 0000000..10a8a83 --- /dev/null +++ b/tests/vodataservice/vodataservice_models_test.py @@ -0,0 +1,371 @@ +"""Tests for VODataService models + +# TODO: This is an incomplete spec, covering only elements needed for VOSITables +# https://github.com/spacetelescope/vo-models/issues/17 +""" + +from unittest import TestCase +from xml.etree.ElementTree import canonicalize + +from vo_models.vodataservice import ( + DataType, + FKColumn, + ForeignKey, + Table, + TableParam, + TableSchema, + TableSet, +) + + +class TestFKColumn(TestCase): + """Test the FKColumn element model""" + + test_xml = """ + + from_column + target_column + + """ + + test_element = FKColumn( + from_column="from_column", + target_column="target_column", + ) + + def test_read_from_xml(self): + """Test reading FKColumn from XML.""" + fk_column = FKColumn.from_xml(self.test_xml) + self.assertEqual(fk_column.from_column, "from_column") + self.assertEqual(fk_column.target_column, "target_column") + + def test_write_to_xml(self): + """Test writing FKColumn to XML.""" + fk_column_xml = self.test_element.to_xml(skip_empty=True) + self.assertEqual( + canonicalize(fk_column_xml, strip_text=True), + canonicalize(self.test_xml, strip_text=True), + ) + + +class TestForeignKey(TestCase): + """Test the ForeignKey element model""" + + test_xml = """ + + target_table + + from_column + target_column + + + """ + + test_element = ForeignKey( + target_table="target_table", + fk_column=[ + FKColumn( + from_column="from_column", + target_column="target_column", + ) + ], + ) + + def test_read_from_xml(self): + """Test reading ForeignKey from XML.""" + foreign_key = ForeignKey.from_xml(self.test_xml) + self.assertEqual(foreign_key.target_table, "target_table") + self.assertEqual(len(foreign_key.fk_column), 1) + self.assertEqual(foreign_key.fk_column[0].from_column, "from_column") + self.assertEqual(foreign_key.fk_column[0].target_column, "target_column") + + def test_write_to_xml(self): + """Test writing ForeignKey to XML.""" + foreign_key_xml = self.test_element.to_xml(skip_empty=True) + self.assertEqual( + canonicalize(foreign_key_xml, strip_text=True), + canonicalize(self.test_xml, strip_text=True), + ) + + +class TestDataType(TestCase): + """Test the DataType element model""" + + test_xml = """ + string + """ + + test_element = DataType( + type="vs:VOTableType", + arraysize="*", + value="string", + ) + + def test_read_from_xml(self): + """Test reading DataType from XML.""" + data_type = DataType.from_xml(self.test_xml) + self.assertEqual(data_type.type, "vs:VOTableType") + self.assertEqual(data_type.arraysize, "*") + self.assertEqual(data_type.value, "string") + + def test_write_to_xml(self): + """Test writing DataType to XML.""" + data_type_xml = self.test_element.to_xml(skip_empty=True) + self.assertEqual( + canonicalize(data_type_xml, strip_text=True), + canonicalize(self.test_xml, strip_text=True), + ) + + +class TestTableParam(TestCase): + """Test the TableParam element model""" + + test_xml = """ + + name + description + unit + ucd + caom2:Artifact.productType + string + flag + """ + + test_element = TableParam( + column_name="name", + description="description", + unit="unit", + ucd="ucd", + utype="caom2:Artifact.productType", + datatype=DataType(type="vs:VOTableType", arraysize="*", value="string"), + flag=["flag"], + ) + + def test_read_from_xml(self): + """Test reading TableParam from XML.""" + table_param = TableParam.from_xml(self.test_xml) + self.assertEqual(table_param.column_name, "name") + self.assertEqual(table_param.description, "description") + self.assertEqual(table_param.unit, "unit") + self.assertEqual(table_param.ucd, "ucd") + self.assertEqual(table_param.utype, "caom2:Artifact.productType") + self.assertEqual(table_param.datatype.type, "vs:VOTableType") + self.assertEqual(table_param.datatype.arraysize, "*") + self.assertEqual(table_param.datatype.value, "string") + self.assertEqual(table_param.flag, ["flag"]) + + def test_write_to_xml(self): + """Test writing TableParam to XML.""" + table_param_xml = self.test_element.to_xml(skip_empty=True) + self.assertEqual( + canonicalize(table_param_xml, strip_text=True), + canonicalize(self.test_xml, strip_text=True), + ) + + +class TestTableElement(TestCase): + """Test the Table element model""" + + test_xml = """ + + tap_schema.schemas + description of schemas in this dataset + + schema_name + Fully qualified schema name + char + std + +
+ """ + + test_element = Table( + table_name="tap_schema.schemas", + table_type="table", + description="description of schemas in this dataset", + column=[ + TableParam( + column_name="schema_name", + description="Fully qualified schema name", + datatype=DataType(type="vs:VOTableType", arraysize="*", value="char"), + flag=["std"], + ) + ], + ) + + def test_read_from_xml(self): + """Test reading Table from XML.""" + table = Table.from_xml(self.test_xml) + self.assertEqual(table.table_name, "tap_schema.schemas") + self.assertEqual(table.table_type, "table") + self.assertEqual(table.description, "description of schemas in this dataset") + self.assertEqual(len(table.column), 1) + self.assertEqual(table.column[0].column_name, "schema_name") + self.assertEqual(table.column[0].description, "Fully qualified schema name") + self.assertEqual(table.column[0].datatype.type, "vs:VOTableType") + self.assertEqual(table.column[0].datatype.arraysize, "*") + self.assertEqual(table.column[0].datatype.value, "char") + self.assertEqual(table.column[0].flag, ["std"]) + + def test_write_to_xml(self): + """Test writing Table to XML.""" + table_xml = self.test_element.to_xml(skip_empty=True) + self.assertEqual( + canonicalize(table_xml, strip_text=True), + canonicalize(self.test_xml, strip_text=True), + ) + + +class TestSchemaElement(TestCase): + """Test the TableSchema element model""" + + test_xml = """ + + tap_schema + schema information for TAP services + + tap_schema.schemas + description of schemas in this dataset +
+ + tap_schema.tables + description of tables in this dataset +
+ + """ + + test_element = TableSchema( + schema_name="tap_schema", + description="schema information for TAP services", + table=[ + Table( + table_name="tap_schema.schemas", + table_type="table", + description="description of schemas in this dataset", + ), + Table( + table_name="tap_schema.tables", + table_type="table", + description="description of tables in this dataset", + ), + ], + ) + + def test_read_from_xml(self): + """Test reading TableSchema from XML.""" + table_schema = TableSchema.from_xml(self.test_xml) + self.assertEqual(table_schema.schema_name, "tap_schema") + self.assertEqual(table_schema.description, "schema information for TAP services") + self.assertEqual(len(table_schema.table), 2) + self.assertEqual(table_schema.table[0].table_name, "tap_schema.schemas") + self.assertEqual(table_schema.table[0].table_type, "table") + self.assertEqual(table_schema.table[0].description, "description of schemas in this dataset") + self.assertEqual(table_schema.table[1].table_name, "tap_schema.tables") + self.assertEqual(table_schema.table[1].table_type, "table") + self.assertEqual(table_schema.table[1].description, "description of tables in this dataset") + + def test_write_to_xml(self): + """Test writing TableSchema to XML.""" + table_schema_xml = self.test_element.to_xml(skip_empty=True) + self.assertEqual( + canonicalize(table_schema_xml, strip_text=True), + canonicalize(self.test_xml, strip_text=True), + ) + + +class TestTableSetElement(TestCase): + """Test the TableSet element model""" + + test_xml = """ + + + tap_schema + schema information for TAP services + + tap_schema.schemas + description of schemas in this dataset +
+ + tap_schema.tables + description of tables in this dataset +
+ + + dbo + ArchiveCatalog Infrastructure, version 1.0 + + dbo.detailedCatalog +
+ + dbo.SumMagAper2Cat +
+ + """ + + test_element = TableSet( + tableset_schema=[ + TableSchema( + schema_name="tap_schema", + description="schema information for TAP services", + table=[ + Table( + table_name="tap_schema.schemas", + table_type="table", + description="description of schemas in this dataset", + ), + Table( + table_name="tap_schema.tables", + table_type="table", + description="description of tables in this dataset", + ), + ], + ), + TableSchema( + schema_name="dbo", + description="ArchiveCatalog Infrastructure, version 1.0", + table=[ + Table( + table_name="dbo.detailedCatalog", + table_type="table", + ), + Table( + table_name="dbo.SumMagAper2Cat", + table_type="table", + ), + ], + ), + ], + ) + + def test_read_from_xml(self): + """Test reading TableSet from XML.""" + tableset = TableSet.from_xml(self.test_xml) + self.assertEqual(len(tableset.tableset_schema), 2) + self.assertEqual(tableset.tableset_schema[0].schema_name, "tap_schema") + self.assertEqual(tableset.tableset_schema[0].description, "schema information for TAP services") + self.assertEqual(len(tableset.tableset_schema[0].table), 2) + self.assertEqual(tableset.tableset_schema[0].table[0].table_name, "tap_schema.schemas") + self.assertEqual(tableset.tableset_schema[0].table[0].table_type, "table") + self.assertEqual(tableset.tableset_schema[0].table[0].description, "description of schemas in this dataset") + self.assertEqual(tableset.tableset_schema[0].table[1].table_name, "tap_schema.tables") + self.assertEqual(tableset.tableset_schema[0].table[1].table_type, "table") + self.assertEqual(tableset.tableset_schema[0].table[1].description, "description of tables in this dataset") + self.assertEqual(tableset.tableset_schema[1].schema_name, "dbo") + self.assertEqual(tableset.tableset_schema[1].description, "ArchiveCatalog Infrastructure, version 1.0") + self.assertEqual(len(tableset.tableset_schema[1].table), 2) + self.assertEqual(tableset.tableset_schema[1].table[0].table_name, "dbo.detailedCatalog") + self.assertEqual(tableset.tableset_schema[1].table[0].table_type, "table") + self.assertEqual(tableset.tableset_schema[1].table[1].table_name, "dbo.SumMagAper2Cat") + self.assertEqual(tableset.tableset_schema[1].table[1].table_type, "table") + + def test_write_to_xml(self): + """Test writing TableSet to XML.""" + tableset_xml = self.test_element.to_xml(skip_empty=True) + self.assertEqual( + canonicalize(tableset_xml, strip_text=True), + canonicalize(self.test_xml, strip_text=True), + ) diff --git a/tests/vosi/VOSITables-v1.1.xsd b/tests/vosi/VOSITables-v1.1.xsd new file mode 100644 index 0000000..47388ff --- /dev/null +++ b/tests/vosi/VOSITables-v1.1.xsd @@ -0,0 +1,46 @@ + + + + + + A schema for formatting table metadata as returned by a + tables resource, defined by the IVOA Support Interfaces + specification (VOSI). + See http://www.ivoa.net/Documents/latest/VOSI.html. + + + + + + + + + + A description of the table metadata supported by the + service associated with a VOSI-enabled resource. + + + + + + + + + A description of a single table supported by the + service associated with a VOSI-enabled resource. + + + + diff --git a/tests/vosi/tables_test.py b/tests/vosi/tables_test.py new file mode 100644 index 0000000..b7ccc9c --- /dev/null +++ b/tests/vosi/tables_test.py @@ -0,0 +1,193 @@ +"""Tests for VOSI-Tables specific pydantic-xml models""" + +# We're only parsing a locally controlled XSD file +from unittest import TestCase +from xml.etree.ElementTree import canonicalize + +from lxml import etree # nosec B410 + +from vo_models.vodataservice.models import DataType, Table, TableParam, TableSchema +from vo_models.vosi.tables import VOSITable, VOSITableSet + +with open("tests/vosi/VOSITables-v1.1.xsd", "r") as schema_file: + vosi_tables_schema = etree.XMLSchema(file=schema_file) + +VOSIT_TABLES_HEADER = """xmlns:vosi='http://www.ivoa.net/xml/VOSITables/v1.0' +xmlns:vr='http://www.ivoa.net/xml/VOResource/v1.0' +xmlns:vs='http://www.ivoa.net/xml/VODataService/v1.1' +xmlns:xsd='http://www.w3.org/2001/XMLSchema' +xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' +""" + + +class TestVOSITableElement(TestCase): + """Test the VOSI Table element""" + + test_xml = ( + f"" + "tap_schema.schemas" + "description of schemas in this dataset" + "" + "schema_name" + "Fully qualified schema name" + "char" + "std" + "" + "" + "description" + "Brief description of the schema" + "char" + "std" + "" + "" + ) + + test_element = VOSITable( + table_type="table", + table_name="tap_schema.schemas", + title=None, + description="description of schemas in this dataset", + utype=None, + nrows=None, + column=[ + TableParam( + column_name="schema_name", + description="Fully qualified schema name", + unit=None, + ucd=None, + datatype=DataType(type_attr="vs:VOTableType", arraysize="*", value="char"), + flag=["std"], + ), + TableParam( + column_name="description", + description="Brief description of the schema", + unit=None, + ucd=None, + datatype=DataType(type_attr="vs:VOTableType", arraysize="*", value="char"), + flag=["std"], + ), + ], + foreign_key=None, + ) + + def test_read_from_xml(self): + """Test reading a Table from XML""" + vosi_table = VOSITable.from_xml(self.test_xml) + self.assertEqual(vosi_table.table_name, "tap_schema.schemas") + self.assertEqual(vosi_table.table_type, "table") + self.assertEqual(vosi_table.description, "description of schemas in this dataset") + self.assertEqual(len(vosi_table.column), 2) + self.assertEqual(vosi_table.column[0].column_name, "schema_name") + self.assertEqual(vosi_table.column[0].description, "Fully qualified schema name") + self.assertEqual(vosi_table.column[0].datatype.type, "vs:VOTableType") + self.assertEqual(vosi_table.column[0].datatype.arraysize, "*") + self.assertEqual(vosi_table.column[0].datatype.value, "char") + self.assertEqual(vosi_table.column[0].flag, ["std"]) + + def test_write_to_xml(self): + """Test writing a Table to XML""" + + tables_xml = self.test_element.to_xml() + self.assertEqual( + canonicalize(tables_xml, strip_text=True), + canonicalize(self.test_xml, strip_text=True), + ) + + def test_validate(self): + """Test validating a Table against the VOSI Tables schema""" + table_xml = etree.fromstring(self.test_element.to_xml(skip_empty=True, encoding=str)) # nosec B320 + vosi_tables_schema.assertValid(table_xml) + + +class TestVOSITableSet(TestCase): + """Test the TableSet element model as specified in VOSI Tables""" + + test_xml = ( + f"" + "" + "tap_schema" + "schema information for TAP services" + "" + "tap_schema.schemas" + "description of schemas in this dataset" + "
" + "" + "tap_schema.tables" + "description of tables in this dataset" + "
" + "" + "" + ) + + test_element = VOSITableSet( + tableset_schema=[ + TableSchema( + schema_name="tap_schema", + title=None, + description="schema information for TAP services", + table=[ + Table( + table_type="table", + table_name="tap_schema.schemas", + title=None, + description="description of schemas in this dataset", + utype=None, + nrows=None, + column=None, + foreign_key=None, + ), + Table( + table_type="table", + table_name="tap_schema.tables", + title=None, + description="description of tables in this dataset", + utype=None, + nrows=None, + column=None, + foreign_key=None, + ), + ], + ), + ] + ) + + def test_vositableset_ns(self): + """Test that the TableSet is specifically namespaced to VOSI""" + vosi_element = VOSITableSet.from_xml(self.test_xml) + self.assertEqual(vosi_element.__xml_ns__, "vosi") + + vosi_xml = vosi_element.to_xml(skip_empty=True, encoding=str) + self.assertIn(" None: + # If what we were given is of the form: + # {'target_table': 'target_table', 'from_column': 'from_column', 'target_column': 'target_column'} + # and we don't have an fk_column, make one + if not data.get("fk_column", None): + if data.get("from_column") and data.get("target_column"): + data["fk_column"] = [ + FKColumn( + from_column=data["from_column"], + target_column=data["target_column"], + ) + ] + super().__init__(**data) + + @field_validator("fk_column", mode="before") + def validate_fk_column(cls, value): + """If we have a single fk_column, make it a list""" + if not isinstance(value, list): + value = [value] + return value + + +class DataType(BaseXmlModel, tag="dataType", nsmap={"xsi": "http://www.w3.org/2001/XMLSchema-instance"}): + """A type of data contained in the column. + + Parameters: + type: + (attr) - A type of data contained in the parameter. + arraysize: + (attr) - The shape of the array that constitutes the value. + value: + (text) - The name of the data type (e.g. 'char', 'int', 'double'). + """ + + type: Optional[str] = attr(name="type", ns="xsi", default="vs:VOTableType") + arraysize: Optional[str] = attr(name="arraysize", default=None) + value: str + + +class TableParam(BaseXmlModel, ns="", tag="column"): + """A description of a table column. + + Parameters: + column_name: + (elem) - The name of the parameter or column. + description: + (elem) - A free-text description of a parameter's or column's contents. + unit: + (elem) - The unit associated with the values in the parameter or column. + ucd: + (elem) - The name of a unified content descriptor that describes the scientific content of the parameter. + utype: + (elem) - An identifier for a concept in a data model that the data in this schema represent. + xtype: + (elem) - The xtype of the column. + datatype: + (elem) - A type of data contained in the column + flag: + (elem) -A keyword representing traits of the column. Recognized values include + “indexed”, “primary”, and “nullable”. + """ + + column_name: str = element(tag="name") + description: Optional[str] = element(tag="description", default=None) + unit: Optional[str] = element(tag="unit", default=None) + ucd: Optional[str] = element(tag="ucd", default=None) + utype: Optional[str] = element(tag="utype", default=None) + xtype: Optional[str] = element(tag="xtype", default=None) + datatype: Optional[DataType] = element(tag="dataType", default=None) + flag: Optional[list[str]] = element(tag="flag", default=None) + + def __init__(__pydantic_self__, **data: Any) -> None: + data["datatype"] = __pydantic_self__.__make_datatype_element(data) + data["flag"] = __pydantic_self__.__make_flags(data) + super().__init__(**data) + + # pylint: disable=unused-private-member + def __make_datatype_element(self, col_data) -> DataType: + """Helper to make datatype element from column data when first created. + + For TAP_SCHEMA.columns tables that record datatype, arraysize as separate columns + """ + if col_data.get("datatype", None): + if isinstance(col_data["datatype"], DataType): + return col_data["datatype"] + + datatype_value = col_data.get("datatype", None) + datatype_arraysize = col_data.get("arraysize", None) + + datatype_elem = DataType( + arraysize=datatype_arraysize, + value=datatype_value, + ) + return datatype_elem + # If no datatype provided, default to char(*) + return DataType(value="char", arraysize="*") + + def __make_flags(self, col_data) -> list[str]: + """Set up the flag elements when creating the column. + + In the case that column flags are boolean values, as may occur in TAP_SCHEMA.columns, parse them into + a list of strings. + """ + if not col_data.get("flag", None): + flag = [flag for flag in ["principal", "indexed", "std"] if col_data.get(flag, None) == 1] + return flag + return col_data["flag"] + + @field_validator("column_name") + def validate_colname(cls, value: str): + """Escape the column name if it is an ADQL reserved word + + See: https://www.ivoa.net/documents/ADQL/20180112/PR-ADQL-2.1-20180112.html#tth_sEc2.1.3 + + value: - The column name to escape. + """ + if value.upper() in ADQL_SQL_KEYWORDS: + value = f'"{value}"' + return value + + @field_validator("description") + def validate_description(cls, value: str): + """Sanitize bad XML values in the description""" + if value: + value = escape(str(value)) + return value + + +class Table(BaseXmlModel, tag="table", ns="", skip_empty=True): + """A model representing a single table element. + + Parameters: + table_type: + (attr) - A name for the role this table plays. + + Recognized values include “output”, indicating this table is output from a query; + “base_table”, indicating a table whose records represent the main subjects of its schema; + and “view”, indicating that the table represents a useful combination or subset of other tables. + Other values are allowed. + table_name: + (elem) - The fully qualified name of the table. + + This name should include all catalogue or schema prefixes needed to sufficiently uniquely + distinguish it in a query. + title: + (elem) - A descriptive, human-interpretable name for the table. + description: + (elem) - A free-text description of the table's contents + utype: + (elem) - An identifier for a concept in a data model that the data in this table represent. + nrows: + (elem) - The approximate size of the table in rows. + column: + (elem) - A description of a table column. + foreign_key: + (elem) - A description of a foreign keys, one or more columns from the current table that can be used to + join with another table. + """ + + table_type: Optional[str] = attr(name="type", default=None) + + table_name: str = element(tag="name", ns="") + title: Optional[str] = element(tag="title", ns="", default=None) + description: Optional[str] = element(tag="description", ns="", default=None) + utype: Optional[str] = element(tag="utype", ns="", default=None) + nrows: Optional[int] = element(tag="nrows", gte=0, ns="", default=None) + column: Optional[list[TableParam]] = element(tag="column", ns="", default=None) + foreign_key: Optional[list[ForeignKey]] = element(tag="foreignKey", ns="", default=None) + + def __init__(__pydantic_self__, **data: Any) -> None: + """Escape any keys that are passed in.""" + for key, val in data.items(): + if isinstance(val, str): + data[key] = escape(val) + super().__init__(**data) + + @field_validator("column", "foreign_key", mode="before") + def validate_lists(cls, value): + """If we have a single column or foreign_key, make it a list""" + if value: + if not isinstance(value, list): + value = [value] + return value + + +class TableSchema(BaseXmlModel, tag="schema", ns="", skip_empty=True): + """A detailed description of a logically related group of tables. + + Parameters: + schema_name: + (elem) - A name for the group of tables. + + If no title is given, this name can be used for display purposes. If there is no appropriate logical + name associated with this group, the name should be explicitly set to “default”. + title: + (elem) - A descriptive, human-interpretable name for the group of tables. + description: + (elem) - A free text description of the group of tables that should explain in general how all of the tables + in the group are related. + utype: + (elem) - An identifier for a concept in a data model that the data in this schema as a whole represent. + table: + (elem) - A description of a table. + + """ + + schema_name: str = element(tag="name", default="default") + title: Optional[str] = element(tag="title", default=None) + description: Optional[str] = element(tag="description", default=None) + utype: Optional[str] = element(tag="utype", default=None) + table: Optional[list[Table]] = element(tag="table", default=None) + + def __init__(__pydantic_self__, **data: Any) -> None: + """Escape any keys that are passed in.""" + for key, val in data.items(): + if isinstance(val, str): + data[key] = escape(val) + super().__init__(**data) + + @field_validator("table", mode="before") + def validate_table(cls, value): + """If we have a single table, make it a list""" + if not isinstance(value, list): + value = [value] + return value + + +class TableSet(BaseXmlModel, tag="tableset", skip_empty=True): + """A description of the tables that are accessible through this service. + + Each schema name must be unique within a tableset. + + Parameters: + tableset_schema: + (elem) - A named description of a group of logically related tables. + + The name given by the “name” child element must be unique within this TableSet instance. + If there is only one schema in this set and/or there is no locally appropriate name to provide, + the name can be set to “default”. + + """ + + tableset_schema: list[TableSchema] = element(tag="schema") + + @field_validator("tableset_schema", mode="before") + def validate_tableset_schema(cls, value): + """If we have a single tableset_schema, make it a list""" + if not isinstance(value, list): + value = [value] + return value diff --git a/vo_models/voresource/types.py b/vo_models/voresource/types.py index 241f71b..c6412ba 100644 --- a/vo_models/voresource/types.py +++ b/vo_models/voresource/types.py @@ -55,7 +55,7 @@ def _validate(cls, value: str): """Validator that expands the pydantic datetime model to include Z UTC identifiers Args: - value (str): datetime string. Comes from either a user's POST (destruction) or from the cache + value: datetime string. Comes from either a user's POST (destruction) or from the cache Returns: UTCTimestamp: VO-compliant datetime subclass @@ -90,8 +90,8 @@ def isoformat(self, sep: str = "T", timespec: str = "milliseconds") -> str: """Overwrites the datetime isoformat output to use a Z UTC indicator Parameters: - sep (str): Separator between date and time (default: 'T') - timespec (str): Resolution of time to include (default: 'milliseconds') + sep: Separator between date and time (default: 'T') + timespec: Resolution of time to include (default: 'milliseconds') Returns: str: VO-compliant ISO-8601 datetime string diff --git a/vo_models/vosi/availability/models.py b/vo_models/vosi/availability/models.py index a47e1c6..c9f3f2d 100644 --- a/vo_models/vosi/availability/models.py +++ b/vo_models/vosi/availability/models.py @@ -17,16 +17,16 @@ class Availability(BaseXmlModel, tag="availability", nsmap=NSMAP, skip_empty=Tru """VOSI Availability complex type. Parameters: - available (bool): + available: (element) - Whether the service is currently available. - up_since (UTCTimestamp): + up_since: (element) - The instant at which the service last became available. - down_at (UTCTimestamp): + down_at: (element) - The instant at which the service is next scheduled to become unavailable. - back_at (UTCTimestamp): + back_at: (element) - The instant at which the service is scheduled to become available again after a period of unavailability. - note (Optional[list[str]]): + note: (element) - A textual note concerning availability. """ diff --git a/vo_models/vosi/tables/__init__.py b/vo_models/vosi/tables/__init__.py new file mode 100644 index 0000000..3911d2a --- /dev/null +++ b/vo_models/vosi/tables/__init__.py @@ -0,0 +1,3 @@ +"""Module containing models for VOSI Tables objects. +""" +from vo_models.vosi.tables.models import VOSITable, VOSITableSet diff --git a/vo_models/vosi/tables/models.py b/vo_models/vosi/tables/models.py new file mode 100644 index 0000000..4a2d9c6 --- /dev/null +++ b/vo_models/vosi/tables/models.py @@ -0,0 +1,55 @@ +"""Pydantic-xml models for the VOSI Tables specification""" +from vo_models.vodataservice import Table, TableSet + +NSMAP = { + "vosi": "http://www.ivoa.net/xml/VOSITables/v1.0", + "vr": "http://www.ivoa.net/xml/VOResource/v1.0", + "vs": "http://www.ivoa.net/xml/VODataService/v1.1", + "xsd": "http://www.w3.org/2001/XMLSchema", + "xsi": "http://www.w3.org/2001/XMLSchema-instance", +} + + +class VOSITable(Table, tag="table", ns="vosi", nsmap=NSMAP): + """A table element as returned by a VOSI /tables request + + Parameters: + table_type: + (attr) - A name for the role this table plays. + + Recognized values include “output”, indicating this table is output from a query; + “base_table”, indicating a table whose records represent the main subjects of its schema; + and “view”, indicating that the table represents a useful combination or subset of other tables. + Other values are allowed. + table_name: + (elem) - The fully qualified name of the table. + + This name should include all catalogue or schema prefixes needed to sufficiently uniquely + distinguish it in a query. + title: + (elem) - A descriptive, human-interpretable name for the table. + description: + (elem) - A free-text description of the table's contents + utype: + (elem) - An identifier for a concept in a data model that the data in this table represent. + nrows: + (elem) - The approximate size of the table in rows. + column: + (elem) - A description of a table column. + foreign_key: + (elem) - A description of a foreign keys, one or more columns from the current table that can be used to + join with another table. + """ + + +class VOSITableSet(TableSet, tag="tableset", ns="vosi", nsmap=NSMAP): + """A tableset element as returned by a VOSI /tables request + + Parameters: + tableset_schema: + (elem) - A named description of a group of logically related tables. + + The name given by the “name” child element must be unique within this TableSet instance. + If there is only one schema in this set and/or there is no locally appropriate name to provide, + the name can be set to “default”. + """