Massive refactoring of CI and flows (#289)

* user/password auth in fixtures; core testing can run on hcd

* WIP: live_test_models and live_provider_info must become a db-dependent tower of fixtures

* wip2, formatting

* wip refactored all (nonvecze) env vars

* restructure conftests to avoid quirks as much as possible

* last before live_test_models becomes a fixture

* full refactoring of the test suites + description in readme

* improve workflows; add hcd workflow; condition nonAstra admin on env var

* skip cursor timeout test on nonAstra (astra test is enough)

.github/workflows/hcd.yaml

@@ -0,0 +1,39 @@
+name: Run idiomatic pytest on HCD
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  test:
+    env:
+      # special setting to not drop any collection (core only)
+      TEST_SKIP_COLLECTION_DELETE: ${{ secrets.TEST_SKIP_COLLECTION_DELETE }}
+      # for admin-related testing if enabled
+      DO_IDIOMATIC_ADMIN_TESTS: ${{ secrets.DO_IDIOMATIC_ADMIN_TESTS }}
+      # hardcoding the target DB
+      DOCKER_COMPOSE_LOCAL_DATA_API: "yes"
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.11
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install poetry
+        poetry install
+
+    - name: Run pytest
+      run: |
+        poetry run pytest tests/idiomatic

.github/workflows/main.yml

@@ -1,4 +1,4 @@
-name: Run Pytest on Astrapy
+name: Run idiomatic pytest on Astra DB
 
 on:
   push:
@@ -16,7 +16,7 @@ jobs:
       ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }}
       ASTRA_DB_KEYSPACE: ${{ secrets.ASTRA_DB_KEYSPACE }}
       ASTRA_DB_SECONDARY_KEYSPACE: ${{ secrets.ASTRA_DB_SECONDARY_KEYSPACE }}
-      # special setting to not drop any collection
+      # special setting to not drop any collection (core only)
       TEST_SKIP_COLLECTION_DELETE: ${{ secrets.TEST_SKIP_COLLECTION_DELETE }}
       # for admin-related testing if enabled
       DO_IDIOMATIC_ADMIN_TESTS: ${{ secrets.DO_IDIOMATIC_ADMIN_TESTS }}

Makefile

@@ -28,14 +28,24 @@ format-fix-src: format-src
 format-fix-tests: FMT_FLAGS=
 format-fix-tests: format-tests
 
-test-idiomatic: test-idiomatic-unit test-idiomatic-integration
+test-idiomatic:
+	poetry run pytest tests/idiomatic -vv
 
 test-idiomatic-unit:
 	poetry run pytest tests/idiomatic/unit -vv
 
 test-idiomatic-integration:
 	poetry run pytest tests/idiomatic/integration -vv
 
+docker-test-idiomatic:
+	DOCKER_COMPOSE_LOCAL_DATA_API="yes" poetry run pytest tests/idiomatic -vv
+
+docker-test-idiomatic-unit:
+	DOCKER_COMPOSE_LOCAL_DATA_API="yes" poetry run pytest tests/idiomatic/unit -vv
+
+docker-test-idiomatic-integration:
+	DOCKER_COMPOSE_LOCAL_DATA_API="yes" poetry run pytest tests/idiomatic/integration -vv
+
 build:
 	rm -f dist/astrapy*
 	poetry build
@@ -53,5 +63,8 @@ help:
 	@echo "test-idiomatic                   run idiomatic tests"
 	@echo "  test-idiomatic-unit              unit only"
 	@echo "  test-idiomatic-integration       integration only"
+	@echo "docker-test-idiomatic            same, on docker container"
+	@echo "  docker-test-idiomatic-unit           same"
+	@echo "  docker-test-idiomatic-integration    same"
 	@echo "build                            build package ready for PyPI"
 	@echo "======================================================================"

README.md

@@ -226,56 +226,136 @@ naming convention and module structure).
 
 ### Running tests
 
-"Full regular" testing requires environment variables:
+Tests are grouped in three _blocks_ (in as many subdirs of `tests/`):
 
-```bash
-export ASTRA_DB_APPLICATION_TOKEN="AstraCS:..."
-export ASTRA_DB_API_ENDPOINT="https://.......apps.astra.datastax.com"
+- **core**: pre-1.0 classes
+- **idiomatic**: all 1.0+ classes and APIs, except...
+- **vectorize**: ... everything making use of `$vectorize` (within the idiomatic classes)
+
+Actually, for convenience, _sub-blocks_ of tests are considered:
+
+- **core regular**: everything except DevOps interactions
+- **core ops**: core DevOps operations
+- **idiomatic regular**: everything except the admin parts
+- **idiomatic admin Astra**: the Astra-specific admin operations
+- **idiomatic admin nonAstra**: the nonAstra-specific admin operations
+- **vectorize in-depth**: many Data API interactions for a single choice of provider/model. This is mostly test the client
+- **vectorize all-providers**: a slightly more shallow test repeated for all providers, models, auth methods etc. This is mostly testing the API
+
+Tests can be run on three types of Data API _targets_ (with slight differences in what is applicable):
+
+- **DockerCompose**: HCD started by the test initialization with `docker-compose`. _Note that in this case you will have to manually destroy the created containers._
+- **nonAstra**: a ready-to-use (user-supplied) local Data API
+- **Astra**: an Astra DB target account (or two, as some tests are specific to dev environment)
+
+Depending on the (sub-block, target) combination, some environment variables may be needed.
+Templates for the environment variables are to be found in `tests/env_templates`.
+
+The general expectation is that idiomatic non-Admin tests, and vectorize in-depth tests, are
+part of the main CI flow; conversely, core, admin and vectorize all-providers are kept as a
+manual task to run (locally in most cases) when circumstances require it (use your judgement).
+
+#### Required environment variables
+
+Below is a detail of the reference template files needed for the various types
+of testing:
+
+- **DockerCompose**: generally no variables needed, except:
+  - **vectorize in-depth**: provide as in `env.vectorize-minimal.template`
+  - **vectorize all-providers**: provide as in `env.vectorize.template`
+  - (also note that _core ops_ and _idiomatic admin Astra_ amount to nothing in this case)
+- **nonAstra**: all tests require as in `env.local.template`, plus:
+  - **vectorize in-depth**: also provide as in `env.vectorize-minimal.template`
+  - **vectorize all-providers**: also provide as in `env.vectorize.template`
+  - (also note that _core ops_ and _idiomatic admin Astra_ amount to nothing in this case)
+- **Astra**: all tests require as in `env.astra.template`, plus:
+  - **core ops**: the token must have at least "Database Administrator" role (possibly through definition of a separate `ASTRA_DB_OPS_APPLICATION_TOKEN`), and `ASTRA_DB_ID` must also be defined
+  - **idiomatic admin Astra**: also provide as in `env.astra.admin.template`
+  - **vectorize in-depth**: also provide as in `env.vectorize-minimal.template`
+  - **vectorize all-providers**: also provide as in `env.vectorize.template`
+  - (also note that _idiomatic admin nonAstra_ amounts to nothing in this case)
+
+#### Sample testing commands
+
+For the **DockerCompose** case, prepend all of the following with `DOCKER_COMPOSE_LOCAL_DATA_API="yes" `.
+
+All the usual `pytest` ways of restricting the test selection hold in addition
+(e.g. `poetry run pytest tests/idiomatic/unit` or `[...] -k <test_name_selector>`).
+
+##### _core regular_:
 
-export ASTRA_DB_KEYSPACE="default_keyspace"
-# Optional:
-export ASTRA_DB_SECONDARY_KEYSPACE="..."
+```
+poetry run pytest tests/core
 ```
 
-#### "Idiomatic" testing
+##### _core ops_:
 
-Tests can be started in various ways: mostly `make tests-idiomatic`, but also:
+Note the special variable needed to actually run this. You will have to manually clean up afterwards.
 
-```bash
-# test the "idiomatic" layer
+```
+TEST_ASTRADBOPS="1" poetry run pytest tests/core/test_ops.py
+```
+
+##### _idiomatic regular_:
+
+Warning: this will also trigger the very long-running _idiomatic admin Astra_ if the vars as in `env.astra.admin.template` are also detected. Likewise, the _idiomatic admin nonAstra_ may start (if `DO_IDIOMATIC_ADMIN_TESTS` is set), which however takes few seconds.
+
+```
 poetry run pytest tests/idiomatic
-poetry run pytest tests/idiomatic/unit
-poetry run pytest tests/idiomatic/integration
+```
 
-# remove logging noise:
-poetry run pytest [...] -o log_cli=0
+##### _idiomatic admin Astra_:
+
+```
+poetry run pytest tests/idiomatic/integration/test_admin.py 
 ```
 
-The above runs the regular testing (i.e. non-Admin, non-core).
-The (idiomatic) Admin part is tested manually by you, on Astra accounts with room
-for up to 3 new databases, possibly both on prod and dev, and uses specific env vars,
-as can be seen on `tests/idiomatic/integration/test_admin.py`.
+##### _idiomatic admin nonAstra_:
 
-#### Other tests
+```
+DO_IDIOMATIC_ADMIN_TESTS="1" poetry run pytest tests/idiomatic/integration/test_nonastra_admin.py
+```
 
-Vectorize tests are confined in `tests/vectorize_idiomatic` and are run
-separately. A separate set of credentials is required to do the full testing:
-refer to `tests/.vectorize.env.template` for the complete listing, including
-the secrets that should be added to the database beforehand, through the UI.
+##### _vectorize in-depth_:
 
-Should you be interested in testing the "core" modules, moreover,
-this is also something for you to run manually (do that if you touch "core"):
+```
+poetry run pytest tests/vectorize_idiomatic/integration/test_vectorize_methods*.py
+```
 
-```bash
-# test the core modules
-poetry run pytest tests/core
+or just:
 
-# do not drop collections:
-TEST_SKIP_COLLECTION_DELETE=1 poetry run pytest [...]
+```
+poetry run pytest tests/vectorize_idiomatic/integration/test_vectorize_methods_sync.py
+```
+
+##### _vectorize all-providers_:
+
+This generates all possible test cases and runs them:
+
+```
+poetry run pytest tests/vectorize_idiomatic
+```
+
+For a spot test, you may restrict to one case, e.g.
 
-# include astrapy.core.ops testing (tester must clean up after that):
-TEST_ASTRADBOPS=1 poetry run pytest [...]
 ```
+EMBEDDING_MODEL_TAGS="openai/text-embedding-3-large/HEADER/0" poetry run pytest tests/vectorize_idiomatic/integration/test_vectorize_providers.py -k test_vectorize_usage_auth_type_header_sync
+```
+
+#### Useful flags for testing
+
+Remove logging noise with:
+
+```
+poetry run pytest [...] -o log_cli=0
+```
+
+Do not drop collections (core):
+
+```
+TEST_SKIP_COLLECTION_DELETE=1 poetry run pytest [...]
+```
+
 
 ## Appendices