Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[uss_qualifier] Add badge concept #78

Merged

Conversation

BenjaminPelletier
Copy link
Member

Many testing situations call for more fine-grained results than a simple binary success/fail -- for instance, there may be optional capabilities a participant may choose to implement or not, and knowing whether each participant has been checked out for each optional capability may be useful. To address this need, this PR introduces the concept of "badges" which are explained in the uss_qualifier README.md changes.

From a technical perspective, available badges are defined by test suites, and those badges are granted to participants at the conclusion of running the test suite in which they are defined. The new participant_badges field in TestSuiteDefinition can be seen in suites/definitions.py and then the core badge data structures can be found in reports/badge_definitions.py. Badges are granted at the conclusion of a test suite run in suites/suite.py which calls out to badge grant logic in reports/badges.py.

To support more precise identification of collections of requirements, RequirementsCollection is created in requirements/definitions.py, and the RequirementsID and RequirementsSetID data structures are moved into that module to support inclusion in other data structures while avoiding circular dependencies. The resolution of a RequirementsCollection into a set of requirements is added to the end of requirements/documentation.py.

RequirementSetIDs get an expanded capability to refer to a particular section of a Markdown document to more flexibly identify requirement collections.

All types of reports receive additional check examination abilities (finding all passed checks, failed checks, relating to a particular participant) to support badge grant logic, and some improvement is made to the way the pseudo-polymorphic TestSuiteActionReport accesses its different contents possibilities.

A bug in jsonpath-ng lead to the discovery that that package has not been updated in years, so it is replaced by a fork that fixes that bug and has been updated much more recently (bc-jsonpath-ng).

Example badges are added to a few test suites to verify functionality.

In a follow-up PR, I hope to add the capability to report details about why a particular badge was not granted to a particular participant to the output report as I suspect investigation of the reasons a badge wasn't granted will be a core user journey for participants under test.

Copy link
Contributor

@barroco barroco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, thanks!

@BenjaminPelletier BenjaminPelletier merged commit af57d19 into interuss:main Apr 4, 2023
@BenjaminPelletier BenjaminPelletier deleted the uss_qualifier/badges branch April 4, 2023 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants