Unit Testing

[benbrastmckie]

This is low priority but want to log details here. I've been working on writing unit tests and am getting the following error when running pytest in the Code directory:

____________________________________________________________________________________________________________ ERROR collecting test/test_examples.py ____________________________________________________________________________________________________________
test/test_examples.py:12: in <module>
    test_models(premises,conclusions,N)
test/test_examples.py:9: in test_models
    mod = make_model_for(length)(prems, cons)
/home/brastmck/.local/lib/python3.9/site-packages/model_checker/model_structure.py:627: in make_relations_and_solve
    mod = ModelStructure(
/home/brastmck/.local/lib/python3.9/site-packages/model_checker/model_structure.py:119: in __init__
    self.prefix_conclusions = [Prefix(con) for con in infix_conclusions]
/home/brastmck/.local/lib/python3.9/site-packages/model_checker/model_structure.py:119: in <listcomp>
    self.prefix_conclusions = [Prefix(con) for con in infix_conclusions]
/home/brastmck/.local/lib/python3.9/site-packages/model_checker/syntax.py:151: in Prefix
    return parse(tokenize(A))
/home/brastmck/.local/lib/python3.9/site-packages/model_checker/syntax.py:146: in parse
    return [op_str, parse(left_expression), parse(right_expression)]
/home/brastmck/.local/lib/python3.9/site-packages/model_checker/syntax.py:146: in parse
    return [op_str, parse(left_expression), parse(right_expression)]
/home/brastmck/.local/lib/python3.9/site-packages/model_checker/syntax.py:130: in parse
    if tokens[0] in unary_operators:
E   IndexError: list index out of range

The relevant lines of code in syntax.py:130 are:

    bin_comp_tokens = binary_comp(tokens)
    # if isinstance(tokens, str):
    #     return tokens
    # if not tokens:
    #     raise SyntaxError(f"Error: '{tokens}' is not a list.")
    if tokens[0] in unary_operators:
        return [tokens[0], parse(tokens[1:])]

The comments are some of what I have tried. Not totally sure what it is unhappy about.

[mbuit82]

What command are you running in the terminal?

[benbrastmckie]

You have to install pytest with pip install pytest and then run pytest (no arguments) from the Code directory where the toml file is.

[mbuit82]

I added a new test file (sat_tests.py in the test folder) with some tests (all taken from test_complete.py). To run them, go to the test folder and run pytest sat_tests.py. You need to install pytest-timeout before running them (pip install pytest-timeout). There should be a total of 14 tests that run. I added timeout limits (which is what pytest-timeout is for); with these limits (which can be turned off by setting the number in the parentheses of each @pytest.mark.timeout(30) statement to 0) the first 7 tests pass on my computer and the last 7 fail (take too long). Without time limits all pass (in like 10 minutes total).

This configuration for testing is based on the tests they'd grade us on for my programming class in the fall semester. There are some issues, most visibly with the imports—right now it's testing the package you've downloaded locally, not the file "next door" from the test file so to speak (see the comment in sat_tests.py for a more thorough explanation).

The syntax of the tests is relatively simple, it should be pretty easy to add more tests, but let me know any problems you run into (I have never made unit tests from scratch lol).

[benbrastmckie]

That's sounds great! I'll start messing around with these. I'm having trouble getting it to test the local files also. I don't have model-checker installed since I've been waiting for it to be added to Nix (my package manager). It has been added but it would be good to get it working to test locally. I'll keep trying to fix this. But great that the tests pass!

[mbuit82]

Now in theory you shouldn't need to have model-checker installed—I (think I) fixed the file path issue that was causing that problem. Try giving it another shot to see if the tests work now

[mbuit82]

Another update—I just put all of the test cases from your paper (from sections 3.2 for the valid rules and 3.3 for the invalid ones) in the file and labeled them to match those in the paper. I think that took care of everything that's in the current version of test_complete.py, though notably there is no Sobel sequence test case yet. Some of them take too long to run on my computer (and it doesn't seem like the time limit thing I put seems to work), so I'm curious to see if you are able to run them on your machine or the MIT server

[benbrastmckie]

Awesome! I got most of the tests working where only 4 were slow. I have copied them below for convenience:

@pytest.mark.timeout(30)
def test_CL_5():
    """SLOW: requires N = 4 and 347 seconds on the MIT server"""
    N = 4
    premises = ['(A \\boxright C)', '(B \\boxright C)']
    conclusions = ['((A \\wedge B) \\boxright C)']
    desired_model_status = True
    check_model_status(premises, conclusions, desired_model_status, N)

@pytest.mark.timeout(30)
def test_CL_6():
    """SLOW: requires N = 4 and 125 seconds on the MIT server"""
    N = 4
    premises = ['(A \\boxright B)', '\\neg B']
    conclusions = ['(\\neg B \\boxright \\neg A)']
    desired_model_status = True
    check_model_status(premises, conclusions, desired_model_status, N)

@pytest.mark.timeout(30)
def test_CL_8():
    """SLOW: MIT servers found a model in 467 seconds"""
    N = 3
    premises = ['(A \\boxright (B \\boxright C))']
    conclusions = ['((A \\wedge B) \\boxright C)']
    desired_model_status = True
    check_model_status(premises, conclusions, desired_model_status, N)

@pytest.mark.timeout(10)
def test_STA_w_negation():
    """SLOW: requires N = 4 and 242 seconds on the MIT server"""
    N = 4
    premises = ['\\neg A', '(A \\boxright C)']
    conclusions = ['((A \\wedge B) \\boxright C)']
    desired_model_status = True
    check_model_status(premises, conclusions, desired_model_status, N)

I got the first one to pass on the MIT server, but the second took ages, and the third and fourth failed. I'm going to comment these out, as well as the R10 which was slow. That way pytest can still be used to check that changes don't break anything.