Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CodeRunner/Runtime: add failure cases to test suite #16

Open
7 tasks
camerondurham opened this issue Jan 27, 2022 · 0 comments
Open
7 tasks

CodeRunner/Runtime: add failure cases to test suite #16

camerondurham opened this issue Jan 27, 2022 · 0 comments
Labels
code runner Code runner handles language config and concurrent requests runtime Engine runtime handles safely executing user submitted code

Comments

@camerondurham
Copy link
Owner

camerondurham commented Jan 27, 2022

This may be blocked by #15

Create test suite of "malicious" code to understand limits of the CodeRunner. This will be helpful for benchmarking and understanding how much progress we've made toward the Code Isolation Milestone.

These test cases can be wrapped in an environment variable check so they do not fail the build and just used in a temporary test container. Example:

func Test_MaliciousCode(t *testing.T) {
  if os.Getenv("TEST_MC") != "" {
    // test stuff that may intentionally break
  }
  // rest of the test
}
  • add set of test cases for the runtime that run separately from unit tests (ideally in Docker)
  • test failure modes
    • throw an unexpected exception
    • spawn many processes
    • "escape" the current directory
    • spawn a shell
    • ... (think of other ideas)
@camerondurham camerondurham pinned this issue Jan 27, 2022
@camerondurham camerondurham unpinned this issue Jan 27, 2022
@camerondurham camerondurham changed the title Runtime: add failure cases to test suite CodeRunner/Runtime: add failure cases to test suite Jan 27, 2022
@camerondurham camerondurham added this to the Code Isolation milestone Jan 27, 2022
@camerondurham camerondurham added enhancement New feature or request code runner Code runner handles language config and concurrent requests runtime Engine runtime handles safely executing user submitted code and removed enhancement New feature or request labels Feb 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code runner Code runner handles language config and concurrent requests runtime Engine runtime handles safely executing user submitted code
Projects
Development

No branches or pull requests

1 participant