-
Notifications
You must be signed in to change notification settings - Fork 8
/
ROADMAP.txt
119 lines (76 loc) · 6.22 KB
/
ROADMAP.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
This file lists the planned features for the Jumi test runner. Those
features which are marked with "!!" have currently the highest priority.
Most of these features can be implemented in any order, so to get your
needs satisfied sooner, come to the Jumi test runner mailing list at
https://groups.google.com/d/forum/jumi-test-runner and be vocal about your
use cases and which features are most important to you.
EXPRESSIVENESS
- Support tests in any kinds files, not just .class files
- Before & after suite events (probably as methods on a runner; somebody can even make a runner which doesn't run tests, but only those hooks, in case a testing framework doesn't itself support them)
- Thread & timestamp metadata for each event ("Spacetime"), in the future maybe also process ID/hostname
!! - Pending/ignored tests
- Must support marking a test pending both before and after starting to execute it (e.g. @Ignore vs. assumeThat())
- Should the pending status be per suite, or per run? (i.e. when a test is run multiple times, can some of them be pending and not others?)
- What use cases are there? Need to investigate current solutions
- Running only individual tests from a test class
- Common abstraction? Or is it too framework specific to be abstracted properly?
- Some frameworks can uniquely identify tests based on TestId, but for others the test name is more reliable
- What are the needs of IDE integration?
- Running a subset of tests based on groups/tags
- Probably string based (http://tech.groups.yahoo.com/group/junit/message/24017) or hybrid (http://tech.groups.yahoo.com/group/junit/message/24024)
- Reporting system properties, hostname, hardware and other metadata about the JVM on which the tests were run
PERFORMANCE
!! - Cache library JARs (implies a persistent daemon process)
- May need to allow excluding some dependencies from caching, in case some mocking or testing frameworks don't work well from a parent class loader
- May need to be able to detect class loader leaks
- Publish a benchmark (video or article) demonstrating the speed. Use Dimdwarf as a sample project. Compare test run times:
- Maven (only the test stage of the build)
- IDEA
- Eclipse
- NetBeans
- sbt
- runs tests in parallel http://www.scala-sbt.org/release/docs/Detailed-Topics/Testing.html#disable-parallel-execution-of-tests
- runs tests in the same JVM (by default no forking), not sure whether does classloader caching http://www.scala-sbt.org/release/docs/Detailed-Topics/Testing.html#forking-tests
- Jumi (1-thread, cold-start)
- Jumi (1-thread, warm-start)
- Jumi (4-thread, cold-start)
- Jumi (4-thread, warm-start)
- Running the tests in a server cluster
- Statistics (required by many other features)
- Test priorization, run first those which will most probably fail, run fast tests first etc.
- Look for the fastest order of running tests from class loading point of view (best-effort thread affinity at test class level, to avoid all threads being blocked by loading the same classes?)
- Detect what classes a test depends on (e.g. run one test in new class loader and spy that what classes it loads) and run only those (or those first)
- IO/CPU boundness checking (to choose how many threads to use)
- Measure how long class loading takes by comparing cold and warm starts: Would that be useful statistics? Would it help identify fat tests?
USABILITY
!! - Debug tray GUI, shows only individual test runs (list of run IDs on left, run log on right; maybe also the test names in a tree, leftmost; print all test run data in log, but no colors or styles)
- Timeline of all events, to see what tests were run concurrently, and which tests were slow (requires thread and timestamp for each event)
- Well designed UI
- Per-module working directory (requires a separate JVM for each module)
- Get a thread dump of tests that appear to be frozen
- Automatically report exceptions in background threads by setting a default handler
- Wait for background threads to finish at the end of a test run, to help notice leaking tests
TEST COMPATIBILITY
- Per-testclass parallelism controls (for bad non-isolated integration tests in a suite with otherwise good unit tests)
- e.g. as annotations: @Parallel (default), @LocallySequential (inside one test file), @GloballySequential (JVM/suite-globality)
- e.g. as one annotation: @Parallelism(ParallelismMode.PARALLEL), LOCALLY_SEQUENTIAL, GLOBALLY_SEQUENTIAL
- what to call it when we have multiple JVMs, or even multiple suites/classloaders? rename "globally sequential" to "statically/processly/JVM sequential"? add a bigger scope "universally sequential"?
- Isolating tests by running them in parallel JVMs (sequentially inside each of them), or by using multiple class loaders for isolating static variables
- Make sure that test code cannot access the classes used by Jumi, except Jumi API; avoid library version conflicts using custom class loaders
- Work around the issue of class loader locking the JARs (it's possible, though rare, that some JAR dependencies are generated by the same build)
- Faking "java.class.path", so that code which relies on it would not be disturbed by the use of multiple classloaders (should be optional and off by default?)
TOOL COMPATIBILITY
- Build tool, IDE and CI integration
- Maven (to gain traction in Java community)
- Run tests as part of a Maven build
- Produce test reports in same format as Surefire (for CI integration)
- Generate bootsrap classes for running tests within an IDE with the same settings as Maven (to compensate for lack of IDE integration)
OK - sbt (to gain traction in Scala community)
- Leiningen (also reloading Clojure namespaces might require additional support inside Jumi)
- IntelliJ IDEA, Eclipse, NetBeans; will probably need help from the community because of the amount of work involved in developing them
- JUnit compatible reports, then later native reporting support: Jenkins, ThoughtWorks Go, JetBrains TeamCity, Atlassian Bamboo, Cruise Control
- Integration test project
- Infrastructure for running tests against specified Jumi version
- Tests for Specsy, Maven plugin etc.
- Document and advertise it to framework developers
- Integration with PIT mutation testing (http://pitest.org/), preferably as an optional plugin to drive the plugin API