Hi all,
I'd need an advice about a framework to test the thread safety of some classes. I know about tools like http://groboutils.sourceforge.net/ (but it looks like a bit old) and http://www-128.ibm.com/developerworks/java/library/j-contest.html (which uses random delays).
Any else?
Thanks in advance,
Michele
Funny that you should mention those two libraries; I came to this forum today specifically to discuss them (more ConTest than GroboUtils).
I originally started writing concurrency tests manually, which was painful, nay, excruciating. Then I discovered GroboUtils' MultiThreadedTestRunner, and it greatly eased my plight.
Then I started noticing that my tests were missing concurrency bugs regardless of how many threads I threw at them. Turns out that testing threads tend to interleave in the same way nearly every time, i.e., the threads always switch in the same places. I wasn't testing all cases, I was testing the same cases over and over again.
ConTest to the rescue. It instrumented classes with random wait and yield calls to produce a different threading interleave with each run. (See the ConTest site for a better description of both the problem and solution.) Many of my tests that succeeded previously now failed when run with ConTest. As a TDD subscriber, this was a welcome sign. It meant that, now that I could finally test for these bugs, my code would be free of them forever (once I fixed them).
I have been using these two technologies with much success ever since.
The question I had for this forum was: Why are these technologies not in the mainstream of the Java community?
I suspect there are two reasons:
1. Concurrency tests are not being written
2. Developers are oblivious to the impotence of their concurrency tests
Hopefully these issues and technologies will garner more attention from the community in the future.
> I suspect there are two reasons:
> 1. Concurrency tests are not being written
I suspect that is too restrictive. Instead it should say....
Tests are not being written.
(And although I wish I was joking I am not.)
> 2. Developers are oblivious to the impotence of their
> concurrency tests
For those that do test then that is certainly possible. However I note that for the tests that I write I do use random factors (one of the things you suggested was missing from your tests.)
You should also note that it is much better to test multi-threaded applications on a multiple CPU machine. I suppose dual core serves some of the same purpose but if so a multiple CPU machine with dual core cpus would then just be that much better.
<This is like talking to my wife.>
I was hoping for some general tips or tricks on randomizing thread interleaves, because besides ConTest I see no obvious way to do it. I suppose you coudl insert a random delay before each thread is started, but that might require some delicate tuning, and wouldn't handle all cases.
I thought you might have something to teach on the subject.
These tools are still in their infancy which is why nothing has yet emerged as "THE Tool" for the community.
There are tools that rewrite the bytecode to introduce yields() and/or sleeps between other bytecodes to try and simulate arbitrary interleavings. But there is no way to tell how well you have tested things, and testing this way can be very slow.
More often people use "stress tests" to try and expose bugs, these just try to hit code as hard as possible without trying to consider any kind of whte-box code path coverage.
Often tests for concurrent programs are harder to write and contain more concurrency bugs then the programs under test. Such is the nature of the beast.
> <This is like talking to my wife.>
>
> I was hoping for some general tips or tricks on
> randomizing thread interleaves, because besides
> ConTest I see no obvious way to do it. I suppose you
> coudl insert a random delay before each thread is
> started, but that might require some delicate tuning,
> and wouldn't handle all cases.
What sort of magic do you think that ConTest does?
>
> I thought you might have something to teach on the
> subject.
I might at that. But that doesn't mean I have time (nor the interest) in writing a book on the subject.
And no I wouldn't normally vary the start time of threads. What I would do is vary (randomly) the behavior of the threads after they are started. How that is implemented depends on what I am trying to simulate and what I am testing.
> What sort of magic do you think that ConTest does?
As far as I understand it, ConTest instruments your class file inserting yields, sleeps, etc. around concurrency sensitive points in the code. The inserts have random behavior such that different runs produce different interleavings. Run repeatedly, and soon all concurrency significant interleavings are exercised. Again, the ConTest site gives a fuller description.
you may actually want ot run thoses tests that are interleved without the random stuff first. If something is wrong it will show up soon and you will be able to find what it is.
Beyond that you wan to introduce random or differing factors,
threads should be random in ordering anyway, they should get out of sync naturally if the test is long enough
I agree that it needs to be done on a properly configured dual-core machine or better otherwise you are not testing much with a single CPU, the threading will be way too predictable to represent real life multi-cpu instruction reordering
Sean
I'm not sure I understand the push-back on this ConTest technology. Why write excessively long tests that must run on multi-processor/core machines when there's simpler, faster, more reliable solution? Please give it a try before coming to judgment.
(BTW, I just realized that I'm sounding like a ConTest sales rep. I'm not. Just a fan.)
> I'm not sure I understand the push-back on this
> ConTest technology. Why write excessively long tests
> that must run on multi-processor/core machines when
> there's simpler, faster, more reliable solution?
> Please give it a try before coming to judgment.
At least from me it was rather your suggestion that developers didn't know what they were doing and thus couldn't be writing correct tests.
Even your suggestion here suggests something similar in that developers can't write tests that are just as reliable. (Noting that I already pointed out that developers are deficient in writing tests of any sort so providing a tool to test a subset of functionality isn't going to help with that.)
Finally I am rather certain that relying on a single cpu (single core even) to do multi-threading testing regardless of the test tool is a really bad idea. If the Sun VM was still using green threads then that might make sense. But since everything is now native threaded that is a really bad idea.