How to measure memory consumption during unit tests?
Hello,
I'm looking for simple tools to automate measurement of overall memory consumption during some memory-sensitive unit tests.
I would like to apply this when running a batch of some test suite targetting tests that exercise memory-sensitive operations.
The intent is, to verify that a modification of code in this area does not introduce regression (raise) of memory consumption.
I would include it in the nightly build, and monitor evolution of summary figure (a-ah, the "userAccount" test suite consumed 615Mb last night, compared to 500Mb the night before... What did we check-in yesterday?)
Running on Win32, the system-level info of memory consumed is known not to be accurate.
Using perfmon is more accurate but it seems an overkill - plus it's difficult to automate, you have to attach it to an existing process...
I've looked in the hprof included in Sun's JDK, but it seems to be targetted at investigating problems rather than discovering them. In particular there isn't a "summary line" of the total memory consumed...
What tools do you use/suggest?
[1132 byte] By [
jdupreza] at [2007-10-1 5:00:54]
Add a test to check the Runtime.getRuntime().totalMemory()
This will only increase when the GC increases it.
An increase in this value indicates that more memory was needed at some point.
You can try more accurate measures but you may find some tests use less memory than the previous test. i.e. consumed a negative amount of memory.
> Add a test to check the
> Runtime.getRuntime().totalMemory()
> This will only increase when the GC increases it.
> An increase in this value indicates that more memory
> was needed at some point.
> You can try more accurate measures but you may find
> some tests use less memory than the previous test.
> i.e. consumed a negative amount of memory.
Thanks.
However this requires manual code in my unit test classes themselves, e.g. in my setUp/tearDown methods.
I was expecting something more orthogonal to the tests, that I could activate or not depending on the purpose of the test.
If I don't have another option, OK I'll wire my own pre/post memory counting, maybe using AOP, and will activate memory measurement only when needed.
Have you actually used your suggestion to automate memory consumption measurement as part of daily builds?
Plus, I did not understand your suggestion, can you elaborate?
- I first assumed you meant freeMemory(), which, as you suggest, is not accurate, since it returns "an approximation of [available memory]"
- I re-read it and now assume you do mean totalMemory(), which unfortunately will grow only when more memory than the initial heap setting is needed.
- Eventually, I may need to inlcude calls to System.gc() but I seem to remember it is best-effort only (endless discussion) and may not help accuracy.
> However this requires manual code in my unit test
> classes themselves, e.g. in my setUp/tearDown
> methods.
> I was expecting something more orthogonal to the
> tests, that I could activate or not depending on the
> purpose of the test.
Some IDEs display mmeory usage and execution time for each test/group of tests.
> If I don't have another option, OK I'll wire my own
> pre/post memory counting, maybe using AOP, and will
> activate memory measurement only when needed.
If you need to check the memory used, I would do this.
You can do the same thing with AOP. Unless you are using an AOP library, I doubt it is worth additional effort.
> Have you actually used your suggestion to automate
> memory consumption measurement as part of daily builds?
Yes, but I have less than a dozen tests which fail if the memory consumption is significantly different.
I have more test which fail if the execution time is siginificantly different.
Rather than use the setUp()/tearDown() approach, I use the testMethod() as a wrapper for the real test and add the check inside it. This is useful as different test will use different amounts of memory.
> Plus, I did not understand your suggestion, can you elaborate?
> - I first assumed you meant freeMemory(), which, as
> you suggest, is not accurate, since it returns "an
> approximation of [available memory]"
freeMemory gives the free memory from the total. The total can change so you need to take the total - free as the memory used.
> - I re-read it and now assume you do mean
> totalMemory(), which unfortunately will grow only
> when more memory than the initial heap setting is
> needed.
more memory is needed when more memory is used. Unless your test uses a significant amount of memory there is no way to measure it reliably. i.e. if a GC is perform during a test, you can have the test appear to use less memory than it consumes.
> - Eventually, I may need to inlcude calls to
> System.gc() but I seem to remember it is best-effort
> only (endless discussion) and may not help accuracy.
if you do a System.gc(); followed by a Thread.yield() at the start it can improve things marginally.
You can download a trial of JProbe for free. JProbe can profile your memory
Reviving this thread as there have been some similar questions lately...
> You can download a trial of JProbe for free. JProbe can profile your memory
Yes, but as far as I've read, it gives info on whole JVM run, not sure how I can find out the breaking-up (of memory consumed) between the various TestSuite classes.
What I need is more something that could output some text or Excel file (2 columns: test suite name, memory consumed), which I could consolidate in a master Excel file to watch evolution over the days.
Thanks Peter for your advices.
>> Have you actually used your suggestion to automate
>> memory consumption measurement as part of daily builds?
>Yes, but I have less than a dozen tests which fail if the memory
>consumption is significantly different.
I didn't plan to fail the tests depending on memory usage : hard to set the threshold, which amount is reasonable or not. Plus, existing unit tests, testing code semantics, already exercise just the code I want, I just want additional info over them.
I merely wanted to log the consumption, and monitor evolution across succesive nightly builds.
> Rather than use the setUp()/tearDown() approach, I use the testMethod() as a wrapper for the real test and add the check inside it. This is useful as different test will use different amounts of memory.
Yeah right.
Indeed I think I just need a coarser breakdown of memory consumption (e.g. per TestSuite), I seem to remember some utility class in some jUnit version lets you specify suite-level pre- and post- processing.
> freeMemory gives the free memory from the total. The total can change so > you need to take the total - free as the memory used.
Right!
> more memory is needed when more memory is used. Unless your test
> uses a significant amount of memory there is no way to measure it
> reliably. i.e. if a GC is perform during a test,
> you can have the test appear to use less memory than it consumes.
> if you do a System.gc(); followed by a Thread.yield() at the start
> it can improve things marginally.
Those uncertainties deterred me to go further for the time being...
We currently stick to stress&load tests on the whole application, that currently exhibit enough memory problems to keep the experts busy!
JProbe/OptimizeIt then come handy at this step.
But these are costly tests, and costly investigations, and come late in the release cycles; I'm sure we could prevent some issues and save time and money, by automating memory measurements at the unit-test level.
Switching project now, I'll come back to this in another life...
