On 5/21/11 5:25 PM, David Laight wrote:
On Sat, May 21, 2011 at 05:14:07PM +0200, Manuel Bouyer wrote:
On Sat, May 21, 2011 at 03:34:22PM +0100, Julio Merino wrote:
Yes, the timeout thing is broken. It should really be a specification of the test case size (e.g. 'small', 'large') and allow the user to define his timeout preferences for every class, because they will vary from machine to machine.
Sure. I ran ATF on a machine with a 1Mhz CPU clock (this is a simulator, the real hardware is expected to run faster :). Lots of tests timed out just because on such hardware they can't complete in less than 5mn.
Perhaps the timeout(s) should be settable units that default
to 1 second.
Them the timeouts can be scaled for very slow (or fast) systems.
Perhaps. But timeouts in a measure of time do not make sense in this context. There won't be any way to scale them in a reliable manner, so this will lead to flaky tests. And anyway, the user will be required to have control over how long these deadlines are.
Again, there is no point in setting test timeouts to 1 second (except for very very rare cases). Timeouts are only used to kill tests that get stuck; nothing should rely on them. If there is many timeouts in your test suite (which will result in failed tests), something is broken and needs fixing.
-- Julio Merino / @jmmv