Running regression tests in batch mode

[James Kanze]

How do you get a memory fault to simply return a bad return code
under Windows?

I'm interested in writing regression test programs which run in
batch mode. At present, if my code has an assertion failure, I
get a pop-up window with an offer to go to the debugger---this
will be worthless on the remote machine (and is a pain even on
my machine, if I happen to hit a sequence of tests which each
trigger the same error).

I'm current calling
SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX);
as the first thing in main, as per
http://msdn.microsoft.com/en-us/library/5z4bw5h5.aspx, but this
has absolutely no effect on the program; I still get my pop-up
with "Debug Assertion Failed!", "Expression: ITERATOR LIST
CORRUPTED!" and "(Press Retry to debug the application)", with
three choices: Abort, Retry or Ignore. This is absolutely
useless for regression tests.

I've also tried converting the structured exception to a C++
exception, as per
http://msdn.microsoft.com/en-us/library/ms680621%28VS.85%29.aspx,
and catching the exception to return EXIT_FAILURE, with no
success either.

So how do you run a suite of regression tests under Windows?

--
James Kanze

[Bruce Wheeler]

On Mon, 15 Feb 2010 02:42:28 -0800 (PST), James Kanze

Post by James Kanze
How do you get a memory fault to simply return a bad return code
under Windows?
I'm interested in writing regression test programs which run in
batch mode. At present, if my code has an assertion failure, I
get a pop-up window with an offer to go to the debugger---this
will be worthless on the remote machine (and is a pain even on
my machine, if I happen to hit a sequence of tests which each
trigger the same error).

The dialog box is called from abort(), which is called by assert().
Maybe there's a way to change this behavior, but I'm not aware of a
way.

A kludge for the assert problem could be something like

void myAssert(int expression)
{
/* text output and possible other processing */
exit(EXIT_FAILURE);
}

and call myAssert() instead of assert(). Note that exit(n) exits with
status n and does not create the dialog box.

(snip)

Post by James Kanze
I've also tried converting the structured exception to a C++
exception, as per
http://msdn.microsoft.com/en-us/library/ms680621%28VS.85%29.aspx,
and catching the exception to return EXIT_FAILURE, with no
success either.

That should probably work. Maybe you aren't catching the exception
properly, and terminate() is being called. Note that terminate() also
calls abort() by default. You can change this by calling
set_terminate() with a function which exits instead of aborting. See
http://msdn.microsoft.com/en-us/library/k0k9c522.aspx
for an example of this.

Post by James Kanze
So how do you run a suite of regression tests under Windows?

myAssert() above would be a possibility if you are only trying to
capture asserts. I tend to just print an error message and call
exit(n) directly. Note that n can (nonportably) have values other than
EXIT_SUCCESS and EXIT_FAILURE under Windows, so a batch file can
branch based on different types of error returns.

Regards,
Bruce Wheeler

[James Kanze]

Post by Bruce Wheeler
On Mon, 15 Feb 2010 02:42:28 -0800 (PST), James Kanze

Post by James Kanze
How do you get a memory fault to simply return a bad return
code under Windows?
I'm interested in writing regression test programs which run
in batch mode. At present, if my code has an assertion
failure, I get a pop-up window with an offer to go to the
debugger---this will be worthless on the remote machine (and
is a pain even on my machine, if I happen to hit a sequence
of tests which each trigger the same error).

The dialog box is called from abort(), which is called by
assert(). Maybe there's a way to change this behavior, but
I'm not aware of a way.

I would expect abort() to raise a signal, as specified in the C
and C++ standards, and that it was the default signal handler
which called up the dialog box, or the system, when a signal is
uncaught.

Post by Bruce Wheeler
A kludge for the assert problem could be something like
void myAssert(int expression)
{
/* text output and possible other processing */
exit(EXIT_FAILURE);
}
and call myAssert() instead of assert(). Note that exit(n)
exits with status n and does not create the dialog box.

The problem isn't just assert---it's anything which would have
caused the program to crash. If a program crashes during a
regression test (which is run as a batch process, often
overnight), I would expect only a specific error status, and to
be able to go on to the next test.

Post by Bruce Wheeler
(snip)

Post by James Kanze
I've also tried converting the structured exception to a C++
exception, as per
http://msdn.microsoft.com/en-us/library/ms680621%28VS.85%29.aspx,
and catching the exception to return EXIT_FAILURE, with no
success either.

That should probably work. Maybe you aren't catching the
exception properly, and terminate() is being called. Note that
terminate() also calls abort() by default. You can change this
by calling set_terminate() with a function which exits instead
of aborting.
See http://msdn.microsoft.com/en-us/library/k0k9c522.aspx for
an example of this.

I know how to change the behavior in case of an uncaught
exception. I intentionally made my exception derive from
std::runtime_error, and caught std::exception, so that I'd fall
into the usual cases.

Anyway, it seems to be working except for assert, so possibly
changing the assert macro would do the trick. (If I find the
original code which caused the problems, I'll repost. And it
may depend on the compiler options.)

Post by Bruce Wheeler

Post by James Kanze
So how do you run a suite of regression tests under Windows?

myAssert() above would be a possibility if you are only trying
to capture asserts.

Asserts are just part of the issue. I'm concerned about
anything which might cause the program to crash.

Post by Bruce Wheeler
I tend to just print an error message and
call exit(n) directly. Note that n can (nonportably) have
values other than EXIT_SUCCESS and EXIT_FAILURE under Windows,
so a batch file can branch based on different types of error
returns.

I know. I use this in my actual code (since I'm only concerned
with portability between Windows and various Unix).

--
James Kanze

[Igor Tandetnik]

Post by James Kanze
I'm current calling
SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX);
as the first thing in main, as per
http://msdn.microsoft.com/en-us/library/5z4bw5h5.aspx, but this
has absolutely no effect on the program; I still get my pop-up
with "Debug Assertion Failed!", "Expression: ITERATOR LIST
CORRUPTED!" and "(Press Retry to debug the application)", with
three choices: Abort, Retry or Ignore. This is absolutely
useless for regression tests.

Try _CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_DEBUG)

[Bruce Wheeler]

On Tue, 16 Feb 2010 07:45:33 -0500, "Igor Tandetnik"

Post by Igor Tandetnik

Post by James Kanze
I'm current calling
SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX);
as the first thing in main, as per
http://msdn.microsoft.com/en-us/library/5z4bw5h5.aspx, but this
has absolutely no effect on the program; I still get my pop-up
with "Debug Assertion Failed!", "Expression: ITERATOR LIST
CORRUPTED!" and "(Press Retry to debug the application)", with
three choices: Abort, Retry or Ignore. This is absolutely
useless for regression tests.

Try _CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_DEBUG)

On my system (I use both VS 2008 and VC6), that doesn't work for a
simple test program which asserts.
However, this does:
_CrtSetReportMode(_CRT_ERROR, _CRTDBG_MODE_DEBUG);

Test program
---------------
#include <assert.h>
#include <crtdbg.h>

int main(int argc, char* argv[])
{
_CrtSetReportMode(_CRT_ERROR, _CRTDBG_MODE_DEBUG);
//_CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_DEBUG);
assert(false);
return 0;
}
---------------

Thanks for the tip. It looks like a bug in VC++ to me that your
suggestion doesn't work.

Regards,
Bruce Wheeler