Project

General

Profile

Feature #2090

redirecting stdout or stderr for testing

Added by Mark Abraham almost 4 years ago. Updated almost 4 years ago.

Status:
New
Priority:
Normal
Assignee:
Category:
testing
Target version:
-
Difficulty:
uncategorized
Close

Description

Getting some kinds of tools (particularly) under test will likely need some crude testing infrastructure so that we can refactor infrastructure to be testable and have solid error handling, and do so with confidence that we aren't generating bugs (e.g. as I did when fixing #1926). Running the tool in the test harness and observing no segfault is useful, but limited. Particualrly if some of the primary output goes to the terminal, one ought to be able to capture it to assert upon - gmx dump is an obvious candidate. One should also be able to test at a lower level than "run the tool from the command line and redirect stdout to file," but it can be reasonable to start at that high level.

One strategy is to do an initial refactoring to write output to a stream that is passed in, rather than hard-code stdout or stderr like we do in some places. In testing, passing in a file stream that the test harness reads back is quite workable, but would likely be too slow to use over wide ranges of tools and their parameter spaces. (Although running such tests only post-commit would help with that.)

An alternative implementation of that strategy is to use a memory stream, e.g. via POSIX open_memstream (http://pubs.opengroup.org/onlinepubs/9699919799/functions/open_memstream.html). This works pretty well for testing gmx dump (code not yet on gerrit). But while GNU libc implements it, none of Windows, Mac or BSD support such a function, so either such tests would need to be disabled, or a backup implementation of open_memstream provided. That's not a trivial proposition, because FILE is an opaque type. Mac and BSD can support an implementation of open_memstream via the BSD extension funopen (and there's MIT-licensed code out there http://piumarta.com/software/memstream, which we could bundle, and which works well for me on Mac). It's also not great to support such things in CMake - if open_memstream or funopen is found, then you can set up a COMPILE_DEFINITIONS for test code to define GMX_USE_OPEN_MEMSTREAM. src/testutils/memopen.{h,cpp} can be set up to do the right thing. Test code needs to disable tests where the infrastructure doesn't support it, but Windows builds on Jenkins will keep that working properly. Having Mac support so people can generate test reference data on Mac laptops seems useful to me (but individual developers could probably arrange to link to memstream library for that convenience).

Questions:
1) Do we care that it's likely we'd never support that on Windows?
2) Do we care enough about developer convenience on Mac to accept ~170 lines of external code to support such tests on Mac and BSD?
3) Do we want to implement CMakery for src/testutils/config.h (and checks like we have for src/config.h) for clarity now, and possible future convenience? (It probably isn't suitable to try to use that for passing GMOCK_COMPILE_DEFINITIONS, though)
4) Is this all too hard and the test harness should just write a temporary file and accept that we run such tests infrequently?

History

#1 Updated by Mark Abraham almost 4 years ago

Mark thinks
1) No
2) Yes (and can do the work)
3) Probably not
4) Acceptable, but we can do better for moderate cost

#2 Updated by Aleksei Iupinov almost 4 years ago

1) No
2) Yes
3) Not me, but would be nice :-)
4) How does "infrequently" factor in here?

#3 Updated by Eric Irrgang almost 4 years ago

Is the refactoring you are considering in the test harness or in the gmx tools?

#4 Updated by Mark Abraham almost 4 years ago

Aleksei Iupinov wrote:

1) No
2) Yes
3) Not me, but would be nice :-)
4) How does "infrequently" factor in here?

We have already agreed that we'll run a small set of Jenkins configurations that must pass before a gerrit change is submitted. But a broader set, perhaps of tests that run for longer will trigger only either on demand or after submission (or even only before releases, for very costly tests). This compromises verification time, test configuration coverage and sysadmin time cost.

#5 Updated by Mark Abraham almost 4 years ago

Eric Irrgang wrote:

Is the refactoring you are considering in the test harness or in the gmx tools?

The tool. Most old school tools use FILE pointers, so one can alter functions so that the normal code passes e.g. stdout in and nothing changes. But a test can pass a memory stream, to either catch or suppress the output, and perhaps assert upon it.

Also available in: Atom PDF