Project

General

Profile

Feature #3277

Allow testing feature that is partly implemented

Added by Artem Zhmurov 6 months ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Difficulty:
uncategorized
Close

Description

Although feature completeness is a good target to have, it is often beneficial to test something that is under development and currently works only in limited number of cases. Current regression tests can deal with that by browsing through the output file in search of a special phrase, that tells the testing framework that this particular case is not supported. This is a work-a-round, rather than a proper solution. Integration tests don't have any solution for such a problem. Consequently, when new feature is developed, we have to either deal with failing tests of force the code to follow old path in yet-not-supported cases. Both solutions are not ideal. Having test failures requires checking what exactly failed, to ensure that no new bugs were introduced. Using old code path in not supported cases means that the user is not firmly informed that the selected options are not valid. One possible better solution would be to throw special exception and made the testing frameworks aware of it, thus not failing the tests even though the execution was halted.


Related issues

Related to GROMACS - Feature #3328: Testing framework for task assignmentNew

History

#1 Updated by Mark Abraham 6 months ago

Wrapping the calls to gmx_mdrun() to catch a newly created ImplementationIncompleteError exception would be a reasonable. It should be distinct from the current NotImplementedError, because we need an exception that does alert developers to a logic issue (much like an assertion does). But perhaps no new type is needed and the harness can inspect the message inside the exception object for clues on whether to fail or pass the test.

#2 Updated by Mark Abraham 6 months ago

That's one of the strengths of exception based error handling, that the binary doesn't have to terminate (and the test harness doesn't have to fork a new process to cater for the possibility of such failure).

It would also cause problems where test runs did not release resources that are not yet handled in RAII style. But those can probably be probed with LeakSanitizer (running as part of ASAN, but not yet used with GPU builds...)

#3 Updated by Artem Zhmurov 6 months ago

Mark Abraham wrote:

It would also cause problems where test runs did not release resources that are not yet handled in RAII style. But those can probably be probed with LeakSanitizer (running as part of ASAN, but not yet used with GPU builds...)

Good point. Didn't think of that.

Can we just add a key-word suffix (e.g. "Feature not supported") to all messages that should terminate the execution without failing tests? This way no new exception is needed. Though having separate exception for this case might be better.

#4 Updated by Mark Abraham 6 months ago

Artem Zhmurov wrote:

Mark Abraham wrote:

It would also cause problems where test runs did not release resources that are not yet handled in RAII style. But those can probably be probed with LeakSanitizer (running as part of ASAN, but not yet used with GPU builds...)

Good point. Didn't think of that.

Can we just add a key-word suffix (e.g. "Feature not supported") to all messages that should terminate the execution without failing tests? This way no new exception is needed. Though having separate exception for this case might be better.

Yeah but more like "Feature implementation incomplete" so that we differentiate cases were nobody plans to extend the implementation

#5 Updated by Paul Bauer 6 months ago

  • Related to Feature #3328: Testing framework for task assignment added

Also available in: Atom PDF