Insoftware testing, a test oracle (or just oracle) is a provider of information that describes correct output based on the input of a test case. Testing with an oracle involves comparing actual results of the system under test (SUT) with the expected results as provided by the oracle.[1]
The term "test oracle" was first introduced in a paper by William E. Howden.[2] Additional work on different kinds of oracles was explored by Elaine Weyuker.[3]
An oracle can operate separately from the SUT; accessed at test runtime, or it can be used before a test is run with expected results encoded into the test logic.[4]
However, method postconditions are part of the SUT, as automated oracles in design by contract models.[5]
Determining the correct output for a given input (and a set of program or system states) is known as the oracle problemortest oracle problem,[6]: 507 which some consider a relatively hard problem, and involves working with problems related to controllability and observability.[7]
A research literature survey covering 1978 to 2012[6] found several potential categories of test oracles.
A specified oracle is typically associated with formalized approaches to software modeling and software code construction. It is connected to formal specification,[8] model-based design which may be used to generate test oracles,[9] state transition specification for which oracles can be derived to aid model-based testing[10] and protocol conformance testing,[11] and design by contract for which the equivalent test oracle is an assertion.
Specified test oracles have a number of challenges. Formal specification relies on abstraction, which in turn may naturally have an element of imprecision as all models cannot capture all behavior.[6]: 514
A derived test oracle differentiates correct and incorrect behavior by using information derived from artifacts of the system. These may include documentation, system execution results and characteristics of versions of the SUT.[6]: 514 Regression test suites (or reports) are an example of a derived test oracle - they are built on the assumption that the result from a previous system version can be used as aid (oracle) for a future system version. Previously measured performance characteristics may be used as an oracle for future system versions, for example, to trigger a question about observed potential performance degradation. Textual documentation from previous system versions may be used as a basis to guide expectations in future system versions.
A pseudo-oracle[6]: 515 falls into the category of derived test oracle. A pseudo-oracle, as defined by Weyuker,[12] is a separately written program which can take the same input as the program or SUT so that their outputs may be compared to understand if there might be a problem to investigate.
A partial oracle[6]: 515 is a hybrid between specified test oracle and derived test oracle. It specifies important (but not complete) properties of the SUT. For example, metamorphic testing exploits such properties, called metamorphic relations, across multiple executions of the system.
An implicit test oracle relies on implied information and assumptions.[6]: 518 For example, there may be some implied conclusion from a program crash, i.e. unwanted behavior - an oracle to determine that there may be a problem. There are a number of ways to search and test for unwanted behavior, whether some call it negative testing, where there are specialized subsets such as fuzzing.
There are limitations in implicit test oracles - as they rely on implied conclusions and assumptions. For example, a program or process crash may not be a priority issue if the system is a fault-tolerant system and so operating under a form of self-healing/self-management. Implicit test oracles may be susceptible to false positives due to environment dependencies. Property based testing relies on implicit oracles.
A human can act as a test oracle.[7] This approach can be categorized as quantitative or qualitative.[6]: 519–520 A quantitative approach aims to find the right amount of information to gather on a SUT (e.g., test results) for a stakeholder to be able to make decisions on fit-for-purpose or the release of the software. A qualitative approach aims to find the representativeness and suitability of the input test data and context of the output from the SUT. An example is using realistic and representative test data and making sense of the results (if they are realistic). These can be guided by heuristic approaches, such as gut instincts, rules of thumb, checklist aids, and experience to help tailor the specific combination selected for the SUT.
Test oracles are most commonly based on specifications and documentation.[13][14] A formal specification used as input to model-based design and model-based testing would be an example of a specified test oracle. The model-based oracle uses the same model to generate and verify system behavior.[15] Documentation that is not a full specification of the product, such as a usage or installation guide, or a record of performance characteristics or minimum machine requirements for the software, would typically be a derived test oracle.
A consistency oracle compares the results of one test execution to another for similarity.[16] This is another example of a derived test oracle.
An oracle for a software program might be a second program that uses a different algorithm to evaluate the same mathematical expression as the product under test. This is an example of a pseudo-oracle, which is a derived test oracle.[12]: 466
During Google search, we do not have a complete oracle to verify whether the number of returned results is correct. We may define a metamorphic relation[17] such that a follow-up narrowed-down search will produce fewer results. This is an example of a partial oracle, which is a hybrid between specified test oracle and derived test oracle.
A statistical oracle uses probabilistic characteristics,[18] for example with image analysis where a range of certainty and uncertainty is defined for the test oracle to pronounce a match or otherwise. This would be an example of a quantitative approach in human test oracle.
A heuristic oracle provides representative or approximate results over a class of test inputs.[19] This would be an example of a qualitative approach in human test oracle.
