Modena OPT++ Suite


Overview



Benefits


Testing Methodology

Upgrades/Evaluation

Customers

Latest release

Performance testing for C++ optimizers

Product overview. The OPT++ Suite tests whether C++ compiler optimziers properly perform various optimizations.

The optimizations checked in version 2.0 of the suite include "traditional" optimizations such as common subexpression elimination, dead code removal, forward substitution, expression simplification, etc., as well as C++ specific optimizations such as return-value optimization, elimination of virtual function calls and copy optimization.

Initial results with various C++ compilers have shown that several C++ compilers fail to optimize several simple code-sequences, especially in the presence of objects that arenot built-in types. The suite also checks each basic "test" for several built-in as well as user-defined data types, allowing users to determine several optimizations which pass for some data types and fail for several other data types.

The OPT++ Suite (like all other test suites from Modena), is set up so that each test includes only one header file and links in with one object file. Each test is completely self-contained. A bourne shell script is supplied to run all of the tests. Individual tests can be run in "standalone" mode by writing a trivial script.


Benefits. The OPT++ Suite can be very effectively used to tune the performance of C++ optimizers. It allows users to pin-point cases in which the optimizer fails to perform common C++ optimizations. The suite also checks C++ optimizers for correctness, by ensuring that and optimized function returns the same value as the unoptimized version of the function.


Testing Methodology. The organization of each test in the OPT++ Suite is explained below.

Each test includes two kernels : a basic test kernel and a hand-optimized source code kernel. The test kernel is run for time_limit seconds, and the number of iterations it takes to get to complete time_limit seconds is recorded. The hand-optimized kernel is then run for the same number of iterations as the test kernel. The time taken is recorded.

Test failure is determined as follows:

if (testKernelTime > (optimization_factor * optTime) then

<<< Assume that the test has failed >>>

else

<<< Assume that the test has passed >>>

Optimization_factor and time_limit are constants that are set in the file "opthdr.h", and can be modified by the user. Optimization_factor should always be a real number greater than 1. For example, if optimization_factor is 1.2, and testKernelTime = 1.0 and optTime = 0.9, then, since 1.0 < (0.9 * 1.2), we assume that the test passes. i.e. the compiler executed the non-optimal test kernel within a time of 20% of the hand-optimized test kernel.

By varying the time_limit and optimization_factor, users can experiment with different runs of the suite. Users can tell the percentage of inefficiency of performance by determining the largest optimization_factor for which a test just passes.

All tests in the OPT++ suite are completely "templatized". i.e., they make extensive use of the C++ class and function template features. Basing the tests on templates makes it much easier to instantiate the tests for various built-in and user-defined types, and exposes several more optimizer bugs.

Maintenance, Upgrades and Evaluation. We back up our software with outstanding support. We try to solve any problems within hours or days. When you call us or send us email, you get one of the developers of the software, who will work with you to solve problems, whether they are yours or ours. We work with customers and welcome their suggessions to continuously improve the quality and usefulness of our product.

Updates to test sources typically include seveal new optimization tests, together with bug fixes. Source updates are made one or two times per year.

Evaluation. Prospective customers with a serious interest in the OPT++ Suite are allowed to examine the complete suite for a period of 30 days. A simple software confidentiality agreement needs to be signed for this purpose.

Please contact us via phone or email to obtain the evaluation NDA.


Customers. The Test++ Suite has already been licensed by several software and computer systems companies. These include

IBM

Silicon Graphics

Siemens-Nixdorf, AG

and several other systems and software companies. Our client list continues to grow, as more companies realize the importance of C++ performance testing.

Latest release. Release 2.0 of the OPT++ suite was made in May 1997. The next major release is scheduled for spring 1998.