The benchmark programs were placed in a test harness which consisted of a loop with a certain number of iterations. The number of iterations was adjusted so that the execution time was approximately 60 s using a DEC (Digital Equipment Corporation) Ada compiler on a MicroVAX   machine for each benchmark program.

This was done to minimize any measurement artifacts due to stopwatch measurements as well as to minimize the noise introduced for executing the loop statements themselves. We accepted timing measurements using the stopwatch technique since we wanted to eliminate all U0 statements as well as the timing statements of Ada which could vary considerably.

These benchmark programs were sent in October 1987 to Ada compiler vendors who executed them on various 16- and 32-bit commercially available microprocessors. Along with the benchmark programs, we sent a questionnaire to assess the host and the target environment. Questions on the host environment included the model number, operating system information, etc. The target environment questions included information on the microprocessor, single board computer used, memory wait states, floating point coprocessor used, etc. Finally, the execution time and the memory required for each of these programs was to be recorded.

Compiler vendors were asked to optimize for minimum execution time.

A total of eight Ada compiler vendors submitted results on thirteen different compilers targeted to six commercial microprocessors.

Table 1 shows the information regarding the target environment

used for each of these compilers.

The eight compiler vendors. Most of the compiler vendors used a stand-alone single board computer or a "bare" machine. All targets used a floating-point coprocessor, which is essential for comparing performances of those benchmarks that are floating-point intensive. Note that the Mil-Std-1750A processors have floating point instructions; hence, a separate coprocessor is not required.

The clock rates for these target machines were different. In order to normalize this fundamental variable in our results, we multiplied the execution time (in seconds) by the clock rate (in cycles/second) to obtain the total number of cycles. The memory wait states were different, too. No attempt has been made to factor in this variable since it would entail finding out for each compiler the number of read and write accesses.

In order to evaluate the maturity of these Ada compilers, we translated the benchmark programs to Pascal and executed them in-house on a standard microprocessor (Motorola 680x0) using a mature optimizing Pascal compiler (Domain Pascal compiler Rev.

7.54). All our results are shown as normalized with respect to Pascal. For the 32-bit microprocessors (68020, 80386), we used the 68020 as the benchmark processor whereas for the 16/32-bit processors (80286,68010, 80186, 1750A), we used the 68010 as the baseline benchmark processor for comparison purposes.

Figure 1 shows the total execution performance of the six benchmark programs relative to Pascal. These six benchmark programs represented the Pascal subset and did not have special Ada features like generics or exception handling. Hence, comparison with respect to Pascal is very realistic. The x axis represents the various Ada compilers. The y axis is the normalized execution performance and was obtained as follows. The total execution time for the six benchmark programs was summed. The sum was then multiplied by the respective clock frequency to obtain the total number of machine cycles. A similar calculation was done for the Pascal version and was used to normalize the Ada results. Hence, the y axis represents the normalized execution performance as a percent of Pascal. Note that in this figure and others that follow, less is better, i.e., if the percent execution time is less than the 100% mark, then that Ada compiler generated more efficient code compared to Pascal. As is evident from Fig. 1, several compilers targeted to various microprocessors have performance that is equal to or better than the Pascal compiler.

In order to evaluate the specific Ada feature of exception handling, two of the eight programs were identical except that one of them included the Ada exception feature and the other had a test and goto construct. Figure 2 shows the execution perfor- mance relative to each other and with respect to Pascal. As shown in this figure, some of the compilers (A323, A325, A162, and A164) incurred a significant penalty for using this Ada feature. On the other hand, some others (A321, A322, A326. A161. And A172) had little or no penalty. Finally. three of the compilers

(A324, A163, and A173) exhibited a somewhat better performance when this feature was used.

Figure 3 shows the effect of using generics in Ada for integer matrix multiplication. One of the compilers (A325) generated code which looped forever. It appears that there is some overhead (compared to Pascal, which has no generics) in using this feature although it is very compiler dependent. Figure 4 shows the amount of application-specific code (excluding the run-time system) generated by these compilers relative to that of Pascal for the six benchmark programs. Some of these compilers generated a huge amount of code which could incur excessive costs for storing these programs. In fact, some of these compilers (A321, A324, and A164) generate efficient code (see Fig. 1) but not compact code. However, there are some Ada compilers (A322, A325, and A162) which produce efficient as well as compact code.

In order to better understand the time evolution of these compilers, we sent out the same benchmarks again in October 1988.