News Course Articles

Performance: Java vs C vs Limbo

Summary
It has been lack of quantitative comparison among different programming languages. In this article, we use some benchmark programs to measure performance of C/C++, Java and Limbo. We will look into some built-in language features, such as file access and network connections. We hope this can shed some lights on where both Java and Limbo can be further improved.

Ever since Java emerges as a mainstream language, there has been all sorts of complaints about its dissappointing performance. There have been some serious efforts in measuring the performance of different Java virtual machines, and, of course, a lot of bragings from different vendors about how their JVMs can beat their competitors'. However, it has been lack of comparisons among different languages. For example, how bad(or good) Java is when it is compared to C or C++.

Java was originally proposed as programming language for embedded systems. When Java has become a general-purpose language for heavy duty use, it still contends for embedded market. JavaSoft has already come up with Personal Java, Embedded Java and even Java Card specifications and reference implementations. This brings up another performance questions: how does Java measure up against its competitors in embeded market, for example, Limbo, which is poised as Java alternative by Bell Lab.

In this article, we use some benchmarks to compare performance of C/C++, Java and Limbo. We are going to use C, Java and Limbo to implement Linpack and RIPEMD, mainly to measure number crunching capability. We will also investigate some built-in features and libraries to understand overall performance of different languages.

In order to achieve objective comparison, we use the same code base, i.e., developing in one language and then porting to the others. We run the executables compiled under different optimization modes with C, Java and Limbo under same hardware and operating system environments to collect the performance data.

1 Some Basic Facts about Languages:

Both Java and Limbo are interpreted languages and have a clearly defined virtual machines for platform-independence purpose. Both have garbage collection built-in. Both have multi-threading built into VM level. For performance reasons, both have implemented Just-In-Time(JIT) or on-the-fly compilation technology.

Java is an object-oriented. Limbo is not, but sometimes it can be touted as object-based. A programmer who gets used to think in object-oriented way enforced by C++ or Java may find Limbo limits expressive power. Limbo remedies this by including more primitive types, such as list(which is very similar to LISP), string, tuples.

Java's VM is stack based. This has resulted some critisism because it is very hard to generate optimized code. On the contrary, Limbo adopts a memory-to-memory design for its VM. So in theory, Limbo's VM may have some potential performance advantages. Moreover, Limbo has an UNIX-alike shell so that the VM can always be on whileas in Java the VM must be loaded every time it is needed. In practice, loading VM appears to be very time consuming.

In general, Java is a more matured language compared to Limbo. Java's class library has much richer contents than Limbo. In addition, Java has well-thought security architecture. On the other hand, Limbo's VM is much smaller compared to a standard JDK VM. It requires relatively smaller memory and takes less time to load. Therefore, it is more suitable for embedded applications. It can be argued that Personal-Java and Embedded-Java may overcome this shortcoming by cutting down some features.

Java has two different graphics packages: awt for simply applications and swing(JFC) for complex ones. Limbo's GUI toolkit is based on Ousterhout's Tk which is very familiar to a lot of programmers.

In terms of industry support, currently, Java is available on almost every conceivable platform whileas Limbo is ported only to Windows NT, Linux and Solaris. As a viable programming language, we have to see more support from hardware and OS vendors. In short, if Java can be seen as a clean C++, then Limbo can be regarded as clean C.

2 What Tests Do We Conduct?

There are several good benchmark programs to measure floating point performance. The most notable one is Linpack. There are C and Java version of Linpack available. For this purpose, I have ported Linpack from Java to Limbo. However, there are not many simple benchmarks for integer performance. Here we are going to use RIPEMD as a simple program to measure integer performance. RIPEMD is a yet another secure hashing algorithm, as an alternative to MD5 and SHA-1.

There are 3 reasons we choose RIPEMD instead of other more often used algorithms such as SHA-1 or MD5:

  1. RIPEMD involves balanced logic and arithmetic operations which are common to a lot of applications, e.g. cryptography(hashing or symmetric ciphers).
  2. Just like other computationally intensive programs, a lot of C MACROs are used in the original RIPEMD implementation, however, there is no such a feature in Java and Limbo. It is interesting to see how this will affect performance.
  3. The fact that there already exist a lot of implementations of SHA-1 and MD5 in different languages makes doing another one not so exciting.
I also benchmarked file access by using a straight-forward program that I have written solely for testing purpose. The test program simply copies a file from an existing file. It can test how well the underlying native code performs.

The network test is conducted in a similar fashion to file access, i.e., read and write a socket. In addition to the client code, a simple server is also implemented in order to response to the client's requests.

3 Test Results

Both Java and Limbo are interpreted language. However, they support the concept of JIT(Just In Time) or on-the-fly compilations. For each language, we try two modes, optimizaion on and off. For C, we use gnu compiler, gcc and optimization switch "gcc -O4".

For Java, we have tested two different JVMs, JDK1.1.6 which we did not install JIT and JDK1.2 which bundled with a JIT. It should be noted that the JIT bundled with JDK1.2 is the one from SunSoft, not the JavaSoft's soon-to-be-released HotSpot. In Java, we only measure the time elapsed after the JVM is loaded. We did not measure the time to load JVM.

For Limbo, we use Lucent's Inferno SDK-2.0. However, this version of Limbo does not support compiler time optimization. The compiler switch "limbo -c" is mainly to inform the virtual machine to turn on JIT mode, not meant for optimization. It has the same effect as turning on JIT for the interpreter.

All benchmarks are conducted on a very modest hardware, Sun Sparc 5 with a 110Mhz CPU, 64MB RAM running Solaris 2.51.

3.1 The test result for linpack

The C version of Linpack is available from here and the Java version is available from here. I ported Linpack to Limbo. The source code can be downloaded from here, all from netlib.org.

The test result for linpack
Language C
(gcc)		 Java
(JDK1.1.6 No JIT)		 Java
(JDK1.2 with JIT)		 Limbo
(No JIT)		 Limbo
(With JIT)
No Optimization 0.16 0.86 0.16 4.46 0.30
With Optimization 0.07 0.86 0.15 N/A N/A

From the above table, we can see clearly that

  1. JDK1.2 with JIT is about twice as slow as C
  2. optimization does not help much to Java because current "java -O" mainly expand inline code and Linpack does not have much to inline.
  3. JDK with JIT is same as C without optimization.
  4. Limbo lags behind

3.2 The test result for RIPEMD

The RIPEMD tests are conducted based mainly on testsuite switch of RIPEMD-160's reference implemention hashtest. It mainly does message digestion on a byte array with length of 1 mega-bytes. The original C implementation is ported to both Java and Limbo.

Compared to Limbo, Java is closer to C/C++ in terms of primitive types. Limbo does not have single precision type float. The real type is more like double in Java.

The following Java code behaves exactly as what C/C++ programmers could expect. Basically, an integer with radix 16 representation of 0xFFFFFFFF will make it decimal "-1". 

public class testInt {

	public static void main(String[] args) throws Exception {
		int i = 0xFFFFFFFF;
		System.out.println("int i = 0xFFFFFFFF; will make i = " + i);
	}

}
However, Limbo tries to widen 0xFFFFFFFF to become 0x00000000FFFFFFFF and assign it to long type, which is big in Limbo. Even stranger, assigning 16rFFFFFFFFFFFFFFFF to variable bigI will make it a 32 bit integer(int). In order to correctly emulate the C/C++ behavior, a casting is needed. See the Limbo example below and look for variable shortI. 
implement testInt;
include "sys.m";
include "draw.m";

sys: Sys;

testInt: module {
	init : fn(ctxt: ref Draw->Context, argv: list of string);
};

init(ctxt: ref Draw->Context, argv: list of string) {
	sys = load Sys Sys->PATH;
	sys->print("test integer: \n");
	i:=16rFFFFFFFF;
	shortI:= int 16rFFFFFFFF;
	bigI:=16rFFFFFFFFFFFFFFFF;
	sys->print("i:=16rFFFFFFFF; will make i = %bd\n", i);
	sys->print("shortI:= int 16rFFFFFFFF; will make shortI = %d\n", shortI);
	sys->print("bigI:=16rFFFFFFFFFFFFFFFF; will make bigI = %d\n", bigI);
}

In Java, there are two right shift operators: logic right shift >>> and arithmetic right shift >>. However, in Limbo, there is only arithmetic right shift. This puts Limbo in a disadvantageous position when a lot of logic operations occur. For example, in cryptographic applications, cyclic shift is a primitive operation. In C, you can write it as (x << n) | (x>>(32-n)) and in Java, you can write it as (x << n) | (x >>> (32-n)). However, in Limbo, you have to write it as 

 (x << n) | (((x >> 1) & 16r7FFFFFFF)>>(31-n))
Essentially, two more operations are needed in order to accompolish the same thing.

It is not surprising to notice that neither Java nor Limbo support cyclic shift operations even though a lot of microprocessors support that. This is inherited from C/C++.

The test result for RIPEMD-160
Language C
(gcc)		 Java
(JDK1.1.6 No JIT)		 Java
(JDK1.2 with JIT)		 Limbo
(No JIT)		 Limbo
(With JIT)
No Optimization 1.65 17.52 6.05 178.14 18.06
With Optimization 0.33 12.05 1.60 N/A N/A

From about test result, we understand

  1. optimization matters a lot to both C and Java. This is mainly because there are a lot of MACROs need to be inlined.
  2. JDK1.2 is about 3-5 times slower than C.
  3. JDK1.2 with JIT is even faster than C without optimization. Surprising?
  4. Limbo lags behind.

3.3 The test result for file access

In this test, we simply make a file copy operation: read from an existing file(with length about 5MB) and write to another file.

The test result for file access
Language C
(gcc)		 Java
(JDK1.1.6 No JIT)		 Java
(JDK1.2 with JIT)		 Limbo
(No JIT)		 Limbo
(With JIT)
No Optimization 1.79 1.39 2.24 0.63 0.63
With Optimization 1.78 1.36 2.17 N/A N/A
From the about result, we have learned that
  1. Java JIT slows things down
  2. Optimization does NOT gain much performance
  3. Both Java(No JIT) and Limbo outperform C in a significant way. It's not hard to explain: both Java and Limbo use system calls to access file I/O but the C testing code uses fread and fwrite which are much costlier.

3.4 The test result for network

In this test, we implement a network client that can read a stream of data from a simple server running on another machine which is connected with the client machine through a 10BaseT ethernet. After the client receives the data, it then writes back to the server. The simple server firstly reads a 1 MB file and stores the data in its buffer. It writes buffered data out every time there is a client connection. We measure the time elapsed for the client to read and write the data from and to the server.

The test result for network
Language C
(gcc)		 Java
(JDK1.1.6 No JIT)		 Java
(JDK1.2 with JIT)		 Limbo
(No JIT)		 Limbo
(With JIT)
No Optimization 2.92 3.48 2.93 2.57 2.56
With Optimization 2.92 3.47 2.93 N/A N/A
It is clear from the above table that:
  1. Optimization does not gain any performance.
  2. There is no speedup for JIT. The difference between JDK1.1.6 and JDK1.2 is mainly caused by more efficient socket read implementation of JDK1.2. A separate read only test can confirm this.
  3. Limbo outperforms Java by about at least 20%.
  4. Both Java and Limbo are good for network.

4 Conclusions

There is still a gap between interpreted languages(such as Java and Limbo) and C in computationally intensive tasks. However, this gap is becoming smaller and smaller. Today, a well-written and optimized Java code can easily beat a lousy C code.

Compiler-time optimization is very important in certain type of applications. Unfortunetely, Limbo does not support this feature. This makes it very hard to make a simple porting from C without some manual optimization.

Java's file access and network performance is reasonable. But it's still not up to Limbo's amazing speed. Actually, JDK1.2 is worse than JDK1.1 in file access. I do not think it is a difficult task because it is just a matter of fine tuning the underlying native code.

In general, I feel that Java is a matured language and offers reasonable performance compared to C or C++. With time passing by, we can expect better and better techniques to improve VM's performance. I will not be surprised that someday Java can rival or even outperform C. And that day is not far.