GCL 2.6.2 tests

GCL 2.6.2 RELEASE NOTES

The GCL team is happy to announce the release of version 2.6.2, the latest achievement in the 'stable' series. While strictly speaking a bug-fix only release, 2.6.2 incorporates several major improvements over the last stable release, 2.5.3. Some highlights:

The development of a 'lisp compiler torture tester' by GCL developer Paul Dietz which repeatedly compiles randomly generated forms of specifiable length to test the compiler for correctness.
The application of several significant corrections to the GCL lisp compiler to remove every known instance of miscompilation uncovered by this tester. To our knowledge, GCL is alone with CLISP in passing this torture test for runs of effectively indefinite length.
Major performance improvements were applied to the lisp compiler to enable it to complete random tests of great length in a reasonable amount of time.
Corrections to the GCL core files to enable very large image sizes in 64 bits, in which more than a billion cons cells can be allocated. Current 64bit options include amd64, ia64, and alpha running most flavors of GNU/Linux.
Corrections to the heap scaling behavior of the garbage collector, resulting in significant performance gains in many instances.
Support for the latest gcc and binutils versions on all platforms but mingw
The elimination of many instances of unnecessary internal garbage generation bringing the associated performance gains
Native support for execstack protected linux kernels, such as on Fedora core systems
Native support for FreeBSD, OpenBSD, and MacOSX
Static function pointer support to stabilize dynamic library usage on Itanium systems
Transparent readline initialization
Support for profiling via gprof
Automatic disabling of SGC (stratified garbage collection) if the image is executed on a kernel not supporting fault address recovery
Remove a memory leak associated with heavy bignum usage via the introduction of SGC contiguous pages
Several significant internal bug fixes, epecially in the mingw port.
Alter the build process to perform a full self compile with full function proclamation at build time.
GCL now compiles Axiom from scratch and carries it to all supported platforms with the current exception of mingw
GCL's ANSI build now in use for its first end-user application -- maxima (current cvs)
New 64bit platform support -- amd64, with full native object relocation

The full changelog can be found in the source tree in the file 'debian/changelog'.

The GCL team has subjected this release to a wide variety of tests and benchmarks. While all such results are necessarily incomplete, one can nevertheless usefully summarize the approximate state of affairs as follows:

GCL is about as portable as CLISP
The GCL lisp compiler is about as robust/correct as that of CLISP, at least as measured by the random tester, which at present only covers a mostly integer subset of lisp.
GCL is about as fast as CMUCL
GCL plays a major role in carrying the primary large open source lisp end user applications to a wide variety of systems
GCL is still the least ANSI compliant of the freely available lisp systems, though a modest level of compliance has been achieved in this release. Much greater compliance has been achieved in the 2.7.x (cvs unstable) series yet to be officially released.

The specific test results are arranged in the following table. Some terms need defining:

BFD
the method of relocating compiled lisp object modules into the running executable using the BFD library

custreloc
the method of relocating compiled lisp object modules into the running executable using the native GCL code. This method as well as the BFD method preserve the module loading across image saving and re-execution

dlopen
the method of dynamically linking in compiled lisp object modules into the existing session only via the system dynamic linker loader, ld.so.

SGC
Stratified Garbage Collection -- an optional accelerated generational garbage collection algorithm employing read-only memory

CLtL1
Common Lisp, the Language vol I, referring to the book of the same name by Steele defining a widely used lisp language standard prior to the ANSI standardization process in 1994.

ANSI
the work in progress image build attempting to eventually extend traditional GCL into full ANSI complaince

Ansi tests
the results of the work in progress ansi compliance test suite written by GCL developer Paul Dietz presented as the number of failures divided by the total number of tests run

Random tests
the results of the random 'compiler torture tester' presented as the number of tests/the size of the random forms/the number of variables passed to the random function

In the table below, green denotes a pass, yellow denotes an as yet unimplemented option, and red indicates failure. Blank cells indicate tests that have not been run.

System	CPU	Self Build	Preferred Linking	SGC	ANSI tests	Random tests	Axiom CVS/CLtL1	pc-nqthm CLtL1
Debian GNU/Linux (sid)	i386		bfd or custreloc		303/ 10697	50000/10000/8 500000/1000/8		(setq si::multiply-stacks 16)
Debian GNU/Linux (sid)	sparc		bfd or custreloc		303/ 10697
Debian GNU/Linux (sid)	powerpc		bfd		303/ 10697
Debian GNU/Linux (sid)	amd64		bfd		303/ 10697
Debian GNU/Linux (sid)	arm		bfd		303/ 10697
Debian GNU/Linux (sid)	m68k		bfd		303/ 10697
Debian GNU/Linux (sid)	s390		bfd		303/ 10697
Debian GNU/Linux (sid)	ia64		dlopen		303/ 10697		(1)
Debian GNU/Linux (sid)	hppa	-O0	dlopen		303/ 10697		(1)
Debian GNU/Linux (sid)	mips		dlopen		303/ 10697		(1)
Debian GNU/Linux (sid)	mipsel		dlopen		303/ 10697		(1)
Debian GNU/Linux (sid)	alpha		dlopen		303/ 10697		(1)
Fedora FC1	i386		bfd or custreloc		303/ 10697	12000/1000/8
Solaris	sparc		bfd or custreloc		303/ 10697	4000/1000/8 (4)
Windows MINGW(a)	i386		custreloc		303/ 10697	57000/1000/8	(2)
MacOSX	powerpc		bfd	(3)	303/ 10697
OpenBSD	i386		bfd		303/ 10697
FreeBSD	i386		custreloc		303/ 10697

Notes:

(1) dlopen builds use file descriptors for each object load. The step in the Axiom build process which regenerates its databases consumes more than the conventional maximum of 1024 file descriptors available by default on most UNIX systems.
(2) An AXIOMsys executable can be produced, and is basically functional, but experiences sporadic errors of a type as yet unknown.
(3) This problem has been recently and inadvertently introduced. We are looking into a fix, which should be straightforward.
(4) On this machine, the underlying gcc was old (3.0) and segfaulted outside of GCL when attempting to compile its produced C code after a few thousand iterations.

(a) The preferred build environment for Mingw/Windows is gcc 3.3.1, binutils 2.14.90, and the latest msys release.

The following table presents the results of the popular gabriel benchmarks of three freely available lisp systems, GCL, CLISP and CMUCL. Times are presented as multiples of the time GCL took in completing the tests. Green indicates tests on which GCL is the fastest, while red indicates tests on which GCLwas not the fastest. The benchmark code can be found in ftp://ftp.ma.utexas.edu/gcl/gabriel.tgz. The number of test iterations has been increased by a factor of 400 to overcome granularity issues on modern machines. The '(print (time ...))' statements around each test iteration were removed, again due to granularity and relative i/o load. Likewise the special init.lsp file conventionally used to preallocate GCL memory in such cases was removed as it is now mostly obsolete. Finally the tests were modified slightly to place the optimization declamations at the top of each file being compiled as suggested by a CMUCL expert.

As with any benchmark, results can vary somewhat with the details of the executing machine. With lisp in particular, the ratios of the cache sizes, cpu speed, and memory bandwidths can impact such tests significantly. We present the results for two popular configurations below. While the precise details of the differences are not as yet known, it is speculated that the first result is more dominated by in-cache cpu performance, while the latter is more dominated by memory access efficiency.

Dual Intel Xeon 2.4Ghz, 512 Mb, Linux 2.4.20	Athlon XP 3000+ (2.1Ghz), 512 Mb, Linux 2.4.26
Benchmark GCL 2.6.2 CMUCL 18e-9 CLISP 2.33-2 BOYER 1.000 2.200 9.869 BROWSE 1.000 2.240 NA CTAK 1.000 0.230 1.890 DDERIV 1.000 2.148 2.909 DERIV 1.000 2.083 3.640 DESTRU-MOD 1.000 2.043 9.880 DESTRU 1.000 1.168 5.743 DIV2 1.000 2.222 3.911 FFT-MOD 1.000 1.585 206.057 FFT 1.000 1.544 176.088 FPRINT 1.000 2.136 3.742 FREAD 1.000 1.746 2.111 FRPOLY 1.000 1.524 5.112 PUZZLE-MOD 1.000 10.824 41.618 PUZZLE 1.000 11.324 37.671 STAK 1.000 1.536 9.836 TAK-MOD 1.000 1.465 15.053 TAK 1.000 1.486 14.629 TAKL 1.000 1.419 14.965 TAKR 1.000 1.933 12.327 TPRINT 1.000 0.937 1.263 TRAVERSE 1.000 0.875 8.378 TRIANG-MOD 1.000 7.067 26.814 TRIANG 1.000 1.281 18.565 GEOMETRIC AVERAGE 1.00 1.86 10.33 MEDIAN 1.00 1.67 9.87	Benchmark GCL 2.6.2 CMUCL 18e CLISP 2.33 BOYER 1.000 0.892 6.316 BROWSE 1.000 0.965 NA CTAK 1.000 0.435 3.489 DDERIV 1.000 0.822 1.579 DERIV 1.000 0.651 1.639 DESTRU-MOD 1.000 0.812 4.779 DESTRU 1.000 0.550 3.239 DIV2 1.000 0.599 1.525 FFT-MOD 1.000 2.655 337.207 FFT 1.000 1.923 251.026 FPRINT 1.000 2.322 3.508 FREAD 1.000 1.890 1.900 FRPOLY 1.000 1.013 3.606 PUZZLE-MOD 1.000 5.976 20.350 PUZZLE 1.000 5.472 19.387 STAK 1.000 1.655 8.064 TAK-MOD 1.000 1.382 14.775 TAK 1.000 1.399 14.514 TAKL 1.000 1.281 12.877 TAKR 1.000 1.735 15.500 TPRINT 1.000 2.008 1.674 TRAVERSE 1.000 0.770 8.013 TRIANG-MOD 1.000 6.639 25.182 TRIANG 1.000 1.186 16.948 GEOMETRIC AVERAGE 1.00 1.40 8.46 MEDIAN 1.00 1.33 8.01

Many improvements are planned for the 2.7.x development series time permitting, the most important of which is to complete the task of building an ANSI compliant GCL image.