Things on the site itself that may be of interest to students or philosophers of any age or generation include complete online books of poetry, various support materials for the study of physics, and links related to beowulfery. All materials on this site that are authored by Robert G. Brown are Copyright 2004. The details of their Open Public License (modified) can be viewed here. If you use or enjoy anything at all on this site -- free textbooks, stories, programs, or other resources, consider hitting to help spread the word so others can find it as well. Note, Robert G. Brown is generally either rgb or rgbatduke on many external sites crosslinked here.
Various links and publications of Robert G. Brown that fit under none of the other categories. All of these materials are published under a modified Open Publication License that permits unlimited free noncommercial and personal use. The materials (books, presentations, or otherwise) may not be published in any form or media that is sold for profit. The details of the license can be viewed here and in each available version viewed below. Stats for this page can be viewed here.
Commercial publishers interested in producing an actual book (or other media form) of the material below are encouraged to contact the author.
gflashcard is a program based on GTK and XML for presenting simple flashcards to students in a standard terminal (e.g. xterm) window. Its license (GPL 2b) can be viewed at the bottom of this page. Its current features include:
gflashcard requires libxml2 and Gtk-2.
flashcard requires libxml2 and ncurses.
Back to topflashcard is a program I wrote to help my second grader drill basic spelling and arithmetic, my seventh grader drill Spanish and vocubulary, and more advanced students get immediate feedback on "conceptual" or memory building multiple choice questions.
flashcard is not a game. No imaginary space aliens are harmed as one learns vocabulary or arithmetic. It is a very, very efficient way to get the practice and immediate feedback necessary to memorize a large number of dry, boring factoids. It never gets tired, impatient, angry, or has to cook dinner. It doesn't require lots of little cutout cards or pieces of paper to get lost. Suitably installed on a local area network of student-accessible computers, it can be used as a memorization aid for nearly any sort of factoid and as a relatively weak "deep" learning tool.
Based on my own personal experience so far with it, young kids initially find it at least intriguing (doing flashcards on the computer is kind of neat), then boring and to be resisted, and finally just another daily homework chore to be done as quickly as possible. Older kids progress through these stages more rapidly as they can see that using flashcard to e.g. learn their vocabulary words beats the hell out of trying to do it by just reviewing their handout list or writing each word three times. By the time any student is in the latter stage, the program is generally "working".
I've been quite pleased with the results. My second (now third) grader went from getting an average of 60% or worse on timed arithmetic tests to getting consistent high-90%s (and occasional 100%s!) in about one month of three or four time a week drill.
His spelling is improving as well, but more slowly as he learned to spell phonetically and is a bit stubborn (literally) about the non-phonetic spellings and multiple possible spellings that abownd in English. He initially exhibited a similar stubbornness about memorizing arithmetic versus using his fingers to compute the answer, but repetition and time pressure eventually take their toll and he ended up memorizing the right answers in spite of himself.
One thing that I'm doing that may or may not be the right thing to do (and may be responsible for his initially relatively slow progress) is having him drill on all the second grade core vocabulary words at once, instead of doing a small list every week in phase with his class. In the short run this leaves him behind, as he sees his current vocabulary words only occasionally and mixed in with all many others. However, I expect that very soon, with consistent drill, he will suddenly know the entire list (this is the way it worked with arithmetic) and be not only ahead but done with learning second grade vocabulary half way through the year.
I'd suggest the following strategies for using flashcard for yourself or with your kids:
There is a significant difference between memorization of factoids and true learning. To speak metaphorically, factoids are the many tiny rocks which are cemented together by experience and conceptualization into the edifice of our full understanding. One should never confuse successful memorization of a large body of factoids with real comprehension.
It is entirely possible to comprehend things deeply without memorizing lots of factoids. Mathematics is not arithmetic, although arithmetic is a useful skill that underlies some mathematics. History is not a bunch of events and their associated dates. Language is not a collection of words. Science is not scientific data. The abstract rule is not the many concrete realizations of the rule. One could embark on a long discussion of semantics and epistemology, semiotics and psychology -- and indeed I'm working slowly on a book on these subjects -- but not here. The main point is to recognize that memorization can be a soul-sucking process for a young mind (or an older one!) when unsupported by any sort of reason.
For many of these subjects, of course, memorizing factoids is one essential step in beginning to comprehend the subject. It is difficult to understand American History without knowing when the American Revolution occurred and whether it occurred before or after (say) the American Civil War. It is difficult to read and write clearly and effectively if one's collection of vocabulary factoids is inadequate. For that reason I think flashcard can be a useful component of teaching and learning, but it does not teach anything like real comprehension of a subject, only its associated and foundational factoids, and its only real virtue here is its efficiency -- by drilling those factoids with a tool, one can quickly build up a base of factual knowledge sufficient to be a foundation for deeper learning.
I would therefore recommend that this tool be used ONLY as a factoid memorization tool, and NOT as a classroom "testing" tool or "teaching" tool, although it does have a timed test mode and other things that might be construed or abused into a classroom role. Don't expect flashcards to be more than they are or do more than they can do.
Welcome to the dieharder distribution website.
Version 3.29.4beta is the current snapshot. Some of the documentation below may not quite be caught up to it, but it should be close.
Dieharder is a random number generator (rng) testing suite. It is intended to test generators, not files of possibly random numbers as the latter is a fallacious view of what it means to be random. Is the number 7 random? If it is generated by a random process, it might be. If it is made up to serve the purpose of some argument (like this one) it is not. Perfect random number generators produce "unlikely" sequences of random numbers -- at exactly the right average rate. Testing a rng is therefore quite subtle.
dieharder is a tool designed to permit one to push a weak generator to unambiguous failure (at the e.g. 0.0001% level), not leave one in the "limbo" of 1% or 5% maybe-failure. It also contains many tests and is extensible so that eventually it will contain many more tests than it already does.
If you are using dieharder for testing rngs either in one of its prebuilt versions (rpm or apt) or built from source (which gives you the ability to e.g. add more tests or integrate your rng directly with dieharder for ease of use) you may want to join either or both of the dieharder-announce or the dieharder-devel mailing lists here. The former should be very low traffic -- basically announcing when a snapshot makes it through development to where I'm proud of it. The latter will be a bit more active, and is a good place to post bug reports, patches, suggestions, fixes, complaints and generally participate in the development process.
At the suggestion of Linas Vepstas on the Gnu Scientific Library (GSL) list this GPL'd suite of random number tests will be named "Dieharder". Using a movie sequel pun for the name is a double tribute to George Marsaglia, whose "Diehard battery of tests" of random number generators has enjoyed years of enduring usefulness as a test suite.
The dieharder suite is more than just the diehard tests cleaned up and given a pretty GPL'd source face in native C. Tests from the Statistical Test Suite (STS) developed by the National Institute for Standards and Technology (NIST) are being incorporated, as are new tests developed by rgb. Where possible or appropriate, all tests that can be parameterized ("cranked up") to where failure, at least, is unambiguous are so parameterized and controllable from the command line.
A further design goal is to provide some indication of why a generator fails a test, where such information can be extracted during the test process and placed in usable form. For example, the bit-distribution tests should (eventually) be able to display the actual histogram for the different bit ntuplets.
Dieharder is by design extensible. It is intended to be the "Swiss army knife of random number test suites", or if you prefer, "the last suite you'll ever ware" for testing random numbers.
Dieharder can be freely downloaded from the Dieharder download site. On this page there should be a long list of previous versions of dieharder, and it should tell you what is the current snapshot. The version numbers have the following specific meaning which is a bit different than usual:
The single-tree dieharder sources (.tgz and .src.rpm) files can be downloaded from this directory. In addition, binary rpm's built on top of Fedora Core whatever (for either i386 or both of x86_64) may be present. Be warned: the GSL is a build requirement. The current packaging builds both the library and the dieharder UI from a single source rpm, or from running "make" in the toplevel directory of the source tarball. With a bit of effort (making a private rpm building tree), "make rpm" should work for you as well in this toplevel directory.
This project is under very active development. Considerable effort is being expended so that the suite will "run out of the box" to produce a reasonably understandable report for any given random number generator it supports via the "-a" flag, in addition to the ability to considerably vary most specific tests as applied to the generator. A brief synopsis of command options to get you started is presented below. In general, though, documentation (including this page, the man page, and built-in documentation) may lag the bleeding edge snapshot by a few days or more.
An rpm installation note from Court Shrock:
I was reading about your work on dieharder. First, some info about getting dieharder working in Gentoo: cd ~ emerge rpm gsl wget http://www.phy.duke.edu/~rgb/General/dieharder/dieharder-0.6.11-1.i386.rpm rpm -i --nodeps dieharder-0.6.11-1.i386.rpm
Rebuilding from tarball source should always work as well, and if you are planning to play a lot with the tool may be a desireable way to proceed as there are some documentation goodies in the ./doc subdirectory and the ./manual subdirectory of the source tarball (such as the original diehard test descriptions and the STS white paper).
George Marsaglia retired from FSU in 1996. For a brief time diehard appeared to have finally disappeared from FSU webspace, but what had really happened is google's favorite path to it had disappeared when his personal home directory was removed. Diehard is still there, at the URL http://www.stat.fsu.edu/pub/diehard as well as at a Hong Kong website. The source code of diehard itself is (of course) Copyright George Marsaglia but Marsaglia did not incorporate an explicit license into his code which muddles the issue of how and when it can be distributed, freely or otherwise. Existing diehard sources are not directly incorporated into dieharder in source form for that reason, to keep authorship and GPL licensing issues clear.
Note that the same is not true about data. Several of the diehard tests require that one use precomputed numbers as e.g. target mean, sigma for some test statistic. Obviously in these cases we use the same numbers as diehard so we get the same, or comparable, results. These numbers were all developed with support from Federal grants and have all been published in the literature, though, and should therefore be in the public domain as far as reuse in a program is concerned.
Note also that most of the diehard tests are modified in dieharder, usually in a way that should improve them. There are three improvements that were basically always made if possible.
Unfortunately, some of the diehard tests that rely on weak inverses of the covariance matrices associated with overlapping samples seem to have errors in their implementation, whether in the original diehard (covariance) data or in dieharder-specific code it is difficult to say. Fortunately, it is no longer necessary to limit the number of random numbers drawn from a generator when running an integrated test, and non-overlapping versions of these same tests do not require any treatment of covariance. For that reason non-overlapping versions of the questionable tests have been provided where possible (in particular testing permutations and sums) and the overlapping versions of those tests are deprecated pending a resolution of the apparent errors.
In a few cases other variations are possible for specific tests. This should be noted in the built-in test documentation for that test where appropriate.
Aside from these major differences, note that the algorithms were independently written more or less from the test descriptions alone (sometimes illuminated by a look at the code implementations, but only to clear up just what was meant by the description). They may well do things in a different (but equally valid) order or using different (but ultimately equivalent) algorithms altogether and hence produce slightly different (but equally valid) results even when run on the same data with the same basic parameters. Then, there may be bugs in the code, which might have the same general effect. Finally, it is always possible that diehard implementations have bugs and can be in error. Your Mileage May Vary. Be Warned.
The primary point of dieharder (like diehard before it) is to make it easy to time and test (pseudo)random number generators, both software and hardware, for a variety of purposes in research and cryptography. The tool is built entirely on top of the GSL's random number generator interface and uses a variety of other GSL tools (e.g. sort, erfc, incomplete gamma, distribution generators) in its operation.
Dieharder differs significantly from diehard in many ways. For example, diehard uses file based sources of random numbers exclusively and by default works with only roughly ten million random numbers in such a file. However, modern random number generators in a typical simulation application can easily need to generate 10^18 or more random numbers, generated from hundreds, thousands, millions of different seeds in independent (parallelized) simulation threads, as the application runs over a period of months to years. Those applications can easily be sensitive to rng weaknesses that might not be revealed by sequences as short as 10^7 uints in length even with excellent and sensitive tests. One of dieharder's primary design goals was to permit tests to be run on very long sequences.
To facilitate this, dieharder prefers to test generators that have been wrapped up in a GSL-compatible interface so that they can return an unbounded stream of random numbers -- as many as any single test or the entire suite of tests might require. Numerous examples are provided of how one can wrap one's own random number generator so that it is can be called via the GSL interface.
Dieharder also supports file-based input three distinct ways. The simplest is to use the (raw binary) stdin interface to pipe a bit stream from any rng, hardware or software, through dieharder for testing. In addition, one can use "direct" file input of either raw binary or ascii formatted (usually uint) random numbers. The man page contains examples of how to do all three of these things, and dieharder itself can generate sample files to use as templates for the appropriate formatting.
Note Well! Dieharder can consume a lot of random numbers in the course of running all the tests! To facilitate this, dieharder should (as of 2.27.11 and beyond) support large file (> 2GB) input, although this is still experimental. Large files are clunky and relatively slow, and the LFS (large file system) in linux/gcc is still relatively new and may have portability issues if dieharder is built with a non-gcc compiler. It is therefore strongly recommended that both hardware and software generators be tested by being wrapped within the GSL interface by emulating the source code examples or that the pipe/stdin interface be used so that they can return an essentially unbounded rng stream.
Dieharder also goes beyond diehard in that it is deliberately extensible. In addition to implementing all of the diehard tests it is expected that dieharder will eventually contain all of the NIST STS and a variety of tests contributed by users, invented by the dieharder authors, or implemented from descriptions in the literature. As a true open source project, dieharder can eventually contain all rng tests that prove useful in one place with a consistent interface that permits one to apply those tests to many generators for purposes of comparison and validation of the tests themselves as much as the generators. In other words, it is intended to be a vehicle for the computer science of random number generation testing as well as a practical test harness for random number generators.
To expand on this, the development of dieharder was motivated by the following, in rough order of importance:
Although this tool is being developed on Linux/GCC-based platforms, it should port with no particular difficulty to other Unix-like environments (at least ones that also support the GSL), with the further warning that certain features (in particular large file support) may require tweaking and that the dieharder authors may not be able to help you perform that tweaking.
If you compile the test or install the provided binary rpm's and run it as:
dieharder -ait should run -a(ll) tests on the default GSL generator.
Choose alternative tests with -g number where:
dieharder -g -1will list all possible numbers known to the current snapshot of the dieharder.
dieharder -lshould list all the tests implemented in the current snapshop of DieHarder. Finally, the venerable and time tested:
dieharder -hprovides a Usage synopsis (which can quite long) and
man dieharderis the (installed) man page, which may or many not be completely up to date as the suite is under active development. For developers, additional documentation is available in the toplevel directory or doc subdirectory of the source tree. Eventually, a complete DieHard manual in printable PDF form will be available both on this website and in /usr/share/doc/dieharder-*/.
List of GSL and user-defined random number generators that can be tested by dieharder:
#=============================================================================# # dieharder version 3.29.4beta Copyright 2003 Robert G. Brown # #=============================================================================# # Id Test Name | Id Test Name | Id Test Name # #=============================================================================# | 000 borosh13 |001 cmrg |002 coveyou | | 003 fishman18 |004 fishman20 |005 fishman2x | | 006 gfsr4 |007 knuthran |008 knuthran2 | | 009 knuthran2002 |010 lecuyer21 |011 minstd | | 012 mrg |013 mt19937 |014 mt19937_1999 | | 015 mt19937_1998 |016 r250 |017 ran0 | | 018 ran1 |019 ran2 |020 ran3 | | 021 rand |022 rand48 |023 random128-bsd | | 024 random128-glibc2 |025 random128-libc5 |026 random256-bsd | | 027 random256-glibc2 |028 random256-libc5 |029 random32-bsd | | 030 random32-glibc2 |031 random32-libc5 |032 random64-bsd | | 033 random64-glibc2 |034 random64-libc5 |035 random8-bsd | | 036 random8-glibc2 |037 random8-libc5 |038 random-bsd | | 039 random-glibc2 |040 random-libc5 |041 randu | | 042 ranf |043 ranlux |044 ranlux389 | | 045 ranlxd1 |046 ranlxd2 |047 ranlxs0 | | 048 ranlxs1 |049 ranlxs2 |050 ranmar | | 051 slatec |052 taus |053 taus2 | | 054 taus113 |055 transputer |056 tt800 | | 057 uni |058 uni32 |059 vax | | 060 waterman14 |061 zuf | | #=============================================================================# | 200 stdin_input_raw |201 file_input_raw |202 file_input | | 203 ca |204 uvag |205 AES_OFB | | 206 Threefish_OFB | | | #=============================================================================# | 400 R_wichmann_hill |401 R_marsaglia_multic. |402 R_super_duper | | 403 R_mersenne_twister |404 R_knuth_taocp |405 R_knuth_taocp2 | #=============================================================================# | 500 /dev/random |501 /dev/urandom | | #=============================================================================# | 600 empty | | | #=============================================================================#
Two "gold standard" generators in particular are provided to "test the test" -- AES_OFB and Threefish_OFB are both cryptographic generators and should be quite random. gfsr4, mt19937, and taus (and several others) are very good generators in the GSL, as well. If you are developing a new rng, it should compare decently with these generators on dieharder test runs.
Note that the stdin_input_raw interface (-g 200) is a "universal" interface. Any generator that can produce a (continuous) stream of presumably random bits can be tested with dieharder. The easiest way to demonstrate this is by running:
dieharder -S 1 -B -o -t 1000000000 | dieharder -g 75 -r 3 -n 2
where the first invocation of dieharder generates a stream of binary bits drawn from the default generator with seed 1 and the second reads those bits from stdin and tests them with the rgb bitdist test on two bit sequences. Compare the output to:
dieharder -S 1 -r 3 -n 2
which runs the same test on the same generator with the same seed internally. They should be the same.
Similarly the file_input generator requires a file of "cooked" (ascii readable) random numbers, one per line, with a header that describes the format to dieharder. Note Well! File or stream input rands (with any of the three methods for input) are delivered to the tests on demand, but if the test needs more than are available dieharder either fails (in the case of a stdin stream) or rewinds the file and cycles through it again, and again, and again as needed. Obviously this significantly reduces the sample space and can lead to completely incorrect results for the p-value histograms unless there are enough rands to run EACH test without repetition (it is harmless to reuse the sequence for different tests). Let the user beware!
List of the CURRENT fully implemented tests (as of the 08/18/08 snapshot):
#=============================================================================# # dieharder version 3.29.4beta Copyright 2003 Robert G. Brown # #=============================================================================# Installed dieharder tests: Test Number Test Name Test Reliability =============================================================================== -d 0 Diehard Birthdays Test Good -d 1 Diehard OPERM5 Test Suspect -d 2 Diehard 32x32 Binary Rank Test Good -d 3 Diehard 6x8 Binary Rank Test Good -d 4 Diehard Bitstream Test Good -d 5 Diehard OPSO Good -d 6 Diehard OQSO Test Good -d 7 Diehard DNA Test Good -d 8 Diehard Count the 1s (stream) Test Good -d 9 Diehard Count the 1s Test (byte) Good -d 10 Diehard Parking Lot Test Good -d 11 Diehard Minimum Distance (2d Circle) Test Good -d 12 Diehard 3d Sphere (Minimum Distance) Test Good -d 13 Diehard Squeeze Test Good -d 14 Diehard Sums Test Do Not Use -d 15 Diehard Runs Test Good -d 16 Diehard Craps Test Good -d 17 Marsaglia and Tsang GCD Test Good -d 100 STS Monobit Test Good -d 101 STS Runs Test Good -d 102 STS Serial Test (Generalized) Good -d 200 RGB Bit Distribution Test Good -d 201 RGB Generalized Minimum Distance Test Good -d 202 RGB Permutations Test Good -d 203 RGB Lagged Sum Test Good -d 204 RGB Kolmogorov-Smirnov Test Test Good
Full descriptions of the tests are available from within the tool. For example, enter:
rgb@lilith|B:1003>./dieharder -d 203 -h OK, what is dtest_num = 203 #================================================================== # RGB Lagged Sums Test # This package contains many very lovely tests. Very few of them, # however, test for lagged correlations -- the possibility that # the random number generator has a bitlevel correlation after # some fixed number of intervening bits. # # The lagged sums test is therefore very simple. One simply adds up # uniform deviates sampled from the rng, skipping lag samples in between # each rand used. The mean of tsamples samples thus summed should be # 0.5*tsamples. The standard deviation should be sqrt(tsamples/12). # The experimental values of the sum are thus converted into a # p-value (using the erf()) and a ks-test applied to psamples of them. #==================================================================
Note that all tests have been independently rewritten from their description, and may be functionally modified or extended relative to the original source code published in the originating suite(s). This has proven to be absolutely necessary; dieharder stresses random number generator tests as much as it stresses random number generators, and tests with imprecise target statistics can return "failure" when the fault is with the test, not the generator.
The author (rgb) bears complete responsibility for these changes, subject to the standard GPL code disclaimer that the code has no warranty. In essence, yes it may be my fault if they don't work but using the tool is at your own risk and you can fix it if it bothers you and/or I don't fix it first.
All tests are encapsulated to be as standard as possible in the way they compute p-values from single statistics or from vectors of statistics, and in the way they implement the underlying KS and chisq tests. Diehard is now complete in dieharder (although two tests are badly broken and should not be used), and attention will turn towards implementing more selected tests from the STS and many other sources. A road map of sorts (with full supporting documentation) is available on request if volunteers wish to work on adding more GPL tests.
Note that a few tests appear to have stubborn bugs. In particular, the diehard operm5 test seems to fail all generators in dieharder. Several users have attempted to help debug this problem, and it tentatively appears that the problem is in the original diehard code and not just dieharder. There is extensive literature on overlapping tests, which are highly non-trivial to implement and involve things like forming the weak inverse of covariance matrices in order to correct for overlapping (non-independent) statistics.
A revised version of overlapping permutations is underway (as an rgb test), but is still buggy. A non-overlapping (rgb) permutations test is provided now that should test much the same thing at the expense of requiring more samples to do it.
Similarly, the diehard sums test appears to produce a systematically non-flat distribution of p-values for all rngs tested, in particular for the "gold standard" cryptographic generators aes and threefish, as well as for the "good" generators in the GSL (mt19937, taus, gfsr4). It seems very unlikely that all of these generators would be flawed in the same way, so this test also should not be used to test your rng.
I hope that even during its development, you find dieharder useful. Remember, it is fully open source, so you can freely modify and redistribute the code according to the rules laid out in the Gnu Public License (version 2b), which might cost you as much as a beer one day. In particular, you can easily add random number generators using the provided examples as templates, or you can add tests of your own by copying the general layout of the existing tests (working toward a p-value per run, cumulating (say) 100 runs, and turning the resulting KS test into an overall p-value). Best of all, you can look inside the code and see how the tests work, which may inspire you to create a new test -- or a new generator that can pass a test.
To conclude, if you have any interest in participating in the development of dieharder, be sure to let me know, especially if you have decent C coding skills (including familiarity with Subversion and the GSL) and a basic knowledge of statistics. I even have documents to help with the latter, if you have the programming skills and want to LEARN statistics. Bug reports or suggestions are also welcome.
Submit bug reports, etc. to
rgb at phy dot duke dot eduBenchmaster is a fairly sophisticated program designed to time and exercise very specific systems functions. It uses the fastest onboard clock that it can find (generally the CPU's cycle counter on x86-derived architectures) to time test "objects", and determines the precision of that timer including the overhead of the timer call.
A test object contains (in addition to test creators/destructor functions) a test routine with two branches -- one "empty" and one "full" -- that are structured to be, as nearly as possibly, identical except for the additional code to be timed in the full loop.
The test harness then determines iteration counts -- the number of times it has to run the empty or full branches to accumulate a time much greater than the timer resolution. It then proceeds to generate a requested number of samples of the timings of the empty and full branches. Finally, it subtracts the average full time from the average empty time to determine the result and evaluates the mean and standard deviations to produce the cumulative expected error.
Finally, the results are printed out in a standard XML based format with an optional header describing the test and certain runtime details. Numbers that are returned include the length of the vector (see discussion of vector tests below), the stride (ditto), and the mean time with error bars, in nanoseconds required to execute just the tested code fragment in the particular context of the test routine. Finally, a "megarate" is returned that is generally the number of million times the test fragment is executed per second. There are a few exceptions to this, see below.
The use of XML in the test output is one of the project's major design goals, as it in principle makes it possible to build e.g. DTDs for the benchmark description language to generate standard reports in a variety of media. Of course this is not yet done -- it is one of the next major design goals. Volunteers/contributions of code welcome... as are comments on the XML itself (which is still pretty malleable until at least one program or transformation process is written that uses it).
Back to topThe version numbers have the following meaning. Note that these aren't necessarily what you might expect, so be sure to at least glance at this.
All benchmaster sources (.tgz and .src.rpm) files can be downloaded from this directory. In addition, i386 binary rpm's (currently built on top of Fedora Core 5) are present.
This project is currently semi-stable. The tests seem to work (at least for me) fairly consistently, there just aren't as many as I'd like there to eventually be. Alas, I have too many projects going at once, and have recently been spending a lot of time with the Dieharder project available elsewhere on this website. If you are interested in seeing the benchmaster project advanced more aggressively, contact me (especially with an offer to help out:-).
Below are descriptions of the different kinds of tests already in benchmaster.
Back to topAs noted above, in addition to being sampled in one loop (with a controllable number of samples) and iterated inside that loop per sample (with an automatically set but user controllable number of iterations) some of the tested code fragments are loops that operate on vectors of numbers. For example, the stream benchmark by John D. McCalpin consists of four simple numerical operations -- a vector copy, scaling a vector, adding a vector, and adding and rescaling a vector. Each of these stream operations is one of the tests already programmed into benchmaster, so benchmaster is capable of "running stream" on a system
In stream, the length of the tested vector is generally fixed -- programmed directly into the test as a compile-time parameter. In benchmaster's stream (and other related vector arithmetic tests) the vector length is a variable and can be selected by the user at runtime. This permits one to directly observe how cache improves performance for various vector lengths and strides.
Most modern CPUs have elaborate optimization mechanisms designed to improve numerical performance on vectors in particular. They prefetch data from memory into the cache in order to ensure that it is waiting there when needed. They have special registers and pipelines that speed repeated operations iterated over a vector. However, sometimes one runs code that (although iterative) does not have such a fortunate memory access pattern. Benchmaster therefore contains a facility for permitting at least some vector tests to be performed in "shuffled order". Basically, a matching vector of vector indices is shuffled and used as the indices for the test. The overhead associated with the shuffling process per se is duplicated in the "empty" code fragment so that only the actual time required to access the memory in shuffled order contributes.
Back to topThe general function and design of the timing harness is explained above and documented in the code itself. This section indicates how to add a test to the suite, as one of its major design goals is to make it easy for you to add your own test fragments and operations.
Before showing you an example of a test (the easiest way to document how to create a new one) let me remark on a few of the many problems that plague "benchmarking". For one, the speed with which code is executed depends on many, many things, some of which are out of our control. For example, there is an obvious dependence on system state -- if your task is swapped off of the CPU on a multitasking system in mid-instruction, your time for that sample will be quite high. There is an uncontrollable dependence on the compiler. I'm assuming that the full and empty branches are both likely to be close together in memory and both likely to be kept resident in cache in order for the timings to be comparable and subtractable. This is likely enough to be true, but there are no guarantees from the compiler.
It is also very difficult to test/time single instructions of just about any kind. A multiply on a system can take a fraction of a clock cycle. The finest-grained timekeeper on the system is the clock cycle counter (typically order of a nanosecond), and the code required to read it takes many nanoseconds to execute. Timing a multiply is thus akin to timing the beating of a hummingbird's wings with an hourglass, a process made worse by the compiler's tendency to optimize away instructions that it can tell are doing nothing and can be compressed.
This is just a warning. As it says in the program's Usage statement and man page, the "Mega-rates" returned by this tool are BOGUS and may not be even approximately correct. When interpreting results of existing tests or adding your own, be cautious and test the tester as much as the code fragment itself until the results make sense in your own mind.
One final warning about hidden optimizations, overflows, etc. Many CPUs are smart enough to use superfast internal arithmetic in order to perform certain operations. For example, multiplying by 0.0 or 1.0 or 2.0 on many CPUs will take much, much less time than multiplying by (say) 3.141592653589. For that reason I typically use this as a number to multiply by whenever I am testing multiplication. Of course if one iterates multiplication by \pi it doesn't take long to overflow a floating point variable, so one needs to use caution in designing test loops to avoid this without using 1.0 as a multiplier or dividing (which takes much longer than multiplication) when one wants to test only multiplication.
With all that said, a test consists of two pieces. An include file that minimally contains the function prototypes for the the test itself, for example (for the stream copy test):
/* * $Id: benchmaster.abs,v 1.7 2004年12月17日 15:31:56 rgb Exp $ * * See copyright in copyright.h and the accompanying file COPYING * */ /* * Goes with stream_copy_test.c. */ void stream_copy_init(Test *newtest); void stream_copy_alloc(); void stream_copy_free(); int stream_copy_test(int full_flag); void stream_copy_results(); void stream_copy_about();
and the stream copy test source for these components:
/*
* $Id: benchmaster.abs,v 1.7 2004年12月17日 15:31:56 rgb Exp $
* See copyright in copyright.h and the accompanying file COPYING
*/
#include "benchmaster.h"
/*
*==================================================================
* This is the "copy" test from the stream suite. It is not
* directly comparable to stream results for a variety of reasons.
* For one, it uses malloc to allocate all vectors so vector
* length may be different from what is compiled into any given copy
* of stream. Also, it uses the average time to determine the rate
* and not the best time. It will therefore generally return
* results that are very SLIGHTLY LOWER/SLOWER than regular stream
* (but which may be more realistic for general purpose code).
*
* It also uses a different timing harness, one that is both
* more accurate (uses a superior timer) and which repeats the
* computation many times, typically order 100, to obtain both a
* mean time and its standard deviation as test results.
*==================================================================
*/
void stream_copy_init(Test *mytest){
int i;
mytest->alloc = stream_copy_alloc;
mytest->free = stream_copy_free;
mytest->test = stream_copy_test;
mytest->results = stream_copy_results;
snprintf(mytest->name,K,"stream copy");
snprintf(mytest->about,K,"d[i] = a[i] (%d byte double vector)",sizeof(double));
if(verbose == VERBOSE || verbose == V_INIT){
printf("# Init for test %s\n",mytest->name);
}
}
void stream_copy_alloc()
{
int i;
/*
* Allocate vector(s) to be tested with and initialize it and all
* associated test-specific variables.
*/
d = (double *) malloc((size_t) (size*sizeof(double)));
a = (double *) malloc((size_t) (size*sizeof(double)));
/*
* Initialize the vector. xtest is set from the command line, default PI.
*/
for(i=0;i < size;i+=stride){
a[i] = xtest;
}
}
void stream_copy_free()
{
int i;
/*
* Free all the memory we just allocated, to be neat and clean and
* all that.
*/
free(a);
free(d);
}
int stream_copy_test(int full_flag)
{
int i;
if(full_flag){
for(i=0;i < size;i+=stride){
d[i] = a[i];
}
return(full_flag);
} else {
return(full_flag);
}
}
void stream_copy_results(Test *mytest)
{
double nanotime_norm;
/*
* This is the number of copy operations in the core loop. We adjust the
* test normalization so it is the SAME as that of stream, which computes
* the rate as "megabytes/seconds": 1.0e-6*2*sizeof(double)*nsize/time
* (in seconds). We measure nanoseconds, so ours is just a teeny bit
* different.
*/
nanotime_norm = (double)size/stride;
mytest->avg_time = fabs(mytest->avg_time_full - mytest->avg_time_empty)/nanotime_norm;
mytest->sigma = (mytest->sigma_time_full + mytest->sigma_time_empty)/nanotime_norm;
mytest->avg_megarate = 1000.0*2*sizeof(double)/mytest->avg_time;
show_results(mytest);
}
Note that xtest is a user-settable value for the real number used to initialize the vector. It defaults to pi, but can be overridden on the command line so you can see for yourself the effect of using 1.0 or 0.0 in certain contexts of certain tests instead of a number that cannot be optimized at the CPU microcode level. Note that the avg_megarate is a bit different than from other tests as it returns a "bandwidth" in Mbytes/sec (to be comparable with stream) instead of a "bogomegarate", which is just the number of millions of iterations of the test fragment completed per second.
This code fragment in my testing harness produces results that are generally a few percent slower than those of off-the-shelf stream. This is understandable -- I use the average time instead of the minimum time to generate the number, so my stream number is average/expected performance over a reasonably large number (default 100) of samples while stream reports the observed peak performance on a fairly small sample size (default 10).
Once you have written a "microbenchmark" test fragment of your own, you have merely to insert it into the overall structure of the benchmaster program. The best way to do that is to follow exactly the way the existing tests are inserted. Put your test's include file and NAME into the enum and include list in tests.h. Initialize the test in startup.c. At that point the code should "just work". Try to use the provided global variables and pointers for things like xtest and vectors, just to keep the primary code from getting too gooped up. You should be able to put globals into the include file for your test if you need to add any new ones for specific tests, and should be sure to add code to parsecl.c and startup as required to manage them.
Note Well! Benchmaster is not written for the purpose of encouraging "benchmarketing" by vendors, so take any publication of results from the benchmark with a grain of salt.
Consider the source! I mean that quite literally. Benchmaster is a fully GPL tool that is available in rpm form. That means that you or a vendor can hack up the source, insert optimized machine code, etc. Note that you should insist on results of unmodified benchmaster source, compiled with the default flags as at least one component of what you consider when you are trying to measure or understand systems performance. Only then is it reasonable to have fun tuning things up to see how fast you can make them.
Back to topAs noted above, benchmaster is designed to be easy to modify to insert your own tests as "test objects". You should feel free to contribute particularly good or revealing tests back to me for inclusion in the primary distribution of the tool. If and when I get so many tests into the tool that the current method of selecting tests by number no longer works, I'll probably have to rewrite the toplevel interface to make it easier to keep track of everything, but that is likely some time off (and should not affect results).
You should also feel free to make suggestions or report bugs or contribute code to the timing harness itself -- I am constantly trying things to see if I can obtain better control over system state (and hence get more reproducible results) and to see if I can finagle accurate timings of those hummingbird wing beats using my hourglass. Perhaps one of you reading this knows just how to go about doing it in userspace code...
Future Plans for benchmaster include:
Participation in benchmaster is openly solicited and encouraged. All code is written in C for efficiency and its tight coupling to the operating system, for all that it is still "object oriented" in general design to keep it scalable and extensible. Contact me at rgbATphy.duke.edu if you are interested in participating.
I hope that you find benchmaster useful.
This is a distribution site for a simple, perl based toolset for tracking Amazon Sales Rank for initially books only -- the current key item indicator is the book's ISBN number -- but eventually this will be broadened to other kinds of content sold via Amazon.
The tool's operation is quite simple. Unpack the tarball into a suitable directory. Make sure that you've installed its dependencies: perl, gd and gd-devel, perl-GD and perl-GDGraph.noarch and elinks. Enter the directory and enter:
on the command line (inside, say, an xterm). You will need internet access on the system
It should connect to amazon, download and parse your book's page, and extract its sales rank. It will write this out into a table and begin to generate a png image of its graph as a function of sample time. Leave the tool running. Every hour it will reconnect with Amazon and update the sales rank (Amazon updates sales ranks only once an hour so there is no point in sampling any more often).
That's all there is to it! You can view the sales rank png with any graphical viewer, e.g. gthumb, xv, or you can edit the amazon_sales_rank.html file provided to make it point to the path of the png and open this page in your favorite browser. It will then automagically update every hour to show you the entire graph to date. The raw data is collected in the amazon_sales_rank_table.isbn file that appears in the working directory, and this can be imported to a spreadsheet or other tools for further processing. This table is preloaded should you need to stop the tool and restart it -- you don't lose the points you already have if you must shut the tool down for a time.
This is a VERY PRELIMINARY release of the tool. It does the very minimum needed to track ASR for your book and display it as a reasonably formatted graph and save it in a reasonably formatted table. Future releases will be a bit slicker, and will probably automate things a lot more. Right now it is recommended that only linux-savvy people attempt to use it, but come back in a few months and you should find a drop in, plug-and play tool that installs in the actual operating system and that runs automatically to generate reports in much the same way that webalize does now.
Submit bug reports, feature requests etc. to
rgb at phy dot duke dot eduAlso, consider visiting the site of my book: The Book of Lilith. If you honor me and the work put into developing this tool by electing to purchase a copy, I think you'll be glad you did!
NOTE WELL! This book is not written by or copyrighted by Robert G. Brown! It is a mirror of an online book on C programming that -- curiously enough -- has a license almost identical to my Gnu Public License, v2b (b for beverage). However, you have to buy the actual authors of the book a beer, not me.
From the book's front page:
This is the online version of The C Book, second edition by Mike Banahan, Declan Brady and Mark Doran, originally published by Addison Wesley in 1991. This version is made freely available.
While this book is no longer in print, it's content is still very relevant today. The C language is still popular, particularly for open source software and embedded programming. We hope this book will be useful, or at least interesting, to people who use C.
If you have any comments about this book, or if you find any bugs in its presentation, please send a message to consulting@gbdirect.co.uk.
This is a draft of a future yum HOWTO, offered up here for private comment by initially Seth and Michael and later by (I imagine) the yum list. If you've found this in the meantime via google, feel free to use it but beware -- it may be openly wrong or misleading in places.
Note Well: This is a Request for Comments only; Use at your own risk, and please return comments and corrections to rgb (or better, the yum mailing list) for encorporation into what I hope to be a dynamic document associated with a project, rather than a static prescription that may, in fact, be heavily flawed.
It was written primarily to teach colleagues and students with whom I collaborate the rudiments of CVS, as I use it as an essential component of the management of any project I'm involved in that generates documents and other data objects (such as teaching a course as part of a team, guiding students in research involving programming, writing papers).
The HOWTO appears to fill a community need -- a quick google reveals only one standard-format HOWTO for CVS, and that one is extremely terse and specific to NetBSD. This particular HOWTO is structured as a tutorial that leads you step by step through the basic commands required to set up CVS root repositories for various purposes and use to CVS to manage project archives within those repositories.
It deliberately omits the setup and use of CVS with pserver (web-based access) as this is documented a variety of places and because we do not use this locally (internal to our department LAN) due to security concerns -- vulnerabilities have been found and patched within the last year, and while the are patched, one always worries about whether there are more. In any event, this requires root privileges to set up and manage and therefore almost by definition is advanced usage (in my opinion) and hence is inappropriate for this document.
In other words, this is very much a Getting Started with CVS sort of document -- CVS is very powerful and hence very complex, and given half a chance it will, as they say, eat your meatloaf for you. However to my own direct experience, well over 95% of all CVS usage in a simple LAN environment is encompassed within the commands tutorially demonstrated in this HOWTO. Many users will never need all that power; users that do will need to master the conventional and relatively simple aspects of CVS documented here before using the actual manual and more advanced documentation to learn how to (for example) manage multiple branches of code anyway.
At any rate, if you are interested in learning to use CVS (and you should be, if you work at all with dynamic information of any type whatsoever) you many find this document useful. Feel free to provide me with feedback, but remember that I did not write CVS and am not a CVS developer, so any actual bug reports or feature requests should probably go to the CVS developers via its home page:
There are several additional sources of tutorial information, manuals, and other documentation on the primary website, but unfortunately none are formatted in the familiar HOWTO style and they tend (in my opinion) to be too simple (omitting enough information to be able to set up or participate in a project as a member of a group) or too complex (an exhaustive manual complete with visuals and lots of detail and instructions for Windows as well as Unices). Nevertheless, new CVS users will find it worthwhile to visit this site and quickly browse and bookmark these resources as they will provide additional support for this document when some feature or detail it omits is needed for a project.
Note Well! This mini-HOWTO was developed on a linux system and
makes the following assumptions:
random_pvm is a C source demo/template for generating random numbers using a PVM master/slave program. It is derived from the C source project template also available from rgb's website. It was written for a Cluster World Magazine column, and although it is freely available and GPL'd users are encouraged to subscribe to the magazine to get all sorts of other goodies that come with every issue.
random_pvm actually installs from the tarball ONLY. In most cases a program template I write will create a workable rpm, but it isn't really desireable to install this demo in a rootspace /usr/share/pvm3 directory so although the make targets are there (and might even work, although I doubt it) I advise against messing with them.
To build it, create or change to your source directory (I use $HOME/Src but suit yourself), put random_pvm.tgz there and unpack it:
tar xvfz random_pvm.tgz
(and it should verbosely unpack).
Change to the random_pvm directory. There is a random_pvm.1 man page there that gives instructions on how to build and install and use the program. (Basically stuff like make, make install, and then running the program.) Remember to start pvm and build a virtual machine (instructions NOT included herein) before trying to run the program, and make sure that the random_pvm_slave program is installed in the appropriate place on all the nodes.
If you have any fundamental trouble getting it working, let me know and I'll try to help you. My email address is rgb@phy.duke.edu.
This is a portable/semimaintainable rpm packaging of jove. Jove stands for Jonathan's Own Version of Emacs, and in my opinion it has been the best text editor available for decades (as emacs, its progenitor, has become ever more crack-ridden until I can no longer stand to use it at all even as a stand-in for jove). Jove is, in particular, a really great editor for e.g. C source code, latex source code, and in general source codes that require an invocation of "make" to build internally. It has all the essential features of emacs without losing its attractive sparseness.
Since I use jove exclusively (having done so for getting on 18 years at this point) and since I also use rpm-based systems exclusively and rpm-centric distribution tools such as yum, I need jove to be neatly packaged. The first thing I ever do on a system is go in and install jove so I can work on it. It needs to be cleanly rpm-buildable and (I think) distributed as prebuilt source rpm if not binary rpm for some of the major distributions.
Jove is currently maintained (as far as I can tell) as a tarball-only product within Toronto's CS department. From their base, I've hacked the Makefile, the spec file, and the versions.h file (trivially) as follows:
These changes SHOULD NOT affect any other build targets or build processes (with the possible exceptions of the specfile changes, where I don't have enough distribution alternatives to test across all of them). Either way, if you want a repository from which to mirror relatively current signed jove rpm's, yum update jove rpms, grab a tarball of jove that has the above make targets for your own local builds, feel free to use this site.
I'm also willing to provide some debugging support if the rpm's on this site don't work for you or rebuild for you. I have to emphasize the some because I have a lot of projects and as long as jove works for me, I'm happy and may be busy as well as happy. However, if you encounter a bug or just need some help feel free to contact me at rgb at phy.duke.edu.
project abstract (in html) .
project_pvm is a C source project template for PVM master/slave projects. It is derived from the C source project template also available from rgb's website and does the usual automagical project maintenance via simple (and not so simple) make targets: make tgz, make rpm, make installweb, make cvs for example. It is worth several hours of work, minimally, in getting a pvm project off the ground, and gets it off on the right foot, with a lot of things you might add "eventually" already there in template form, ready to fill in.
project_pvm actually installs from the tarball ONLY -- or if you prefer, the tarball IS the project template ready to use -- but this page and distribution set is more or less automagically created by the make installweb target, so it seems worthwhile to include the rpm's even if they only install a trivial (cluster parallelized) "Hello World!" program.
To use this template, create or change to your source directory (I use $HOME/Src but suit yourself), put the project_pvm.tgz there and unpack it:
tar xvfz project_pvm.tgz
(and it should verbosely unpack).
Change to the project directory. There is a README that gives instructions on how to use to the template. Most of the mucky part of the process is encapsulated in a script called "newproject" that you can read to see how it works. To use this script for a new "rgb standard" PVM project, start pvm and build a virtual machine (instructions NOT included herein) and enter:
newproject projectname
cd ../projectname
cvs import -m "initial revision" projectname projectname start
cd ..
/bin/rm -rf projectname
cvs checkout projectname
cd projectname
make
./projectname(and Hello World example should run).
This presumes that you've got CVS setup (a functioning CVSROOT). If you want to use the make rpm targets, additionally you must:
./rpm_tree \ |-BUILD |-RPMS |-SOURCES |-SPECS |-SRPMS(all empty and chmod 755). Note that this is in USERSPACE. You don't need to be root to build root-installed rpm's.
Optionally edit the man page template, the README, the abstract (this file), the php file, Remember, the man page is your friend. Also remember to update/modify the "Usage" statement in parsecl.c as you add new command line commands, if any.
If you grab this project template and have any fundamental trouble getting it working, let me know and I'll try to help you. My email address is rgb@phy.duke.edu.
This is a reusable template for latex projects. Download the tarball
and unpack it with e.g.
tar xvfz latex.proj-0.4.1.tgz cd latex.proj-0.4.1Place it under e.g. Subversion control so that you can back off any changes you make to it.
Look over the Makefile to familiarize yourself with the various targets. Not all of them are guaranteed to work, and some may not work for you without some editing, at least. Do not modify the Makefile yet, though -- work at first on a copy created as below and then backport the changes to the original Makefile carefully when you are sure they work.
In this directory, run the "newproject" command:
./newproject projectname cd ../projectname
Place the new project under your choice of version control -- subversion is currently supported by a couple of targets in the Makefile that will require a bit of editing to make work for you but CVS is equally possible with a bit more editing.
Note that newproject SHOULD have changed all the basic filenames around so that they correspond to your project name. Only make changes in the template itself if you want to make them permanent features of a new latex project!
You should be able to type "make" in the new project directory at any
time and have it just work to build dvi, pdf, and a4pdf (a pdf for A4
sized paper commonly used in Europe). Bring up the dvi file for preview
with a command such as:
xdvi projectname &Then start up your favorite editor on the projectname.tex source file. If it is emacs or jove (or any editor that permits you to invoke make from inside the editor) you should be able to make simple changes to the latex source, invoke make via e.g. ^x ^e, and bring the preview screen to the foreground, where it will automatically refresh to the current/new dvi file! This makes the edit, make/debug, view/debug, edit cycle quite painless and generally faster than most latex IDE GUI tools.
Good luck! Feel free to contact me with e.g. bug reports or problems.
This is a HOWTO project template, using the linuxdoc dtd. It isn't intended to be very complete at the display side, but in the actual compressed tarball there are complete templates for both dtd and linuxdoc straight from the tldp.org website. The wrapping might prove useful as well.
This is the online latex manual. It is here for my own use (the Nasa site where I found it can be pretty busy) but it is also freely available to others if you should find it and want to use it here.