Re: why benchmarks need salt
[
Date Prev][
Date Next][
Thread Prev][
Thread Next]
[
Date Index]
[
Thread Index]
- Subject: Re: why benchmarks need salt
- From: Enrico Colombini <erix@...>
- Date: 2015年7月21日 15:15:05 +0200
On 20-Jul-15 21:12, Elias Barrionovo wrote:
Soon:
"Abstract: In this paper we present a new optimization technique,
where we use an artificial neural network together with other
heuristics to find a suitable name for (...)"
While you are busy writing that paper ;-) I ran a few tests under Linux
with a freshly compiled 5.3.1, randomizing the executable name length
from 1 to 10 characters.
I raised 'rep' to 10000 in Roberto's code, to minimize os.clock()
granularity influence.
On 64-bit xubuntu (i7), len=1..10, 100 repetitions:
min=1.074592, max=1.223682, avg=1.08894445
On 32-bit xubuntu (Pentium 4 HT), len=1..10, 10 repetitions:
min=5.215278, max=9.036871 (!?!), avg=5.8223072
Strangely enough, the above pathological result (that happened in two
instances with length=6, towards the end of the 10 runs) did not happen
again with 100 repetitions. Now on the 32-bit machine I got:
min=5.214088, max=5.517392, avg=5.2637858
(quick & dirty code attached - of course it could add its own problems)
Two notes:
- could swapping and/or other processes be involved?
(e.g. something making heavy use of the system heap)
- my test program runs from the same intepreter that it calls
to run 'temp'.
Now I *have* to turn off the old Pentium: 120 W in the middle of a heat
wave...
--
Enrico
local rep = string.rep
local rnd = math.random
local floor = math.floor
local exebase = 'a' -- Lua executable, no extension
local exeext = '' -- (Linux version)
local luaexe = exebase .. exeext
local nrep = tonumber(arg[1])
local maxlen = tonumber(arg[2])
if (nrep == nil) or (maxlen == nil)
or (nrep <= 0) or (maxlen <= #luaexe) then
print('usage: lua test.lua <nrep> <maxlen>')
print(' (nrep > 0, maxlen > ' .. #luaexe .. ')')
os.exit(1)
end
-- table of execution times
local t = {}
math.randomseed(os.clock())
for i = 1, nrep do
-- make a copy of Lua executable with random name length
local len = maxlen - #luaexe
assert(maxlen >= 0)
exename = exebase .. rep('x', floor(rnd(len))) .. exeext
os.execute('cp ' .. luaexe .. ' ' .. exename)
-- run the copy, get the result
local f = io.popen('./' .. exename .. ' temp')
local r = f:read()
f:close()
-- remove the copy
os.remove(exename)
-- add result to table (len includes extension, if any)
t[#t + 1] = {len=#exename, time=r}
-- show it is doing something
io.write('time=', r, ', len=', #exename, '\n')
end
io.write('\n')
-- process results
local maxtime = t[1].time
local mintime = maxtime
local totaltime = 0
for i = 1, #t do
local entry = t[i]
local len = entry.len
local time = entry.time
if time > maxtime then
maxtime = time
elseif time < mintime then
mintime = time
end
totaltime = totaltime + time
end
local avg = totaltime / #t
io.write('min=', mintime, ', max=', maxtime, ', avg=', avg, '\n')