Re: Looking for benchmark test tool
- From: Jean-David Beyer <jeandavid8@xxxxxxxxxxx>
- Date: Fri, 14 Jul 2006 02:04:08 GMT
dbone wrote:
I'm looking for a benchmark test tool to compare the performance andI used to work on the Bell Labs optimizer for the C compiler.
reliability of different PCs running LInux centOS 4. I'm looking to
measure CPU, memory, disk IO, thread count etc. I don't mind paying for
a tool that can give complete reports. Any recommendations? Thanks.
Our marketing department gave us a bunch of about a dozen benchmarks to run
so we could prove our compiler, running code for our processor, was faster
and better than the competition. We could easily do this.
But the thing is that those benchmarks were pathetic and showed lack of
understanding. About half of them were copied from BYTE Magazine (IIRC), and
the editor who though them up had no clue what compilers did. So they tended
to have a loop executed 10,000 times that did what they thought they were
testing. But I wrote a Loop Invariant Code Motion optimization that noticed
if the same thing was computed each time through a loop, and if so, it moved
it out of the loop. Thus the test was run only once instead of 10,000x. Then
my partner's Live-Dead Analysis program noticed that the stuff outside the
loop, and the loop code itself, were dead after being computed (i.e., never
used) so it eliminated them too. Thus a benchmark that the editor thought
measured something 10,000x executed in about 12 microseconds (we had 10 MHz
processors in those days). USELESS.
There was a benchmark called Whetstone that was appropriate in the 1960s,
but is still used today. It is thought to measure floating point
performance, which, with a decent optimizer, it does not. That benchmark was
designed to measure all the heavily-used code types generated by the
Whetstone Algol-60 Interpreter. One of the test modules, the only one with
any floating point computation, was used to measure call and return
overhead. It worked by calling a function 10,000 times (or so). Well, my
partner's Inline Function Expansion expanded that function inline (so it no
longer tested what it was meant to test), and then my LICM moved the
computation out of the loop, after which Live-Dead Analysis removed the last
reminant of that test. So any decent optimizer (we are talking about mid
1980s, by the way) would ensure that no floating point computation was done
at all in that benchmark.
We had another idea for making better benchmarks. Our company had a large
System/360 or the Amdahl equivalent running UNIX and we just profiled the
thing to see which programs were using the most processor time for a week.
We then took the top 10 of those (no surprise: troff headed the list). We
then took those 10 programs and re-coded them to do no IO but to get their
data from built-in tables, and write their output to built-in tables.
Those were more real benchmarks for the load our main-frame UNIX systems
were running. Unfortunately, our optimizer could not improve them by factors
of 10000:1 like is the stupid benchmarks provided by our marketing
department. We did improve them some. But we had to compete with our
competition, and they used the BYTE benchmarks, too, so we clobbered them.
But not in sales. People did not believe our tests, though the tests were OK
on the benchmarks provided. Well, Bell Labs is no longer in the computer
business in any way. That is what happens when bean-counters reign supreme.
--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key: 9A2FC99A Registered Machine 241939.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 21:50:01 up 94 days, 11:23, 4 users, load average: 4.40, 4.44, 4.21
.
- References:
- Looking for benchmark test tool
- From: dbone
- Looking for benchmark test tool
- Prev by Date: Re: tar default to what device?
- Next by Date: Re: tar versus cpio
- Previous by thread: Re: Looking for benchmark test tool
- Next by thread: Latex file eps include problem
- Index(es):
Relevant Pages
|
Loading
