Re: Benchmarking an IO library

On Feb 27, 6:19 am, David Schwartz <dav...@xxxxxxxxxxxxx> wrote:
On Feb 26, 1:34 am, mathieu <mathieu.malate...@xxxxxxxxx> wrote:

I am currently working on an open source IO library for DICOM files.
I have been trying to optimize the reading time, and found something I
do not understand. The first time I am reading a file, the time to
read (as measured with the unix 'time' command) can be around ~24s,
while if I read it back again immediately it can be as low as ~3s.

1. Is this something known ?
2. What is this ?

Are you being serious? It's hard to imagine any even marginally

Nobody's perfect :/

competent programmer who doesn't understand the most basic information
about memory access. The first time, the data must be read from disk.
The second time, it is still in memory. (The operating system has no
reason not to keep this data in memory. If it is never needed again,
no harm is done. If it is used again, there is a huge benefit. If the
memory is needed for some other purpose, the operating system can
simply use it for that purpose.)

3. Is there a way to control that from my application (let say I know
the list of files that need to be read next) ?

Try 'man fadvise'. Depending on what you're doing, 'mmap' and
'madvise' may be better.

Anyway, since I am running linux, I decided to go for the readahead
approach and implemented a daemon (much like the readahead-list
package) instead of the fadvice+ suggested fd. I do not understand
what exactly the difference is (apparently one is blocking the other
not, but that does not mean anything for me). And since I can now
assume my file are in memory and not on disk I am now able to read a
series of files from differents threads.

Ref:
http://gdcm.svn.sourceforge.net/viewvc/gdcm/trunk/Applications/gdcmreadahead.c?view=markup


This makes my life so much easier now to benchmark the whole thing
using valgrind !

Thanks
-Mathieu
.

Relevant Pages

  • Re: Benchmarking an IO library
    ... I have been trying to optimize the reading time, ... The first time, the data must be read from disk. ... it is still in memory. ... memory is needed for some other purpose, ...
    (comp.os.linux.development.apps)
  • RE: Memory Error
    ... first time it shows the follwoing error message ... If you get Memory Cannot be read or written. ... Under General Tab clear your History, Internet Files and Cookies. ... You may have a bad RAM try to test your RAM by running Memtest by ...
    (microsoft.public.windowsxp.general)
  • Re: Assembly.LoadFrom and machine restart
    ... Most probably your assembly is cached in memory by windows the first time, ... > dll xrt.dll for which I have not maintained versioning... ... > When I launch my application after a machine reboot (xrt.dll already ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Hardware Affect
    ... First time I ever saw somebody measure RAM in milligrams. ... RAM will only make visual effects faster if lack of memory was the ...
    (comp.os.linux.hardware)
  • Re: Memory allocation problem
    ... Your statement of "But reading it gets garbage. ... First time." ... I had a similar problem when dealing with linked lists - I malloced enough memory for a header structure plus some string memory after it to which an entry in the structure pointed; however, due to a misplaced cast, memory was not be allocated to the size I thought it was. ...
    (alt.os.linux)