Re: lseek/write

On Nov 20, 12:00 pm, David Schwartz <dav...@xxxxxxxxxxxxx> wrote:
On Nov 19, 7:39 pm,golden<i...@xxxxxxxxxxxxx> wrote:

What I am trying to determine is why my writes are faster to my local
disk than to my SAN ( all things being equal) .

I have no idea what you mean. How are you measuring? How much faster?

If I remove the
fsync, my write is faster to the SAN.

Faster than with the 'fsync'? Or faster than to the local disk?
Measured how? By how much?

I assume that the OS handles
the writes far more effeciently in this case. When I fsync after each
write i remove any performance enhancements the OS can provide. So
The only conclusion I can come up with is that there are more hops to
the SAN. The transfer from memory to local disk cache is faster than
the fiberchannel path.

Or your local disk is not really supporting the 'fsync' (for example,
committing it only to the disk's buffer cache) and your SAN is
respecting it (fully committing it to permanent media before allowing
your program to continue). I would have to see your evidence to
support your conclusion.

DS

Hi.
My user is complaining that his writes to SAN are slower than local
disk. He provided me a program to test this, I modified it and am
including it below. I ran the test writing 800 bytes 50000 times.
When I wrote to SAN, it took ~ 60 seconds. Local disk took ~40
seconds. If I remove the fsync, the overall operation is faster to
SAN than to local disk.

My conclusion is that increasing the number of operations, increases
the number of fsyncs which is slower to the NAS.

Of course this removes any buffering or caching that the kernel
performs. My hardware is a DL585 G1 with a raid controller.

I have spoken to some of our storage folks and even a guy at emc.
There comment is that the majore reasons are that 1) there is a larger
distance between memory and the raid controller than to the SAN. 2)
There is more contention on the SAN controller. If I conclude that
the fsync operation is 5 milliseconds faster to local disk than to
SAN, on 50,000 operations that would equate to 25 seconds wich fits
nicely into my equation, but I know that is not a good way to
measure.

here is the code that I used to test this hypothesis:

#include <sys/types.h>
#include <sys/time.h>


#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>


int main(int argc, char **argv)
{
struct timeval start, end;
double usecs;
long val;
int ch, fd, idx, ops, numThreads;
char *fname= "";
int filesize = 40000000;
int bytes = 0;
bool dosync = true, doSeek=false;


bytes = 0;
ops = 0;
char *buf = new char[bytes];
fname = argv[1];


while (( ch = getopt(argc,argv, "b:o:f:sl")) != EOF)
switch (ch) {
case 'b' :
bytes = atoi(optarg);
break;
case 'o' :
ops = atoi(optarg);
break;
case 'f' :
fname = (optarg);
break;
case 's' :
dosync = false;
break;
case 'l' :
doSeek = true;
break;
}
argc -= optind;
argv += optind;


gettimeofday(&start,NULL);


memset(buf,0,bytes);
if ( dosync ) {
printf("Processing %d bytes with %d Operations of fsync :
\t",
bytes,ops);
} else {
printf("Processing %d bytes with %d Operations of fsync :
\t",
bytes,1);
}


// unlink(fname);
if ((fd = open(fname, O_RDWR | O_CREAT, 0666)) == -1)
{
int errNum = errno;
printf("ERROR: failed to open %s: n",fname);
return(0);
}


for ( int idx(0) ; idx < ops ; idx++)
{


if (write(fd, buf, bytes) != bytes)
{
printf("write: \n");
exit (1);
}


if ( dosync ) {
if (fsync(fd) != 0)
{
printf("fsync: \n");
exit (1);
}
}
if ( doSeek )
{
if (lseek(fd, (off_t)0, SEEK_SET) == -1)
{
printf("lseek: %s\n",
strerror(errno));
exit (1);
}
}


}


// One last sync


if (fsync(fd) != 0)
{
printf("fsync: \n");
exit (1);
}
gettimeofday(&end,NULL);


int totalSec = 0;
long totalUSec = 0;


if (start.tv_usec > end.tv_usec) {
end.tv_usec += 1000000;
end.tv_sec--;
}


totalSec = end.tv_sec - start.tv_sec;
totalUSec = end.tv_usec - start.tv_usec;
int t = totalSec + (totalUSec / 1000000);


printf("%ld Hours ",t / ( 60 * 60));
t %= (60*60);
printf("%ld Minutes ",t / 60);
t %= 60;
printf("%ld.%ld Seconds ",t ,totalUSec);
printf("%ld.%ld Seconds\n ",totalSec ,totalUSec);
.

Relevant Pages

  • Re: lseek/write
    ... disk than to my SAN. ... How are you measuring? ... fsync, my write is faster to the SAN. ...
    (comp.os.linux.development.system)
  • Re: lseek/write
    ... My user is complaining that his writes to SAN are slower than local ... I ran the test writing 800 bytes 50000 times. ... Local disk took ~40 ... What's missing is any evidence or proof that the affect of 'fsync' is ...
    (comp.os.linux.development.system)
  • Re: File and Print Server Cluster- Active/Active SAN Attached
    ... It has to be on the SAN, local disk is not supported. ... MVP - Windows Server - Clustering ...
    (microsoft.public.windows.server.clustering)
  • Re: Booting from SAN or local?
    ... Microsoft will not support it. ... >>> SAN. ... >> using local disk connecting using a local bus. ... >>> likely lose significant functionality by switching to local disk. ...
    (microsoft.public.windows.server.clustering)
  • Re: How to verify/fix High Disk Read Latencies in Exch2003 ?
    ... Exchange production servers are, the SAN is an EMC CX600. ... >>> current log file on disk and continues until data in the log buffers ... Comingling occurs whe two or more LUNs reside ...
    (microsoft.public.exchange.admin)