Re: linux, sockets, c

lsg wrote: 
Hi,

I'm new to linix programming and I need some help. I'm writing a program
to collect data from remote devices using sockets.
My program:
-connects to remote device (sockets,TCP)
(5x){
-sends some data,
-receives some bytes
}
-decode data
-write to mysql database
-close connection

It takes ~20s. There are n devices (~50). I need to do every 5 min.
The ideal solutions is to read all devices at the same time. How to
impement that? Fork() ? Threads ? Select() or write one client and with
crontab run program ip1, program ip2.... ? Any suggestions, pointers,
help or links would be very helpful. Maybe it could be done like port
scanner?

Thanks a lot
Leo
Think a little about the logical message protocol you are using a bit. Design the message structure so that:

a) use a minimum of request/reply message handshakes.

b) block the data up as much as possible.

c) minimize the number of actual send()/recv() calls it takes.

d) write and read into large buffers and buffer up what you are sending/receiving. Then do parsing or reading from the buffers. You want to leverage > 1,024 byte send()/recv()'s whenever possible to let TCP send fully loaded packets.

e) try to use a streaming protocol as much as possible. Latency with request/reply type messages really really adds up in a big hurry.

f) lastly, if you need to, you can use a library like bzip2 or libz (or is it zlib) to compress data on the fly on both the sending and reading sides if your data is really bulky. However, that can take a while to get exactly right (been there on a couple projects) so do all you can with (a) through (e) before you dive into compression. Compression on the fly gives you the most bang if you are streaming the data. Also, on a local LAN, you may trade network volume bottlenecks for CPU load bottlenecks. It may not be worth the effort. Over long hauls, steaming with on-the-fly compression can really crank up the throughput.

I've just been working on a project to maximize message throughput, and every one of these was critical to getting maximum performance.


Rick.
.