automatically restarting dying daemons?

From: Will Trillich (will_at_serensoft.com)
Date: 06/30/04 

Date: Wed, 30 Jun 2004 16:34:06 -0500
To: debian-user@lists.debian.org

problem: xinetd, after working just fine and dandy for weeks at
        a time, gets dozens of "unexpected signal" (source unknown)
        and gives up the ghost.

questions:
        1) what's the best way (e.g. debian way) to monitor active
           daemons and restart them when necessary? maybe some
           utility already exists for this? or /proc/something?
           or `ps ax`?
        2) how can i track down the source of the signals specific
           to this case and make it stop?

xinetd chugs along nicely for the most part, and then -- poof!
-- it dies a sudden death:

root@boss# cd /var/log
root@boss# grep xinetd daemon.log

Jun 30 13:39:13 boss xinetd[21953]: {general_handler} (21953) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:13 boss xinetd[21953]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:13 boss xinetd[4873]: Resetting...
Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:13 boss xinetd[4873]: Resetting...
Jun 30 13:39:25 boss xinetd[21964]: {general_handler} (21964) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:25 boss xinetd[21964]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:25 boss xinetd[4873]: Resetting...
Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:25 boss xinetd[4873]: Resetting...
Jun 30 13:41:27 boss xinetd[21990]: {general_handler} (21990) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:41:27 boss xinetd[21990]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
Jun 30 13:41:27 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:41:27 boss xinetd[4873]: Resetting...
Jun 30 13:41:27 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:41:27 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:41:27 boss xinetd[4873]: {bad_signal} Received 50 bad signals. Exiting...
Jun 30 15:51:56 boss xinetd[23062]: xinetd Version 2.3.4 started with libwrap loadavg options compiled in.
Jun 30 15:51:56 boss xinetd[23062]: Started working: 8 available services

so that shows when xinetd died. here's all the activity at
that time plus-or-minus a second or two (13:39:12 - :14 and
13:39:24 - :28):

root@boss# find . -type f \
        | xargs grep 'Jun 30 13:39' \
        | egrep ':1[234] |:2[45678] '

./syslog:Jun 30 13:39:13 boss xinetd[21953]: {general_handler} (21953) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:13 boss last message repeated 9 times
./syslog:Jun 30 13:39:13 boss xinetd[21953]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
./syslog:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:13 boss last message repeated 9 times
./syslog:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:13 boss postgres[21954]: [1] DEBUG: pq_recvbuf: recv() failed: Connection reset by peer
./syslog:Jun 30 13:39:13 boss postgres[21954]: [2] DEBUG: incomplete startup packet
./syslog:Jun 30 13:39:13 boss -f[21958]: (v4.0.4) Unable to get canonical name of client 127.12.21.44: Unknown host (1) [pop_init.c:1075]
./syslog:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:13 boss last message repeated 8 times
./syslog:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:13 boss -f[21958]: (null) at 127.12.21.44 (127.12.21.44): -ERR POP EOF or I/O Error [popper.c:820]
./syslog:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./syslog:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./syslog:Jun 30 13:39:25 boss postgres[21962]: [1] DEBUG: pq_recvbuf: recv() failed: Connection reset by peer
./syslog:Jun 30 13:39:25 boss postgres[21962]: [2] DEBUG: incomplete startup packet
./syslog:Jun 30 13:39:25 boss xinetd[21964]: {general_handler} (21964) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:25 boss last message repeated 9 times
./syslog:Jun 30 13:39:25 boss xinetd[21964]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
./syslog:Jun 30 13:39:25 boss spamd[371]: connection from localhost [127.0.0.1] at port 4706
./syslog:Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:25 boss last message repeated 9 times
./syslog:Jun 30 13:39:25 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./syslog:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./syslog:Jun 30 13:39:25 boss spamd[21967]: SIGPIPE received - reopening log socket
./syslog:Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:25 boss last message repeated 8 times
./syslog:Jun 30 13:39:25 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:26 boss -f[21968]: (null) at localhost (127.0.0.1): -ERR POP EOF or I/O Error [popper.c:820]
./syslog:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./syslog:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./auth.log:Jun 30 13:39:25 boss sshd[21963]: warning: can't get client address: Connection reset by peer
./auth.log:Jun 30 13:39:25 boss sshd[21963]: Could not write ident string to 127.0.0.1
./daemon.log:Jun 30 13:39:13 boss xinetd[21953]: {general_handler} (21953) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:13 boss last message repeated 9 times
./daemon.log:Jun 30 13:39:13 boss xinetd[21953]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
./daemon.log:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:13 boss last message repeated 9 times
./daemon.log:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./daemon.log:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:13 boss last message repeated 8 times
./daemon.log:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./daemon.log:Jun 30 13:39:25 boss xinetd[21964]: {general_handler} (21964) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:25 boss last message repeated 9 times
./daemon.log:Jun 30 13:39:25 boss xinetd[21964]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
./daemon.log:Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:25 boss last message repeated 9 times
./daemon.log:Jun 30 13:39:25 boss xinetd[4873]: Resetting...
./daemon.log:Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:25 boss last message repeated 8 times
./daemon.log:Jun 30 13:39:25 boss xinetd[4873]: Resetting...
./mail.log:Jun 30 13:39:13 boss -f[21958]: (v4.0.4) Unable to get canonical name of client 127.12.21.44: Unknown host (1) [pop_init.c:1075]
./mail.log:Jun 30 13:39:13 boss -f[21958]: (null) at 127.12.21.44 (127.12.21.44): -ERR POP EOF or I/O Error [popper.c:820]
./mail.log:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./mail.log:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./mail.log:Jun 30 13:39:25 boss spamd[371]: connection from localhost [127.0.0.1] at port 4706
./mail.log:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./mail.log:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./mail.log:Jun 30 13:39:25 boss spamd[21967]: SIGPIPE received - reopening log socket
./mail.log:Jun 30 13:39:26 boss -f[21968]: (null) at localhost (127.0.0.1): -ERR POP EOF or I/O Error [popper.c:820]
./mail.log:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./mail.log:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./mail.info:Jun 30 13:39:13 boss -f[21958]: (null) at 127.12.21.44 (127.12.21.44): -ERR POP EOF or I/O Error [popper.c:820]
./mail.info:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./mail.info:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./mail.info:Jun 30 13:39:25 boss spamd[371]: connection from localhost [127.0.0.1] at port 4706
./mail.info:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./mail.info:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./mail.info:Jun 30 13:39:25 boss spamd[21967]: SIGPIPE received - reopening log socket
./mail.info:Jun 30 13:39:26 boss -f[21968]: (null) at localhost (127.0.0.1): -ERR POP EOF or I/O Error [popper.c:820]
./mail.info:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./mail.info:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./mail.warn:Jun 30 13:39:25 boss spamd[21967]: SIGPIPE received - reopening log socket
./debug:Jun 30 13:39:13 boss postgres[21954]: [1] DEBUG: pq_recvbuf: recv() failed: Connection reset by peer
./debug:Jun 30 13:39:13 boss postgres[21954]: [2] DEBUG: incomplete startup packet
./debug:Jun 30 13:39:25 boss postgres[21962]: [1] DEBUG: pq_recvbuf: recv() failed: Connection reset by peer
./debug:Jun 30 13:39:25 boss postgres[21962]: [2] DEBUG: incomplete startup packet

as you can see below, i'm using kernelt 2.4.18-bf2.4...

are there any likely suspects in there? are there any likely
suspects to be found elsewhere? 

-- 
I use Debian/GNU Linux version 3.0;
Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown
 
DEBIAN NEWBIE TIP #51 from Will Trillich <will@serensoft.com>
:
Interested in CUSTOMIZING MUTT to work the way you'd like?
Visit Tom Gilbert's site at http://linuxbrit.co.uk/mutt/ and
download his .muttrc to your home directory (save it under a
different name if you're paranoid like I am, then tell mutt
":source file/path/here" to give it a whirl). Wow!
Also see http://newbieDoc.sourceForge.net/ ...
-- 
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org