RE: Server hacked?

Johan, NoOp,

Thanks for your comments.
It seems I can be quite lucky, as the damage seems to be rather
contained to a very limited set of my system. The processes are of the
user www-data. So it seems a web site has been hacked instead. (Count
myself lucky this time)

Evidence:
root@shushan:/proc/29965# ls -l
total 0
[snip]
lrwxrwxrwx 1 www-data www-data 0 2008-01-01 23:07 exe -> /usr/bin/perl
[snip]
root@shushan:/proc/29965# ls fd -l
total 10
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 0 -> socket:[543920]
l-wx------ 1 www-data www-data 64 2008-01-01 23:01 1 -> pipe:[543929]
l-wx------ 1 www-data www-data 64 2008-01-01 23:01 2 ->
/var/log/apache2/error.log
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 3 -> /tmp/.apc.TeL4il
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 4 -> /tmp/.apc.MpL1Yi
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 5 -> /tmp/.apc.2d14Oh
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 6 -> /tmp/.apc.5zT9Eg
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 7 -> /tmp/.apc.Njjgvf
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 8 -> socket:[525267]
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 9 -> socket:[784974]

It seems it limited to the www-data user. Of course the fd/3 file gives
a very good hint that it is indeed apache. The apc comes from PHP. Of
course its odd that the webpage runs perl, since all processes are
supposed to be PHP.
Up to a certain point, this restores my faite (hope) that the system can
be trusted (up to some extend).

Some oddities are these on a site running Joomla are:
===
htdocs/.wonk/motd/USER1.MOTD.old::arcor.de.eu.dal.net 372 _mugo1`dealz
:- You can always reach this server by typing /server arcor.dal.net
6667
htdocs/.wonk/motd/USER2.MOTD.old::arcor.de.eu.dal.net 372 [X]enzo :-
You can always reach this server by typing /server arcor.dal.net 6667
htdocs/.wonk/src/p_client.c:
ap_snprintf(irccontent,sizeof(irccontent),"6667");
htdocs/.wonk/src/p_inifunc.c: if (ern != 0) { user(usernum)->port =
6667; } else { user(usernum)->port = atoi(value); }
Binary file htdocs/.wonk/src/p_client.o matches
Binary file htdocs/.wonk/vi matches
htdocs/dimenso:my $porta="6667";
htdocs/dimenso.1:my $porta="6667";
htdocs/scanner.pl:my $porta="6667";
===

It also has wonk.tar.gz from 2007-03-18.

Anyone familiar with this?

I hope to diagnose when the incident occurred an how to protect against
it better in the future.


At least there are some lessons in this:

* Use one-user-per-website only (easier auditing).
* Deploy you firewall with strict rules.
* Do auditing & automated monitoring.
* Keep longer logs if you don't automatically monitor your systems.
* Keep ALL software up to date (something automatic for websites?)

* Its good policy to deny traffic, except if required for system/website
operation.


I'll be moving all stuff to a new box with Xen and more isolation, these
seem good lessons to get started. Still it seems to be quite a lot of
work for doing it right (or at least better) the second time. Any
suggestions from experienced people that might help me?


Thanks,


- Joris

-----Original Message-----
From: ubuntu-users-bounces@xxxxxxxxxxxxxxxx
[mailto:ubuntu-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of johanb
Sent: Tuesday, 1 January 2008 23:26
To: Ubuntu user technical support,not for general discussions
Subject: Re: Server hacked?


Joris,

There seems to be some kind of rootkit running on your server.
It concerns syn_sent , towars numbering devices, which
inclines scans for open dcc-servers, to sent packets to. What
I would advice you, is to install a package searching for
installed rootkits and scan your system with ex. chkrootkit or
rkhunter.
To answer your question : yes according to my humble opinion
it concerns some kind of rootkit or trojan. (rather rootkit
then trojan) greetz, Johan

Joris Dobbelsteen schreef:
Dear,

I'm having a Linux server running Ubuntu 6.06 LTS (under vmware) and
it seems to have some quite weird behaviour. For some reason (or
another) the box seems to create OUTGOING connections to an
IRC server
from a supposed kernel address. Below is a snapshot of the
netstat output...

Can anyone provide some detailed information about this?
(i.e. confirm my suspicion it is indeed a trojan or something)

If so, I'm desiring to do some more diagnosis/forensics on
the box to
get to know what may caused this strange behaviour. Can
anyone provide
some help on this?

The box has PostFix, PowerDNS, Apache2 and SSH exposed to
the Internet.
Unfortunally its connected to the single LAN segment I have at home.
Fortunally I have a strict firewall that doesn't allow IRC out (I
don't use it, so I do not need to allow it).

Thanks in advance...

- Joris


root@shushan:~# netstat -tnp
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 1 192.168.10.xx:52020 216.152.66.54:6667
SYN_SENT 18687/[kjournald]
tcp 0 1 192.168.10.xx:49383 216.152.66.48:6667
SYN_SENT 18599/[kjournald]
tcp 0 1 192.168.10.xx:32936 216.152.66.46:6667
SYN_SENT 11304/[kjournald]
tcp 0 1 192.168.10.xx:56901 216.152.67.49:6667
SYN_SENT 29965/[kjournald]
tcp 0 1 192.168.10.xx:54354 216.152.67.30:6667
SYN_SENT 24235/[kjournald]
tcp 0 1 192.168.10.xx:60278 216.152.66.47:6667
SYN_SENT 15412/[kjournald]
[trusted entries removed]





--
ubuntu-users mailing list
ubuntu-users@xxxxxxxxxxxxxxxx
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-users



--
ubuntu-users mailing list
ubuntu-users@xxxxxxxxxxxxxxxx
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users