Compromised Solaris 9

I have Solaris 9 installed on E420R server. It has oracle DB and Application server running. This server behind firewall and only selective ports are opened to access application from Internet. Recently I found some suspicious files and process running on system. It looks my server has been compromised.

bash-2.05#cd /dev/cua/...

bash-2.05# ls -la

total 1548

drwxr-xr-x6 rootroot 512 Feb 28 06:19 .

drwxr-xr-x3 rootroot 512 Feb 28 06:15 ..

-rw-r--r--1 rootroot18124 Oct 28 2000 CHANGES

-rw-1 rootroot17982 May 15 1997 COPYING

-rw-r--r--1 rootroot2660 Aug 8 2000 FAQ

-rw-r--r--1 rootroot35874 Oct 28 2000 README

-rw-r--r--1 rootroot 76 Oct 28 2000 TODO

drwxr-xr-x2 rootroot2048 Apr 7 2002 help

drwxr-xr-x2 rootroot 512 Feb 28 06:16 log

drwxr-xr-x2 rootroot 512 Apr 7 2002 motd

-rwxr-xr-x1 rootroot248708 Oct 28 2000 nfsd

-rwxr-xr-x1 rootroot 331 Apr 25 2002 ntpstats

-rwxr-xr-x1 rootroot430080 Feb 28 06:15 psy.tar

-rw-r--r--1 rootroot 76 Feb 28 06:15 psybnc.conf

-rw-1 rootroot7 Feb 28 06:16 psybnc.pid

drwxr-xr-x3 rootroot 512 Apr 7 2002 scripts

and following process:

bash-2.05# ps -ef|grep cua

root 198061 0Feb 28 ?0:00 /dev/cua/.../solbnc

bash-2.05#cd /xfn

bash-2.05# ls -la

total 13

dr-xr-xr-x3 rootroot3 Mar 6 10:11 .

drwxr-xr-x 40 rootroot1024 Mar 2 10:51 ..

dr-xr-xr-x1 rootroot1 Mar 6 10:14...

dr-xr-xr-x1 rootroot1 Mar 6 10:14 _dns

dr-xr-xr-x1 rootroot1 Mar 6 10:14 _orgunit

dr-xr-xr-x1 rootroot1 Mar 6 10:14 _thisens

dr-xr-xr-x1 rootroot1 Mar 6 10:13 _thisorgunit

dr-xr-xr-x1 rootroot1 Mar 6 10:14 _x500

dr-xr-xr-x1 rootroot1 Mar 6 10:14 org

dr-xr-xr-x1 rootroot1 Mar 6 10:14 orgunit

dr-xr-xr-x1 rootroot1 Mar 6 10:14 thisens

dr-xr-xr-x1 rootroot1 Mar 6 10:13 thisorgunit

There is evidence in root's .bash_history that /dev/cua/.../solbnc has been renamed as nfsd.

Request all of you to suggest how can I deal with this situation. What are the implications and whether really system compromised.

Shoud I delete all these stuff and stop solbnc process?

[2242 byte] By [Vijendra.Jaina] at [2007-11-26 20:37:27]
# 1

you need to review the ports that you have opened form to the internet.

Please list them! The best way to check all the open ports is to use a port scanner, this might also help to identify ports that an attacker might have opned on the node. Scan from both inside the network and also form the DMZ.

You have a problem and need to do some serious investigation, I would say got in. Also check with other admins, if any.

Do you have SNORT/SANCP/SGUIL/Cisco MARS etc. configured on for your DMZ? Check with your network admin.

this might help:

http://www.ucl.ac.uk/cert/nix_intrusion.pdf

HvRa at 2007-7-10 1:53:39 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...
# 2

A good place to start is:

http://www.sun.com/bigadmin/collections/security.html

Know your Enemies

Know your Enemies 2

Know your Enemies 3

These articles have some good info as to where to start. Always remember that even though you have your firewall and systems locked down...Most attacks happen from within :(

Good luck!

Lee_McCreerya at 2007-7-10 1:53:39 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...
# 3

We have following ports are opened and all these ports are required to access application installed on this server:

Ports:

7777, 5631, 5632, 5801, 1521, 4041, 5051, 7778, 500, 8888,

Also, SSH (22) is opened for few selective IPs.

We don't have any tools like SNORT/SANCP etc. configured.

IF any one can suggest some site/docs that will help to fix the problem. I can not re-install the system. Do you aware about tools that will help to find such problems. I have couple of more servers in same network and doubt whetgher they also compromised.

Vijendra.Jaina at 2007-7-10 1:53:39 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...
# 4
I strongly recommend that you run "Rootkit Hunter" http://www.rootkit.nl/You will have to compile it for Solaris
HvRa at 2007-7-10 1:53:39 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...
# 5

Since this server is exposed to the Internet, you should not have ftp, tftp, telnet, rsh, rlogin and rcp and their respectiv daemons available on the system -- that is, the executables should be removed (this is not an exhaustive list). Rely solely on using ssh to access the system for maintenance, and scp for file transfers.

Make sure you have removed anonymous, and uucp and other none essential users from passwd and shadow.

Even if there is a firewall between tha server and the Internet, there should also be a firewall between it and the rest of your network that acts like a one-way mirror, only letting inside servers to access the exposed server, not allowing the exposed server to reach inside.

Since this is an Oracle server, write the system backups to a filesystem that is on appropriate Enterprise storage, then either use scp to pull the data inside for backup to tape, or use the storage's internal copying capability to bring the data to a separate secure set of volumes that can be mounted inside for backup to tape. Restoration works the same way, either push to the landing site using scp or land to disk internally, copy to external landing disk, then mount those disk and use to recover the database.

Chasmana at 2007-7-10 1:53:39 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...
# 6

As I told I can not format and re-install system. I want to do fic this issue as much as possible. Using check rootkit and manual inspection logs, processes, I identified the unwanted processes, files in system and made a list of that.

Now, should I remove all unwanted file, programs and processes manually, replace compromise binaries with good binaries from CD? Just wanted to know whether is theie any harm doing that. Worrying about logic bomb that can be triggered while doing cleaning task and can do more damage.

Vijendra.Jaina at 2007-7-10 1:53:40 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...
# 7

Seriously, your best option is to completely re-install. There's no telling what kind of things the miscreant left behind on your system and what backdoors he installed for himself (that have not been detected).

Also, go ahead and change -all- the passwords on the system (or your network if those passwords are the same for more than one system). It's a safe bet the guy's took your /etc/shadow and that he's already gotten all the passwords in there.

Cailin_Coilleacha at 2007-7-10 1:53:40 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...