Shm00 Splunk Fu - Part 1

I'm going to start writing a series on Splunk Fu and how to use it to detect malicious/nefarious activity within your network.

The structure will vary.

I am not going to write a tutorial on installing Splunk, there are plenty of those online.

I'm going to give you specific examples for dashboards and alerts that you can create for your own environment.

Today's example will be creating a query for detecting new services that are installed on a machine.


Why is this useful?

• If you have a Systems Engineering team, they *should* know what is being installed on the servers.
• If something gets installed and they don't know what it is, it could be malicious.
• If it is not malicious, it could simply be unintentional, which is another problem.
• You can detect if malware is being installed on multiple machines, and stop a small problem from turning into a big problem.
• After a certain sample size, you can compile a pseudo-white list of approved services, so you gain a better understand of what actually exists on your servers and network.

How do I prepare?

• Splunk needs to be on all servers.
• GPO settings for Advanced Auditing needs to be enabled for "Security System Extension" so that Event ID 4697 is being logged by the Windows Servers.
• Install a driver on the machine and verify within Windows Event Viewer that a message "A service was installed in the system." is populated.

I am going to show you the way I do this, you can craft your own that works within your environment.

First, I create a button that will indicate the number of services installed, and will change colors depending on the severity.

If this is not lighting up, I don't even have to think about services being installed.

You create this panel within a dashboard by utilizing the "Single" function w/in Splunk's dashboard XML. 

My XML looks like this:

<row>

    <single>

      <searchString>index="Windows" sourcetype=WinEventLog:System "Message=A service was installed in the system." | stats count by Service_File_Name | stats sum(count) as total_count | fillnull value=0 AS total_count | rangemap field=total_count low=0-0 elevated=1-3 severe=4-10000</searchString>

      <title>Number of unauthorized services installed</title>

      <earliestTime>-24h@h</earliestTime>

      <latestTime>now</latestTime>

      <option name="afterLabel">services.</option>

      <option name="beforeLabel"/>

      <option name="charting.chart.nullValueMode">gaps</option>

      <option name="charting.chart.rangeValues">[0,1,5,100]</option>

      <option name="charting.chart.stackMode">default</option>

      <option name="charting.chart.style">minimal</option>

      <option name="charting.gaugeColors">[0x84e900,0xffe800,0xbf3030]</option>

      <option name="charting.layout.splitSeries">false</option>

      <option name="charting.legend.placement">right</option>

      <option name="charting.secondaryAxis.maximumNumber">""</option>

      <option name="charting.secondaryAxis.minimumNumber">""</option>

      <option name="charting.secondaryAxis.scale">""</option>

      <option name="classField">range</option>

      <option name="count">50</option>

      <option name="displayRowNumbers">true</option>

    </single>

  </row>

Then, I create an additional dashboard panel that gives me the details of the installed services.

Click to View Larger Image

My XML for this panel looks like this:

<row>

    <table>

      <searchString>index="Windows" sourcetype=WinEventLog:System "Message=A service was installed in the system." | table _time, User,  Service_File_Name, host, Message</searchString>

      <title>Detailed view of unauthorized services installed .</title>

      <earliestTime>-24h@h</earliestTime>

      <latestTime>now</latestTime>

      <option name="charting.axisTitleX.text">File Name</option>

      <option name="charting.chart.rangeValues">[-1,0,1,15]</option>

      <option name="charting.gaugeColors">[0x84e900,0xffe800,0xbf3030]</option>

      <option name="charting.primaryAxisTitle.text"/>

      <option name="count">10</option>

      <option name="dataOverlayMode">highlow</option>

      <option name="displayRowNumbers">true</option>

    </table>

  </row>

In the next post we will talk about using lookup lists.

Updates coming...

Took a hiatus, will start doing weekly updates.

Artillery

I have known about honeypots for a little while now, but haven't had any professional experience with them. When I first read about the idea of a honeypot I thought it sounded really fun. I don't know about you, but I have a passion for information security because it is exciting. There is sort of a secret agent appeal to a lot of the industry, honeypots being one of them.

As defined by ISACA: A specially configured server, also known as a decoy server, designed to attract and monitor intruders in a manner such that their actions do not affect production systems. 

Scope Notes: Also known as "decoy server".

So, when I was looking at the Social Engineering Toolkit, Dave Kennedy also promotes his honeypot, Artillery.

Artillery is a honeypot/monitoring/prevention tool used to protect Linux-based systems. Artillery will setup multiple ports on the nix system and if anything touches it will automatically blacklist them. In addition, it monitors the file system for changes and e-mails the changes back to you. It also detects SSH brute force attacks and automatically blocks them as well.

I went ahead and downloaded it and let it take its natural course. Well, you won't really notice anything happen until you configure it properly. So, let's walk through a few of the steps you'll take to configure your honeypot.

1) Download Artillery.

root@:/root/# svn co http://svn.secmaniac.com/artillery artillery/

2) Configure Artillery.

root@:/root/# gedit /var/artillery/config



Feel free to use whatever editor you use, nano, pico, vi, etc.


There are some pretty self explanatory options that you need to configure.


# DETERMINE IF YOU WANT TO MONITOR OR NOT
MONITOR=YES



# THESE ARE THE FOLDERS TO MONITOR, TO ADD MORE, JUST DO "/root","/var/", etc.
MONITOR_FOLDERS="/var/www","/etc/","/root/"

# BASED ON SECONDS, 2 = 2 seconds.
MONITOR_FREQUENCY=60

# DO YOU WANT TO TURN ON THE HONEYPOT
HONEYPOT=YES

# DO YOU WANT TO AUTOMATICALLY BAN ON THE HONEYPOT
HONEYPOT_BAN=YES

# WHITELIST IP ADDRESSES, SPECIFY BY COMMAS ON WHAT IP ADDRESSES YOU WANT TO WHITELIST
WHITELIST_IP=127.0.0.1,localhost

Those sort of begin the basic functionality of the honeypot/monitoring tool. There are some preferential options after this. Which will be up to your discretion and how it fits your scenario. If you have a router properly filtering traffic to your network, you likely will not be seeing much action. However, if you want to see how much traffic would be getting through or are just curious to see Artillery in action, you could forward port 80 from your router to your local device. It can be quite alarming to see the random wanderings and probing that are happening all the time.




Kudos to SECMANIAC for Artillery, a great product.

MySQL .sock error resolved

After a few days of troubleshooting, installing, re-installing, purging, auto-removing, cleaning, auto-cleaning, updating, and upgrading, I have solved my MySQL woes.

Indeed, the problem appeared that MySQL service was not starting up properly on boot and manually starting the service was either hanging or just not getting the job done.

So, I looked at chkconfig and mysql was not enabled to start.

I opened:


sudo gedit /etc/rc.local


and added the line:

start mysql


right before the exit command.

Lo and behold, upon reboot and no other commands entered, I was able to connect to MySQL without any sock error.

MySQL continued

Ok so, after typing all that up and going through a hundred or more forum posts, I was actually still having the same problem of MySQL-Server hanging when I was trying to install.

Fortunately, I documented everything as I found potential solutions, and referenced all of it to fix my problem.

Again, my symptoms were:

• apt (advanced packaging tool) could not complete fresh install of MySQL. It would hang on the line: Setting up MySQL-Server-5.1
• Because apt would freeze, I would have to force close, which would give me /var/lib/dpkg/lock issues, aka error: "Could not get lock /var/lib/apt/lists/lock"
• Any attempt to purge/remove MySQL would end erroneously, either because of missing dependencies, or other errors. 
• If I was able to trick MySQL into kind of finishing install, I would receive the error: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'

To solve these problems:

Problem #1: apt (advanced packaging tool) could not complete fresh install of MySQL. It would hang on the line: Setting up MySQL-Server-5.1

Solution: I found that if I went into a new terminal tab while that was "setting up" and typed in:

  service mysql stop

It allowed the install to complete.

Problem #2: Because apt would freeze, I would have to force close, which would give me /var/lib/dpkg/lock issues, aka error: "Could not get lock /var/lib/apt/lists/lock"

Solution: First command I would type in: 

sudo fuser -vki /var/lib/dpkg/lock

Then, I entered every command verbatim (including the prefix LANG=C;) from Ubuntu's Help Site (Step 5):

sudo fuser -vvv /var/lib/dpkg/lock

cat /etc/lsb-release

uname -a

sudo rm /var/lib/apt/lists/lock 

sudo cp -arf /var/lib/dpkg /var/lib/dpkg.backup

sudo cp /var/lib/dpkg/status-old /var/lib/dpkg/status

sudo cp /var/lib/dpkg/available-old /var/lib/dpkg/available

sudo rm -rf /var/lib/dpkg/updates/*

sudo rm -rf /var/lib/apt/lists

sudo rm /var/cache/apt/*.bin

sudo mkdir /var/lib/apt/lists

sudo mkdir /var/lib/apt/lists/partial

LANG=C;sudo apt-get clean

LANG=C;sudo apt-get autoclean

LANG=C;sudo apt-get --purge autoremove

LANG=C;sudo apt-get update -o APT::Cache-Limit=25165824

sudo dpkg --configure -a

sudo dpkg --clear-avail

LANG=C;sudo apt-get -f install

LANG=C;sudo apt-get --fix-missing install

LANG=C;sudo apt-get update -o APT::Cache-Limit=25165824 && sudo apt-get dist-upgrade

This cleared up my apt-get/dpkg issues, including partially installed packages.

Problem #3: Any attempt to purge/remove MySQL would end erroneously, either because of missing dependencies, or other errors. 

Solution: Perform the previous step before you go here. 

First I kill any mysql process, check: 

ps -e | grep mysql

And make sure mysql/mysqld is not running. Then:

apt-get --purge autoremove mysql-common

apt-get --purge autoremove mysql-server

In addition to this, I deleted /var/lib/mysql, because that appeared to contain some user information and configuration files that were overlapping into the "fresh installs". I did a locate mysql and looked around at some of the folders and deleted any that I thought contained configuration files or user info. 

Problem #4: If I was able to trick MySQL into kind of finishing install, I would receive the error: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'

Solution: As long as you delete /var/lib/mysql, purge all MySQL packages and perform a fresh install, you should no longer receive this error. Just make sure your mysqld (MySQL daemon) service is running and your /etc/my.cnf file is pointing to the right location. But, since you theoretically did a fresh install, the default my.cnf settings should be correct and working, they were for me.

Good luck.

MySQL and dpkg lock Troubleshooting

Well, I'm having problems with my LAMP setup, so I am going to document some of it. Hopefully, if someone comes across similar errors/symptoms, this will help solve their problems as well. I am running Ubuntu 11.10, no significant mods.


Symptoms:

• Initial MySQL error: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
• Attempts to fix/repair MySQL installation fail, leading to decision to uninstall/re-install MySQL.
• Upon attempt to re-install MySQL, Ubuntu hangs while "Setting up mysql-server-5.1 (5.1.58-1ubuntu1)", and can not Ctrl+C out of it, forced to close terminal. (Even after waiting 30+ minutes.)
• Attempts to re-install anything at this point leads to error message "Could not get lock /var/lib/apt/lists/lock". Or any variation of /var/lib/lock.

Steps Taken:

1. Unfortunately, the "lock" error is very persistent and even following many common troubleshooting steps, can be difficult to alleviate. 

What is happening when the /var/lib/apt/lists/lock, or any other lock file is preventing apt (advanced packaging tool)?

This error appears to occur when you have halted apt in the middle of upgrading/installing/purging without letting it complete on its own. It can also occur when you have apt and Synaptic Package Manager running at the same time. There may be other instances of this occurring independent of prematurely ending apt. 

How do I fix this? (MOST SUCCESSFUL SOLUTION: fuser -vki /var/lib/dpkg/lock)

Now that you understand why the error is occurring, you can evaluate your unique problem. 

Basic Possible Solution #1 & #1a: Check to make sure there is only one package manager running at a time. Ubuntu could have a few things going on, including Update Manager scanning for updates, Software Center installing something, Synaptic. If you can soft-close these, it will be better than force-closing/killing processes. Reboot. If you have already done this, go to step 2. This apparently solves the problem for some people.

Possible Solution #2: In a terminal, enter in the command line: ps -e | grep apt

This will show you any processes with the keyword apt (our likely culprit for the source of this problem) and you can then kill the process(es) with the command line input: sudo kill 8488 (change 8488 to the corresponding process PID). Once you complete this, you can check to see if this solved your problem by entering: apt-get update or apt-get upgrade.

If your apt command is working correctly, you have solved your problem. It may give you an error message that says: E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. If that is your situation, go ahead and type that into the command line. This will try to finish off where apt got hung up. If it completes its task and returns you to command line prompt, you should be in the clear. If it continues to hang or returns a /var/lib/lock error, try another solution. If you tried ps -e | grep apt and could not locate the process using the lock, try another approach: sudo fuser -vki /var/lib/dpkg/lock (not only will this find the process, the -k flag will kill it) or sudo lsof | grep /var/lib/dpkg/lock (this will only locate the process using the lock) (change the location of the lock file to fit your situation).

Possible Solution #3: You can try a less preferred method of removing the lock files causing the lockup. Because your problem is unique, your /lock will likely be in a unique file location. To begin, if your error is, for example, Could not get lock /var/lib/apt/lists/lock, you can try typing into the command line: sudo rm -f /var/lib/apt/lists/lock. Again, check if this fixes your problem by typing: apt-get update or apt-get upgrade. Additionally, a bit of a rough method, you can try: sudo killall -9 apt-get aptitude. I personally did not find much success in just outright deleting the lock file, I even went as far as to locate lock, and removed every instance of the file I could find. That did not get me any further. I include these examples because some people claim it solved their issue. I call these methods rough or less preferred, because you are not being very precise with your troubleshooting. It is more of a cluster-attack. Theoretically, there is one underlying problem here that you can eradicate with precision. It is just a matter of signal flow and finding where that problem begins and ends.

Possible Solution #4: Official Steps from Ubuntu's Help Site:

sudo fuser -vvv /var/lib/dpkg/lock

cat /etc/lsb-release

uname -a

sudo rm /var/lib/apt/lists/lock 

sudo cp -arf /var/lib/dpkg /var/lib/dpkg.backup

sudo cp /var/lib/dpkg/status-old /var/lib/dpkg/status

sudo cp /var/lib/dpkg/available-old /var/lib/dpkg/available

sudo rm -rf /var/lib/dpkg/updates/*

sudo rm -rf /var/lib/apt/lists

sudo rm /var/cache/apt/*.bin

sudo mkdir /var/lib/apt/lists

sudo mkdir /var/lib/apt/lists/partial

LANG=C;sudo apt-get clean

LANG=C;sudo apt-get autoclean

LANG=C;sudo apt-get --purge autoremove

LANG=C;sudo apt-get update -o APT::Cache-Limit=25165824

sudo dpkg --configure -a

sudo dpkg --clear-avail

LANG=C;sudo apt-get -f install

LANG=C;sudo apt-get --fix-missing install

LANG=C;sudo apt-get update -o APT::Cache-Limit=25165824 && sudo apt-get dist-upgrade

Steps Taken(continued)

Hopefully through that, you are able to get your lock cleared up, and now we can move on to repairing our MySQL setup.

Why is MySQL giving me a  'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)' error in the first place? 

Potential Causes:

• mysqld.sock file becomes corrupt.
• no disk space to write the .sock file to.
• mysqld is not running. (verify with: ps -ef | grep mysqld and ps -ef | grep mysql)
• the Linux user does not have permission to access the file.
• MySQL is expecting to find the socket in a location where it is not.

Possible Solution #1: Verify the user has permission to access the file. Do this by: 

usermod -G usernamehere

chmod g+rwx /var/lib/mysql

Possible Solution #2: Edit /etc/my.cnf to specify the correct location of the .sock file. If your /etc/my.cnf is pointing to /var/lib/mysql/mysql.sock, try changing it to /tmp/mysql.sock

Possible Solution #3: service mysql start     or     /etc/init.d/mysqld start

Steps Taken(conclusion)

Ultimately, I was able to fix every problem I was having through one or more methods described above. If you got through everything and still having MySQL problems, I recommend a fresh install.

Feel free to comment in any corrections/questions/suggestions/things that worked for you.

Israel responds to credit card hack

BBC News - Israel vows to retaliate after credit cards are hacked

" Israel has said it will respond to cyber-attacks in the same way it responds to violent "terrorist" acts after the credit card details of thousands of its citizens were published online.

A hacker named OxOmar claiming to be Saudi said on Thursday he had leaked the private information.

Credit card companies say at least 6,000 valid cards have been exposed.

Reports say OxOmar may be a 19-year-old living in Mexico.

Such cyber-attacks are "a breach of sovereignty comparable to a terrorist operation, and must be treated as such", Deputy Foreign Minister Danny Ayalon has said.

"Israel has active capabilities for striking at those who are trying to harm it, and no agency or hacker will be immune from retaliatory action," he added, without giving further details.

An aide to Mr Ayalon said Israel was aware of the report OxOmar may be in Mexico, but had not yet requested help from the Mexican authorities, Reuters news agency reports."

Indian hackers

Indian hacker lords have Symantec antivirus code