Divine Lunacy

As far as I know, there’s really nothing that does this — at least nothing that does it the way I want to. As I have mentioned, Aanval has this functionality, but it is … horrible… at best. I’ve been working on writing a web-based signature management interface for the last few weeks. I currently have something that is stable, though slightly buggy. It’s not nearly done — I’ll likely make many changes in the near future. (Most noteably, support for multiple policies with inheritence.) Just in case anyone stumbles upon this and wants to help/try it, I’ve uploaded the code to http://www.uri.edu/security/software/sigmanager.tar.gz. …

Due to a cron job doing exactly what I told it to do, I lost all of my work on the signature management system I was writing :(. On the bright side, I uploaded the parser with my last post. (the working copy had some modifications, but nothing huge.). I’m about 90% done rewriting it, so not a *huge* loss, but it was shocking when I got to work and saw an emtpy directory. I was *going* to back it up today. he he he.

For the previous year and a half, I have worked for ITS Security at The University of Rhode Island doing such things as managing Snort sensors, being a code monkey and such. When i started here, we were using BASE to view our snort alerts. At the time, our snort signature management was of the “a file is either enabled or disabled” variety; there was no further discretion. As such, filtering through all of the noise was a challenge. More challenging, perhaps, than at other sites running snort: our residential network plays host to a variety of traffic, the likes of which was surely not in the thoughts of people who wrote the rules. (Someone fetches a server list for an online game, and it triggers the storm worm rules? Oh boy.) My first project here was to write software to aide us in filtering through the hoise. The result of that project was SMAC: Snort Management and Assessment Center. (We come up with an acronym, then figure out what it means; the M really has no meaning.) SMAC was made up of profiles, and each profile contained a list of ips and signatures which were considered high fidelity. As an example, one could have a SMAC profile containing ip addresses of known botnet servers, and a selection of high-fidelity rules for detecting command and control channels. Upon going to SMAC, one could select a profile, and then begin viewing alerts in that profile. It is important to note that SMAC was not an alert viewer entire of itself, but rather, an enhanced searching interface to BASE; alerts were still viewed in BASE, and there were many links to BASE. At the time, SMAC seemed like a step in the right direction. And so we continued to use SMAC for about a year. Over that year, features were added to SMAC. My favorite added feature was the addition of a “GROUP BY” to SMAC’s main query. One could select fields to group on: source ip,destination ip, signature. The GROUP by allowed us to “ignore” repeat offenses and greatly aided in log analysis. Using the groups, SMAC would query as normal, but note the number of alerts returned in a GROUP. Clicking on this number brought the user to a page where he could view the alerts in that group. As a final level of drilling down, one could still view the alert in BASE, search for an IP in BASE, or search for a Signature from BASE.

Although SMAC was great, we had a huge problem: BASE was too slow. BASE is designed to work in your environemnt; it is not designed for maximum speed. When our database grew upwards of 1 million alerts, BASE would crumble under the pressure. The solution we finally arrived at was aanval, which we are using to date. Aanval’s lack of advanced searching features prompted me to overhaul my rule management strategies: instead of “ignoring” the noise, I had to STOP the noise. I accomplished this by using Aanval’s sensor management tools and the built-in signature management. The signature management seemed pretty awesome for the first few seconds, but I quickly realized that it was not. To begin with, aanval does not know how to count: it displays an incorrect rule count for many of our rules files. I deduced that this was partially due to the fact that the aanval people don’t know how to parse rules: they failed to account for the fact that there can be whitespace between sid: and the sid. (this was never the case with the community rules, and was inconsistent in the Emerting Threats rules.) They also failed to account for newlnes (slash-escaped) in rules. Rules that aanval could not parse (if that’s what you call it) could not be enabled. I submitted a bug report with the specifics on this, and was told that it would be fixed. I also suspsect that aanval does not delete an SID from the enabled list when it is no longer in the file. Over time, this would skew the enabled count. Meanwhile, I continued using the signature management in aanval. This lead to other gripes: Aanval (stupidly) keeps track of an enabled list of SIDs, as opposed to a disabled list. When the signatures were updated, I had to go through them and enable the new ones. This might sound simple, but it required remembering which ones I had intentionally disabled.

And then Aanval 4 was released. Oh boy was I happy! The aanval team spent months hyping this up; it was rumored to be the best thing since silced bread! I installed Aanval 4 and even tried it for about 5 minutes. I was not impressed. It was clear to me that the aanval people are just random peopole making a security-related application. One might assume that they have experience doing the task that their application is supposed to accomplish, but it is clear looking at aanval 4 that they haven’t a clue as to what security is. The new features included the ability to change my background, and ohhhhhhh a flash-based interface? So now my IDS looked pretty, but it appeared to have 1/2 the functionality as before. The signature management system which I had loved so much was GONE. Let’s recap: Aanval is now UGLY and USELESS. A few weeks later, the signautre management system was re-released. I guess they jumped the gun on the inital release. I was happy about this, as I figured those pesky bugs were finally fixed. Nope. The code was exactly the same, and had exactly the same flaws as the previous version. In fact, most of Aanval 4 seemed exactly the same as aanval 3…. well, excluding the clunky, revammped, shit-pile GUI. I resubmitted the bug report, this time, including a *patch*. They’d have to be pathetically lazy not to fix it then. Update: It is partially fixed. The rules spanning multiple lines is still not fixed, but the (rev|sid):\s*[0-9]+; is *sort* of fixed. They used [ ]* where they should have used \s*, so tabs are not accounted for. *sigh*.

Still not fixed, and my snort rules are only getting more out-of-date. The time came, last week, to take matters into my own hands. I wrote my own signature parser class in PHP, using the Snort source code as an example (actually, that was second rewrite. First time, I did it however the hell I wanted. Second time, I parsed it character-by-charcter, used states, et cetera. That version is far more “intelligent” than the snort signature parser. The third time, I finally decided to read the snort code and figure out how they did it.). As it turns out, snort rule parsing is very odd. It essentially breaks the rule options into an array of tokens by splitting at semicolons — even if the semicolon is inside a quoted value. As it turns out, quotes don’t really have any syntactical meaning in snort rules. I discovered this while trying to figure out how a rule was still being parsed with a regex that contained a doulbe quote (example — pcre:”/”/”; is parsed “correctly” by snort.) When i was coding a stateful parser, I expected these nested quotes to have syntactical meaning, but upon learning that they did not, I figured that snort kept track of an in_pcre state or something, so I added this so quotes had no syntactical meaning inside regex. I did this by using the / as a delimiter for the regex, but later learned that it’s also okay to use an unescaped / in the regex. It is not, however, okay to use an unescaped semi-colon in the regex. A semi-colon would cause snort’s mSplit(…) function to split the regex. ODD.

Armed with my signature parser and KISS, I set about writing my own signature management system. That was about four hours ago; I’m “done” now. About as done as I’ll be for today, anyway. It’s prettier than aanval’s. It works. (also different from aanval!). It’s simple for now, though I plan on unsimplifying it in the future. The code could use some neatening as well. But for now, it works. See scripts for the signature parsing code (I’ll continue to update that, most likely). As for the signature management, it needs work. Maybe I’ll release it when i’m ready, or if there’s demand, which there won’t be because no one will read this anyway :). proofread… later.

Last Friday, I had to run some complicated SQL queries at work. I was dealing with 300k rows of history items that I had to select based on an aggregate function. Unfortunately, WHERE clauses cannot contain the result of an aggregate function. Having clauses are like where clauses, but they’re applied last, and without optimization–and they can contain the result of an aggregate function. Here’s an example:

mysql> SELECT `col1`, COUNT(distinct `col2`) as as distcol2 , COUNT(distinct HOUR(`date`)) as hours FROM `sometable` WHERE foo GROUP BY `col1` HAVING `hours` = 2 and `distcol2` = 2 order by rand() DESC limit 100;


While reading my Gmail yesterday, I saw a sponsored link titled “The PHP Doctor“. Being a self-proclaimed PHP expert myself, I was interested to see exactly how good this PHP Doctor is. He has a blog–a PHP tips and tricks blog. The first post was about functions, and their use in writing modular code. Although the post used a solid example of functions, I couldn’t help but notice a few sloppy performance issues. The following are tips that I have created after reading that post, and the second post on the page.

1. Where possible, do not use functions in the entrance condition for loops. Consider the following example: (From The PHP Doctor*)
   for( $x = 0 ; $x < strlen( $InString ); $x++ ) {
    if (strstr( $OKChars , $InString[$x] ) ) $OutString .= $InString[$x];
}

   Note that for every iteration of that loop, the value strlen( $InString ) remains the same. However, on each iteration of the loop, the function is run. To avoid running this function each iteration, the value should be assigned to a variable before the first iteration:
   
for ( $x = 0,$c = strlen( $InString ); $x < $c; $x++) {
    if ( strstr( $OKChars , $InString[$x] ) ) $OutString .= $InString[$x];
}

2. ++$i, $i++, $i += 1, and $i = $i + 1 all have the same basic effect, but they are indeed different. Using using $i++ or ++$i is faster than using$i += 1 or $i = $i + 1. For those who don’t remember, $i++ and ++$i are respectively, the post increment and the pre increment operators. The difference between the two is that pre increment increments $i, then returns the incremented value, whereas post increment returns the value of $i, then increments $i.
3. Although PHP doesn’t really care, try to avoid implicitly casting numerical types to strings when it is not necessary. Consider a function that takes a numer, $Number, and rounds it to the nearest hundredth. (Note: this can be done via PHP’s round function: round($Number,2)). Here is an example from The PHP Doctor*:
   
function roundToTwo( $Number ) {
    //pow(10, -3) = .001
    $Number += pow( 10 , -3 ); //Make sure the thousandths place is populated
    $Number = round( $Number * pow( 10 , 2 ) ) / pow( 10 , 2 );
    //Make sure the thousandths place is populated
    $Number += pow( 10 , -3 ); //pow() returns float or int (false on error)
    //Take the decimal point, and the following two characters.
    $Number = substr( $Number , 0 , strpos( $Number , '.' ) + 3 );
    return $Number;
}

   Rounding is a mathematical function, but the preceding function relies on properties of strings. One downside of this is performance; another is formatting: roundToTwo() will return a string, and because strings have no concept of significant digits, non-significant digits will be returned. (return value could be something like 123.00) Anyway, with a little more thought, we can find a way to do this without using strings. When round is passed only one argument, it will round the argument to the nearest integer. It follows that round($Number*100)/100 will function the same as roundToTow($Number). But suppose we want to avoid using round()… Recall that if we are rounding to n decimal places, we round up when the digit in the (n+1)th decimal place is greater than or equal to 5, and down otherwise. Adding 5 to this digit will accomplish this. Thus, we can now use the floor() function instead of round(): floor($Number*100 + .5)/100.
4. In the previous example, not the use of pow(x,y) where x and y are constant. For better performance, calculate the value pow(x,y), and use that value in your program. In some cases, this will also lead to more readable code.

In the past, writing has not been my strong point, though some of my former instructors may disagree. I have always struggled through writing assignments, especially those that were literary in nature. Research papers, or papers that involved writing about and interpreting factual information have, as far as writing goes, been my strong point. Though I have never gone so far as to create a portfolio, I have archived all of my mathematical reports, some of my creative pieces, and in general, those pieces that I actually liked. (Which aside from the math reports, number few.)

As a quiet, shy, and very independent person, I have never liked peer editing, and don’t tend to take it seriously. This is partially because I feel uncomfortable pointing out others’ grammatical errors and suggesting revisions to wording, et cetera. (unless it is someone with whom I am friendly.) In the past, this has led my instructors to repremand me for my blatant disregard for the peer editing process. As someone who’s bedroom’s entropy exceeds a large room full of helium gas, portfolios are not exactly my forte. My room’s disorder has overflowed into my desk, my backpack, and most importantly, my mind. One folder has always been enough for all my classes. Inside the folder, chaos prevails. The papers therein are organized on demand, and until there is such demand, disorder shall endure. December fourteenth will likely find me up into the wee hours of the night, anxiously trying to complete my portfolio.

It started as a quest to find a piece of paper that details the channels for each row of house lights. Now, it is as though I am bringing closure to the last four years of my life. I have gone through several huge stacks of paper and found things dating as far back as my Freshman year. I have chemistry notes from my sophomore year, Spanish notes from my freshmen and sophomore years, English notes from my sophomore year, and history notes from my sophomore year. (I still haven’t found that stupid paper. Every time I write that stuff down, I end up losing the paper and writing it all down again next time I need it.) “Summer of ‘69″, by Bryan Adams just came on shuffle. We played that song during the slide show at our eighth grade class supper. I have thus far manged to Distrubute the last four years in to four folders, and one huge pile, as follows: High School Math, High School Math - POW’s, Math League (I saved all of the papers.), Stuff from High School, and the “Why didn’t I throw this crap away a long time ago” pile.

I *secretly* never completed some of those essays for the superintendent. Oops. My bizzle.

Once upon a time, someone messed the board up. This has actually happened many times, but it was even more messed up this time. Actually, it wasn’t the board, but we didn’t know that before we spent an hour and a half playing with it. Anyway, the night we went to fix it, we opened up the back of the board and installed a hardware upgrade kit that had been sitting around in my house for a while. We did not have permission to do this. We were not qualified personnel. Until now, very few people knew about this. That same night, after discovering ze rack (you may have heard of this.), we reprogrammed all of the submasters. As part of this, I wrote down which rows of houselights are controlled by which channels. (channels 97 and 98 do rows 1 & 2, respectively, but it’s a little screwy after that. 113 is the exit lights.). I lost the paper. ::sigh:: Happens every time. I’ll guess I’ll never completely complete that diagram. This whole night has been named operation cupcake. We also played with macros. They’re fun. OMG, I wonder if we ever tried to make a recursive macro. I wonder how one would stop that…

Graduation. It always seemed so far off in the future that I managed to convince myself it would never come. Now it is in past. Graduated. I spent every day of my high school career awaiting the end. Now that the end has come and gone, I find myself slightly depressed. About 169 seniors graduated last night. I probably won’t see many of them again, but I didn’t like most of them anyway. I’ll be in touch with those who are worth staying in touch with. What distresses me is that I am an alumnus. For four years, high school was the only life I knew. When summer came, I knew what September would hold. When September came, the student entrance always welcomed me. Though I may walk through those doors again, I will never again do so as a student. Those years have come and gone as quickly as …. I no longer know what September holds, but I have come to accept this. The student entrance is closed to me, but in front of me lies a new path. This may be an end to a lifestyle that I have come to know and love, but it is also the beginning of something greater. I’LL BE BACK!

In other news, I pwned Bashful at Badminton, but he’s afraid to admit it. (he can’t admit to being pwned by the fat kid.)

DO NOT eat the crumbs that are in the bottom of the bag. OUCH. You have been warned.