To DB or Not To DB.........

Man do I really love to use databases. When you have a decent database and a good design there is nothing that you can not accomplish. Now when I say databases you are probably thinking Oracle, DB2, Sql Server, Mysql, etc... Those are all great databases with rich features but I am thinking more along the lines of an embedded database. What I usually use is Sqlite, it is a embedded relational database that is small and fast and supports most of SQL92. By combining Sqlite and perl I can do many things. Some examples of what I can do are as follows:

Store data from log files and report on them based on different criteria.

Load data and use sql to generate commands, ie: load up file names and then use sql to generate rename commands for the files.

Load multiple log files and types and correlate the data into a comprehensive report.

I will now show you what I am talking about. I will use one of Harlan Carveys   Cpan scripts that reads the event logs. I will use the lsevt3.pl program and make a few modifications to insert the records into a Sqlite database. The initial program looks like this:

#! c:\perl\bin\perl.exe

use strict;
use File::ReadEvt;

my $file = shift || die "You must enter a filename.\n";
die "$file not found.\n" unless (-e $file);

my $evt = File::ReadEvt::new($file);
my %hdr = ();
if (%hdr = $evt->parseHeader()) {
# no need to do anything...
}
else {
print "Error : ".$evt->getError()."\n";
die;
}

my $ofs = $evt->getFirstRecordOffset();

while ($ofs) {

my %record = $evt->readEventRecord($ofs);
print "Record Number : ".$record{rec_num}."\n";
print "Source : ".$record{source}."\n";
print "Computer Name : ".$record{computername}."\n";
print "Event ID : ".$record{evt_id}."\n";
print "Event Type : ".$record{evt_type}."\n";
print "Time Generated: ".gmtime($record{time_gen})."\n";
print "Time Written : ".gmtime($record{time_wrt})."\n";
print "SID : ".$record{sid}."\n" if ($record{sid_len} > 0);
print "Message Str : ".$record{strings}."\n" if ($record{num_str} > 0);
print "Message Data : ".$record{data}."\n" if ($record{data_len} > 0);
print "\n";

# length of record is $record{length}...skip forward that far
$ofs = $evt->locateNextRecord($record{length});
# printf "Current Offset = 0x%x\n",$evt->getCurrOfs();
}
$evt->close();

One of the programs I use to create the database is SqliteSpy. This is a nice gui to create and view the data that you load into the database. What I did was create a table with the following definition:

CREATE TABLE events
( file_name text,
Record_Number number,
Source text,
Computer_Name text,
Event_ID number,
Event_Type text,
Time_Generated text,
time_generated_unix number,
Time_Written text,
time_written_unix number,
SID text,
Message_Str text,
Message_Data text);

You can compare this definition to the $record in the lsevt3.pl script. I have added 3 extra columns to make the table more flexible, they are:

file_name which is the event file name that is being loaded, this allows for multiple event logs to be inserted into the database.

time_generated_unix and time_written_unix were added to allow for easier selecting and sorting of timestamps.

The following is the changed lsevt3 program that does the inserts into the database (Added lines in Bold):

#! c:\perl\bin\perl.exe

use strict;
use File::ReadEvt;

use DBD::SQLite;

# Attributes to pass to DBI to manually check for errors
my %attr = (
PrintError => 0,
RaiseError => 0
);

# Create the connecton to the database
my $dbh = DBI->connect("dbi:SQLite:events.db3","","",\%attr);


my $file = shift || die "You must enter a filename.\n";
die "$file not found.\n" unless (-e $file);

my $evt = File::ReadEvt::new($file);
my %hdr = ();

my $sid = "";
my $message = "";
my $data = "";

if (%hdr = $evt->parseHeader()) {
# no need to do anything...
}
else {
print "Error : ".$evt->getError()."\n";
die;
}

my $ofs = $evt->getFirstRecordOffset();

# Make it so Inserts run in a batch mode
$dbh->do("Begin Transaction");

while ($ofs) {

my %record = $evt->readEventRecord($ofs);

# Convert data and check type to be inserted
if ($record{sid_len} > 0) {
$sid = $record{sid};
} else {
$sid = "";
}
if ($record{num_str} > 0) {
$message = $record{strings};
} else {
$message = "";
}
if ($record{data_str} > 0) {
$data = $record{data};
} else {
$data = "";
}

# Insert statement for the data into the events tables Use prepate and execute to handle quotes in the string fields
my $sql_stmt = qq{Insert into events values ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)};
my $sth = $dbh->prepare( $sql_stmt);
$sth-> execute( $file, $record{rec_num}, $record{source}, $record{computername},
$record{evt_id}, $record{evt_type}, $time_gen, $record{time_gen},
$time_wrt, $record{time_wrt}, $sid, $message, $data);

# Check for any errors in the insert statement
my $err_desc = $dbh->errstr();
if (($err_desc =~ m/not\sunique/) || ($err_desc eq "")) {
} else {
print "Error in Database $err_desc\n";
print "loading Record ".$record{rec_num}."\n";
}



# length of record is $record{length}...skip forward that far
$ofs = $evt->locateNextRecord($record{length});
# printf "Current Offset = 0x%x\n",$evt->getCurrOfs();
}

# Commit the Batch
$dbh->do("Commit");

$evt->close();


By running this program from the command line, lsevt3_db.pl Sysevent.evt, the events will now be loaded into the Sqlite database. You can then load multiple event logs into the table and report on them through sqlite.

The following is an example of a query to show when the Removable Storage Service wrote to the event log:

select * from events where source like 'Remov%';

or

To show the when the computer was started and stopped.

select * from events where event_id in (6009, 6006) order by time_generated_unix desc;

If you were to add the application events then you can see everything that happened during a specific time period as well (now you will see why the unix time is important to have since it is much easier to use and sort by).

select * from events where time_generated_unix between 1168484317 and 1168516719 order by time_generated_unix desc;


Now if you use x-ways forensics you can define the perl script under the external viewer programs and when you select a file you can have it run this program and it will load up the database as if you were running the program from the command line.

If there is interest I can post a generic perl script to print out reports from the database, just leave some comments and I will put one out there.

Hopefully I did not confuse you to much, if I did then let me know and I will try and make it less confusing.