Tag Archives: log2timeline

This article is a quick exercise and a small introduction to the world of Linux forensics. Below, I perform a series of steps in order to analyze a disk that was obtained from a compromised system that was running a Red Hat operating system. I start by recognizing the file system, mounting the different partitions, creating a super timeline and a file system timeline. I also take a quick look at the artifacts and then unmount the different partitions. The process of how to obtain the disk will be skipped but here are some old but good notes on how to obtain a disk image from a VMware ESX host.

When obtaining the different disk files from the ESX host, you will need the VMDK files. Then you move them to your Lab which could be simple as your laptop running a VM with SIFT workstation. To analyze the VMDK files you could use the “libvmdk-utils” package that contain tools to access data store in VMDK files. However, another approach would be to convert the VMDK file format into RAW format. In this way, it will be easier to run the different tools such as the tools from The Sleuth Kit – which will be heavily used – against the image. To perform the conversion, you could use the QEMU disk image utility. The picture below shows this step.

Following that, you could list the partition table from the disk image and obtain information about where each partition starts (sectors) using the “mmls” utility. Then, use the starting sector and query the details associated with the file system using the “fsstat” utility. As you could see in the image, the “mmls” and “fsstat” utilities are able to identify the first partition “/boot” which is of type 0x83 (ext4). However, “fsstat” does not recognize the second partition that starts on sector 1050624.

This is due to the fact that this partition is of type 0x8e (Logical Volume Manager). Nowadays, many Linux distributions use LVM (Logical Volume Manager) scheme as default. The LVM uses an abstraction layer that allows a hard drive or a set of hard drives to be allocated to a physical volume. The physical volumes are combined into logical volume groups which by its turn can be divided into logical volumes which have mount points and have a file system type like ext4.

With the “dd” utility you could easily see that you are in the presence of LVM2 volumes. To make them usable for our different forensic tools we will need to create device maps from the LVM partition table. To perform this operation, we start with “kpartx” which will automate the creation of the partition devices by creating loopback devices and mapping them. Then, we use the different utilities that manage LVM volumes such as “pvs”, “vgscan” abd “vgchange”. The figure below illustrates the necessary steps to perform this operation.

After activating the LVM volume group, we have six devices that map to six mount points that make the file system structure for this disk. Next step, mount the different volumes as read-only as we would mount a normal device for forensic analysis. For this is important to create a folder structure that will match the partition scheme.

After mounting the disk, you normally start your forensics analysis and investigation by creating a timeline. This is a crucial step and very useful because it includes information about files that were modified, accessed, changed and created in a human readable format, known as MAC time evidence (Modified, Accessed, Changed). This activity helps finding the particular time an event took place and in which order.

Before we create our timeline, noteworthy, that on Linux file systems like ext2 and ext3 there is no timestamp about the creation/birth time of a file. There are only 3 timestamps. The creation timestamp was introduced on ext4. The book “The Forensic Discovery 1st Edition”from Dan Farmer and Wietse Venema outlines the different timestamps:

Last Modification time. For directories, the last time an entry was added, renamed or removed. For other file types, the last time the file was written to.

Last Access (read) time. For directories, the last time it was searched. For other file types, the last time the file was read.

Last status Change. Examples of status change are: change of owner, change of access permission, change of hard link count, or an explicit change of any of the MACtimes.

Deletion time. Ext2fs and Ext3fs record the time a file was deleted in the dtime stamp.the filesystem layer but not all tools support it.

Creation time: Ext4fs records the time the file was created in crtime stamp but not all tools support it.

The different timestamps are stored in the metadata contained in the inodes. Inodes are the equivalent of MFT entry number on a Windows world. One way to read the file metadata on a Linux system is to first get the inode number using, for example, the command “ls -i file” and then you use “istat” against the partition device and specify the inode number. This will show you the different metadata attributes which include the time stamps, the file size, owners group and user id, permissions and the blocks that contains the actual data.

Ok, so, let’s start by creating a super timeline. We will use Plaso to create it. For contextualization Plaso is a Python-based rewrite of the Perl-based log2timeline initially created by Kristinn Gudjonsson and enhanced by others. The creation of a super timeline is an easy process and it applies to different operating systems. However, the interpretation is hard. The last version of Plaso engine is able to parse the EXT version 4 and also parse different type of artifacts such as syslog messages, audit, utmp and others. To create the Super timeline we will launch log2timeline against the mounted disk folder and use the Linux parsers. This process will take some time and when its finished you will have a timeline with the different artifacts in plaso database format. Then you can convert them to CSV format using “psort.py” utility. The figure below outlines the steps necessary to perform this operation.

Before you start looking at the super timeline which combines different artifacts, you can also create a traditional timeline for the ext file system layer containing data about allocated and deleted files and unallocated inodes. This is done is two steps. First you generate a body file using “fls” tool from TSK. Then you use “mactime” to sort its contents and present the results in human readable format. You can perform this operation against each one of the device maps that were created with “kpartx”. For sake of brevity the image below only shows this step for the “/” partition. You will need to do it for each one of the other mapped devices.

Before we start the analysis, is important to mention that on Linux systems there is a wide range of files and logs that would be relevant for an investigation. The amount of data available to collect and investigate might vary depending on the configured settings and also on the function/role performed by the system. Also, the different flavors of Linux operating systems follow a filesystem structure that arranges the different files and directories in a common standard. This is known as the Filesystem Hierarchy Standart (FHS) and is maintained here. Its beneficial to be familiar with this structure in order to spot anomalies. There would be too much things to cover in terms of things to look but one thing you might want to run is the “chkrootkit” tool against the mounted disk. Chrootkit is a collection of scripts created by Nelson Murilo and Klaus Steding-Jessen that allow you to check the disk for presence of any known kernel-mode and user-mode rootkits. The last version is 0.52 and contains an extensive list of known bad files.

Now, with the supertimeline and timeline produced we can start the analysis. In this case, we go directly to timeline analysis and we have a hint that something might have happened in the first days of April.

During the analysis, it helps to be meticulous, patience and it facilitates if you have comprehensive file systems and operating system artifacts knowledge. One thing that helps the analysis of a (super)timeline is to have some kind of lead about when the event did happen. In this case, we got a hint that something might have happened in the beginning of April. With this information, we start to reduce the time frame of the (super)timeline and narrowing it down. Essentially, we will be looking for artifacts of interest that have a temporal proximity with the date. The goal is to be able to recreate what happen based on the different artifacts.

After back and forth with the timelines, we found some suspicious activity. The figure below illustrates the timeline output that was produced using “fls” and “mactime”. Someone deleted a folder named “/tmp/k” and renamed common binaries such as “ping” and “ls” and files with the same name were placed in the “/usr/bin” folder.

This needs to be looked further. Looking at the timeline we can see that the output of “fls” shows that the entry has been deleted. Because the inode wasn’t reallocated we can try to see if a backup of the file still resides in the journaling. The journaling concept was introduced on ext3 file system. On ext4, by default, journaling is enabled and uses the mode “data=ordered”. You can see the different modes here. In this case, we could also check the options used to mount the file system. To do this just look at “/etc/fstab”. In this case we could see that the defaults were used. This means we might have a chance of recovering data from deleted files in case the gap between the time when the directory was deleted and the image was obtained is short. File system journaling is a complex topic but well explained in books like “File system forensics” from Brian Carrier. This SANS GCFA paper from Gregorio Narváez also covers it well. One way you could attempt to recover deleted data is using the tool “extundelete”. The image below shows this step.

The recovered files would be very useful to understand more about what happened and further help the investigation. We can compute the files MD5’s, verify its contents and if they are known to the NSLR database or Virustotal. If it’s a binary we can do strings against the binary and deduce functionality with tools like “objdump” and “readelf”. Moving on, we also obtain and look at the different files that were created on “/usr/sbin” as seen in the timeline. Checking its MD5, we found that they are legitimate operating system files distributed with Red Hat. However, the files in “/bin” folder such as “ping” and “ls” are not, and they match the MD5 of the files recovered from “/tmp/k”. Because some of the files are ELF binaries, we copy them to an isolated system in order to perform quick analysis. The topic of Linux ELF binary analysis would be for other time but we can easily launch the binary using “ltrace -i” and “strace -i” which will intercept and record the different function/system calls. Looking at the output we can easily spot that something is wrong. This binary doesn’t look the normal “ping” command, It calls the fopen() function to read the file “/usr/include/a.h” and writes to a file in /tmp folder where the name is generated with tmpnam(). Finally, it generates a segmentation fault. The figure below shows this behavior.

Provided with this information, we go back and see that this file “/usr/include/a.h” was modified moments before the file “ping” was moved/deleted. So, we can check when this “a.h” file was created – new timestamp of ext4 file system – using the “stat” command. By default, the “stat” doesn’t show the crtime timestamp but you can use it in conjunction with “debugfs” to get it. We also checked that the contents of this strange file are gibberish.

So, now we know that someone created this “a.h” file on April 8, 2017 at 16:34 and we were able to recover several other files that were deleted. In addition we found that some system binaries seem to be misplaced and at least the “ping” command expects to read from something from this “a.h” file. With this information we go back and look at the super timeline in order to find other events that might have happened around this time. As I did mention, super timeline is able to parse different artifacts from Linux operating system. In this case, after some cleanup we could see that we have artifacts from audit.log and WTMP at the time of interest. The Linux audit.log tracks security-relevant information on Red Hat systems. Based on pre-configured rules, Audit daemon generates log entries to record as much information about the events that are happening on your system as possible. The WTMP records information about the logins and logouts to the system.

The logs shows that someone logged into the system from the IP 213.30.114.42 (fake IP) using root credentials moments before the file “a.h” was created and the “ping” and “ls” binaries misplaced.

And now we have a network indicator. Next step we would be to start looking at our proxy and firewall logs for traces about that IP address. In parallel, we could continue our timeline analysis to find additional artifacts of interest and also perform in-depth binary analysis of the files found, create IOC’s and, for example, Yara signatures which will help find more compromised systems in the environment.

That’s it for today. After you finish the analysis and forensic work you can umount the partitions, deactivate the volume group and delete the device mappers. The below picture shows this steps.

Linux forensics is a different and fascinating world compared with Microsoft Windows forensics. The interesting part (investigation) is to get familiar with Linux system artifacts. Install a pristine Linux system, obtain the disk and look at the different artifacts. Then compromise the machine using some tool/exploit and obtain the disk and analyze it again. This allows you to get practice. Practice these kind of skills, share your experiences, get feedback, repeat the practice, and improve until you are satisfied with your performance. If you want to look further into this topic, you can read “The Law Enforcement and Forensic Examiner’s Introduction to Linux” written by Barry J. Grundy. This is not being updated anymore but is a good overview. In addition, Hal Pomeranz has several presentations here and a series of articles written in the SANS blog, specially the 5 articles writtenabout EXT4.

In these series of articles about performing file system forensics on a Windows system we covered the evidence acquisition in the first article. The second article was about processing the evidence and creating a timeline of the NTFS metadata. The one below will be split in two parts and will cover the analysis of a Super Timeline and the different artifacts. The main focus will be on parsing and analyzing Windows Event Logs.

In this article we will not cover the creation of the super timeline because has been covered already in this article using Plaso engine. We will move on directly to the analysis and artifacts.

For contextualization Plaso is a Python-based rewrite of the Perl-based log2timeline initially created by Kristinn Gudjonsson and enhanced by others. The creation of a super timeline is an easy process and it applies to different Microsoft Windows operating systems. However, the interpretation is hard. The Plaso engine is able to parse different type of artifacts such as Windows Event Logs, Windows Registry, Windows Prefetch files and many others. During the analysis it helps to be meticulous, patience and it facilitates if you have comprehensive file systems, operating system artifacts and registry knowledge.

One thing that facilitates the analysis of a Super Timeline is to have some kind of lead about when the event did happen. Then you start to reduce the time frame of Super Timeline and narrowing it down. Essentially we will be looking for artifacts of interest that have a temporal proximity with the event. The goal is to be able to recreate what happen based on the different artifacts.

A good starting point is looking at the different Windows Event Logs that were recorded in the Super Timeline. But before we dig into our Super Timeline and the WinETX artifacts let’s review some concepts about the Windows Event Logs.

The Windows Event Logs are a good source of information when looking for potential attacks or misuse of the system. The Windows Event Logs include all the records that are a result of the enforced auditing policy i.e., the amount of information is dependent on the Audit Policy settings. For example on a standalone Windows 7 default installation the Audit Policies state that there is no auditing. This can been seen in the local group policy settings as you could see in the below image.

However, there is some auditing enable and you could verify it using the command line auditpol.exe tool. The below picture shows the auditpol.exe output in the same Windows installation from where the group policy stated that there was no auditing enable. This inconsistency is describe by Microsoft here.

From Windows Vista onwards the Windows Event Logs are stored in the %SystemRoot%\System32\Winevt\Logs folder and contain a .EVTX extension. This new format called Windows XML Event Log (EVTX) format super seeds the EVT format used in Windows XP. Apart of Event Logs from Vista onward there are Application and Service logs that record events about a particular component or application rather then system. On a Windows 7/2008 System many event log files can be found depending on the roles performed by the system. In here you can even find application event logs. For example if the system has Symantec Endpoint you will have a “Symantec Endpoint Protection Client.evtx” file. The different categories are described here. Nonetheless, there are three important files. The description was taken from Microsoft Technet website.

Application.evtx – “The Application log contains events logged by applications or programs. For example, a database program might record a file error in the application log. Program developers decide which events to log.”

System.evtx – “The Security log contains events such as valid and invalid logon attempts, as well as events related to resource use, such as creating, opening, or deleting files or other objects. Administrators can specify what events are recorded in the security log. For example, if you have enabled logon auditing, attempts to log on to the system are recorded in the security log.”

Security.evtx – “The System log contains events logged by Windows system components. For example, the failure of a driver or other system component to load during startup is recorded in the system log. The event types logged by system components are predetermined by Windows.”

One tool is the Plaso engine but that we already covered in another post. Another is the Python-evtx created by Willi Ballenthin. This tool written in Python has the ability to read the EVTX format. The tool is available on Github and contains different modules. One is the evtxdump.py that allows us to dump the contests of the Event Log files. The following picture shows the installation and a simple usage of this tool in the SIFT workstation.

The below pictures illustrates an example where we map a network drive to point to our evidence that is mounted on the SIFT workstation. Then we use wevutil.exe to query the Event Logs. First example with no filter and second example with a XML filter that was created using the Windows Event Viewer.

Please note that in this exercise the Security.evtx contains logon events because the system is not part of a Active Directory Domain. If that was the case the logon events will be recorded on the Domain Controllers.

Now, that we briefly covered the Windows Event Logs let’s review Audit Policy Settings. This is important because will allow us to to know what to expect on the Event Logs. The Audit Policy Settings control which events will be recorded on the Event Logs.

To be able to read the Audit Policy settings we need to read a special Registry key. At the time of this writing I could not find an open source tool that allowed us to retrieve these settings from a forensic image. The Audit Policy settings are stored in the registry under the HKEY_LOCAL_MACHINE\Security\Policy\PolAdtEv key. RegRipper is able to parse and output the contents of this key on Windows XP but seems to have issues on Windows 7 as you could on the below image. Nonetheless, it can retrieve the contents of the key. By having the contents of the registry key we can use the paper created by Teru Yamazaki that describes the structure of this key on newer Microsoft operating systems. With the information we can understand which audit settings are defined. The below image illustrates a manual mapping between the different events and the possible settings.

Teru Yamazaki also has an online tool on his website that allows you to parse this information automatically. You can get the audit policy from the SECURITY registry file using RegRipper and paste the contents into his tool. The output will be in Japanese but you can use Google Translator. In this exercise as seen by the Audit Policy settings we have Success and Failure turned on for all the event types.

Now that we know which kind of auditing settings we can expect and we reviewed some of the tools we could use to read the Event Logs we can start analyzing our Super Timeline and WinEVTX artifacts. This will be done on part II.

[This is my second post on a series of articles that I would like to cover different tools and techniques to perform file system forensics of a Windows system. The first article was about acquiring a disk image in Expert Witness Format and then mount it using the SIFT workstation. The below one will be about processing the disk image and creating a timeline from the NTFS metadata. LR]

After evidence acquisition, you normally start your forensics analysis and investigation by doing a timeline analysis. This is a crucial step and very useful because it includes information on when files were modified, accessed, changed and created in a human readable format, known as MAC time evidence. This activity helps finding the particular time an event took place and in which order. Different techniques and tools exist to create timelines. In recent years an approach known as super timeline is very popular due to ability to bring together different sources of data. However, in this article we will focus on creating a timeline from a single source. The Master File Table file.

Before we move to the hand-on exercise let’s review some concepts behind the Master File Table. The Master File Table is a special system file that resides on the root of every NTFS partition. This file contains a wealth of forensic evidence. The file is named $MFT and is not accessible via user mode API’s but can been seen when you have raw access to the disk e.g, forensic image. This special file contains entries for every file and directory including itself. As written by Brian Carrier the MFT is the heart of NTFS. Each entry of the $MFT contains a series of attributes about a file, directory and indicates where it resides on the physical disk and if is active or inactive. The active/inactive attribute is the flag that tracks deleted files. If a file gets deleted, its MFT record becomes inactive and is ready for reuse. The size of these entries are usually 1Kb. Because each record doesn’t fill 1Kb each entry contains an attribute stating if contains resident data or not. Due to file system optimization, NTFS might store files directly on MFT records. A good example of this are Internet cookie files. Microsoft reserves the first 16 MFT entries for special metadata files. These entries point to a special file that begins with $. The $Bitmap and $LogFile are examples of such files. A list of the first MFT entries are shown in the below picture. As well, it shows how to read the MFT record of a disk image on SIFT workstation using istat. The 0 at the end of the command is the record number you want to read for this partition that starts at offset 206848. The record 0 is the $MFT file itself.

Each record contains a set of attributes. Some of the most important attributes in a MFT entry are the $STANDART_INFORMATION, $FILENAME and $DATA. The first two are rather important because among other things they contain the file time stamps. Each MFT entry for a given file or directory will contain 8 time stamps. 4 in the $STANDARD_INFORMATION and another 4 in the $FILENAME. These time stamps are known as MACE.

M – Modified : When the contents of a file were last changed.

A – Accessed : When the contents of a file were accessed/read.

C – Created : When the file was created.

E – Entry Modified : When the MFT record associated with the file changed.

For our exercise, this small introduction will suffice. Please see the references for great books on NTFS.

Now that we have reviewed some initial concepts on MFT let’s move to our hands-on exercise. For this exercise we will need the SIFT workstation with our evidence mounted – this was done on previous article. Then we need a Windows machine where we will access the mounted evidence on the SIFT workstation using a network drive. Finally, we will need the Mft2Csv tool from Joakim Schicht on the Windows machine to read, parse and produce the MFT timeline.

To start we share the mounted evidence on our SIFT workstation. In this case its /mnt/windows1 and was mounted on previous article. To perform this we edit the smb.conf and we add the lines as shown in the below figure. Then we restart the SMB deamon.

Next, from your windows machine, which needs to be in the same network segment as your SIFT workstation. you can view the shares by using the net view command. Then using the net use command you can map a drive letter. With this step on our Windows machine we will have access to our mounted evidence over the Z: drive. Next step is to run Mft2Csv tool. Mft2Csv is a powerful and granular tool developed by Joakim Schicht. For those who are not familiar with Joakim Schicht, he is a brilliant engineer who has enormously contributed to the Forensics community with many powerful tools.The tool has the ability to read $MFT from a variety of sources, including live system acquisition. It runs on Windows and has GUI and CLI capabilities and needs admin rights. The tool can be downloaded from here. As we speak the last version is v2.0.0.33. In this case, we will launch it from our Windows machine. The command line parameters define from where you are reading the $MFT file and the Time zone. The output by default will be saved in a CSV format but could be saved in a log2timeline or bodyfile. If you are familiar with the log2timeline format than you could use /OutputFormat:l2t. Below picture illustrate this step. The command executed is Mft2Csv.exe /MftFile:Z:\$MFT /TimeZone:0.00 /OutputFormat:l2t When the command is finished you can open the timeline in Excel or copy it to SIFT workstation and use grep, awk and sed to review the entries. Another approach to create a timeline of the MFT metadata is using an old version of log2timeline which is still available on the SIFT workstation. This old version has a MFT parser. You can use log2timeline directly on the mounted evidence. First we capture the Time Zone information from the mounted evidence using Registry Ripper – which we will cover on another post. Then we run log2timeline with -f MFT suffix to read and parse the $MFT file. The -z defines the time zone and the -m is a marker that will show prepended to the output of the filenames.

Or if you don’t have the evidence mounted you can export the $MFT using icat from TheSleuthKit.

Below picture illustrates the output of both tools using the l2t format. In this case the cache.txt is an executable file part of a system that has been compromised with w32.morto worm.

That’s it! In this article we reviewed some introductory concepts about the Master File Table and we used Mft2Csv and Log2timeline to read, parse and create a timeline of it. The techniques and tools are not new. However, they are relevant and used in today’s digital forensic analysis. Next step, review more NTFS metadata.

After evidence acquisition, you normally start your forensics analysis and investigation by doing a timeline analysis. This is a crucial step and very useful because it includes information on when files were modified, accessed, changed and created in a human readable format, known as MAC time evidence. This activity helps finding the particular time an event took place and in which order. The traditional timeline analysis is done to the file system and has been used for several years and folks like Rob Lee and others have been championing it for years. The data is gathered using a variety of tools and is extracted from the metadata layer of the file system (inode on Linux or MFT records on Windows) and then parsed and sorted in order to be analyzed. The end goal is to generate a snapshot of the activity done in the system including its date, the artifact involved, action and source. The creation is an easy process but the interpretation is hard. During the interpretation it helps to be meticulous and patience and it facilitates if you have comprehensive file systems and operating system artifacts knowledge. To accomplish this step several commercial or open source tools exists such as the SANS Investigate Forensic Toolkit (SIFT) that is freely available and frequently updated.

In June 2010, the SANS reading room published a paper from Kristinn Gudjonsson as part of his GCFA gold certification introducing a new method called super timeline and a tool called log2timeline. The super timeline goes beyond the traditional file system timeline creation based on metadata extracted from acquired images by extending it with more sources of data in an automated fashion. As stated on Kristinn paper the super timeline includes more artifacts that provide vital information to the investigation. Basically the super timeline is a timeline of timelines that are gathered from the file system, registry keys and windows artifacts producing a single correlated timeline. To create the super timeline, Kristinn originally developed a framework that is materialized in the log2timeline tool written in Perl. The tool uses a modular approach that makes easier for someone to contribute to the project by developing a new module. A new module could be for example a new parser for a specific windows artifact. The first release introduced several parsers for windows artifacts such as the ability to create time stamped data of Chrome, Firefox, Opera and Internet Explorer browser history, Mcafee Antivirus log file, Open XML used in the Microsoft Office documents metadata, PDF metadata, pcap network captures, UserAssist registry key and others. The log2timeline has been actively developed by Kristinn and others and its backend is currently being replaced by a new framework named Plaso that still contains the log2timeline tool as a frontend and other many other tools written in Python.

In order to practice the production of a super timeline I created a realistic scenario that consists of a Windows XP system that has been compromised by w32/Morto worm just by having remote desktop service exposed to the internet with poor administrator password. Moments after being compromised the machine was taken offline and a forensic image bit-by-bit was created. The image was then moved to the SIFT workstation for analysis.

5 steps are needed to create the super timeline using the SIFT workstation and the logt2timeline.py front-end tool from the plaso suite.

Now with the evidence sorted and reduced I can start doing my analysis, investigation and looks for signs of Evil using for example Excel. An piece of the super timeline is showed in the following picture.

The creation of a super timeline is an easy process and it applies to different windows operating systems but the interpretation is hard. During the interpretation it helps to be meticulous and patience and it facilitates if you have comprehensive file systems, operating system artifacts and registry knowledge. With these type of methodology one can practice and improve their ability to determine past actions which have taken place and understand all kinds of artifacts which occur within the super timeline. On the SANS website there is another great article on how to create a super timeline with log2timeline. Here you can also view a 2011 webcast that looks at what is and how to do super timeline analysis. These and other steps to create a super timeline are well detailed in the log2timeline cheat sheet created by David Nides. The analysis and investigation will be part of other post.