In the previous Blog we had gone through what our expectations should be from Application & File I/O Analytics. And as promised, we had hinted about a tool which can provide us with all the answers…..yes all… Well, here I introduce “PerfAccel”, the Patent-Pending Next Generation Enterprise Storage & Analytics Solution from Datagres Technologies Inc.

PerfAccel is a patent-pending state-of-the-art solution for Advanced application I/O analytics at multiple granularity and uses this intelligence for Dynamic policy based data placement. The various features provide in-depth visibility of the file and directory level data usage and performance thus providing comprehensive data intelligence and dynamic data management within mission critical technology environments. PerfAccel is the next generation of active data management platform that accelerates, manages, and scales active application data and is extremely valuable in large scale grid level compute environment with complete transparency and global visibility.

We will keep this walk-through simple and dive into complex topics individually separately. The goal of this walk-through is to gain a quick insight on how powerful PerfAccel is in terms of its Rich Analytical & Caching features.

Getting PerfAccel up and running in 2 minutes

1. Register

Register for a account in www.datagres.com and login using the credentials. Then, visit the PerfAccel Editions page or click on the Free Trials or Download Now buttons available in the site. You will have a choice to request either a Community Edition License (restricted set of functionalities) or a fully functional Enterprise Trial Editions License

Clicking on the Download button in the PerfAccel Editions page, will request the corresponding license for you in the form of a Customer ID. You can view your licenses by visiting the “Your Licenses” page, whose link is available under the menu displaying your Login Name in the top right side. Once you have a license, now you can download the binaries/rpms. For that you can click the “Downloads” Link on the top Downloads menu or click on the “Download Binaries & RPMs” button on the license page.

2. Download : The PerfAccel binaries for your version of OS can be downloaded from www.datagres.com Please download the binaries which match the kernel version of your operating system completely

Ex: For CentOS/RHEL 6.5, the one of the kernel versions is 2.6.32-431.17.1.el6.x86_64 So, download the PerfAccel binary which supports this kernel

dgres-perfaccel-3.6.0.1-rhel_6.5-2.6.32-431-x86_64.bin_07032017

Once the binary is downloaded, copy it to the node where you want to install it

3. Installation & Activation :

PerfAccel Installation is very simple and takes a minute to complete. You can refer the Installation Guide or Getting started guide for more details on this process

This shows that our node is now Licensed for PerfAccel useNow lets get back to our main agenda i.e how PerfAccel can help us in Analyzing Application I/O and performance.

To proceed further, we need to know briefly about few terminologies from the PerfAccel Glossarya. Source : Any mounted directory i.e. either from local EXT3, EXT4, XFS or NFS or Autofs, which we want to Analyze or Accelerateb. Cache : A Logical Cache unit created from a High Speed Device like SSD. Even SATA devices will work but as we know the performance will be lower than SSDs

Now, lets:a. Create a Cache which can be used for multiple sources (we can allocate how much cache space each source can use)

b. Create a source on the directory that we want to Analyze This step will also involve adding cache space to the source. Ex. 10% of cache device or 10GB space

c. By default the Source will have a Write-through policy i.e. First time read of data will cache it and 2nd time onwards all reads will be served from the cache All writes happening for uncached data will go to Source side only All writes happening for cached data will go to both Source & Cache sides

Once the above is done, we are now ready to analyze the workload

Create PerfAccel CacheWe have a local disk (its a VM ) /dev/md127 which we will use as a cache device

# dgpctl -cache create ssd_cache /dev/md127
This operation will delete existing data from the device /dev/md127 and create a PerfAccel cache.
Do you want to continue? [y/n]: y
Initializing cache device /dev/md127. This operation can take several minutes. Please wait…
Cache ssd_cache successfully created

Create PerfAccel Source

We have a NFS share of a critical Ecad application, mounted on /app_data

We can see above that a Cache is created with total capacity 23.6GB and from that 10GB is allocated to a source. And we have created a Source on the NFS share directory and have allocated 10GB of cache space to it

So, in the above df output, a new mount point has appeared on the same path /app_data. This is a PerfAccel source mount.
Note: The beauty of PerfAccel is that it provides all benefits without even a slight change in the Application data path

Note: This demo is on a Virtual Machine, so local disk I/O speeds may seem to shooting up much more as compared to real physical disks. Network I/O speeds won’t be affected though. For exact accuracy of the numbers, you can run the demo sequence on a Physical machine

We can clearly see that 1GB was written to the Source i.e. on NFS side.
So, this is a network write. There was no Cache I/O that happened during his dd (Remember !! Write-through policy)

b. 1 GB dd Read

Before doing the read, we should note the following facts

a. The file was created just now, so all Pages are in memory and if we read them now, they will be fetched from the system memory itself.

b. So, we need to throw them out of the memory by using

echo “3” > /proc/sys/vm/drop_caches This will throw all non-dirty pages, dentries, inodes etc from the system cache.

Note: We are doing this only for Demo Purpose here to make sure I/Os happen from disk and no System Page cache is used …Its not at all needed or recommended by PerfAccel to manually throw system cache in any of the real world deployments

Clearly, we can see that there is a Source side read of 1GB i.e. Network Read only. As this is the first read, so as per Write-through policy, the data read should have been cached now. Well how to find out what is cached ? Don’t worry we have lots of Analytical traces for that…

The beauty of PerfAccel is that when I/O happens on a File, it does not cache the whole file (which can be 1GB, 10GB or 100Gb or even 1TB …)
Rather, it caches only the hot data in segments of 2MB each. That’s why we can see that there are 512 segments of 2 MB each i.e. A total of
1024 MB or 1GB of cached data. This matches with the 1GB of first time read we had done just now.

When we did the 1st 1GB write, that would have resulted in 262144 Write Misses, as the file was not cached at that time. So, all writes went to the Source side only.
The Read/Write misses signify that the I/O happened on the Source instead of the Cache.
Whereas, Read/Write hits signify that the I/O happened on the Cache a direct cache hit

Then, the 262144 Read Misses would have resulted when we did the 1st 1GB dd Read.

This one is another important Analytics data point, the Attribute access details of a file. Its’ built out of the operations like getattr, setattr arising out of commands like chown, stat, ls, chmod etc. Apart from I/O, these attributes are also important during any I/O analysis.

Here, we can see that there were 283 Attribute read misses i.e. 283 times attributes were served from the NFS Source itself, as the file was not yet cached. But, what is the Attribute Read Hits ?? Well, this is another beauty of PerfAccel. Apart from hot data, PerfAccel caches File Attributes as well. This will improve the performance of Network workload a lot as they don’t have to do a Round-trip on the Network for attributes.

Due to that, when the Read happened and the file was cached, the attributes were also cached and the subsequent attribute reads were serviced from the cache. That’s what the 4 Attribute Read Hits signify

Same for the Attribute Write Hits & Misses. We have done no such operation, so the values are 0. We will do it later

Apart from the above, there are many other groups of information also (like Links, Directories, Evictions etc.) in this trace, but we will not go through them right now.

c. 1GB dd Re-readNow, lets Neutralize the system cache using drop_caches and do a Re-Read.

See the dd read throughput. Its a whopping 275 MB/s (The first time dd Read gave us 55.9 MB/s). The difference is that, this time we are reading from PerfAccel cache. This is definitely going to boost the Application performance# dgpctl -cache trace io

In the IO trace we can see that there has been a 1GB write on both the Source as well as Cache. This is because the file is cached. So that data has to be updated in both ends (Write-through policy !!)

Its Resident set time is 3598 secs i.e. time since when it was cached or the first read

Number of Read & Write system calls on the file are 524288 & 524288 respectively

Lets drill down even more on the File Analytics and see what PerfAccel provides usThe number of file analytics metrics provided are more than 30. So, it wont fit even a 42 inch screen…just kiddingBut, its really long, so just for reference purpose i am putting the command below

# dgpctl -cache trace file –long

Note: The trace output of this command extends beyond my screen , I had to reduce the font size to fit it in a line (so much analytical data here 🙂 )

Was the file created after the source was created or it was existing old file

e. File Offset based I/O Analytics

Ok… that’s lot of metrics. But, what if i want to see offset based analytics of the file. I mean, which offsets are cached,which are having more read hits, more write hits, which offset was accessed recently etc. Well, PerfAccel provides that as well

Now, as we know, we have a 1GB file with 512 segments of 2MB each. This trace shows us the metrics for each of the 2MB chunk
In the above output, i have skipped most of the middle chunks and only showing the beginning and end of the output.

Here, for each chunk i.e. 2MB worth of offset, we can see various metrics likea. Offset startb. Read Hitsc. Write Hitsd. Amount of data cached for the segment. It happens many times that in the 2MB segment, only 100KB or something like that is cached.e. Time the offset range was last accessed – very helpful if we want to find out the specific areas of the file which the Application is constantly accessingf. Resident time of the chunk in the cache

What about directory operations on the source ? mkdir, rmdir, lookup etc. Well, here it is …

This gives us the meta operation metrics for both source and cache side.The cache side part will be applicable when the file is cached and the attributes come from the cache itself (like the 4 getattr ops above)

Well, that’s it for this walkthrough. Too much of Analytics as compared to the first blog…right ?

In the next blog, we will go through how PerfAccel metrics look for a little more less-simpler (i wont call it complex) environment involving :a. Multiple Sources (local & network)b. Multiple cachesc. Multiple files in each sourced. I/O as well as Metadata access

In the meantime, you can go ahead and Download PerfAccel right now and get hands on with the exciting features.