> Has anyone run Accumulo on a single server straight from memory? Probably> using something like a Fusion IO drive. We are trying to use it without> using an SSD or any spinning discs.****>> ** **>> *Matthew Moore*****>> Systems Engineer****>> SAIC, ISBU****>> Columbia, MD****>> 410-312-2542****>> ** **>

> Has anyone run Accumulo on a single server straight from memory? Probably> using something like a Fusion IO drive. We are trying to use it without> using an SSD or any spinning discs.****>> ** **>> *Matthew Moore*****>> Systems Engineer****>> SAIC, ISBU****>> Columbia, MD****>> 410-312-2542****>> ** **>

> You could mount a RAM disk and point HDFS to it.>>> On Tue, Sep 11, 2012 at 9:02 AM, Moore, Matthew J. <> [EMAIL PROTECTED]> wrote:>>> Has anyone run Accumulo on a single server straight from memory?>> Probably using something like a Fusion IO drive. We are trying to use it>> without using an SSD or any spinning discs.****>>>> ** **>>>> *Matthew Moore*****>>>> Systems Engineer****>>>> SAIC, ISBU****>>>> Columbia, MD****>>>> 410-312-2542****>>>> ** **>>>>

> Have you tried it where you’re writing to straight block memory? Not> using any file system or SATA controller.****>> ** **>> Matt****>>> *Sent:* Tuesday, September 11, 2012 12:19 PM> *To:* [EMAIL PROTECTED]> *Subject:* Re: Running Accumulo straight from Memory****>> ** **>> I have run a small cluster with HDFS writing only to a RAM disk. Is that> the sort of thing you are interested in?****>> ** **>> -Eric****>> On Tue, Sep 11, 2012 at 12:02 PM, Moore, Matthew J. <> [EMAIL PROTECTED]> wrote:****>> Has anyone run Accumulo on a single server straight from memory? Probably> using something like a Fusion IO drive. We are trying to use it without> using an SSD or any spinning discs.****>> ****>> *Matthew Moore*****>> Systems Engineer****>> SAIC, ISBU****>> Columbia, MD****>> 410-312-2542****>> ****>> ** **>

I don't know of anyone who has done this, but I believe you could:1. mount a RAM disk2. point the hdfs core-site.xml fs.default.name property to file:///3. point the accumulo-site.xml instance.dfs.dir property to a directory onthe RAM disk4. disable the WAL for all tables by setting the accumulo-site.xmltable.walog.enabled to false5. initialize and start up accumulo as you regularly would and cross yourfingers

Of course, the "you may lose data" and "this is not an officially supportedconfiguration" caveats apply. Out of curiosity, what would you be trying toaccomplish with this configuration?

> Has anyone run Accumulo on a single server straight from memory? Probably> using something like a Fusion IO drive. We are trying to use it without> using an SSD or any spinning discs.****>> ** **>> *Matthew Moore*****>> Systems Engineer****>> SAIC, ISBU****>> Columbia, MD****>> 410-312-2542****>> ** **>

It does look like we are the first to try this. We are trying to keepeverything in memory and as a result there is no minor compactions, andprobably major compactions to make tables larger. We tried this on SSDsusing a file system and we were not getting the processing speeds thatwe had wanted.

It does look like we are the first to try this. We are trying to keep everything in memory and as a result there is no minor compactions, and probably major compactions to make tables larger. We tried this on SSDs using a file system and we were not getting the processing speeds that we had wanted.

I don't know of anyone who has done this, but I believe you could: 1. mount a RAM disk 2. point the hdfs core-site.xml fs.default.name property to file:/// 3. point the accumulo-site.xml instance.dfs.dir property to a directory on the RAM disk 4. disable the WAL for all tables by setting the accumulo-site.xml table.walog.enabled to false 5. initialize and start up accumulo as you regularly would and cross your fingers

Of course, the "you may lose data" and "this is not an officially supported configuration" caveats apply. Out of curiosity, what would you be trying to accomplish with this configuration?

Even if you are just using memory, minor and major compactions areimportant to get compression, handle deletes, get sequential access (cacheline efficiency), use iterators, and introduce locality groups.

> Adam,****>> It does look like we are the first to try this. We are trying to keep> everything in memory and as a result there is no minor compactions, and> probably major compactions to make tables larger. We tried this on SSDs> using a file system and we were not getting the processing speeds that we> had wanted.****>> ** **>> Matt****>> ** **>> ** **>> *From:* [EMAIL PROTECTED][mailto:> [EMAIL PROTECTED]] *On Behalf> Of *Adam Fuchs> *Sent:* Tuesday, September 11, 2012 5:30 PM> *To:* [EMAIL PROTECTED]> *Subject:* Re: Running Accumulo straight from Memory****>> ** **>> Matthew,****>> ** **>> I don't know of anyone who has done this, but I believe you could:****>> 1. mount a RAM disk****>> 2. point the hdfs core-site.xml fs.default.name property to file:///****>> 3. point the accumulo-site.xml instance.dfs.dir property to a directory on> the RAM disk****>> 4. disable the WAL for all tables by setting the accumulo-site.xml> table.walog.enabled to false****>> 5. initialize and start up accumulo as you regularly would and cross your> fingers>> Of course, the "you may lose data" and "this is not an officially> supported configuration" caveats apply. Out of curiosity, what would you be> trying to accomplish with this configuration?****>> ** **>> Adam****>> ** **>> ** **>> On Tue, Sep 11, 2012 at 12:02 PM, Moore, Matthew J. <> [EMAIL PROTECTED]> wrote:****>> Has anyone run Accumulo on a single server straight from memory? Probably> using something like a Fusion IO drive. We are trying to use it without> using an SSD or any spinning discs.****>> ****>> *Matthew Moore*****>> Systems Engineer****>> SAIC, ISBU****>> Columbia, MD****>> 410-312-2542****>> ****>> ** **>

On Wed, Sep 12, 2012 at 4:53 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:> Even if you are just using memory, minor and major compactions are important> to get compression, handle deletes, get sequential access (cache line> efficiency), use iterators, and introduce locality groups.

Yes, the effect of locality groups would be about the same in an in memorysystem. The only exception would be if you're not using locality groups andare fetching a particular column, the automatic seeking behavior of thecolumn filtering iterator would be more efficient with in memory rfiles.

> Why would locality groups be useful in an in-memory system?>> On Wed, Sep 12, 2012 at 4:53 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:> > Even if you are just using memory, minor and major compactions are> important> > to get compression, handle deletes, get sequential access (cache line> > efficiency), use iterators, and introduce locality groups.>

Yes, the effect of locality groups would be about the same in an inmemory system. The only exception would be if you're not using localitygroups and are fetching a particular column, the automatic seekingbehavior of the column filtering iterator would be more efficient within memory rfiles.

Adam

On Sep 12, 2012 5:20 PM, "David Medinets" <[EMAIL PROTECTED]>wrote:

Why would locality groups be useful in an in-memory system?

On Wed, Sep 12, 2012 at 4:53 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:> Even if you are just using memory, minor and major compactions areimportant> to get compression, handle deletes, get sequential access (cache line> efficiency), use iterators, and introduce locality groups.

On Wed, Sep 12, 2012 at 5:20 PM, David Medinets<[EMAIL PROTECTED]> wrote:> Why would locality groups be useful in an in-memory system?

Memory is fast, yet we still organize data in memory to make it reallyfast (e.g. hash maps, sorted maps, bloom filters, etc) Localitygroups are no different. If using that data organization will makewhat you are attempting to do faster, then you would probably use it.Assume you have two locality groups and one contains 1% of your databy volume and the other 99%. Scanning just the locality group with1% of the data will be faster than not having locality groups. Itcuts down on the amount of data you have to read and processes frommemory.

>> On Wed, Sep 12, 2012 at 4:53 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:>> Even if you are just using memory, minor and major compactions are important>> to get compression, handle deletes, get sequential access (cache line>> efficiency), use iterators, and introduce locality groups.

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext