Pages

Wednesday, October 05, 2016

RubyConf 2015 - The Hitchhiker's Guide to Ruby GC by Eric Weinstein

me okay?Good?Rock on, I think we'll go ahead and get started.So yes, The Hitchhiker's Guide to Ruby GC.I, like Justin, speak very quickly,so if I do speak, I get going, I get excited,cause Ruby's exciting and garbage collections is cutting,if I start talking way too fast, you guyscould just kinda like, I don't know like,some kind of, something,yeah, exactly, please.That'd be awesome.So, unlike Justin, however,I don't have any TenX insights for you.I do make silly jokes like Aaron does,but unlike Aaron, I'm not gonna offer youlife changing insights into the Ruby virtual machine.My talk is in fact, literally garbage, or about garbage.Speaking of, did you guys hear aboutthis controversy about the show Dirty Jobs?Maybe it was just me, basically I guessit has something to do with the hugenumber of microservices, that they're employing.Thank you, somebody laughed at it, alright good.(laughter) Good, good.I also don't like enjoy my punslike nearly as much as Aaron does,so I'm just gonna put a line through that.I'm not gonna do that anymore.(laughter) Anyway, alright cool.I should say I'm learning a lot.That was a terrible joke.I'm learning a lot at this conference,I'm learning a lot at RubyConf,and I'm trying to apply it immediately.Gary had a great talk about ideology and beliefand it's one of those things that I'msort of trying to internalize.But you know, it's hard, right?We don't know what we don't know,and even when we know it,sometimes we are unable to disarm ourselves.So this is my first time giving this talk.This is actually my second talk ever.I spoke earlier this year at a RailsConf.So I'm going to try to confront my own impostordomand also channel Aaron and not die.And also, you'll see that my nervousness manifestsin obnoxious slide transitions, so there's that.Anyway, so, my name's Eric Weinstein.I work at a company called Condé Nast.I don't know if you guys have heard of Condé,you're probably familiar with the various brands,so there's Wired, the New Yorker, Vogue, GQ, etc.I really like writing Ruby.I write JavaScript a lot at work,so it's nice to be able to write Rubyfor some of those projects and on my own time as well.The Condé Nast entertainment folksactually are all using Ruby and Rails.So we are hiring, I feel obligated to say that we're hiring.And you might be interested, sort of, by this strange hash,and why is it a hash and not an object, like a person.new?And that is foreshadowing, so you will see.There's a lot of literary devices in this talk.So, also obligatory, self promotion.I wrote the Ruby curriculum on Code Academya couple years ago,and I also wrote this book called Ruby Wizardry.It's for kids ages 8-12.There's a really great birds of a featherorganized by Jay McGavrenwho actually is doing a talk on method lookuplater in this very room, so I highlyencourage you to go to that talk as well.And if you're interested in learning Rubyor teaching Ruby, kids Ruby, nounsplease come see me after the show.And I am, so no starch has been delightfuland is offering a 40% off so if anyonedoes want the book RUBYCONF2015,and it'll be 40% off I think for the week.Garbage collection, so there's a lot of mythologyaround GC and GC tuning and sort of how it all works,it's actually not as bad as it seems.And for those not familiar the don't panic!Is kind of like emblazoned in large friendly letterson the cover of the Hitchhiker's Guide to the Galaxy.This also as much for my benefit as yours.(laughter) So,part zero, cause you know this isa computer type conference and we should start at zero.Ruby is not slow.Okay yes, sort of, sometimes depending, it can be.But not for the reasons that you think.So people will say okay well my Ruby programis slow it must be a database thing.Or it must be there's some superlinearcrazy four-deep nested loop and I'm doingsomething bananas in there that I shouldn't be doing.Or Ruby's an interpreted languageand can't possibly be fast and we should all beusing Go or Java or something.And so really what I've foundis that when Ruby programs slow downthese aren't always or often the culprits.The object space in GC is actuallya rich part of the language.Not surprisingly, when you have a lotof richness there's a lot of nuanceand performance bugs can hide in there.These things are true, that are on the screen.They sort of do make some operations slowerthan you know Java or compiled language.But the reason that we have all thisfor better and for worse is causeeverything is an object.Not everything you know blocks,we're not objects but, you get the idea.So let's take some time and talkabout the history of garbage collection in Ruby.First we're talking about MRI,CRuby, we're not talking aboutRubinius or JRuby, which have differentgarbage collectors, I'd wanted to include themin this talk but unfortunately I only have so much timeand they ended up on the cutting room floor.I also don't know nearly as much about them,but if you are interested in Rubinius or JRubyor have some cool stuff to shareplease come find me.I make make comparisons to these alternate implementationsif the comparisons make sense so they mightkind of make guest appearancesbut by and large the talk is about MRI.So let's talk about Ruby 1.8.7.1.8.7 uses tracing,as opposed to say reference counting.GC and GC traces our sort of like,you're looking for reachable objectsin the object grab.So if by traversing objects I can get to onethat object is reachable, and great.That should not be collectedthere are still references to it.As demonstrated in the graph.And if they're not reachableif an object is unreachable it is eligible for collection.1.8.7 uses very simple mark & sweepmark & sweep was invented by,John McCarthy that's right.It's not Alexander Graham Link,for LISP in 1959 and it's astounding that Rubyhad gotten, had gone so far with just simple mark & sweep.Garbage collection, and garbage collectionworks this way in mark & sweep.Ruby will allocate some amount of memory,we'll talk about details in a little bit.When there's no more free memory,Ruby says okay great, I'm gonna go throughI'm gonna look for all of theactive objects that I can find,I'm gonna mark them as active,and then anything that's inactiveI can sweep onto a new list and that'swhere I'll go when I need more stuff.So this is what Ruby is doing.This was a fun animation, I like the arrows moving.Anyway, the important thing to realize thoughwhen you're marking & sweepingand marking & sweeping and having a great time is thateverything stops.Because reachability in the object graphcan change from you know,when you are marking and then sweeping.It can change while the program is executingand it does change while the program is executing.So in order to mark & sweepthe collector has to stop the world.So if you hear people talking aboutstop the world garbage collection.major garbage collection this is what they're talking about.Everything stops, Ruby marks, sweeps,everything's great and then continues execution.So in 1.9.3 we got some improvements.We got lazy mark & sweep.And this is an improvement because it increasesor rather reduces the pause time by sayingokay I'm actually just going to sweep in phases.It doesn't really do anything to the amountof time you spend collecting garbagebut if you say I'm gonna garbage collectand it's gonna take half a second,and I'm gonna do it like ten timesinstead of one second and doing it five times,you know you don't get these unacceptable hangtimeswhile garbage collection is happening.And if you do have like a highly eventedor io driven application, you have a web server,you have something with a GUI,you can't just like sit there for a couple secondswhile you're collecting garbage?This is a big win.Unfortunately 1.8.7 and 1.9.3 bothsubmit native copy-on-write which we willalso talk about it'll be on the quiz.That's a huge lie there's no quiz.Just more information.Anyway so this doesn't reduce the kind of overallpain of stopping the world but it does sort of amortizethat over more sweeps.So in 2.0 we got bitmap marking.So we are no longer marking objects directlyrather we have a bitmap that representsthe state of objects and their eligibility for collection.This'll be extremely important laterbecause this is what sort of allows us to usecopy-on-write and have a great time.I'll be covering all these in depth pretty soonI just kind of want to show youwhere we're going on the journey.That is this talk.So yes if like Aaron I don't dieduring this presentation I'm gonna be turning 30 in March,which is kind of like when I become old I guess,I guess each decade is a collection.I'm talking broadly about Ruby 2.1,so we have generational garbage collection,two generations a young one, and an old one.And if you survive three collectionsthen 2.1 you become old.And this is sort of based on theweak generational hypothesis, objects die young.You have a lot of objects that appearand do something and then are gone.And so based on the fact that objectstend to die young, it makes sense to dominor GC frequently and the slowerstop-the-world collections less frequently.If you're interested in the RGenGC algorithm,Koichi has a talk from EuRuKo a few years ago.Which is excellent and there's a linkat the end of this presentation.So that was sorta the history of Ruby 1.8.71.9 and then 2, nice 2.1 and now we're gonna talka little bit more in depth about 2.2.2but it's 2.2 in general.And we'll talk a little bit more aboutcopy-on-write, bitmap markingall the stuff I mentioned, but hereI'm just gonna focus quickly on two things,incremental and symbol GC which we have in 2.2.We'll also talk a little bit aboutgarbage collection tuning.Which is scary and amazing.So Symbol GC, you have readhas an excellent writeup on Symbol GC.And if you've read that or if you've beenwriting large Rails app, Rails and Ruby applicationsyou're familiar with this notionof like symbol denial service, right?You have a bunch of symbols, they never get collectedbecause they just don't, they're alive forever.And if you were to do something strangebut not on the face of it's silly,which is to let users generate somethingthat will create a symbol, and someone createsthousands and thousands and thousands of symbolsyou will eventually run out of memory.That is bad, it turns out.And so what we have now is the abilityto reclaim some symbols, not all of them,and allow those to be collectedwhen there are no more references.When they're no longer needed.And the reason I say some not allis because Ruby internally will generatesymbols for say method names,and you could easily DOS yourselfagain if you did something likeI guess dynamically generatea lot of methods that all get their own symbol.That's unusual, anyway, so we have that which is nice.But the really cool thing is we have incremental major GC.Again there's another algorithm thatthere's excellent blog posts and paperson and I encourage you to look up.The cool thing here is this sort of tricolor marking.And many of you are probably familiar with this.But the idea is that we have three types of objects.We have white, which are unmarked,grey, which are marked, and may refer to some white objects,and then black, which are both marked,are marked but do not refer to any white objects.And here's how the algorithm probably works.You say okay, all of my objects are white.Everything that's alive, obviously alive,like things on the stack, those are grey.And I'm going to start picking,I'm gonna pick one grey object,and I'm gonna keep doing this for all of them.I'm gonna visit every single reference,and color that grey.And then when I'm done with thisI'm gonna go back to that first grey objectand say okay this is now black,since it does not refer to any white objects.I'm gonna keep doing this and keep doing thisuntil I only have black and white objects.And this tells me okay great, I have black objectswhich are obviously aliveand white which are obviously available for collection.So I'm gonna go through and sweep,and that's sort of how the algorithm works.This third part, the part where we go throughand we do all the color changingand we change the original node back to black,this is sort of what's going onin the incremental part of the algorithm.Like Ruby will do some execution,and then it will do some sweepingand then some execution, and so on.But there is a bug.And I learned this from Aaron, emoji.So yes I'm learning a lot at this conference.I was gonna make it like dance aroundbut like eh, that was a bad idea.So anyway so what's the bug?The problem is what if you have a white object appearso we create a new object, and there's nogrey objects with references to it,because there are only black and white objects,we are going to inadvertently reclaimthis live object by mistake.It will be white when the algorithm runs,the next time it finishes executingsome Ruby code, and says oh,this is a white object, it is availablefor me to collect, and our liveobject that we wanted is gone.This is a bad thing.So what do we do, he said.So there's this cool thing called Write Barriers.They are super effective.You know it because it says so on the screen.Basically,for a lot of complicated reasonsthere are insufficient write barriers in CRuby.And that is a think you should ask Koichi aboutor somebody who knows much more aboutwrite barriers than I do.I am not unfortunately a write barrier scientist.But basically to sort of circumvent this we now havewrite barrier protected and writebarrier unprotected objects.And what we, the plan here is okayI'm gonna, because the pause timeis relative to the number of livingwrite barrier unprotected objects,and most objects are user-definedone strings, arrays, hashes, things like that.Are write barrier protected, the pause timeis not gonna be that bad.So what is the actual fix?If we're like okay this is notgonna be that tricky, we can applythis to kinda fix the problem.Essentially after all the black & whiteobjects are identified,but before we collect the white ones,we wanna go and say okayI'm gonna scan from all of theunprotected black objects we can actually guaranteethat the ones that are protected are managed,and just kind of do another check.Like has anything basically changed,have new white objects appearedsince I last checked?And by doing this we kinda sayokay great now I'm not gonna havethis bug where I inadvertantly collectobjects that I shouldn't.Or I'm not gonna have a problemwith losing things that I didn'treally wanna allow to be collected.So that is sort of a brief historyof GC from 1.8.7 to 2.2.We can now talk about GC tuning.And if there's sort of, if there's one TL;DRif there's one thing you take awayfrom this talk, it should be this slide.Do not do it.Or rather, for experts only, don't do it yet.And this is a paraphrasing of Michael Jackson.Not the Michael Jackson, Michael A. Jackson.Well I guess he's also a Michael Jackson.But basically he said yes, (laughter)in terms of program optimization,any kind of optimization, don't do it,and if you really know what you're doingokay fine but don't do it yet.So what happens if we decide, we're just going to try this?So here are a few variables that youcan modify in Ruby to sort of affecthow garbage collection is performed.These first two, heap_growth_factorand heap_growth_max_slotslowering either of these will triggermore frequent young object garbage collection.You're essentially saying hey, when Iget new heaps, which we'll talk about in a second,you can I think the default's in 1.9 or 2,it may still be 1.8, but at one time it was 1.8so you know you have 10,000 and thenanother 10,000, and then okay it's gonna be 1.8I'm gonna get 18,000.And this will say as you need more memorygreat here's larger and larger chunks of memory to use.So you can lower this and then you run out of memoryfaster and then you're forced to collect.The same thing with the growth_max_slots.The latter three, malloc_limit, malloc_limit_max,and the limit_growth_factor, loweringany of these three will tell Rubythat it's not allowed to allocate as muchoff-heap memory before running minor GC.Which also triggers more frequent object allocations.Or collections, rather.And you can sort of do this with old malloc,which is I think the namespace for larger, like,major GC, but this is probably a bad ideabecause as we've seen, if you starttriggering a lot of major garbage collection,you're gonna be sitting there collecting garbage,instead of you know, sending emails to your usersor something of that nature.The important thing to realizeis this is not a silver bullet.There are no silver bullets.There are a lot of werewolves.There are no silver bullets.It's a bad time, so there's, you knowthis is not as helpful as it may seem.There'll be times when you say yes,there's a lot of objects there's a lot of stuffthat's going, we're churning, and garbage collectionis a problem, we've identified thisis kind of a thing that we should do,you can inadvertently make things worse for yourself.By modifying these variables and not doing a lotof measuring and being very careful.Right this is also one of those mythsit's very attractive, it's the source of those likeweird old tricks and tips about x, y, z.There's a lot of mythology around things likebittwiddling and garbage collectionand I advise you that if you do decideyes, I'm gonna tune the garbage collector,this is something you just wanna measurea lot beforehand, after, make surethat what you're measuring is actuallywhat you're measuring becauseoften times that is not the case.Just kind of, proceed with caution.So, that said, we're gonna move onto I think the sort of the bulk,which is this case study part three.So, couple of years ago I was workingat a company, and we had a Ruby application.It was interesting in so far as itwas not in a Rails application.It was a suite of seven Sinatra applicationsthat had been sort of smashed together.In interesting ways.And the way that this worked is users wouldbrowse through the site, they wouldgo to the web layer, and the Ruby applicationwould make requests to a number of Java services.The Java services all spoke JSON,so the Java service would say hey,I have some JSON for you,and Ruby would say awesome, that's great,I love JSON, and would kind of inflatethem into hashes, and then pass them around.So you would just have these huge hashesfloating around and you know,people would just start picking stuff out of themor mutating them, cause that's a thing you can do.And it was a bad time.And so someone got this really crazy idea,and said why don't we take some objectsand orient our code around them?And I suspect this is a fad,but we were like alright, we'll try it.Because this hash thing is not working out,and now, you see why earlier,there was a hash and not a personthat got newed up because hashes are a thing.So, anyway.We were talking about doing this,and we started doing it,and performance tanked.We were constantly running out of memory,we were sluggish, we were havingall sorts of issues in new relic,everyone was having a hard time,the business was yelling at us,we were yelling at each other,and we couldn't figure out whywe decided to write object oriented Ruby,but we shot ourselves in the foot.So let's talk about memory.And again this is all in the context of 1.9.3,at the time we were writing Ruby 1.9.3and we were sort of trying to figure outwhat's going on and this is sort of howthe switch to 2 happened.So the memory model looks something like this.Ruby objects are 40-byte RValue structures,which Ruby allocates into heaps of 16KBs each.This is for 64-bit architecture, I supposefor 32-bit it's gonna be somewherelike 20 or 24 but we'll say for argument's sakethat we're all using 64-bit machines.So you're gonna get somewhere around the orderof 400 Ruby objects per heap because you have 16,000bytes divided among you have 40 bytes per RValue structure.And so, Ruby will give you about 150 heaps to start.Which is awesome.And so you can kind of test this yourself here,we're inside this Sinatra app,and you can see the object space,there's somewhere, there's about 408objects per heap.So.In that 400 ballpark.And it turned out we were makinga huge, and this is a little bit hard to read,I apologize, a huge number of objects.So at the top you see bxr consoleit's just my alias for bundle exec rake,the console task starts IRB with the application loaded.We say okay GC_start, great,and then GC_stat, and the interesting things in hereare we see the number of total GC runs.We see the number of heaps with at least one used slot.The heap length is the total number of heaps.And heap increment is how many moreheaps to ask for, which I think yeah in this,version of Ruby it was 1.8 times the previous number.But the most interesting thing,is this, the heap live number, wherethere were half a million objects.Just to start, we weren't even doing anything.We just started the web server and saidhey how many objects are there.There's half a million.I would say that for comparison,the average Rails app would be somewherecloser to 400,000 but that's sort of a nonsensestatistic right, like there's no averageRails app we all have different business needs,we have different versions of Rails,we're running different versions of Ruby,or different Ruby implementations.Different web servers like Puma or UnicornBut this seems completely wrong,for this Sinatra app, or this likesmashed together Frankenstein's monster Sinatra app.And we weren't even using ActiveRecordwe were just you know, talkingto the services and getting responses.So we, let's talk a little bit about these RValues.For small values, it restores them directly on the object.And you might've heard, via Pat Shaughnessywho has an excellent book, Ruby Under The Microscope,that there's in and this is very interestingagain it's one of those weird old tipsright, it's like, you don't want stringsthat are over 23 characters.This is crazy, it doesn't matter.If you actually really do careabout that level of performanceand optimization, please come find meafter the talk because I have a tremendousamount to learn from you.That would be awesome, but, essentially,what happens is yeah, 23 charactershere you have the value directly 24 and larger,you have a pointer to some other location memoryand this is somewhat slower.So for large values,for whether it's a string or a hash or whatever,the value is actually a pointer.So the RValue itself has a flags field,which contains FL_MARK, which we'll talk about.Object contents, which is the value,or it's either the actual objector it's a pointer to one,or and you have next, which is a pointerto the next RStruct.And there's RString for strings,RHash for hashes, RObjectfor custom objects and user space.And these are all the same size.So we talked about heaps.So in 1.9.3, we were seeingwe had 10,000 slots.Which was one heap.We got the second heap, which was another 10,000 slots.And every time we needed another heap,we were multiplying by 1.8.So the third time Ruby says hey I need more memory,great, we get another 18,000 so we have 38,000after the initial 20,000.And then if we have to ask againwe go from 38,000 to 106,400because we're doing this kind of doublingalmost, this 1.8 factor.And this could be tricky if, and this is anothercase for tuning, and this is one thingthat we did do, because you know,I really only need like 50,000 objects.And so I ask for, or 50,000 slots ratherand I ask for more, and now I have 106,400.And I'm never actually going to use these.And you realize you know,Ruby is not going to give this back to the operating systemuntil the process exits, so.Asking for more memory than you needcan be tricky and there's a caseto be made for doing the kind of tuningI warned you about.With that regard.So let's talk more about Mark & Sweep.So Ruby heaps comprised linked lists of RValues,linked lists were in fact invented by Alexander Graham Link.In 1836.I realize that if you didn't see Aaron's docthis just sounds like crazy stuffso I'm sorry about that.So when there are no more free RValues,Ruby 1.9 will set FL_MARK on all active Ruby objects.This is the marking phase,and then relinks, kind of sweeps theminto a single linked list, and this is called the free list.Here's the free list,so you can kind of imagine it in your mind,you have a linked list, marked objects,some are not marked, and the onesthat are not marked are availableto be reclaimed, so they get swept,and used again.I just love that animation, anyway.So copy on write.We talked about this a little bit ago.Copy on write, the idea is,when your production process,or any process calls fork, the new child processshares all memory with the parent,and then copies are only made when a write is forced.Which is cool, this is great.You don't actually have to write anything,unless there's something different.And so the fork process and the parent,as long as you're not mutating stuff,they can share memory and it'syou know you have this sort of persistentdata structure and it's really cool.The problem is if you mark an object directly,and say you are marked, you are available.And then there's you know, a child processsomewhere that object is not marked.So you have a marked and not marked,you have these objects that sort of proliferatethat only differ in their eligibility for collection.And this is bad.You kind of get a proliferation of objects,this kind of subversion of nativecopy on write,and it's not necessarily true in all places,on Linux boxes and production,you know, on your Linux machine or OSX machine,they do leverage copy on write, we'll assumethat it's true everywhere, there are some instanceswhere it's not, but basically now we haveall these objects floating aroundand for a web server like Unicornthat does concurrency via forking,the more Unicorns you have, the worseyour problem gets, because you'reconstantly writing new objects.So this is the Iridad portionof the presentation you can tell,your friends and colleagues and coworkersthat you learned something from my talk.Which is a quote from Shakespeare.So this is from The Tempest, the idea beingokay we don't need to do all these writes,we don't need to have all this, so let's not.And so this leads us to memory and bitmap markingas mentioned before in Ruby 2.0.So every heap now has a header,that points to a bitmap, and you can kinda,it's a little hard to see but these ones and zeroscorrespond to marked or not markedin the actual heap slots, so youno longer have to mark an object itself as marked.You can just update the bitmapand that bitmap keeps track of who is availablefor collection and who is not.So this header is one is marked and zero is unmarked.And so Ruby 2.0 with this header just kind of simplifiesthis and allows us to actually leverage copy on write.The GC mark no longer modifies objects and we onlyhave one object, if it doesn't changein terms of its eligibility for collectionwe don't have to write it amonga bunch of different processeswe just have the one.And is mentioned, because Unicornmanages everything by forking,the more Unicorns you have,the worse the problem gets.Now this is not a bad thing.You know, Unicorn is a popular web server,forking is not intrinsically bad,Unicorn is not intrinsically bad,Ruby is certainly not intrinsically bad.Ruby 1.9.3's garbage collectionalgorithm was inadvertently doing a bad thing.Which has been fixed.Which is great.So as mentioned as it increases,the problem gets worse.So here are some numbers,hopefully they're somewhat illuminating.Loading the application in 1.9.3 invoked122 GC runs and took about 4.4 seconds.Changing nothing else, simply loading the app in 2.0invoked 66 GC runs and took about three seconds.So in this case 1.9.3 and we were looking at the GC stanceand kind of instrumenting and carefully checkingand there was some variation in termsof Gem versions and stuff like that,we tried to get all that jiggle out.And it was about 47% more timesimply collecting garbage in this application.So given this information, what did we do?That was much faster than I expected.So number one, is we upgraded to Ruby 2.0so we could leverage copy on write,turns out require is also faster.It's hard to see the line there,this was new at the time, it's notso new now, but there were cool featureslike Module#prepend, lazy Enumerators,refinements, and there's a talk about refinements,which is awesome, unfortunately you are missing it,but there will be a recording, so that's good.And it was just about time.Like Ruby 1.9.3 reached the end of life in Februaryand you know we were talking about thisI think the previous December.So we had really a couple of monthsto really get racked up, and yeah.So that was the first thing that we didwas just make sure we were on Ruby 2.0.So another takeaway is just update Ruby.So this Christmas you're getting Ruby 2.3and it will be awesome.Number two is we profiled, so I mentionedthat we tried to find and eliminate sources of bloat,we spent a lot of time doing native GC profiling,this is the Ruby 2 docs, there's alsodocs for 2.1 and 2.2, I encourage you to take a lookat GC and sort of the methods that are available there.There's a lot of information about what Ruby is doing.Much like how Aaron kind of showed in his talklike how you can kind of pull apartand see all the instructions in thatYARD was executing, you can alsogo through and see what objectsare being allocated, how long they're alivehow many slots you have, things of that natureand it's really very cool.We also use the ruby-prof Gem, which helped usand I can say, I don't have enough good thingsto say about it, it's it was awesome.And the third thing yes, I know I said don't do it,but we reasonably sure we knewwhat we were doing and it turned outwell so far so good.Basically we tuned the GC.So the three variables that we touched therewas the Malloc_limit, which controlswhen you perform a full GC run,the default was 8MB, which I think was setit was chosen in like 1995, which made a ton of sensefor 1995 but didn't make sense for us.So we tuned number one to get more memoryand sort of punt on full stop the world GC becausewe could, you do more stuff before we hadto actually do that.Number two, the Heap_min_slots,that controls the slots per heapwhich defaults to 10,000.We tuned it a little bit to get more objects per heap.I wanna say we only modified itto about maybe it was 12,000 something of that nature.And then Ruby 3, sorry number three,Ruby_heap_slots_growth_factor as mentioned,the growth factor was 1.8 so wewere to ask for more and more and morewe kind of looked at the graph and realizedthat we were kind of plateauing a little bitand we didn't really need to do 1.8I think we collected 1.1 and 1.2.So again lowering that in Ruby 2will give you more frequent kind of GC runsfor the young generation, for the you knowminor GC, you know have the full stop the world stuff.So some credits and some further readingI'm deeply, deeply indebted to Pat Shaughnessywho I said has an excellent bookRuby Under A Microscope, was very helpfulin kind of pulling Ruby apart and seeing what was happening.Sam Saffron has written a great post,Demystifying the Ruby GC it's a couple years oldbut it's still a great read.Pat's book, Alexander Dymo has a bookcalled Ruby Performance Optimization.Which goes much more in depth in the stuffI was kind of talked about at the beginning.Where a lot of the issues that we seein our Ruby applications have to dowith the richness of the space and garbage collectionand sort of what is around, and is less relatednecessarily to you know, bad algorithmsor bad datas queries althoughthose things do exist.And then again, Koichi's talk is just astoundingI encourage you to look that up as well.Blog post on the Heroku Engineering blog,videos on Youtube, they're all excellent.So really thanks so much for your time,I really appreciate your taking time fromthe RubyConf to come listen to me talk about stuff.And again I guess obligatory, shamelessself-promotion, this is me, this is who I work for.That's the book that I wrote, and yeah.Thanks so much. (applause)I'm sorry I don't understand the question.Oh out of band GC, that's a great question.So that was something that we were tryingto figure out for this application,so the question is, out of band GC.What do I think about it?We, the idea is you essentially tellthe garbage collector, you're not in chargeof when you run, I will tell youwhen you're allowed to run.This is a thing that is usefulif you have something like a web applicationwhere there are requests coming inand someone could be in the middleof a very complicated requestand Ruby will say, hey timeoutI'm out of memory I need to collectI'll be right back, and then just do a majorgarbage collection right in the middle.You don't want that to happen.So the idea is you say, I'm goingto only allow garbage collectionin between requests, like maybeevery ten requests or every 100 requestsor every 300 requests, that's when I'm gonna likedo some garbage collection and then back to the show.For Unicorn at the time the tools availablewere Unicorn worker killer, and I thinkthere was another out of NGC Librarythat were using that sort of like allowed usto maintain memory well it's kind of snipingbig bloated old Unicorns cause that would happensince you were telling GC not to run anymore.Like I will tell you when you're allowed to stop.And it was a lot of overhead, doing that memory management.Like you essentially give up all of the nice, cool likeautomatic memory management that Ruby gives youand it's a lot to fit in your head.So while I think it's, there are certainly timeswhen it makes a lot of sense,and it can be very very useful,we found it was too hard for usto get right, which does not meanit's not too hard for you and your team to get right.How long should a major garbage collection take?That is an excellent question.I feel like I,I feel like I don't know the answer.So, major garbage collection like certainlyyou don't want to be like spendingseveral seconds collecting garbage.As Gary mentioned in his talk yesterdaylike this was sort of like why he was not convincedthat garbage collection was necessarybecause garbage collection in 2002was taking several seconds, and this was unacceptable.With the newer stuff in 2.2 and the abilityto kind of incrementally do it,I feel like there's the applicationsI've worked on it was like less than a second,like more than a second and we were kinda likeoh this is taking a long time like what's actuallygoing on, but I think it's gonna have to dependon your application and your needs yeah.Where do you, oh the book yes,like I said the flashing buy now button.So you can get the book from your local booksellerit's published by no starch pess,it's available on Amazon, it's available at Barnes & Noble.I wish I had copies with me but they were,they were extremely heavy, like it'ssomething like 340 pages, it's surprisingly big.There are illustrations, so, yeahit's pretty cool I'd be happy to show anybodylike a PDF for some sample pages on my machine,later today, like I said I wish I had a copybut I'm not good at marketing, it turns out.Correct, yeah so this is a yes, to be clearthat's exactly right, that's a good question.So the question is, this codeis for no starch, and that's correct.So this is not something you can usewith any bookseller, if you go to nostarch.comand you say I wanna buy Ruby Wizardrythere's a little box that says do you have a promo code,and if you put in RUBYCONF2015,it will be 40% off.Like I said it's for kids ages 8-12,certainly as young as six or sevenif they've got, you know if they'rereally motivated and their parentsare willing to help, cause it can be kinda tricky.And I suppose up to high school,but it's sort of like the Phantom Tollboothlike there's a time to read it,and that time is not maybe when you're 16.But I've had adults tell me thatyou know, they liked it, so.If you wanna learn Ruby,maybe it's helpful for you too.Rock on, well again, thanks so much.