I recently launched a new Ruby on Rails application that worked well in development mode. After the launch I have been experiencing the memory being used is constantly increasing:

UPDATED: When this screen dump (the one below) from New Relic was taken. I have scheduled a web dyno restart every hour (one out of two web dynos). Thus, it does not reach the 500Mb-crash level and it actually gets a bit of a sig saw pattern. The problem is not at all resolved by this though, only some of the symptoms. As you can see the morning is not so busy but the afternoon is more busy. I made an upload at 11.30 for a small detail, it could not have affected the problem even though it appears as such in the stats.

It could be noted as well that it is the MIN memory that keeps on increasing even though the graph shows AVG memory. Even when the graph seems to go down temporarily in the graph, the min memory stays the same or increases. The MIN memory never decreases!

The app would (without dyno restarts) increase in memory until it reached the maximum level at Heroku and the app crashes with execution expired-types of errors.

I am not a great programmer but I have made a few apps before without having this type of problem.

Troubleshooting performed

A. I thought the problem would lie within the before_filter in the application_controller (http://stackoverflow.com/questions/12561409/will-variables-in-application-controller-cause-a-memory-leak-in-rails/12577964) but that wasn't the problem.

B. I installed oink but it does not give any results (at all). It creates an oink.log but does not give any results when I run "heroku run oink -m log/oink.log", no matter what threshold.

C. I tried bleak_house but it was deprecated and could not be installed

D. I have googled and read most articles in the topic but I am none the wiser.

E. I would love to test memprof but I can't install it (I have Ruby 1.9x and don't really know how to downgrade it to 1.8x)

My questions:

Q1. What I really would love to know is the name(s) of the variable(s) that are increasing for each request, or at least which controller is using the most memory.

it is used in the generator#show controller. When some gifts have been generated through my gift-generating-engine I save the input in a session (in case they want to use that info in a later stage). I never kill or expire these sessions so in case this could cause memory increase.

Updated again: I removed this piece of code but the memory still increases, so I guess this part is not it but similar code might causing the error?

Have you tried new relic? They are a heroku add-on and their basic version is free. I think their tools would help debug this issue or at least narrow it down.
–
JohnSep 25 '12 at 8:17

Yeah, the graph above is from New Relic. It has helped me understand the app is crashing due to memory problems but I haven't been able to narrow it down any further.
–
ChristofferSep 25 '12 at 8:36

Assuming category_from_feed returns a string (or a symbol perhaps), a magnitude of 300MB increase is quite unlikely. You can roughly arrive at this by profiling this: 4_000_000.times {related_feed_categories << "Loooooooooooooong string" } This snippet would shoot the memory usage up by about 250MB. I'd look at DB connections or methods that read a file and then don't close it.
–
KashyapSep 27 '12 at 13:39

In the 2.9 part, you have an explanation to destroy sessions, unused for a certain time.

Instead of storing objects in sessions, i suggest you store the url giving the search results. You may even store it in database, offering the possibility to save few research to your customer, and/or by default load the last used.

But at these stage we are still, not totally sure that sessions are the culprits. In order to be sure, you may try on a test server, to stress test your application, with expiring sessions. So basically, you create a large number of sessions, and maybe 20 min later rails has to suppress them. If you find any difference in memory consumption, it will narrow things.

Second case : The memory increase at a faster rate, but don't drop when sessions expires, you know that it is user related, but not session related.

Third case : nothing change(memory increase at usual), so you know it do not depend on the number of user. But i don't know what could cause this.

When i said stress tests, i mean a significant number of sessions, not really a stress test. The number of sessions you need, depends on your average numbers of users. If you had 50 users, before your app crashed, 20 -30 sessions may be sginificant. So if you had them by hand, configure a higher expire time limit. We are just looking for differences in memory comsuption.

Update 2 :

So this is most likely a memory leak. So use object space, it has a count_objects method, which will display all the objets currently used. It should narrow things. Use it when memory have already increased a lot.

Otherwise, you have bleak_house, a gem able to find memory leaks, still ruby tools for memory leaks are not as efficient as java ones, but it's worth a try.

In short, symbols are keep in memory until your restart ruby. So if symbols are created with random name, memory will grow, until your app crash. This don't happen with Strings, the are GCed.

Bit old, but valid for ruby 1.9.x Try this : Symbol.all_symbols.size

Update 4:

So, your symbols are probably the memory leak. Now we still have to find where it occurs. Use Symbol.all_symbols. It gives you the list. I guess you may store this somewhere, and make a diff with the new array, in order to see what was added.

It may be i18n, or it may be something else generating in an implicit way like i18n. But anyway, this is probably generating symbols with random data in the name. And then these symbols are never used again.

The website is a "gift idea"-website. I have about 1.000 gifts and each have a product image stored on the server. I also have additional 24.000 products, fetched from feeds, where I fetch the images externally (www.otherdomain.com/images/productfromfeed.jpg). Outside of images, cached files and the actual code, I do not have any other files on the server.
–
ChristofferSep 27 '12 at 15:32

I am very unused to sessions so it is quite likely I am not handling this correctly. What should I be looking for here? Are not session data stored locally on each users computer (like cookies)?
–
ChristofferSep 27 '12 at 15:33

Regarding your link. I had looked into that question as well and it could be related I guess but I don't think so. The reason for this is that 1. I have not gotten this problem with my other apps 2. I don't know much about memory optimization so it is quite likely I have done something wrong.
–
ChristofferSep 27 '12 at 15:49

Thanks for the update, I did not know how to try the stress test but I removed all sessions from the app (they were just extra features) and the memory is still increasing so I assume sessions was not the case either!
–
ChristofferSep 28 '12 at 11:40

Did it increase at te same rate ? Or was it different ? Are you using something else implying memory ? You may have memory leaks, it happens even if the ruby'gc is pretty smart.
–
PerelloSep 28 '12 at 13:07

Thanks. I am reading feeds but only once a week and the memory is increasing every minute. If I have a controller with the code given above (the one with related_feed_categories). Do I need to "close it" with related_feed_categories = nil ? Or is that handled automatically once the code has been read? I mean, is that something I need to look into?
–
ChristofferSep 27 '12 at 15:29

Btw, how can you tell this snippet uses 110 Mb. What tool do you use for it?
–
ChristofferSep 27 '12 at 15:40

Well, that's just an approximation. "Looong string".bytesize gives you 13 ~ 1 byte per character (depends on encoding) which means that the string in the answer which is 28 bytes is added to the array which means the array size is roughly 28*4million ~ 106MB.
–
KashyapSep 27 '12 at 18:02