Presently I'm running a Rails 3.1.x app atop Heroku Celadon Cedar and it seems that log verbosity is very much lacking. I have set the log level to DEBUG a la heroku config:add LOG_LEVEL=DEBUG --app app_name, which matched up to their recommendation, however beyond that I cannot seem to pull in the log/* file contents.

Changing from Thin to Unicorn did increase verbosity slightly, but only in web worker requests. I still cannot pull down the db requests and so forth.

What is the best way to maximize log verbosity via the heroku "drain" mechanism so that one can pull all instance logs into one cohesive log?

(Ideally I'd like to include a method to dump this into one of my own log servers as this is just a pain the rear not being able to readily look at specific events and surrounding conditions in time.)

Interesting. I'd suggest you contact support on this. Have you tried logging into your app via the shell: heroku run bash to see if the production.log exists or if there is a permission issue with that folder? I've only set this on staging.rb env before so there may be a constraint preventing this on production... Doesn't make sense though off the top.
–
ylluminateFeb 1 '12 at 4:05

Tried heroku bash and found that no log exist. Maybe there has been a change since then. Will contact support. BTW, I tried to "puts xyz_whtaever" into my controller files and those were showing up in the heroku logs. But that is not ideal. Thanks for your quick resp.
–
GeorgeWFeb 1 '12 at 13:00

I think the ActiveRecord logger object sends all SQL query logs with a DEBUG severity level. You may need to adjust the Rails.logger log level to get those in production too.

Since you are interested in dumping all Rails and Heroku logs to an external service, I can suggest you try the Progstr Logger free add-on. (Disclaimer, I work for Progstr). The add-on has a Ruby gem that will collect all Rails logs (everything that usually ends up in log/*). It will also set up a log drain that will fetch all system-level Heroku logs. In addition you get a cool web UI that lets you easily search for logs and some other nice features.

I have added Progstr, however I'm still not seeing additional verbosity in my log output (ie, things that I think that I should see from log/*). Is there something that I'm missing in terms of the config/environments/<env>.rb? I wonder, for example, even though I have set heroku config:add LOG_LEVEL=DEBUG, and I have "config.consider_all_requests_local = true" set for seeing exceptions more clearly, I still cannot pull out database queries and seemingly all of the additionally verbose info that I should be seeing in log/*. Obviously logging in via the shell in Cedar isn't sufficient.
–
ylluminateNov 8 '11 at 16:40

@ylluminate, the problem is that the ActiveRecord logger will log SQL statements with a DEBUG level. Rails goes out of its way to stop logging debug events in production (which is a good thing really). You can still make it log the SQL debug statements by tweaking the Rails logger and its log device. Or you can add the progstr-ruby gem and set up the Progstr::RailsLogger class as described here: docs.progstr.com/#local_setup . Note that Progstr::RailsLogger delivers events straight to our servers and does not route them to the syslog drain. Ex: i.imgur.com/DxF2B.png
–
Hristo DeshevNov 8 '11 at 20:45

I have installed progstr, however I very much want to see this additional info spewed forth for all staging events (ie, not production). To set this up, I simply copied the production.rb over to staging.rb and tweaked a few settings, but I suspect there is a way to coerce staging to act like development in this logging regard?
–
ylluminateNov 8 '11 at 21:31

You can set this up for the staging environment too - it's just a matter of copying the Progstr::RailsLogger part to your staging.rb file: i.imgur.com/VHlBg.png
–
Hristo DeshevNov 9 '11 at 6:36

Thanks very much for the feedback here. I ended up finding the "correct" way to solve this after a lot of effort. I'll post that as the correct answer now for everyone's reference. Turns out that there is some gem overriding normal behavior and if this happens, it must be forced. Seems that this only happens on Heroku though.
–
ylluminateNov 10 '11 at 6:07