setting up bric_queued

I've read through the messages I could find on the topic and it seems running /bin/bric_queued/ as a daemon is the best way to set up queued publishing in Bricolage 2.0. I used the bric_queuedctl script that David wrote in 2009 - to start the script (shouldn't that be included with bric_queued - in a folder perhaps?).

I also set in the bricolage.conf:

queue_publish_job = on

unfortunately it's not publishing anything in the job queue. I check the error messages log file (bric_queued.log) and this is repeated:

voila it cleared the job queue. (I did recall it working on Friday so I was surprised it wasn't working now)

However I did the same thing on Friday and fully expected bric_queued to keep working.

Is there a better way for me to set it up to make sure it keeps running?

thanks Dawn

On 2011-05-01, at 8:32 PM, Dawn Buie wrote:

> Hi > > I've read through the messages I could find on the topic and it seems running /bin/bric_queued/ as a daemon is the best way to set up queued publishing in Bricolage 2.0. > I used the bric_queuedctl script that David wrote in 2009 - to start the script (shouldn't that be included with bric_queued - in a folder perhaps?). > > I also set in the bricolage.conf: > > queue_publish_job = on > > unfortunately it's not publishing anything in the job queue. I check the error messages log file (bric_queued.log) and this is repeated: > > [/home/canadianart/bricolage/lib/Bric.pm:931] > [/home/canadianart/bricolage/lib/Bric/Biz/Category.pm:1230] > [/home/canadianart/bricolage/data/burn/comp/oc_1025/util/keyword_list.mc:97] > [/home/canadianart/bricolage/data/burn/comp/oc_1025/util/keyword_list.mc:123] > [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Component.pm:135] > [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:1279] > [/home/canadianart/bricolage/data/burn/comp/oc_1025/util/meta_keywords.mc:48] > [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Component.pm:135] > [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:1284] > [/home/canadianart/bricolage/data/burn/comp/oc_1/autohandler:31] > [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Component.pm:135] > [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:1279] > [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:473] > [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Interp.pm:342] > [/home/canadianart/bricolage/lib/Bric/Util/Burner/Mason.pm:261] > [/home/canadianart/bricolage/lib/Bric/Util/Burner.pm:1600] > [/home/canadianart/bricolage/lib/Bric/Util/Burner.pm:1340] > [/home/canadianart/bricolage/lib/Bric/Util/Job/Pub.pm:187] > [/home/canadianart/bricolage/lib/Bric/Util/Job.pm:1886] > [/home/canadianart/bricolage/bin/bric_queued:246] > [/home/canadianart/bricolage/bin/bric_queued:212] > Bric::Biz::Category->keywords has been deprecated. > Use Bric::Biz::Category->get_keywords instead at /home/canadianart/bricolage/lib/Bric/Biz/Category.pm line 1230. > > I am trying to publish a story which had been successfully published before. > > Any thoughts on what is wrong with my settings? > > thank you > Dawn >

It sounds like bric_queued had not been started, or it crashed. This happens to us sometimes. It leaks memory until a certain point, then dies. I set up a cron job which restarts bric_queued every 12 hours to avoid this problem.

Thanks, Nick

On 5/1/2011 8:42 PM, Dawn Buie wrote: > Update: > > OK I logged into the bricolage server and typed this command > > /home/sitename/bricolage/bin/bric_queuedctl start > > voila it cleared the job queue. (I did recall it working on Friday so I was surprised it wasn't working now) > > However I did the same thing on Friday and fully expected bric_queued to keep working. > > Is there a better way for me to set it up to make sure it keeps running? > > thanks > Dawn

There should be two instances of bric_queued running at any time. One of them is for publishing and the other one is for distribution. The publishing one should be checked for memory use from time to time and sent a term signal then restarted if it goes over. Allow some time for the restart.

On May 2, 2011 1:46 PM, "Nick Legg" <leggn [at] denison> wrote: > Hi Dawn, > > It sounds like bric_queued had not been started, or it crashed. This > happens to us sometimes. It leaks memory until a certain point, then > dies. I set up a cron job which restarts bric_queued every 12 hours to > avoid this problem. > > Thanks, > Nick > > On 5/1/2011 8:42 PM, Dawn Buie wrote: >> Update: >> >> OK I logged into the bricolage server and typed this command >> >> /home/sitename/bricolage/bin/bric_queuedctl start >> >> voila it cleared the job queue. (I did recall it working on Friday so I was surprised it wasn't working now) >> >> However I did the same thing on Friday and fully expected bric_queued to keep working. >> >> Is there a better way for me to set it up to make sure it keeps running? >> >> thanks >> Dawn

> Hi Dawn, > > It sounds like bric_queued had not been started, or it crashed. This happens to us sometimes. It leaks memory until a certain point, then dies. I set up a cron job which restarts bric_queued every 12 hours to avoid this problem.

Oh that's interesting. And what does memory leak really mean?

I've asked Gossamer Hosts to make sure it's always running. But if they don't have a way to do that I'lll move to the cronjob -

thanks Dawn

> > Thanks, > Nick > > On 5/1/2011 8:42 PM, Dawn Buie wrote: >> Update: >> >> OK I logged into the bricolage server and typed this command >> >> /home/sitename/bricolage/bin/bric_queuedctl start >> >> voila it cleared the job queue. (I did recall it working on Friday so I was surprised it wasn't working now) >> >> However I did the same thing on Friday and fully expected bric_queued to keep working. >> >> Is there a better way for me to set it up to make sure it keeps running? >> >> thanks >> Dawn >

A memory leak is when a program requests memory from the system, and then fails to give it back when it's done. As I understand there is some controversy over whether the behaviour of Perl is considered a memory leak or not, since apparently it fails to give the memory back by design.

At least that's my recollection.

In practice what it means is that you can't have a long-running perl daemon that does anything very interesting. You have to kill it off from time to time and restart it.

> > On 2011-05-02, at 7:46 AM, Nick Legg wrote: > > > Hi Dawn, > > > > It sounds like bric_queued had not been started, or it crashed. This happens to us sometimes. It leaks memory until a certain point, then dies. I set up a cron job which restarts bric_queued every 12 hours to avoid this problem. > > Oh that's interesting. And what does memory leak really mean? > > I've asked Gossamer Hosts to make sure it's always running. But if they don't have a way to do that I'lll move to the cronjob - > > thanks > Dawn > > > > > Thanks, > > Nick > > > > On 5/1/2011 8:42 PM, Dawn Buie wrote: > >> Update: > >> > >> OK I logged into the bricolage server and typed this command > >> > >> /home/sitename/bricolage/bin/bric_queuedctl start > >> > >> voila it cleared the job queue. (I did recall it working on Friday so I was surprised it wasn't working now) > >> > >> However I did the same thing on Friday and fully expected bric_queued to keep working. > >> > >> Is there a better way for me to set it up to make sure it keeps running? > >> > >> thanks > >> Dawn > > >

-- .......................................................................... : Mark Jaroski : Room 9016 : World Health Organization : +41 22 791 16 65 : .......................................................................... More people are killed every year by pigs than by sharks, which shows you how good we are at evaluating risk. -- Bruce Schneier ..........................................................................

> A memory leak is when a program requests memory from the system, and then > fails to give it back when it's done. As I understand there is some > controversy over whether the behaviour of Perl is considered a memory leak > or not, since apparently it fails to give the memory back by design. > > At least that's my recollection. > > In practice what it means is that you can't have a long-running perl > daemon that does anything very interesting. You have to kill it off from > time to time and restart it.

Perl doesn't release the memory allocated, but does reuse it. So you can have Perl daemon's running for a long period of time and doing interesting things, just can't have them keep growing in memory (much like any language). =)

That's a very good point. Maybe we should consider hunting the leak down with

On Mon, May 2, 2011 at 17:47, Alex Krohn <alex [at] gt> wrote:

> Hi Mark, > > > A memory leak is when a program requests memory from the system, and then > > fails to give it back when it's done. As I understand there is some > > controversy over whether the behaviour of Perl is considered a memory > leak > > or not, since apparently it fails to give the memory back by design. > > > > At least that's my recollection. > > > > In practice what it means is that you can't have a long-running perl > > daemon that does anything very interesting. You have to kill it off from > > time to time and restart it. > > Perl doesn't release the memory allocated, but does reuse it. So you can > have Perl daemon's running for a long period of time and doing > interesting things, just can't have them keep growing in memory (much > like any language). =) > > Cheers, > > Alex > > -- > Alex Krohn <alex [at] gt> > >

If I had to guess I'd say bric_queued would be an easier debugging environment than mod_perl, and there's a pretty good chance that if we can fix the leak under the one it will cover the other as well, since all bric_queued does is call a burner.

> Hi Alex, > > That's a very good point. Maybe we should consider hunting the leak down > with > > > On Mon, May 2, 2011 at 17:47, Alex Krohn <alex [at] gt> wrote: > >> Hi Mark, >> >> > A memory leak is when a program requests memory from the system, and >> then >> > fails to give it back when it's done. As I understand there is some >> > controversy over whether the behaviour of Perl is considered a memory >> leak >> > or not, since apparently it fails to give the memory back by design. >> > >> > At least that's my recollection. >> > >> > In practice what it means is that you can't have a long-running perl >> > daemon that does anything very interesting. You have to kill it off from >> > time to time and restart it. >> >> Perl doesn't release the memory allocated, but does reuse it. So you can >> have Perl daemon's running for a long period of time and doing >> interesting things, just can't have them keep growing in memory (much >> like any language). =) >> >> Cheers, >> >> Alex >> >> -- >> Alex Krohn <alex [at] gt> >> >> >

> http://search.cpan.org/dist/Devel-LeakTrace-Fast/lib/Devel/LeakTrace/Fast.pm> > If I had to guess I'd say bric_queued would be an easier debugging > environment than mod_perl, and there's a pretty good chance that if we can > fix the leak under the one it will cover the other as well, since all > bric_queued does is call a burner. > > I wonder if it's worth giving up a few nights or a weekend for this.

> > There should be two instances of bric_queued running at any time. One of > them is for publishing and the other one is for distribution. The publishing > one should be checked for memory use from time to time and sent a term > signal then restarted if it goes over. Allow some time for the restart.

Interesting. I think it would be good to hunt down and kill the 'memory leak' and then to better document how to use bric_queued. I could help with the later at bric hack hack day.

Dawn

> > > On May 2, 2011 1:46 PM, "Nick Legg" <leggn [at] denison> wrote: >> Hi Dawn, >> >> It sounds like bric_queued had not been started, or it crashed. This >> happens to us sometimes. It leaks memory until a certain point, then >> dies. I set up a cron job which restarts bric_queued every 12 hours to >> avoid this problem. >> >> Thanks, >> Nick >> >> On 5/1/2011 8:42 PM, Dawn Buie wrote: >>> Update: >>> >>> OK I logged into the bricolage server and typed this command >>> >>> /home/sitename/bricolage/bin/bric_queuedctl start >>> >>> voila it cleared the job queue. (I did recall it working on Friday so I > was surprised it wasn't working now) >>> >>> However I did the same thing on Friday and fully expected bric_queued to > keep working. >>> >>> Is there a better way for me to set it up to make sure it keeps running? >>> >>> thanks >>> Dawn

> http://search.cpan.org/dist/Devel-LeakTrace-Fast/lib/Devel/LeakTrace/Fast.pm> > If I had to guess I'd say bric_queued would be an easier debugging > environment than mod_perl, and there's a pretty good chance that if we can > fix the leak under the one it will cover the other as well, since all > bric_queued does is call a burner. > > I wonder if it's worth giving up a few nights or a weekend for this.

Definitely. CMS Hack Day is May 9th. =)

The problems we've seen have always been in the parent process (which does the Publishing queue). The child proc handling the dist queue never grows.

One workaround would be to re-archictecht it as a parent process that forks off a child process for each publish. That way the memory is released (similar to setting a MaxRequestsPerchild in Apache). If people are interested in that, it's probably not too much work to do and isolated to just changed bric_queued.

> > There should be two instances of bric_queued running at any time. One of > > them is for publishing and the other one is for distribution. The publishing > > one should be checked for memory use from time to time and sent a term > > signal then restarted if it goes over. Allow some time for the restart. > > Interesting. I think it would be good to hunt down and kill the 'memory > leak' and then to better document how to use bric_queued. I could help > with the later at bric hack hack day.

It can be very difficult to track down, especially if there are circular references around.

We've got some nice mod_perl handlers for doing memory audits (i.e. logs memory status before and after each request, differences in %INC, all the request details, etc).

> On May 2, 2011, at 11:25 AM, Mark Jaroski wrote: > >> http://search.cpan.org/dist/Devel-LeakTrace-Fast/lib/Devel/LeakTrace/Fast.pm>> >> If I had to guess I'd say bric_queued would be an easier debugging >> environment than mod_perl, and there's a pretty good chance that if we can >> fix the leak under the one it will cover the other as well, since all >> bric_queued does is call a burner. >> >> I wonder if it's worth giving up a few nights or a weekend for this. > > +1

Well, running child processes in their own environment isn't *that*different from just killing bric_queued when it gets too big. I guess it's a bit cleaner, but really it would be awesome not to have to limit the memory use under Apache either. So, Monday, may 9th, hmmmm. I'll see if I can get my schedule clear of IAM duties.