From todd at snappl.co.uk Thu Aug 7 12:54:05 2008
From: todd at snappl.co.uk (Todd Tyree)
Date: Thu, 7 Aug 2008 17:54:05 +0100
Subject: [Backgroundrb-devel] Jabber namespace strangeness in workers when
config.gem 'xmpp4r' is set
Message-ID: <44f20a950808070954w5996065dqe4103cbdfb866676@mail.gmail.com>
Just spent three bloody hours looking for the cause of this, so I thought
I'd save anyone else having the same problem the trouble:
In the process of porting our existing app to 2.1.0 and decided I'd use the
config.gem functionality. I've got a legacy worker?working that passes off
messages to our ejabberd server and works fine. It depends on xmpp4r.
Yesterday, as part of a larger set of changes, I checked in environment.rb
with the line: config.gem 'xmpp4r', :version => '0.3.2'. Today, on
restarting backgroundrb, the xmpp worker suddenly starts throwing the
following exception:
uninitialized constant XmppWorker::Jabber (NameError)
I went through everything I could think of before finally digging back
through the svn diff and noticing the config.gem 'xmpp4r'. Commented it out
and, hey presto!, everything went back to normal.
I'm hoping the complete silence of google on the topic means no one else has
encountered it. However, if not, I hope this helps.
Best,
Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From leomayleomay at gmail.com Sun Aug 10 01:37:02 2008
From: leomayleomay at gmail.com (=?GB2312?B?wfXquw==?=)
Date: Sun, 10 Aug 2008 13:37:02 +0800
Subject: [Backgroundrb-devel] uninitialized constant BackgrounDRb (NameError)
Message-ID: <72b9504f0808092237r3947b2fpe1c1a02e9a1f3d96@mail.gmail.com>
hi, guys,
I installed the bdrb from the scratch with the following steps:
1. cd vendor/plugin
2. svn co http://svn.devjavu.com/backgroundrb/trunk backgroudrb
3. cd ../..
4. rake backgroundrb:setup
5. ./script/generate worker example
6. ./script/backgroundrb start
7. ruby lib/workers/example_worker.rb
Error msg given out, says:
lib/workers/example_worker.rb:1: uninitialized constant BackgrounDRb
(NameError)
Anybody has any idea about this error?
All the best,
Hao Liu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From robl at mail.pigdestroyer.co.uk Sun Aug 10 02:44:29 2008
From: robl at mail.pigdestroyer.co.uk (Rob Lacey)
Date: Sun, 10 Aug 2008 07:44:29 +0100
Subject: [Backgroundrb-devel] uninitialized constant BackgrounDRb
(NameError)
In-Reply-To: <72b9504f0808092237r3947b2fpe1c1a02e9a1f3d96@mail.gmail.com>
References: <72b9504f0808092237r3947b2fpe1c1a02e9a1f3d96@mail.gmail.com>
Message-ID: <489E8E4D.4030808@mail.pigdestroyer.co.uk>
Hao,
Trying to run your worker in step 7 simply won't work. The BackgrounDRb
class won't get loaded automatically, and the worker is complaining that
it doesn't know what BackgrounDRb is. You will get the same error if you
try `ruby app/controllers/application.rb` for example. What are you
trying to acheive? Are you simply tyrying to test it is installed properly?
Rob Lacey
?? wrote:
> hi, guys,
> I installed the bdrb from the scratch with the following steps:
>
> 1. cd vendor/plugin
> 2. svn co http://svn.devjavu.com/backgroundrb/trunk backgroudrb
> 3. cd ../..
> 4. rake backgroundrb:setup
> 5. ./script/generate worker example
> 6. ./script/backgroundrb start
> 7. ruby lib/workers/example_worker.rb
>
> Error msg given out, says:
> lib/workers/example_worker.rb:1: uninitialized constant BackgrounDRb
> (NameError)
>
> Anybody has any idea about this error?
>
>
> All the best,
> Hao Liu
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
From lists at wildgooses.com Sun Aug 10 05:05:25 2008
From: lists at wildgooses.com (Ed W)
Date: Sun, 10 Aug 2008 10:05:25 +0100
Subject: [Backgroundrb-devel] Jabber namespace strangeness in workers
when config.gem 'xmpp4r' is set
In-Reply-To: <44f20a950808070954w5996065dqe4103cbdfb866676@mail.gmail.com>
References: <44f20a950808070954w5996065dqe4103cbdfb866676@mail.gmail.com>
Message-ID: <489EAF55.8080905@wildgooses.com>
Todd Tyree wrote:
> Just spent three bloody hours looking for the cause of this, so I
> thought I'd save anyone else having the same problem the trouble:
>
> In the process of porting our existing app to 2.1.0 and decided I'd
> use the config.gem functionality. I've got a legacy worker?working
> that passes off messages to our ejabberd server and works fine. It
> depends on xmpp4r. Yesterday, as part of a larger set of changes, I
> checked in environment.rb with the line: config.gem 'xmpp4r', :version
> => '0.3.2'. Today, on restarting backgroundrb, the xmpp worker
> suddenly starts throwing the following exception:
>
> uninitialized constant XmppWorker::Jabber (NameError)
>
> I went through everything I could think of before finally digging back
> through the svn diff and noticing the config.gem 'xmpp4r'. Commented
> it out and, hey presto!, everything went back to normal.
Could it possibly be causing the gems to load in a different order to
normal (hence triggering the error due to some dependency issue?)
Ed W
From leomayleomay at gmail.com Sun Aug 10 06:22:41 2008
From: leomayleomay at gmail.com (=?GB2312?B?wfXquw==?=)
Date: Sun, 10 Aug 2008 18:22:41 +0800
Subject: [Backgroundrb-devel] How to keep from generating more than one
worker?
Message-ID: <72b9504f0808100322w3f5c43balf3a912b02ad1fc88@mail.gmail.com>
Hi, guys,
I only want one BackgrounDRB worker for my rails app except for the
log_worker baked in.
And how to assign a worker_key to a worker?
All the best,
Hao Liu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From gethemant at gmail.com Sun Aug 10 06:30:33 2008
From: gethemant at gmail.com (hemant)
Date: Sun, 10 Aug 2008 16:00:33 +0530
Subject: [Backgroundrb-devel] How to keep from generating more than one
worker?
In-Reply-To: <72b9504f0808100322w3f5c43balf3a912b02ad1fc88@mail.gmail.com>
References: <72b9504f0808100322w3f5c43balf3a912b02ad1fc88@mail.gmail.com>
Message-ID:
On 8/10/08, ?? wrote:
> Hi, guys,
> I only want one BackgrounDRB worker for my rails app except for the
> log_worker baked in.
put
:debug_log: false
in your backgroundrb.yml file, like:
:backgroundrb:
:ip: 0.0.0.0
:port: 11006
:debug_log: false
> And how to assign a worker_key to a worker?
worker_key can be only assigned to workers that are created using
MiddleMan.new_worker(:worker => :hello_worker,:worker_key => "boy")
--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.
http://gnufied.org
From gethemant at gmail.com Sun Aug 10 07:10:33 2008
From: gethemant at gmail.com (hemant)
Date: Sun, 10 Aug 2008 16:40:33 +0530
Subject: [Backgroundrb-devel] How to keep from generating more than one
worker?
In-Reply-To: <72b9504f0808100336v7bde4dccncf77ed9d39ee1aa9@mail.gmail.com>
References: <72b9504f0808100322w3f5c43balf3a912b02ad1fc88@mail.gmail.com>
<72b9504f0808100336v7bde4dccncf77ed9d39ee1aa9@mail.gmail.com>
Message-ID:
On 8/10/08, ?? wrote:
>
>
> Thank you for your reply.
Please, keep the converation on the mailing list. Lots of folks, try
to search archives for similar questions and hence could be useful to
them.
>
> I tried to create a new worker and assign a worker key to it, following is
> what I did in ./script/console:
>
> >> MiddleMan.new_worker(:worker => :hello_worker,:worker_key => "boy")
> => nil
> >> MiddleMan.worker(:hello_worker).worker_info
> => {:status=>:stopped, :worker=>:hello_worker, :worker_key=>nil}
When, you say:
MiddleMan.worker(:hello_worker).worker_info
you are asking worker_info for worker "hello_worker" which is running
without any worker_key and hence result is expected. Try:
MiddleMan.all_worker_info
--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.
http://gnufied.org
From leomayleomay at gmail.com Sun Aug 10 09:31:48 2008
From: leomayleomay at gmail.com (=?GB2312?B?wfXquw==?=)
Date: Sun, 10 Aug 2008 21:31:48 +0800
Subject: [Backgroundrb-devel] How to keep from generating more than one
worker?
In-Reply-To:
References: <72b9504f0808100322w3f5c43balf3a912b02ad1fc88@mail.gmail.com>
<72b9504f0808100336v7bde4dccncf77ed9d39ee1aa9@mail.gmail.com>
Message-ID: <72b9504f0808100631m2fc49e50n25a0b1490482ccb6@mail.gmail.com>
2008/8/10 hemant
> On 8/10/08, ?? wrote:
> >
> >
> > Thank you for your reply.
>
> Please, keep the converation on the mailing list. Lots of folks, try
> to search archives for similar questions and hence could be useful to
> them.
>
> >
> > I tried to create a new worker and assign a worker key to it, following
> is
> > what I did in ./script/console:
> >
> > >> MiddleMan.new_worker(:worker => :hello_worker,:worker_key => "boy")
> > => nil
> > >> MiddleMan.worker(:hello_worker).worker_info
> > => {:status=>:stopped, :worker=>:hello_worker, :worker_key=>nil}
>
> When, you say:
> MiddleMan.worker(:hello_worker).worker_info
>
> you are asking worker_info for worker "hello_worker" which is running
> without any worker_key and hence result is expected. Try:
>
> MiddleMan.all_worker_info
>
>
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
>
oops, sorry, I wrongly chose 'Reply' but 'Reply all'.
Two things:
1. How come "hello_worker" doesn't have a worker_key? I assigned "boy" as
its worker_key.
2. Can I use worker_key to identify the sole worker in my rails app? say,
can I use the following code to keep only one worker working in my app?
unless MiddleMan.worker(:alert_worker, :alerter)
MiddleMan.new_worker(:worker => :alert_worker, :worker_key => :alerter)
end
Thank you,
All the best,
Hao Liu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From todd at snappl.co.uk Mon Aug 11 07:17:57 2008
From: todd at snappl.co.uk (Todd Tyree)
Date: Mon, 11 Aug 2008 12:17:57 +0100
Subject: [Backgroundrb-devel] Jabber namespace strangeness in workers
when config.gem 'xmpp4r' is set
In-Reply-To: <489EAF55.8080905@wildgooses.com>
References: <44f20a950808070954w5996065dqe4103cbdfb866676@mail.gmail.com>
<489EAF55.8080905@wildgooses.com>
Message-ID: <44f20a950808110417w1106a93sb4b2141679d81913@mail.gmail.com>
> Could it possibly be causing the gems to load in a different order to
> normal (hence triggering the error due to some dependency issue?)
>
> Ed W
>
Seems reasonable. When I have a bit more time I'll investigate. In the
meantime, I'll just have to keep track of the dependency manually.
--T
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From todd at snappl.co.uk Mon Aug 11 07:45:43 2008
From: todd at snappl.co.uk (Todd Tyree)
Date: Mon, 11 Aug 2008 12:45:43 +0100
Subject: [Backgroundrb-devel] Too many clients already
In-Reply-To:
References:
Message-ID: <44f20a950808110445y58d227f0h3adfecc93632267a@mail.gmail.com>
>
> http://pastie.org/222713
>
> We get this error when running the test several times in a row. It
> could be that our test is not properly closing the connection to
> backgroundrb. The test reports that the error occurs in the
> before(:all) block in an RSpec.
>
> Thanks for your help!
>
> dan
>
> Hi Dan,
I'm a bit late to the party on this one, and I'm afraid I can't get in to
pastie at the moment, but I've just had a similar-sounding problem myself.
If you're doing anything with the model in your worker the error is probably
caused by the sql connection not being released when the thread is
complete. Try adding:
ActiveRecord::Base.verify_active_connections!
to the end of your worker method (and/or upping the number of concurrent
connections available on whatever sql server you are using).
Best,
Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From ramon.tayag at gmail.com Tue Aug 12 03:40:36 2008
From: ramon.tayag at gmail.com (Ramon Miguel M. Tayag)
Date: Tue, 12 Aug 2008 15:40:36 +0800
Subject: [Backgroundrb-devel] Starting It In Production?
Message-ID:
Hi everyone!
backgroundrb works fine in my local machine in development mode. When
I first tried it out in my VPS in production tho, I ran into some
problems. It seems that it keeps looking for the development database
even if I've explicitly told it to start the production environment.
Here's how I started it, then what my config/backgroundrb.yml file looks like:
http://pastebin.com/m4b7af1b6
I did some searching then saw that this was a problem 2 years ago, but
was supposedly fixed:
http://rubyforge.org/pipermail/backgroundrb-devel/2006-September/000359.html
What I did for now was put the production database details in
database.yml under "development" so it's pointing to the same
database. When I tried to start backgroundrb this time, I get:
http://pastebin.com/m4d0084ab
I checked and make sure ~/path/to/app/tmp/pids directory existed
though. Does this mean backgroundrb is unable to make its own *.pid
files?
Thank you for your time,
--
Ramon Tayag
From mengkuan at gmail.com Tue Aug 12 03:46:11 2008
From: mengkuan at gmail.com (Meng Kuan)
Date: Tue, 12 Aug 2008 15:46:11 +0800
Subject: [Backgroundrb-devel] Starting It In Production?
In-Reply-To:
References:
Message-ID:
On Aug 12, 2008, at 3:40 PM, Ramon Miguel M. Tayag wrote:
> backgroundrb works fine in my local machine in development mode. When
> I first tried it out in my VPS in production tho, I ran into some
> problems. It seems that it keeps looking for the development database
> even if I've explicitly told it to start the production environment.
>
You can try starting the daemon by prefixing with
"RAILS_ENV=production", i.e.
RAILS_ENV=production script/backgroundrb start
From gethemant at gmail.com Tue Aug 12 03:54:32 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 12 Aug 2008 13:24:32 +0530
Subject: [Backgroundrb-devel] Starting It In Production?
In-Reply-To:
References:
Message-ID:
On Tue, Aug 12, 2008 at 1:10 PM, Ramon Miguel M. Tayag
wrote:
> Hi everyone!
>
> backgroundrb works fine in my local machine in development mode. When
> I first tried it out in my VPS in production tho, I ran into some
> problems. It seems that it keeps looking for the development database
> even if I've explicitly told it to start the production environment.
>
Which version of BackgrounDRb? whats the output of?
./script/backgroundrb --version
(preferred version is the git master copy)
You can also try:
./script/backgroundrb --environment=production
> Here's how I started it, then what my config/backgroundrb.yml file looks like:
> http://pastebin.com/m4b7af1b6
>
No problems here. :)
> I did some searching then saw that this was a problem 2 years ago, but
> was supposedly fixed:
> http://rubyforge.org/pipermail/backgroundrb-devel/2006-September/000359.html
Anything thats two years old should be discarded.
>
> What I did for now was put the production database details in
> database.yml under "development" so it's pointing to the same
> database. When I tried to start backgroundrb this time, I get:
> http://pastebin.com/m4d0084ab
>
> I checked and make sure ~/path/to/app/tmp/pids directory existed
> though. Does this mean backgroundrb is unable to make its own *.pid
> files?
Check if you have permissons to write to that directory. Usually cap
set symlinks, so as tmp and public are shared between different
deployments. Double check if path is correct.
From gethemant at gmail.com Tue Aug 12 04:26:30 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 12 Aug 2008 13:56:30 +0530
Subject: [Backgroundrb-devel] Fwd: Starting It In Production?
In-Reply-To:
References:
Message-ID:
Please keep the discussion on the list, its immensely useful for folks
looking for solutions to similar problems.
---------------------------------------------------------------
Yup, I just reinstalled it today and got the instructions from the
website. Got it from git.
version: 1.0.4
It seems that when I do a simple script/backgroundrb I get permission
errors. However, I see that the current account I'm in is the owner
of script/backgroundrb, tmp, and tmp/pids (I did an "ls -l" to check)
. What I end up having to run is "RAILS_ENV=production ruby
script/backgroundrb start" instead. Not sure why...
-e production or --environment=production don't work.
But you're right. I didn't have the "tmp" folder in /home/ramon/myapp/shared
Thank you :)
On Tue, Aug 12, 2008 at 3:54 PM, hemant wrote:
>
> Which version of BackgrounDRb? whats the output of?
>
> ./script/backgroundrb --version
>
> (preferred version is the git master copy)
>
> You can also try:
>
> ./script/backgroundrb --environment=production
>
> No problems here. :)
>
> Anything thats two years old should be discarded.
>
> Check if you have permissons to write to that directory. Usually cap
> set symlinks, so as tmp and public are shared between different
> deployments. Double check if path is correct.
--
Ramon Tayag
--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.
http://gnufied.org
From iain.adams.1985 at googlemail.com Thu Aug 14 07:41:24 2008
From: iain.adams.1985 at googlemail.com (Iain)
Date: Thu, 14 Aug 2008 12:41:24 +0100
Subject: [Backgroundrb-devel] Strange behaviour (worker only running on 3 or
more attempts)
Message-ID: <48A419E4.3040807@googlemail.com>
Dear Mailing List,
I am trying to use backgroundRB to process an uploaded CSV file and save
it in the database. I have successfully done this, with only one problem
that seems very unusual. After starting the server, it takes three tries
before my script is run.
For example, in my controller I have the code:
def authenticate_import
@dataset = Dataset.new params[:dataset]
@dataset.campaign = @campaign
#Check the normal info is valid
if !@dataset.valid?
render :action => 'import'
else
#Start the worker
MiddleMan.worker(:dataset_worker).async_process(:arg =>
@dataset)
@progress =
MiddleMan.worker(:dataset_worker).ask_result(:progress)
render :action => 'progress'
end
end
This progress is then fed into a progress bar which periodically updates
itself by calling:
@progress = MiddleMan.worker(:dataset_worker).ask_result(:progress)
The problem is that the first two times I attempt uploading (.i.e
calling this code) nothing happens. No output happens at all. After that
every call works fine and all the debugging info etc is written to the
log files etc, everything is saved fine.
Does anyone have any ideas why this may be happening? any help would be
greatly appreciated
My worker is below:
class DatasetWorker < BackgrounDRb::MetaWorker
set_worker_name :dataset_worker
def create(args = nil)
logger.info "Dataset Worker setup"
end
def process(dataset)
thread_pool.defer(:process_csv, dataset)
end
def process_csv(dataset)
logger.info "Started to Process Dataset (title: #{dataset.name})"
cache[:progress] = 0.00
.
. #Code to save stuff into database and updating cache[:progress]
.
end
end
From gethemant at gmail.com Thu Aug 14 07:50:57 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 14 Aug 2008 17:20:57 +0530
Subject: [Backgroundrb-devel] Strange behaviour (worker only running on
3 or more attempts)
In-Reply-To: <48A419E4.3040807@googlemail.com>
References: <48A419E4.3040807@googlemail.com>
Message-ID:
On Thu, Aug 14, 2008 at 5:11 PM, Iain wrote:
> I am trying to use backgroundRB to process an uploaded CSV file and save it
> in the database. I have successfully done this, with only one problem that
> seems very unusual. After starting the server, it takes three tries before
> my script is run.
Which OS and what version of packet?
From gethemant at gmail.com Thu Aug 14 10:18:24 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 14 Aug 2008 19:48:24 +0530
Subject: [Backgroundrb-devel] Strange behaviour (worker only running on
3 or more attempts)
In-Reply-To: <48A41EC3.3060105@googlemail.com>
References: <48A419E4.3040807@googlemail.com>
<48A41EC3.3060105@googlemail.com>
Message-ID:
On 8/14/08, Iain wrote:
> hemant wrote:
> I'm using Packet-0.1.10 and OS is ubuntu 8.04 and yep bdrb 1.0.4
It shouldn't happen, I will check later today and reply.
From zduniak at gmail.com Thu Aug 14 10:25:46 2008
From: zduniak at gmail.com (Marcin Zduniak)
Date: Thu, 14 Aug 2008 16:25:46 +0200
Subject: [Backgroundrb-devel] best approach to managing workers and
getting status
Message-ID: <5f5d62210808140725k7d6e6b62t35ceb38629cfd8f8@mail.gmail.com>
Hi Hemant,
You are writing:
"When a task is pulled out of queue, its flagged as taken and you can
specify a timeout period while creating a task. When you invoke
"#finish!" task is marked finished. There is no, "error", because, if
you a task is "taken" and not "finished!" within specified period, its
automatically considered in error state."
I am wondering what will happen with the task that was somehow
shutdown after it was marked "taken" but before it was marked
"finished". Will it be redone ? When ?
Thank you and best regards,
Marcin Zduniak
Mobile Technology Lab, www.mobtechlab.com
Private: www.zduniak.com
---------- Forwarded message ----------
From: hemant
Date: Thu, Jul 17, 2008 at 2:36 PM
Subject: Re: [Backgroundrb-devel] best approach to managing workers
and getting status
To: Frank Schwach
On Thu, Jul 17, 2008 at 1:46 PM, Frank Schwach wrote:
> Thanks Hemant,
>
> Yes, I saw the part about the persistent job queue but I have a couple
> of questions about this:
>
> What does the table for the job queue look like?
>
You can open mysql or whatever db you are using and run:
desc bdrb_job_queues;
to see, whats the schema of the table.
> How do I set other states of the job like "error"? Is the job status
> changed from "pending" to "running" automatically in the table so that I
> can query this table for pending/running/completed jobs?
When a task is pulled out of queue, its flagged as taken and you can
specify a timeout period while creating a task. When you invoke
"#finish!" task is marked finished. There is no, "error", because, if
you a task is "taken" and not "finished!" within specified period, its
automatically considered in error state.
>
> Can I get the jobs ID from within the worker so that I can update the
> job queue table "manually" too? If I want to record a "percentage
> completion" I guess I would need that.
Sure, from anywhere in your worker code, you can use, 'persistent_job'
attribute to get task thats dequed from job queue and is currently
running.
> Can I specify the (remote) host with enq_some_job like I can with the
> async_enque_job method?
You can't, because for persistent tasks, it doesn't matter, the worker
which fetches the task first, gets to execute it anyway.
--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.
http://gnufied.org
From ramon.tayag at gmail.com Thu Aug 14 10:27:56 2008
From: ramon.tayag at gmail.com (Ramon Miguel M. Tayag)
Date: Thu, 14 Aug 2008 22:27:56 +0800
Subject: [Backgroundrb-devel] Best Way Automatically Start BDRB?
Message-ID:
Hey everyone!
What would be the best way to automatically start backgroundrb? Like
if the server crashes or reboots or something...? When I do it
manually, I have to start it like this from the rails root directory:
$ RAILS_ENV=production ruby script/backgroundrb start
It seems that if I don't put the RAILS_ENV var, it looks for
development database. This is even if I edited
config/backgroundrb.yml to use production.
Thanks!
--
Ramon Tayag
From gethemant at gmail.com Thu Aug 14 10:36:43 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 14 Aug 2008 20:06:43 +0530
Subject: [Backgroundrb-devel] best approach to managing workers and
getting status
In-Reply-To: <5f5d62210808140725k7d6e6b62t35ceb38629cfd8f8@mail.gmail.com>
References: <5f5d62210808140725k7d6e6b62t35ceb38629cfd8f8@mail.gmail.com>
Message-ID:
On 8/14/08, Marcin Zduniak wrote:
> Hi Hemant,
>
> I am wondering what will happen with the task that was somehow
> shutdown after it was marked "taken" but before it was marked
> "finished". Will it be redone ? When ?
Not automatically, I was hoping that, if tasks that are in such state,
programmer will look for them in db and decide what todo with them
(for example, why they didn't finish, what went wrong, whether they
are safe to be rerun again and stuff).
From gethemant at gmail.com Thu Aug 14 10:39:38 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 14 Aug 2008 20:09:38 +0530
Subject: [Backgroundrb-devel] Best Way Automatically Start BDRB?
In-Reply-To:
References:
Message-ID:
On 8/14/08, Ramon Miguel M. Tayag wrote:
> Hey everyone!
>
> What would be the best way to automatically start backgroundrb? Like
> if the server crashes or reboots or something...? When I do it
> manually, I have to start it like this from the rails root directory:
>
Use monit or something.
> $ RAILS_ENV=production ruby script/backgroundrb start
>
> It seems that if I don't put the RAILS_ENV var, it looks for
> development database. This is even if I edited
> config/backgroundrb.yml to use production.
>
If thats happening, its a bug. But, in git version, I have put some
effort for making sure, this kinda thing never happens. What version
you are using?
If you are still facing this problem. send me your sample app as a
tar ball (i.e, don't send me your main project, create a sample rails
app and try to simulate what happens).
From ramon.tayag at gmail.com Thu Aug 14 10:50:26 2008
From: ramon.tayag at gmail.com (Ramon Miguel M. Tayag)
Date: Thu, 14 Aug 2008 22:50:26 +0800
Subject: [Backgroundrb-devel] Best Way Automatically Start BDRB?
In-Reply-To:
References:
Message-ID:
Monit sounds great, but I got an error saying "RAILS_ENV=production"
wasn't a service or something like that.
I'm using the git version, just downloaded it the other day. :o
I'll send you that Rails app off-list :)
On Thu, Aug 14, 2008 at 10:39 PM, hemant wrote:
> On 8/14/08, Ramon Miguel M. Tayag wrote:
>> Hey everyone!
>>
>> What would be the best way to automatically start backgroundrb? Like
>> if the server crashes or reboots or something...? When I do it
>> manually, I have to start it like this from the rails root directory:
>>
>
> Use monit or something.
>
>> $ RAILS_ENV=production ruby script/backgroundrb start
>>
>> It seems that if I don't put the RAILS_ENV var, it looks for
>> development database. This is even if I edited
>> config/backgroundrb.yml to use production.
>>
>
> If thats happening, its a bug. But, in git version, I have put some
> effort for making sure, this kinda thing never happens. What version
> you are using?
>
> If you are still facing this problem. send me your sample app as a
> tar ball (i.e, don't send me your main project, create a sample rails
> app and try to simulate what happens).
>
--
Ramon Tayag
From jonathan.wallace at gmail.com Thu Aug 14 13:32:55 2008
From: jonathan.wallace at gmail.com (Jonathan Wallace)
Date: Thu, 14 Aug 2008 13:32:55 -0400
Subject: [Backgroundrb-devel] Best Way Automatically Start BDRB?
In-Reply-To:
References:
Message-ID:
Here's a link to a god configuration file I created for backgroundrb
that has a custom condition for monitoring a particular worker as
well.
http://robotpoke.blogspot.com/2008/08/monitoring-backgroundrb-workers-with.html
Jonathan
On Thu, Aug 14, 2008 at 10:39 AM, hemant wrote:
> On 8/14/08, Ramon Miguel M. Tayag wrote:
>> Hey everyone!
>>
>> What would be the best way to automatically start backgroundrb? Like
>> if the server crashes or reboots or something...? When I do it
>> manually, I have to start it like this from the rails root directory:
>>
>
> Use monit or something.
>
>> $ RAILS_ENV=production ruby script/backgroundrb start
>>
>> It seems that if I don't put the RAILS_ENV var, it looks for
>> development database. This is even if I edited
>> config/backgroundrb.yml to use production.
>>
>
> If thats happening, its a bug. But, in git version, I have put some
> effort for making sure, this kinda thing never happens. What version
> you are using?
>
> If you are still facing this problem. send me your sample app as a
> tar ball (i.e, don't send me your main project, create a sample rails
> app and try to simulate what happens).
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
From ramon.tayag at gmail.com Thu Aug 14 15:01:55 2008
From: ramon.tayag at gmail.com (Ramon Miguel M. Tayag)
Date: Fri, 15 Aug 2008 03:01:55 +0800
Subject: [Backgroundrb-devel] Best Way Automatically Start BDRB?
In-Reply-To:
References:
Message-ID:
Thanks Jonathan, I'll read up on that too!
On Fri, Aug 15, 2008 at 1:32 AM, Jonathan Wallace
wrote:
> Here's a link to a god configuration file I created for backgroundrb
> that has a custom condition for monitoring a particular worker as
> well.
>
> http://robotpoke.blogspot.com/2008/08/monitoring-backgroundrb-workers-with.html
> Jonathan
>
> On Thu, Aug 14, 2008 at 10:39 AM, hemant wrote:
>> On 8/14/08, Ramon Miguel M. Tayag wrote:
>>> Hey everyone!
>>>
>>> What would be the best way to automatically start backgroundrb? Like
>>> if the server crashes or reboots or something...? When I do it
>>> manually, I have to start it like this from the rails root directory:
>>>
>>
>> Use monit or something.
>>
>>> $ RAILS_ENV=production ruby script/backgroundrb start
>>>
>>> It seems that if I don't put the RAILS_ENV var, it looks for
>>> development database. This is even if I edited
>>> config/backgroundrb.yml to use production.
>>>
>>
>> If thats happening, its a bug. But, in git version, I have put some
>> effort for making sure, this kinda thing never happens. What version
>> you are using?
>>
>> If you are still facing this problem. send me your sample app as a
>> tar ball (i.e, don't send me your main project, create a sample rails
>> app and try to simulate what happens).
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>
--
Ramon Tayag
From woody at crystalcommerce.com Thu Aug 14 15:02:28 2008
From: woody at crystalcommerce.com (Woody Peterson)
Date: Thu, 14 Aug 2008 12:02:28 -0700
Subject: [Backgroundrb-devel] multiple database design question
Message-ID:
Hi all. I have a few questions, but I'll start a thread per question.
My first is a design issue that's fairly specific to my application,
but I thought someone might be willing to give out some insights.
My application has multiple databases, one per client, and I hijack
the database connection at the beginning of a request and connect it
to the appropriate database. So, I have to pass the database to any
workers so that they know which database to operate on. I have two
choices here:
1) Have a single worker, pass in the database to all worker methods,
and have the method connect to the correct database before beginning
the real processing
2) Create multiple workers, one per database, and have them connect to
the appropriate database when initialized via the 'create' method.
Rails can then call methods on the appropriate database specific
worker. without having to pass the database in to each method,
although it will have to use the database as a worker key when getting
the correct worker.
I like the organization of the second method, but the simplicity of
the first. I'm thinking I'll go with the first. Any thoughts?
From woody at crystalcommerce.com Thu Aug 14 15:10:11 2008
From: woody at crystalcommerce.com (Woody Peterson)
Date: Thu, 14 Aug 2008 12:10:11 -0700
Subject: [Backgroundrb-devel] enqueued job polling configurability
Message-ID: <7FB1E9DC-5EA6-4A09-BF36-ECEA7E4ED95A@crystalcommerce.com>
I don't know if this is where I should be submitting patches, but it's
also an idea/feature request, or something.
It bugs me that every worker polls the database every 5 seconds - If I
wanted something done quickly, I'd just call it with the async prefix,
no? If I'm willing to offload it to 5 seconds later, maybe I'm willing
to let it sit for 10, 20, maybe a full minute before being run.
Anyways, this would make that configurable.
line 136 of vendor/plugins/backgroundrb/server/lib/meta_worker.rb :
- add_periodic_timer(5) { check_for_enqueued_tasks }
+ add_periodic_timer(BDRB_CONFIG[:backgroundrb][:enq_poll_time].nil? ?
5 : BDRB_CONFIG[:backgroundrb][:enq_poll_time])
{ check_for_enqueued_tasks }
Same functionality, but then you could say
:backgroundrb:
:enq_poll_time: 30
And enqueued tasks would poll a lot less. I guess it's really not that
big of a hit on the database, it just bugs me for some reason. Also,
my app in particular might have lots and lots of workers, depending on
how I design the backgroundrb stuff, and 30 workers polling every 5
seconds would be a bit much.
-Woody
From woody at crystalcommerce.com Thu Aug 14 15:20:32 2008
From: woody at crystalcommerce.com (Woody Peterson)
Date: Thu, 14 Aug 2008 12:20:32 -0700
Subject: [Backgroundrb-devel] purpose of delayed persistent jobs
Message-ID:
We used to use persistent jobs implemented ourselves for an older
version of backgroundrb (not sure what version, something from 2006),
but they weren't delayed and didn't require polling. Polling isn't
really that big of a deal as far as hits to the database go, but I see
negatives of this system, and can't think of the benefits. The
downsides to delayed persistent jobs are your background tasks don't
execute immediately, and you have more hits to your database
(although small). Shopify released a plugin to do exactly this, so it
has to be useful for something, but when you have scheduling software
like bdrb that does stuff immediately and without polling (and you can
make it save results to the database if you want), what would be an
example of something that benefits from this model of delayed database-
backed jobs?
-Woody
(and sorry for the flood of questions. hopefully someone else in the
vastness of the internet has the same ones :-)
From kieran776 at gmail.com Thu Aug 14 17:36:17 2008
From: kieran776 at gmail.com (Kieran P)
Date: Fri, 15 Aug 2008 09:36:17 +1200
Subject: [Backgroundrb-devel] enqueued job polling configurability
In-Reply-To: <7FB1E9DC-5EA6-4A09-BF36-ECEA7E4ED95A@crystalcommerce.com>
References: <7FB1E9DC-5EA6-4A09-BF36-ECEA7E4ED95A@crystalcommerce.com>
Message-ID:
Hello,
I have finished this exact type of thing (plus the ability to disable it all
together) and pushed to Github last night, and sent a pull request to
gnufied.
I'm waiting for him to merge it in to his copy (he said he was going to
about 8 hours or so ago, but I guess something came up).
Regards
Kieran
On Fri, Aug 15, 2008 at 7:10 AM, Woody Peterson
wrote:
> I don't know if this is where I should be submitting patches, but it's also
> an idea/feature request, or something.
>
> It bugs me that every worker polls the database every 5 seconds - If I
> wanted something done quickly, I'd just call it with the async prefix, no?
> If I'm willing to offload it to 5 seconds later, maybe I'm willing to let it
> sit for 10, 20, maybe a full minute before being run. Anyways, this would
> make that configurable.
>
> line 136 of vendor/plugins/backgroundrb/server/lib/meta_worker.rb :
> - add_periodic_timer(5) { check_for_enqueued_tasks }
> + add_periodic_timer(BDRB_CONFIG[:backgroundrb][:enq_poll_time].nil? ? 5 :
> BDRB_CONFIG[:backgroundrb][:enq_poll_time]) { check_for_enqueued_tasks }
>
> Same functionality, but then you could say
>
> :backgroundrb:
> :enq_poll_time: 30
>
> And enqueued tasks would poll a lot less. I guess it's really not that big
> of a hit on the database, it just bugs me for some reason. Also, my app in
> particular might have lots and lots of workers, depending on how I design
> the backgroundrb stuff, and 30 workers polling every 5 seconds would be a
> bit much.
>
> -Woody
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From lists at wildgooses.com Thu Aug 14 18:12:59 2008
From: lists at wildgooses.com (Ed W)
Date: Thu, 14 Aug 2008 23:12:59 +0100
Subject: [Backgroundrb-devel] enqueued job polling configurability
In-Reply-To:
References: <7FB1E9DC-5EA6-4A09-BF36-ECEA7E4ED95A@crystalcommerce.com>
Message-ID: <48A4ADEB.6030909@wildgooses.com>
> On Fri, Aug 15, 2008 at 7:10 AM, Woody Peterson
> > wrote:
>
> I don't know if this is where I should be submitting patches, but
> it's also an idea/feature request, or something.
>
> It bugs me that every worker polls the database every 5 seconds -
> If I wanted something done quickly, I'd just call it with the
> async prefix, no? If I'm willing to offload it to 5 seconds later,
> maybe I'm willing to let it sit for 10, 20, maybe a full minute
> before being run. Anyways, this would make that configurable.
>
What needs to be changed with the queries to ensure they drop into query
cache?
I think the rough rules for mysql are:
- any updates/deletes/inserts on that table trash the entire cache
(separate status updates into a separate table if these cause query
cache to be purged too regularly)
- no column level privs or caching is bypassed
- no queries with volatile functions, eg "NOW()"
Possibly the database cost is extremely small if it can regularly hit
the query cache?
Ed W
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From woody at crystalcommerce.com Thu Aug 14 21:31:30 2008
From: woody at crystalcommerce.com (Woody Peterson)
Date: Thu, 14 Aug 2008 18:31:30 -0700
Subject: [Backgroundrb-devel] stopping backgroundrb safely
Message-ID: <3E90E096-F2E6-4899-9FC9-6C6CD0FBB41A@crystalcommerce.com>
I know from experience that mongrel finishes its requests before
stopping; does backgroundrb do the same with jobs? If not, what would
be the recommended way to restart it without stopping in the middle of
a job?
Thanks,
-Woody
From gethemant at gmail.com Thu Aug 14 22:24:10 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 15 Aug 2008 07:54:10 +0530
Subject: [Backgroundrb-devel] Strange behaviour (worker only running on
3 or more attempts)
In-Reply-To: <48A43F20.10303@googlemail.com>
References: <48A419E4.3040807@googlemail.com>
<48A41EC3.3060105@googlemail.com>
<48A43F20.10303@googlemail.com>
Message-ID:
Hi,
>>> hemant wrote:
>>> I'm using Packet-0.1.10 and OS is ubuntu 8.04 and yep bdrb 1.0.4
>>>
>>
>> It shouldn't happen, I will check later today and reply.
>>
>
> Thanks very much Herman. I am using default config except for :log:
> foreground
>
I tried to reproduce the problem, But I couldn't, here is the code, I tried:
class WowWorker < BackgrounDRb::MetaWorker
set_worker_name :wow_worker
def create(args = nil)
# we are using memcache for storing results and hence, lets reset the cache
# memcache will persist the results between backgroundrb server restarts
end
def run_wow user_args
thread_pool.defer(:some_crap,user_args)
end
def some_crap user_args
# we are using memcache for storing results and hence, lets reset the cache.
# memcache will persist the results between backgroundrb server restarts
# Also, if you are running lots of tasks parallely, don't forget
to use unique keys
cache[:progress] = 0.00
5.times do |i|
sleep(2)
logger.info "starting some_crap #{i}"
cache[:progress] = ((i+1)/5.00)*100
end
end
end
# And I called using:
MiddleMan.worker(:wow_worker).async_run_wow(:arg => "hello world")
MiddleMan.worker(:wow_worker).ask_result(:progress)
So, I wonder, whats wrong there. You can send me a minimized version
of your application in tar ball and i will see, if I can reproduce the
problem.
Also, upgrade to git version if you haven't and don't use inbuilt
cache, try replacing it with memcache (again refer the docs).
From gethemant at gmail.com Thu Aug 14 23:13:26 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 15 Aug 2008 08:43:26 +0530
Subject: [Backgroundrb-devel] multiple database design question
In-Reply-To:
References:
Message-ID:
On Fri, Aug 15, 2008 at 12:32 AM, Woody Peterson
wrote:
> Hi all. I have a few questions, but I'll start a thread per question. My
> first is a design issue that's fairly specific to my application, but I
> thought someone might be willing to give out some insights.
>
> My application has multiple databases, one per client, and I hijack the
> database connection at the beginning of a request and connect it to the
> appropriate database. So, I have to pass the database to any workers so that
> they know which database to operate on. I have two choices here:
>
> 1) Have a single worker, pass in the database to all worker methods, and
> have the method connect to the correct database before beginning the real
> processing
>
> 2) Create multiple workers, one per database, and have them connect to the
> appropriate database when initialized via the 'create' method. Rails can
> then call methods on the appropriate database specific worker. without
> having to pass the database in to each method, although it will have to use
> the database as a worker key when getting the correct worker.
>
> I like the organization of the second method, but the simplicity of the
> first. I'm thinking I'll go with the first. Any thoughts?
Second version, certainly looks cleaner, besides, I wouldn't know, how
will you be passing db connections to worker.
From gethemant at gmail.com Thu Aug 14 23:14:46 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 15 Aug 2008 08:44:46 +0530
Subject: [Backgroundrb-devel] purpose of delayed persistent jobs
In-Reply-To:
References:
Message-ID:
On Fri, Aug 15, 2008 at 12:50 AM, Woody Peterson
wrote:
> We used to use persistent jobs implemented ourselves for an older version of
> backgroundrb (not sure what version, something from 2006), but they weren't
> delayed and didn't require polling. Polling isn't really that big of a deal
> as far as hits to the database go, but I see negatives of this system, and
> can't think of the benefits. The downsides to delayed persistent jobs are
> your background tasks don't execute immediately, and you have more hits to
> your database (although small). Shopify released a plugin to do exactly
> this, so it has to be useful for something, but when you have scheduling
> software like bdrb that does stuff immediately and without polling (and you
> can make it save results to the database if you want), what would be an
> example of something that benefits from this model of delayed
> database-backed jobs?
Something thats super critical, shouldn't be lost if backgroundrb
server crashes or anything like that. Can be run manually if that
kinda thing happens.
From gethemant at gmail.com Thu Aug 14 23:15:35 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 15 Aug 2008 08:45:35 +0530
Subject: [Backgroundrb-devel] stopping backgroundrb safely
In-Reply-To: <3E90E096-F2E6-4899-9FC9-6C6CD0FBB41A@crystalcommerce.com>
References: <3E90E096-F2E6-4899-9FC9-6C6CD0FBB41A@crystalcommerce.com>
Message-ID:
On Fri, Aug 15, 2008 at 7:01 AM, Woody Peterson
wrote:
> I know from experience that mongrel finishes its requests before stopping;
> does backgroundrb do the same with jobs? If not, what would be the
> recommended way to restart it without stopping in the middle of a job?
>
I do not think, thats the case now. I will welcome a patch before I
get time to do myself.
From woody at crystalcommerce.com Fri Aug 15 02:11:24 2008
From: woody at crystalcommerce.com (Woody Peterson)
Date: Thu, 14 Aug 2008 23:11:24 -0700
Subject: [Backgroundrb-devel] multiple database design question
In-Reply-To:
References:
Message-ID: <208061CC-50A4-4F8E-8506-9F9B633C75A6@crystalcommerce.com>
On Aug 14, 2008, at 8:13 PM, hemant wrote:
> On Fri, Aug 15, 2008 at 12:32 AM, Woody Peterson
> wrote:
>> Hi all. I have a few questions, but I'll start a thread per
>> question. My
>> first is a design issue that's fairly specific to my application,
>> but I
>> thought someone might be willing to give out some insights.
>>
>> My application has multiple databases, one per client, and I hijack
>> the
>> database connection at the beginning of a request and connect it to
>> the
>> appropriate database. So, I have to pass the database to any
>> workers so that
>> they know which database to operate on. I have two choices here:
>>
>> 1) Have a single worker, pass in the database to all worker
>> methods, and
>> have the method connect to the correct database before beginning
>> the real
>> processing
>>
>> 2) Create multiple workers, one per database, and have them connect
>> to the
>> appropriate database when initialized via the 'create' method.
>> Rails can
>> then call methods on the appropriate database specific worker.
>> without
>> having to pass the database in to each method, although it will
>> have to use
>> the database as a worker key when getting the correct worker.
>>
>> I like the organization of the second method, but the simplicity of
>> the
>> first. I'm thinking I'll go with the first. Any thoughts?
>
> Second version, certainly looks cleaner, besides, I wouldn't know, how
> will you be passing db connections to worker.
I wouldn't pass the connection, just the database name via :arg =>
{:database => ActiveRecord::Base.connection.current_database}, then
call our hijack method to create a new connection once inside the
worker method. The second method is cleaner, but on a resource level,
what happens when you have lots of workers? If successful, we'll
eventually have hundreds of clients, thus hundreds of workers. Seems
to reason that it'd take more resources for that scenario. A benefit
though is that for persistent jobs each worker would poll it's
respective database, and I wouldn't have to implement some special
poll-all-databases code for the one worker.
Anyways, thanks for all your feedback!
-Woody
From kieran776 at gmail.com Fri Aug 15 02:31:35 2008
From: kieran776 at gmail.com (Kieran P)
Date: Fri, 15 Aug 2008 18:31:35 +1200
Subject: [Backgroundrb-devel] Starting It In Production?
In-Reply-To:
References:
Message-ID:
Hello,
gnufied pushed a patch earlier today that seems to have fixed this problem
for me.
Do you still get this problem on your local copy if you update to the latest
Git?
Regards
Kieran
On Tue, Aug 12, 2008 at 7:40 PM, Ramon Miguel M. Tayag <
ramon.tayag at gmail.com> wrote:
> Hi everyone!
>
> backgroundrb works fine in my local machine in development mode. When
> I first tried it out in my VPS in production tho, I ran into some
> problems. It seems that it keeps looking for the development database
> even if I've explicitly told it to start the production environment.
>
> Here's how I started it, then what my config/backgroundrb.yml file looks
> like:
> http://pastebin.com/m4b7af1b6
>
> I did some searching then saw that this was a problem 2 years ago, but
> was supposedly fixed:
>
> http://rubyforge.org/pipermail/backgroundrb-devel/2006-September/000359.html
>
> What I did for now was put the production database details in
> database.yml under "development" so it's pointing to the same
> database. When I tried to start backgroundrb this time, I get:
> http://pastebin.com/m4d0084ab
>
> I checked and make sure ~/path/to/app/tmp/pids directory existed
> though. Does this mean backgroundrb is unable to make its own *.pid
> files?
>
> Thank you for your time,
> --
> Ramon Tayag
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From ramon.tayag at gmail.com Fri Aug 15 02:33:03 2008
From: ramon.tayag at gmail.com (Ramon Miguel M. Tayag)
Date: Fri, 15 Aug 2008 14:33:03 +0800
Subject: [Backgroundrb-devel] Starting It In Production?
In-Reply-To:
References:
Message-ID:
Not anymore :) Works great!!
On Fri, Aug 15, 2008 at 2:31 PM, Kieran P wrote:
> Hello,
>
> gnufied pushed a patch earlier today that seems to have fixed this problem
> for me.
>
> Do you still get this problem on your local copy if you update to the latest
> Git?
>
> Regards
> Kieran
>
>
> On Tue, Aug 12, 2008 at 7:40 PM, Ramon Miguel M. Tayag
> wrote:
>>
>> Hi everyone!
>>
>> backgroundrb works fine in my local machine in development mode. When
>> I first tried it out in my VPS in production tho, I ran into some
>> problems. It seems that it keeps looking for the development database
>> even if I've explicitly told it to start the production environment.
>>
>> Here's how I started it, then what my config/backgroundrb.yml file looks
>> like:
>> http://pastebin.com/m4b7af1b6
>>
>> I did some searching then saw that this was a problem 2 years ago, but
>> was supposedly fixed:
>>
>> http://rubyforge.org/pipermail/backgroundrb-devel/2006-September/000359.html
>>
>> What I did for now was put the production database details in
>> database.yml under "development" so it's pointing to the same
>> database. When I tried to start backgroundrb this time, I get:
>> http://pastebin.com/m4d0084ab
>>
>> I checked and make sure ~/path/to/app/tmp/pids directory existed
>> though. Does this mean backgroundrb is unable to make its own *.pid
>> files?
>>
>> Thank you for your time,
>> --
>> Ramon Tayag
--
Ramon Tayag
From chadart at gmail.com Sat Aug 16 05:42:03 2008
From: chadart at gmail.com (Marcos Piccinini)
Date: Sat, 16 Aug 2008 06:42:03 -0300
Subject: [Backgroundrb-devel] Testing with rspec
Message-ID: <9E002EBC-6870-4440-B860-BB96FBFD7388@gmail.com>
Morning gentleman,
I would like to ask if someone can post a example ( or maybe a rspec
bdrb_test_helper ) for a rspec testing a backgroundrb worker.
Im kinda confused.. heh
Thanks,
And btw, thanks to the developers, been using backgroundrb for some
time now with great results =D
From todd at snappl.co.uk Sun Aug 17 10:13:12 2008
From: todd at snappl.co.uk (Todd Tyree)
Date: Sun, 17 Aug 2008 15:13:12 +0100
Subject: [Backgroundrb-devel] purpose of delayed persistent jobs
In-Reply-To:
References:
Message-ID: <44f20a950808170713l1f4afa31s2aaee48643596743@mail.gmail.com>
On Fri, Aug 15, 2008 at 4:14 AM, hemant wrote:
> On Fri, Aug 15, 2008 at 12:50 AM, Woody Peterson
> what would be an
> > example of something that benefits from this model of delayed
> > database-backed jobs?
>
> Something thats super critical, shouldn't be lost if backgroundrb
> server crashes or anything like that. Can be run manually if that
> kinda thing happens.
>
I've forked the main repository here:
git://github.com/tatyree/backgroundrb.git
and made a couple of changes to make the job_queue consume all outstanding
jobs at once, which is what I understand Woody to be commenting on (and
something I've needed for a while).
I'm afraid I've got a conflict on the specs as I have a hard dependency on
Rspec's internal mocks, and so I'm having some trouble making the specs run
in my environment (mocha just causes everything to fall apart badly, which
is ashame). As a result, I'm afraid the level of testing is a little (OK,
completely...it works for me) unsophisticated. That said, I'm pretty new to
Rspec, and am probably missing/misunderstanding something.
I started to re-add the test for RAILS_ENV that KieranP removed ( Line 27 of
bdrb_config.rb - Object.const_set("RAILS_ENV",environment) unless
defined?(RAILS_ENV) ) as bdrb complains on startup without it. However, I'm
not at all clear why he removed it so I decided to leave it alone.
Hope I've done everything all right. I'm new to Git and Rspec, but not new
to development or Rails.
Best,
Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From gethemant at gmail.com Sun Aug 17 12:45:59 2008
From: gethemant at gmail.com (hemant)
Date: Sun, 17 Aug 2008 22:15:59 +0530
Subject: [Backgroundrb-devel] Testing with rspec
In-Reply-To: <9E002EBC-6870-4440-B860-BB96FBFD7388@gmail.com>
References: <9E002EBC-6870-4440-B860-BB96FBFD7388@gmail.com>
Message-ID:
On Sat, Aug 16, 2008 at 3:12 PM, Marcos Piccinini wrote:
> Morning gentleman,
>
> I would like to ask if someone can post a example ( or maybe a rspec
> bdrb_test_helper ) for a rspec testing a backgroundrb worker.
>
> Im kinda confused.. heh
>
That spec_helper is for test/spec, but the basic idea is to not use
actual MetaWorker class while writing test cases for your workers.
From kieran776 at gmail.com Sun Aug 17 17:18:04 2008
From: kieran776 at gmail.com (Kieran P)
Date: Mon, 18 Aug 2008 09:18:04 +1200
Subject: [Backgroundrb-devel] purpose of delayed persistent jobs
In-Reply-To: <44f20a950808170713l1f4afa31s2aaee48643596743@mail.gmail.com>
References:
<44f20a950808170713l1f4afa31s2aaee48643596743@mail.gmail.com>
Message-ID:
Hello Todd,
gnufied actually removed that line, and for good reason. When RAILS_ENV is
set, which it appears is the case when it gets to that line, then it won't
overwrite it with that you want from the config file or the -e command line
option, so you're always stuck in development mode, no matter what you
specify.
That complaining it makes is just a warning, and can be safely ignored.
Regards
KieranP
On Mon, Aug 18, 2008 at 2:13 AM, Todd Tyree wrote:
>
> I started to re-add the test for RAILS_ENV that KieranP removed ( Line 27
> of bdrb_config.rb - Object.const_set("RAILS_ENV",environment) unless
> defined?(RAILS_ENV) ) as bdrb complains on startup without it. However, I'm
> not at all clear why he removed it so I decided to leave it alone.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From gethemant at gmail.com Sun Aug 17 17:14:48 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 18 Aug 2008 02:44:48 +0530
Subject: [Backgroundrb-devel] purpose of delayed persistent jobs
In-Reply-To: <44f20a950808170713l1f4afa31s2aaee48643596743@mail.gmail.com>
References:
<44f20a950808170713l1f4afa31s2aaee48643596743@mail.gmail.com>
Message-ID:
On Sun, Aug 17, 2008 at 7:43 PM, Todd Tyree wrote:
>
> I've forked the main repository here:
>
> git://github.com/tatyree/backgroundrb.git
>
> and made a couple of changes to make the job_queue consume all outstanding
> jobs at once, which is what I understand Woody to be commenting on (and
> something I've needed for a while).
>
> I'm afraid I've got a conflict on the specs as I have a hard dependency on
> Rspec's internal mocks, and so I'm having some trouble making the specs run
> in my environment (mocha just causes everything to fall apart badly, which
> is ashame). As a result, I'm afraid the level of testing is a little (OK,
> completely...it works for me) unsophisticated. That said, I'm pretty new to
> Rspec, and am probably missing/misunderstanding something.
>
> I started to re-add the test for RAILS_ENV that KieranP removed ( Line 27 of
> bdrb_config.rb - Object.const_set("RAILS_ENV",environment) unless
> defined?(RAILS_ENV) ) as bdrb complains on startup without it. However, I'm
> not at all clear why he removed it so I decided to leave it alone.
>
> Hope I've done everything all right. I'm new to Git and Rspec, but not new
> to development or Rails.
We can't add this, mainly because pulling all tasks out of queue at
once will essentially block the worker for a real long time. I do not
think, that will be a good idea, also if your worker is doing nothing
except running tasks from queue, you are essentially achieving the
same result(except for the part where worker is actually responding to
external events).
Thoughts?
PS: Good work, btw. Also, I am using test/spec not rspec there. ;)
From kevin at yardbarker.com Sun Aug 17 22:47:33 2008
From: kevin at yardbarker.com (kevin olsen)
Date: Sun, 17 Aug 2008 20:47:33 -0600
Subject: [Backgroundrb-devel] Testing with rspec
In-Reply-To:
References: <9E002EBC-6870-4440-B860-BB96FBFD7388@gmail.com>
Message-ID: <7B3EC329-D64A-4ED5-BD0B-6C43A71E2BB0@yardbarker.com>
hemant: could you explain?
I too would like to see an example of someone testing a worker,
whether it is with rspec or test/[unit,spec]
cheers
On Aug 17, 2008, at 10:45 AM, hemant wrote:
> On Sat, Aug 16, 2008 at 3:12 PM, Marcos Piccinini
> wrote:
>> Morning gentleman,
>>
>> I would like to ask if someone can post a example ( or maybe a rspec
>> bdrb_test_helper ) for a rspec testing a backgroundrb worker.
>>
>> Im kinda confused.. heh
>>
>
> That spec_helper is for test/spec, but the basic idea is to not use
> actual MetaWorker class while writing test cases for your workers.
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
From rue at thinlayer.co.uk Mon Aug 18 10:37:52 2008
From: rue at thinlayer.co.uk (Rue Turner)
Date: Mon, 18 Aug 2008 15:37:52 +0100
Subject: [Backgroundrb-devel] bdrb without rails?
Message-ID: <200808181537.52531.rue@thinlayer.co.uk>
Hi,
Just wondering how you setup and get bdrb running without rails?
Many thanks
From akyrho at gmail.com Tue Aug 19 04:38:40 2008
From: akyrho at gmail.com (=?ISO-8859-1?Q?C=E9dric?=)
Date: Tue, 19 Aug 2008 10:38:40 +0200
Subject: [Backgroundrb-devel] Managing with multiple worker and processor
load
Message-ID: <67b30a7b0808190138ud62b7aesff4e56f02ee68008@mail.gmail.com>
I wonder how to manage with multiple workers and processor's charges. Here's
the case :
There are two processes. The first is scanning web pages in order to find
some flash movies. If found, the process scan also for title, tags, and so
on. This process is called "crawler" and is running permanently.
In my model, the method "after_create" is calling another worker, which
receive some URL(s). The worker imports the given JPG image, crops it, and
saves it localy.
Thus, things could happen like this : the crawler is scanning a lot of web
pages, importing a lot of flash movies, which are starting the second worker
for something like 30 pictures to crop... each... And my computer is
lacking.
In the docs, i read something about queuing job, but i must confess i didn't
understand everything.
Any help would be appreciate.
Code :
[1] videothumbnailer_worker : http://pastie.org/255529
--
Bousmanne C?dric
Jabber / XMPP : akyrho at gmail.com
Mail : akyrho at gmail.com
Blog : http://www.parenthese.be/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From me at retrodict.com Tue Aug 19 10:23:55 2008
From: me at retrodict.com (P Baker)
Date: Tue, 19 Aug 2008 10:23:55 -0400
Subject: [Backgroundrb-devel] Persistent worker scheduled_at
Message-ID: <526944450808190723i176722e0tc95cd2c2ae8a7526@mail.gmail.com>
Hemant would you be interested in a patch adding an 'scheduled_at' attribute
to BdrbJobQueue objects? I'm considering the best way to run quick,
asynchronous scheduled events targeting future events. Is this an
appropriate use of the persistent worker queue mechanism or is there a
better method already built in or do you have a better idea of how to
implement something like this?
- P Baker
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From gethemant at gmail.com Tue Aug 19 14:34:35 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 20 Aug 2008 00:04:35 +0530
Subject: [Backgroundrb-devel] Testing with rspec
In-Reply-To: <7B3EC329-D64A-4ED5-BD0B-6C43A71E2BB0@yardbarker.com>
References: <9E002EBC-6870-4440-B860-BB96FBFD7388@gmail.com>
<7B3EC329-D64A-4ED5-BD0B-6C43A71E2BB0@yardbarker.com>
Message-ID:
Okay, Here is a very very simple example:
For a worker foo_worker, create foo
require File.join(File.dirname(__FILE__),"..","bdrb_test_helper")
require "foo_worker"
context "For Foo worker" do
setup do
@foo_worker = FooWorker.new
end
specify "should add two numbers" do
a = @foo_worker.add_number(:num1 => 10,:num2 => 20)
a.should == 30
end
end
And run it with:
specrb test/unit/foo_worker_test.rb
On Mon, Aug 18, 2008 at 8:17 AM, kevin olsen wrote:
> hemant: could you explain?
>
> I too would like to see an example of someone testing a worker, whether it
> is with rspec or test/[unit,spec]
>
> cheers
>
> On Aug 17, 2008, at 10:45 AM, hemant wrote:
>
>> On Sat, Aug 16, 2008 at 3:12 PM, Marcos Piccinini
>> wrote:
>>>
>>> Morning gentleman,
>>>
>>> I would like to ask if someone can post a example ( or maybe a rspec
>>> bdrb_test_helper ) for a rspec testing a backgroundrb worker.
>>>
>>> Im kinda confused.. heh
>>>
>>
>> That spec_helper is for test/spec, but the basic idea is to not use
>> actual MetaWorker class while writing test cases for your workers.
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.
http://gnufied.org
From gethemant at gmail.com Tue Aug 19 14:46:04 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 20 Aug 2008 00:16:04 +0530
Subject: [Backgroundrb-devel] Persistent worker scheduled_at
In-Reply-To: <526944450808190723i176722e0tc95cd2c2ae8a7526@mail.gmail.com>
References: <526944450808190723i176722e0tc95cd2c2ae8a7526@mail.gmail.com>
Message-ID:
Persistent workers will be best for this and yes, i will be interested
in the patch (just don't forget to write testcases for it, in past,
not having test cases, was a big problem, i don't want to go that
memory lane again).
On Tue, Aug 19, 2008 at 7:53 PM, P Baker wrote:
> Hemant would you be interested in a patch adding an 'scheduled_at' attribute
> to BdrbJobQueue objects? I'm considering the best way to run quick,
> asynchronous scheduled events targeting future events. Is this an
> appropriate use of the persistent worker queue mechanism or is there a
> better method already built in or do you have a better idea of how to
> implement something like this?
>
> - P Baker
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.
http://gnufied.org
From lists at wildgooses.com Tue Aug 19 18:28:33 2008
From: lists at wildgooses.com (Ed W)
Date: Tue, 19 Aug 2008 23:28:33 +0100
Subject: [Backgroundrb-devel] Persistent worker scheduled_at
In-Reply-To: <526944450808190723i176722e0tc95cd2c2ae8a7526@mail.gmail.com>
References: <526944450808190723i176722e0tc95cd2c2ae8a7526@mail.gmail.com>
Message-ID: <48AB4911.9040509@wildgooses.com>
P Baker wrote:
> Hemant would you be interested in a patch adding an 'scheduled_at'
> attribute to BdrbJobQueue objects? I'm considering the best way to run
> quick, asynchronous scheduled events targeting future events. Is this
> an appropriate use of the persistent worker queue mechanism or is
> there a better method already built in or do you have a better idea of
> how to implement something like this?
"me to". I would be interested in this functionality...
Cheers
Ed W
From smith.benjamin at gmail.com Thu Aug 21 19:36:23 2008
From: smith.benjamin at gmail.com (benjamin smith)
Date: Thu, 21 Aug 2008 16:36:23 -0700
Subject: [Backgroundrb-devel] Multiple worker threads per type per server?
Message-ID:
Hello,
I've recently integrated backgroundrb into my rails application and I
am very happy with how it is performing. We strictly use the 'enq' job
queuing functionality and it works well for us. We have 2 clustered
web/app servers, both of which backgroundrb is running on. When
starting up backgroundrb, it looks for all the workers classes in the
default location (lib/workers), and starts up a packet worker for
each.
However, I am not clear on how to spin up multiple instances of a
worker on the same machine. One type of worker is very taxed and I'd
like to create several of him to facilitate processing queue contents
quicker. The documents seem to suggest using MiddleMan.new_worker, but
this has not worked well for me. I'm able to get the middleman to
create a new worker instance, but it doesn't do any work to pull tasks
out of the queue.
How do you recommend I spin up multiples of one type of worker on one
machine for parallel processing of a job queue?
Thanks in advance,
Benjamin Smith
From gethemant at gmail.com Thu Aug 21 22:56:12 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 22 Aug 2008 08:26:12 +0530
Subject: [Backgroundrb-devel] Multiple worker threads per type per
server?
In-Reply-To:
References:
Message-ID:
On Fri, Aug 22, 2008 at 5:06 AM, benjamin smith
wrote:
> Hello,
>
> I've recently integrated backgroundrb into my rails application and I
> am very happy with how it is performing. We strictly use the 'enq' job
> queuing functionality and it works well for us. We have 2 clustered
> web/app servers, both of which backgroundrb is running on. When
> starting up backgroundrb, it looks for all the workers classes in the
> default location (lib/workers), and starts up a packet worker for
> each.
>
> However, I am not clear on how to spin up multiple instances of a
> worker on the same machine. One type of worker is very taxed and I'd
> like to create several of him to facilitate processing queue contents
> quicker. The documents seem to suggest using MiddleMan.new_worker, but
> this has not worked well for me. I'm able to get the middleman to
> create a new worker instance, but it doesn't do any work to pull tasks
> out of the queue.
>
> How do you recommend I spin up multiples of one type of worker on one
> machine for parallel processing of a job queue?
Thats correct, since each enqueued task belongs to a worker with a
name and worker_key, it won't be pulled out by workers, other than for
what it was created. Now, since each newly started worker, will have
its own new key, the arrangement won't work.
You can perhaps, patch up bdrb_job_queue.rb and remove the worker_key
match restriction, make it configurable and submit a patch. ;)
From smith.benjamin at gmail.com Fri Aug 22 00:49:42 2008
From: smith.benjamin at gmail.com (benjamin smith)
Date: Thu, 21 Aug 2008 21:49:42 -0700
Subject: [Backgroundrb-devel] Multiple worker threads per type per
server?
In-Reply-To:
References:
Message-ID:
Oooo, i think i get it now. I wasn't explicitly setting a worker_key
when enqueuing jobs, so an empty string was being placed into the
bdrb_job_queues.worker_key field. When starting up the default worker
threads, they get started with a worker_key of "" (empty string), thus
there is a match. I see!
Now understanding that, after a bit of experimentation, it seems that
it is possible to start new workers with duplicate worker_keys. So, I
just used the console to start up another worker of the type I wanted
with a worker_key of empty string, and voila!
MiddleMan.new_worker(:worker=>:foo_worker, :worker_key=>"")
Now, I just need to figure out how to start multiples by default when
BackgroundRb spins up! Or, perhaps it makes better sense to have a
monitoring function set on a schedule that will spin up another worker
instances once the queue grows to a certain length...
Regardless, thanks for triggering the synapses needed for me to figure this out!
Ben
From leomayleomay at gmail.com Sat Aug 23 12:07:02 2008
From: leomayleomay at gmail.com (Hao Liu)
Date: Sun, 24 Aug 2008 00:07:02 +0800
Subject: [Backgroundrb-devel] Can backgroundrb work with the rails 2.1.0?
Message-ID: <48B035A6.7070002@gmail.com>
Hi, guys,
bdrb works well with rails 1.2.6, but after I upgrade rails to 2.1.0,
there seems to be some problems with bdrb and rails.
I tried to create t a new worker with
MiddleMan.new_worker(:worker => :alert_worker, :worker_key => 'alerter')
in my application controller, after tt, I tried to start backgroundrb
server with
./script/backgroundrb start
in my terminal, but error msg given out, said:
/home/hliu/Sandbox/blah/trunk/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_cluster_connection.rb:138:in
`new_worker': No BackgrounDRb server is found running
(BackgrounDRb::NoServerAvailable)
from /home/hliu/Sandbox/blah/trunk/app/controllers/application.rb:8
Can someone tell me what's going wrong here? Thank you.
All the best,
Hao Liu
From iain.adams.1985 at googlemail.com Tue Aug 26 13:32:54 2008
From: iain.adams.1985 at googlemail.com (Iain)
Date: Tue, 26 Aug 2008 18:32:54 +0100
Subject: [Backgroundrb-devel] Does Backgroundrb work with frozen gems??
Message-ID: <48B43E46.8050705@googlemail.com>
Hi,
I have written an application that uses backgroundrb and everything was
working fine. Now I am ready for deployment I have frozen the gems into
the vendors directory of my rails app. I then removed the gems from my
local machine as a test to see if backgroundrb was using the frozen
gems. Now I am getting errors. It says it can't find the source. Does
this mean that backgroundrb does not work with frozen gems? Has anyone
come across this problem before?
Thanks
Iain
From gethemant at gmail.com Tue Aug 26 17:48:23 2008
From: gethemant at gmail.com (hemant kumar)
Date: Wed, 27 Aug 2008 03:18:23 +0530
Subject: [Backgroundrb-devel] Does Backgroundrb work with frozen gems??
In-Reply-To: <48B43E46.8050705@googlemail.com>
References: <48B43E46.8050705@googlemail.com>
Message-ID: <1219787303.29022.4.camel@shire>
Folks, have reported some problems with running frozen gems, but it
should be rather trivial to fix. Let me know, if you figured it out(in
that case, I welcome a patch) or I will look into asap.
On Tue, 2008-08-26 at 18:32 +0100, Iain wrote:
> Hi,
>
> I have written an application that uses backgroundrb and everything was
> working fine. Now I am ready for deployment I have frozen the gems into
> the vendors directory of my rails app. I then removed the gems from my
> local machine as a test to see if backgroundrb was using the frozen
> gems. Now I am getting errors. It says it can't find the source. Does
> this mean that backgroundrb does not work with frozen gems? Has anyone
> come across this problem before?
>
> Thanks
>
> Iain
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
From gethemant at gmail.com Thu Aug 28 14:05:06 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 28 Aug 2008 23:35:06 +0530
Subject: [Backgroundrb-devel] Does Backgroundrb work with frozen gems??
In-Reply-To: <48B67095.2020109@googlemail.com>
References: <48B43E46.8050705@googlemail.com> <1219787303.29022.4.camel@shire>
<48B67095.2020109@googlemail.com>
Message-ID:
What was the error? (Any exception or error is most welcome, for
faster response). What you basically need is that gem to be in path
and packet_worker_runner to be in PATH. There might be some issues
with relative paths, but i think, it should just work.
Try running backgroundrb without daemonizing it, and post the exceptions.
On Thu, Aug 28, 2008 at 3:02 PM, Iain wrote:
> hemant kumar wrote:
>>
>> Folks, have reported some problems with running frozen gems, but it
>> should be rather trivial to fix. Let me know, if you figured it out(in
>> that case, I welcome a patch) or I will look into asap.
>>
>>
>> On Tue, 2008-08-26 at 18:32 +0100, Iain wrote:
>>
>>>
>>> Hi,
>>>
>>> I have written an application that uses backgroundrb and everything was
>>> working fine. Now I am ready for deployment I have frozen the gems into the
>>> vendors directory of my rails app. I then removed the gems from my local
>>> machine as a test to see if backgroundrb was using the frozen gems. Now I am
>>> getting errors. It says it can't find the source. Does this mean that
>>> backgroundrb does not work with frozen gems? Has anyone come across this
>>> problem before?
>>>
>>> Thanks
>>>
>>> Iain
>>> _______________________________________________
>>> Backgroundrb-devel mailing list
>>> Backgroundrb-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>
>>
>>
>
> Hermant,
>
> The only Gem I am having trouble with is Packet. I think this is because it
> adds packet_worker_runner to the /usr/bin directory. I thought I could
> reference the gem copy in the PATH variable but doesn't seem to work. Any
> ideas?
>
> Thanks
>
> Iain
>
--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.
http://gnufied.org
From epugh at opensourceconnections.com Thu Aug 28 17:08:34 2008
From: epugh at opensourceconnections.com (Eric Pugh)
Date: Thu, 28 Aug 2008 17:08:34 -0400
Subject: [Backgroundrb-devel] Recipt for using with daemon_controller
Message-ID:
Has anyone played with Daemon_controller?
I was reading through
http://blog.phusion.nl/2008/08/25/daemon_controller-a-library-for-robust-daemon-management/
and they explicitly mention backgroundrb, but no samples...
Eric
From thomas.a.wood at uconn.edu Fri Aug 29 13:03:58 2008
From: thomas.a.wood at uconn.edu (Tom Wood)
Date: Fri, 29 Aug 2008 13:03:58 -0400
Subject: [Backgroundrb-devel] 1.0.4, memcache, ask_result,
and reload_on_schedule
Message-ID: <8C81AA7D3B12F4408C6B3359AEB001CC05784292@LIB-EMarks.library.lib.uconn.edu>
Hi,
Am finally getting around to upgrading our backgroundrb from 1.0.1 to
the current SVN release (1.0.4?). Most things are working quite well.
But ... I was looking forward to using the reload_on_schedule option on
several memory hungry workers that don't run that often, but am running
into a problem: even using memcached for result storage, I'm unable to
see cached results from workers started with 'reload_on_schedule true'.
I created two nearly identical workers:
class HarvesterWorker < BackgrounDRb::MetaWorker
set_worker_name :harvester_worker
reload_on_schedule true
def create(args = nil)
@counter = 0
logger.info "Harvester created (pid #{Process.pid})"
cache['result'] = @counter
end
def run(args = nil)
@counter += 1
logger.info "Harvester run called: #{@counter}"
cache['result'] = @counter
end
end
and
class Harvester2Worker < BackgrounDRb::MetaWorker
set_worker_name :harvester2_worker
def create(args = nil)
@counter = 0
logger.info "Harvester2 created (pid #{Process.pid})"
cache['result'] = @counter
end
def run(args = nil)
@counter += 1
logger.info "Harvester2 run called: #{@counter}"
cache['result'] = @counter
end
end
The only real difference between the workers is that HarvesterWorker
uses the reload_on_schedule option, and Harvester2Worker does not.
I scheduled each worker to exec the run method every 15 seconds, and
configured backgroundrb to use memcached for result storage. The log
file shows the workers are running correctly. However:
MiddleMan.worker(:harvester_worker).ask_result('result') always returns
nil, while
MiddleMan.worker(:harvester2_worker).ask_result('result') returns a
correct result.
I.e., the worker running with 'reload_on_schedule true' doesn't seem to
be caching the result.
Any thoughts? Thanks!
Tom Wood
thomas.a.wood at uconn.edu
ITS Applications Developer
University of Connecticut Libraries
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From gethemant at gmail.com Fri Aug 29 18:37:27 2008
From: gethemant at gmail.com (hemant)
Date: Sat, 30 Aug 2008 04:07:27 +0530
Subject: [Backgroundrb-devel] 1.0.4, memcache, ask_result,
and reload_on_schedule
In-Reply-To: <8C81AA7D3B12F4408C6B3359AEB001CC05784292@LIB-EMarks.library.lib.uconn.edu>
References: <8C81AA7D3B12F4408C6B3359AEB001CC05784292@LIB-EMarks.library.lib.uconn.edu>
Message-ID:
Well this is because, when a worker is restarted on schedule, its
normally started with a unique worker key and what gets used in
memcache as key is a combination of worker name, worker key(if it
exists). So for workers with reload_on_schedule workers, you don't
really know the key and hence you can't query results, just using
worker name.
Its quite possible to change this behaviour. For example, you can
modify master_proxy.rb so as it doesn't use any unique key for workers
started on schedule (see load_and_invoke method).
But this will be obtrusive, hence, I will try to think of a mechanism
so as it co-exists cleanly.
On Fri, Aug 29, 2008 at 10:33 PM, Tom Wood wrote:
> Hi,
>
>
>
> Am finally getting around to upgrading our backgroundrb from 1.0.1 to the
> current SVN release (1.0.4?). Most things are working quite well.
>
>
>
> But ? I was looking forward to using the reload_on_schedule option on
> several memory hungry workers that don't run that often, but am running into
> a problem: even using memcached for result storage, I'm unable to see cached
> results from workers started with 'reload_on_schedule true'.
>
>
>
> I created two nearly identical workers:
>
>
>
> class HarvesterWorker < BackgrounDRb::MetaWorker
>
> set_worker_name :harvester_worker
>
> reload_on_schedule true
>
>
>
> def create(args = nil)
>
> @counter = 0
>
> logger.info "Harvester created (pid #{Process.pid})"
>
> cache['result'] = @counter
>
> end
>
>
>
> def run(args = nil)
>
> @counter += 1
>
> logger.info "Harvester run called: #{@counter}"
>
> cache['result'] = @counter
>
> end
>
> end
>
>
>
> and
>
>
>
> class Harvester2Worker < BackgrounDRb::MetaWorker
>
> set_worker_name :harvester2_worker
>
>
>
> def create(args = nil)
>
> @counter = 0
>
> logger.info "Harvester2 created (pid #{Process.pid})"
>
> cache['result'] = @counter
>
> end
>
>
>
> def run(args = nil)
>
> @counter += 1
>
> logger.info "Harvester2 run called: #{@counter}"
>
> cache['result'] = @counter
>
> end
>
> end
>
>
>
> The only real difference between the workers is that HarvesterWorker uses
> the reload_on_schedule option, and Harvester2Worker does not.
>
> I scheduled each worker to exec the run method every 15 seconds, and
> configured backgroundrb to use memcached for result storage. The log file
> shows the workers are running correctly. However:
>
>
>
> MiddleMan.worker(:harvester_worker).ask_result('result') always returns nil,
> while
>
>
>
> MiddleMan.worker(:harvester2_worker).ask_result('result') returns a correct
> result.
>
>
>
> I.e., the worker running with 'reload_on_schedule true' doesn't seem to be
> caching the result.
>
>
>
> Any thoughts? Thanks!
>
>
>
> Tom Wood
>
> thomas.a.wood at uconn.edu
>
> ITS Applications Developer
>
> University of Connecticut Libraries
>
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.
http://gnufied.org