Background: Synology NAS servers use optware package manager called ipkg, and only certain versions of programs are available as precompiled libraries. The only version of bacula that they have to offer is the 3.0.3, which is precompiled for use with SQlite, which I understand is NOT recommended for use with Bacula-5.0.3. I prefer and am more familar with MySQL, so I decided to build it from source. This is a 64 bit processor (dual core Intel Atom D510).
In order to get bacula-5.0.3 to compile correctly on my Synology DS411+ NAS server, I had to #undef HAVE_CHFLAGS in both attribs.c and in create_file.c, both in the bacula-5.0.3/src/findlib directory.
It was being defined, yet nowhere in any of my sources, including all my standard .h files, was that a member of the structure "struct stat" that contained a member named st_flags.
This was causing compilation of findlib to fail, and subsequently filed, dired, stored and tools also failed to compile. Most were related to the fact the libbacfind.la was not being built.
Since I'm unfamiliar the codebase, could someone tell me what effect changing HAVE_CHFLAGS to undefined might have on operation of bacula and associated programs.
I'm hoping that it's an innocuous change, but I'd like to be reassured before deploying.
TIA for all replies,
-- Steve Downie
s.d@...

Hi Steve,
I don't know if you receive an answer about your setup and configuration.
Do you still need help ?
Regards
On Wed, Feb 16, 2011 at 7:00 PM, Steve Downie <s.d@...> wrote:
> I need some help setting up Bacula to run on a small network, consisting of
> 3 Linux boxes and 2 Win7 and 2 Win XP boxes, a mix of 32 and 64 bit
> machines.
>
> Bacula is installed, but I'm fighting the configurations to get the setup
> working. I'm in Cupertino (San Francisco Bay area) and It would be best if
> this could be done on site. I'm using a NAS server that is running as the
> main Bacula machine, with the other machines to be setup as Clients.
>
> Please email me if this is something you'd consider doing along with your
> rate (hourly or fixed).
>
> I'm an experienced software engineer, but I've spend a bit of time already
> trying to get the setup right and I need to get back to more pressing
> software issues.
>
> TIA for all replies,
>
> -- Steve Downie
> s.d@...
>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> Bacula-devel mailing list
> Bacula-devel@...
> https://lists.sourceforge.net/lists/listinfo/bacula-devel
>

On Thursday 24 February 2011 05.30:49 Dan Langille wrote:
> Oops, I originally sent this to Bacula Users...
Well, I have been a bit undecided on the subject of SQLite especially since
some people still use it or find it simpler to use, so what you wrote is OK.
So, I think not officially supporting it, but continuing to include the code
and doing a few tests against it is a reasonable compromise.
>
> A recent bug report (http://bugs.bacula.org/view.php?id=1688) which
> indicated that we are no longer supporting SQLite.
>
> I feel it's safe to stop regression testing against it.
Yes, it is better to test with something other than SQLite, but testing with
SQLite does no harm. We are able to rather quickly discount the problems that
it has.
Best regards,
Kern

I need some help setting up Bacula to run on a small network, consisting of 3 Linux boxes and 2 Win7 and 2 Win XP boxes, a mix of 32 and 64 bit machines.
Bacula is installed, but I'm fighting the configurations to get the setup working. I'm in Cupertino (San Francisco Bay area) and It would be best if this could be done on site. I'm using a NAS server that is running as the main Bacula machine, with the other machines to be setup as Clients.
Please email me if this is something you'd consider doing along with your rate (hourly or fixed).
I'm an experienced software engineer, but I've spend a bit of time already trying to get the setup right and I need to get back to more pressing software issues.
TIA for all replies,
-- Steve Downie
s.d@...

On Sunday 13 February 2011 12.00:37 rst wrote:
> Thank you for your prompt reply. I'll try to answer both Kern and
> Randy in this message.
>
> To Kern:
>
> I thought bacula is developed for some specific version of mysql
> and changing it could be very risky.
That was the case 10 years ago, but since 5-6 years, both MySQL and PostgreSQL
have been very stable for the features Bacula uses. We seldom find problems.
What you are speaking about below (version 4.1.22...) for us are rather old
versions -- I don't think they are even supported by MySQL any more.
Regards,
Kern
>
>
> To Randy:
>
>
> There are the following two types of incompatible changes:
>
> 1) "controlled" incompatible changes introduced by the mylql
> software development team. I want to provide two examples
> here:
>
> a) Some changes in version 4.1.23 are incompatible with
> the 4.1.22 version of MySQL.
>
> http://dev.mysql.com/doc/refman/4.1/en/news-4-1-23.html
>
> b) Some changes in version 4.1.13 are not compatible
> with 4.1.12
>
> http://dev.mysql.com/doc/refman/4.1/en/news-4-1-13.html
>
> 2) This case is rare but not unusual and thus it must be taken
> into account in case you need high reliability.
>
> There are incompatible changes created due to bug
> corrections. Let's assume a feature F of mysql works well
> but it has some bug that manifests in a predictible and
> known manner. Bacula (or another potential user) of F can
> decide to use the feature by learning how to avoid the
> drawbacks introduced by the bug. In this such case, the
> way bacula uses the feature F is dependent on the bug.
> Thus, correcting the bug may be not a benefit for bacula.

On Sunday 13 February 2011 13.48:34 Frank Sweetser wrote:
> On 2/13/2011 6:00 AM, rst wrote:
> > Thank you for your prompt reply. I'll try to answer both Kern and
> > Randy in this message.
>
> One more point I wanted to throw out there - while Kern and the rest of the
> Bacula developers do an excellent job of making sure Bacula works on a wide
> range of configurations, you don't have to take their word for it.
> Instead, you can set up and run the regression suite:
>
> http://bacula.org/5.0.x-manuals/en/developers/developers/Bacula_Regression_
>Testing.html
>
> This way you can put Bacula through its paces on your own exact
> configuration and verify its functionality before trying to go live. In
> addition, if you do encounter any problems, the results will be visible to
> on the regression upload site, helping troubleshooting.
Thanks for mentioning this Frank. When we are in the final phases of a release
(generally the last month) as we are currently, we look at all the results of
the regression tests. If we find something wrong, we can often change the
debug level or other parameter of a test and get the information we need to
solve problems without involving the testers ...
Regards,
Kern

>
> One more point I wanted to throw out there - while Kern and the rest of the
> Bacula developers do an excellent job of making sure Bacula works on a wide
> range of configurations, you don't have to take their word for it. Instead,
> you can set up and run the regression suite:
>
>
> http://bacula.org/5.0.x-manuals/en/developers/developers/Bacula_Regression_Testing.html
>
> This way you can put Bacula through its paces on your own exact
> configuration and verify its functionality before trying to go live. In
> addition, if you do encounter any problems, the results will be visible to
> on the regression upload site, helping troubleshooting.
>
> thx :)
Very good suggestion.
regards
cristi
> --
> Frank Sweetser fs at wpi.edu | For every problem, there is a solution
> that
> WPI Senior Network Engineer | is simple, elegant, and wrong. - HL
> Mencken
> GPG fingerprint = 6174 1257 129E 0D21 D8D4 E8A3 8E39 29E3 E2E8 8CEC
>

On 2/13/2011 6:00 AM, rst wrote:
> Thank you for your prompt reply. I'll try to answer both Kern and
> Randy in this message.
One more point I wanted to throw out there - while Kern and the rest of the
Bacula developers do an excellent job of making sure Bacula works on a wide
range of configurations, you don't have to take their word for it. Instead, you
can set up and run the regression suite:
http://bacula.org/5.0.x-manuals/en/developers/developers/Bacula_Regression_Testing.html
This way you can put Bacula through its paces on your own exact configuration
and verify its functionality before trying to go live. In addition, if you do
encounter any problems, the results will be visible to on the regression upload
site, helping troubleshooting.
--
Frank Sweetser fs at wpi.edu | For every problem, there is a solution that
WPI Senior Network Engineer | is simple, elegant, and wrong. - HL Mencken
GPG fingerprint = 6174 1257 129E 0D21 D8D4 E8A3 8E39 29E3 E2E8 8CEC

Thank you for your prompt reply. I'll try to answer both Kern and
Randy in this message.
To Kern:
I thought bacula is developed for some specific version of mysql
and changing it could be very risky.
To Randy:
There are the following two types of incompatible changes:
1) "controlled" incompatible changes introduced by the mylql
software development team. I want to provide two examples
here:
a) Some changes in version 4.1.23 are incompatible with
the 4.1.22 version of MySQL.
http://dev.mysql.com/doc/refman/4.1/en/news-4-1-23.html
b) Some changes in version 4.1.13 are not compatible
with 4.1.12
http://dev.mysql.com/doc/refman/4.1/en/news-4-1-13.html
2) This case is rare but not unusual and thus it must be taken
into account in case you need high reliability.
There are incompatible changes created due to bug
corrections. Let's assume a feature F of mysql works well
but it has some bug that manifests in a predictible and
known manner. Bacula (or another potential user) of F can
decide to use the feature by learning how to avoid the
drawbacks introduced by the bug. In this such case, the
way bacula uses the feature F is dependent on the bug.
Thus, correcting the bug may be not a benefit for bacula.

On Saturday 12 February 2011 14.41:40 rst wrote:
> Hello all,
>
> Is it possible to know the exact version of mysql that
> was used for testing bacula 5.0.3? I'm asking this question
> because some patches of mysql introduce incompatible
> changes.
Aside from binaries which require the version on the platform for which the
binary is built, there is no specific version of MySQL that we use or we test.
In my case, it was probably version 5.1.x, which is loaded on my development
machine, but on other platforms, it could well be other versions. Other than
a problem with the one version of MySQL (5.5?) that has made MaxValue a
keyword, there are no problems. That particular problem is corrected (worked
around) in the Source Forge git repository Branch-5.0, and it only occurs if
you are creating a new database from scratch.
Regards,
Kern
>
> best regards
> cristian
>
> ---------------------------------------------------------------------------
>--- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio
> XE: Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> Bacula-devel mailing list
> Bacula-devel@...
> https://lists.sourceforge.net/lists/listinfo/bacula-devel

Hello all,
Is it possible to know the exact version of mysql that
was used for testing bacula 5.0.3? I'm asking this question
because some patches of mysql introduce incompatible
changes.
best regards
cristian

Hello Alan,
Le jeudi 10 février 2011 16:06:24, Alan Brown a écrit :
> There was discussion on the devel list sometime backabout using
> fadvise() on the filedaemon.
>
> Can anyone tell me what the outcome was? Is it used? I can only see
> references to it in the storage daemon's code.
We use it for years now, take a look to findlib/bfile.c.
Bye
--
Need professional help and support for Bacula ?
Visit http://www.baculasystems.com

There was discussion on the devel list sometime backabout using
fadvise() on the filedaemon.
Can anyone tell me what the outcome was? Is it used? I can only see
references to it in the storage daemon's code.
AB

Linux has a couple of "interesting" ioctls which in one case would allow
faster + Sparse-accurate file handling in one go.
FIEMAP
Recommended LWN article: SEEK_HOLE or FIEMAP?
http://lwn.net/Articles/260795/
When an application wants to know how a file is store in the disk (for
example, a backup application that wants to know if a file is a sparse
file and wants to avoid backing up the hole) it uses the fibmap ioctl.
But this ioctl is suboptimal - the ioctl can only be asked for a block
at a time, which is too expensive for big files. The FIEMAP ioctl, in
the other hand, returns a list of extents.
(It's aimed at handling sparse files without the overheads.)
Other OSes also have this ioctl, or an equivalent version.
Is anyone interested/able to code for it?

(I suspect that this should be setup in the FAQ somewhere)
Now that I've got a better understanding of the problem, I see two
options for fabric connected tape drives (usually inside a library)
1: Send unlock commands to all instances of the tape drive - this
requires that all instances are known to bacula or the scripts it calls.
This can be hacked up fairly quickly and I may be able to create some
udev rules to reduce the necessity to scan for tapedrive WWIDs at each
mtx-changer call.
OR
2: Have an option not to lock the drive in the first place (bacula-sd)
Here's a better explanation of the problem as reported on HP's forums.
=====
https://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1466961
"...prevent/allow settings are per initiator/target connection so a
single host can have multiple connections and send a prevent over one
then for some reason switch to a different port - the allow has to come
from the same port as the prevent"
"The SCSI standards require that the drive logically OR all of the PAMR
settings for the different initiators and if any of them are set to 1
then media removal is prevented and in this case media removal is
prevented by the host with the WWN shown above."
====

Hi all,
there are two problems in Bacula that makes logwatch unusable under
some circumstances.
First bug is in dispatch_message function. If you have defined catalog
directive in config file, all subsequent destinations incorrectly uses
catalog date format because catalog output overwrites date string.
This is clear bug and it is easy to fix it.
But there is a second issue that is connected with locales.
The bstrftime_ny function uses %d-%b %H:%M format which contains month
in string format. Problem is that this string varies under different
locales. The logwatch forces locale to C and it is not able to correctly
find log entries by date.
I propose to change date format. It shouldn't use month in string format
or it should be forced to C locale.
What are your opinions? I can create a patch but I don't know which
solution is better.
Jan
More informations in Fedora Bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=669758

-------- Original Message --------
Subject: [Bacula-users] webpage - what tapes are in my tape library?
Date: Wed, 09 Feb 2011 21:26:29 -0500
From: Dan Langille <dan@...>
Organization: The FreeBSD Diary
To: bacula-users <bacula-users@...>
Over the past week, I've been playing with a webpage that displays the
following information:
* what is ready for copy from disk to tape
* the status of the tapes in the tape library
* how much data is on tape
* the status of the tapes not in the tape library
The current status is at https://services.unixathome.org/tapes.php
This is what it looked like earlier tonight, before I replaced some tapes:
http://www.freebsddiary.org/tapes.html
After inserting the new tapes, one of which was unknown to Bacula. By
this time, I had run 'update slots'.
http://www.freebsddiary.org/tapes.unknown.html
Then, after I ran 'label barcodes', the output looked like this:
http://www.freebsddiary.org/tapes.after.label.html
The source code for this is at:
http://www.freebsddiary.org/samples/tapes.txt
The SQL is PostgreSQL specific. I'm sure it can be adapted to
MySQL/SQLite. The code is also specific to the disk and tape pools that
I am interested in.
--
Dan Langille - http://langille.org/
------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
Bacula-users mailing list
Bacula-users@...
https://lists.sourceforge.net/lists/listinfo/bacula-users

Firstly, thanks for the response. I appreciate your time to feedback to
me.
On Tue, 2011-02-08 at 18:44 +0100, Eric Bollengier wrote:
> Hello Matthew,
>
> Le jeudi 20 janvier 2011 15:23:21, Matthew Ife a écrit :
> > The patch below adds native support for quotas in bacula 5.0.3.
>
> Thanks for your interest, this feature is really interesting for big hosting
> providers.
>
> > We resell bacula and require a method to prevent resource abuse of the
> > service we provide. We have used another method to do this in the past
> > (Max Volume Jobs and Max Volume Bytes). However, this solution does not
> > scale to our needs, leads to avoidable delays in backups due to waiting
> > for volumes to become free and is difficult to administratively manage.
>
> An other quick way to handle this is to have a RunScript that checks for the
> quota when the job starts, and prevents or not the job to run. Perhaps more
> flexible, but not so well integrated.
That might of saved me a couple of weekends of coding if I had thought
of that :). I suspect because I was focused on the problem of what to do
when a job runs this was never something I considered palatable.
>
> > Personally I believe volume management should not be responsible for this
> > type of work, and that it belongs in job management. Thus the patch below
> > provides this functionality.
> >
> > Quotas attempt to mostly emulate the functionality of filesystem quotas,
> > providing soft and hard limits with grace times.
> >
> > I've tried to document where I thin necessary. I would like the patch to be
> > looked into and criticized - I am not a developer by trade and imagine
> > there is some things that could be done better! I am open to suggestions.
>
> No problem, you code is clean (next time, I will appreciate a git diff or
> diff -Naur which are easier to read)
>
I'll bear that in mind when I look at changes.
> > A "Quota" is determined by the sum total of all JobBytes values within the
> > JobRetention period. Thus increasing your job retention increases the
> > scope for which quota is evaluated. In addition deleting or purging your
> > jobs has the effect to modifying the reported quota value.
> >
> > Quotas are checked at two spots. The first is when the job is started.
> > Quotas are checked and if you exceed quota your job is terminated before
> > connections to bacula-fd are initialized.
>
> Ok, nice.
>
> > The second place they are
> > checked is in the Job Monitor job. Every minute those jobs which have
> > quotas enabled are checked again for their quotas against their running
> > jobs.
>
>
> I don't like the idea to spawn a storage daemon connexion every minutes for
> every jobs in the watchdog, this is a big overhead, what are the benefits to
> check the quota during the job running time?
> I can understand that it's more "strict", but the cost associated seems to be
> very high for a very small benefit. For example, you can have cases where the
> job will be almost completed and you will cancel it for few bytes and have to
> restart the whole job later, this job will use resources and won't be usable
> for restore (directly).
Well, its this only occurs on those that have quota enabled. But I agree
doing this incurs a lot of cost in complexity of implementation.
I've tried to make this as passive as possible. The SD connection needed
to do job status doesn't signal a fatal end if the connection does not
work. Rather it returns without updating the JobBytes value. What this
does do though is potentialy fill up the max-sd connections, although
the worst case I can envision is a job being delayed from starting
because of it.
I like the idea of performing this check in case, for example; the job
being ran is a full - your quota is almost full and the new job run now
takes up 200 extra gigs of space.
Is the idea of perhaps adding a per-client config option to enable
in-job quota checks (perhaps being ran every five minutes instead) more
digestable, or is the scheme fundamentally problematic?
>
> If you absolutely don't want to use more resources than expected, you can
> probably estimate the size of the next job by looking previous job in the
> catalog.
>From a hosted provider experience I am not sure explaining to one of our
customers that we didnt take a backup because it *might* be over is a
very tenable position for us to be in really. Other suggestions like
perhaps doing something with an estimate job might work though..
>
>
> > This is done by taking the sum total of all jobs so far, plus the
> > bytes value (as reported by the storage daemon for that job) and
> > performing the quota check against that.
> >
> > To enable this facility in the Client resource of a bacula config file the
> > following four items have been introduced.
> >
> > Hard Quota - (takes a byte size) The absolute ceiling limit of size a total
> > quota can be. Soft Quota - (takes a byte size) The limit of quota you
> > can have once you have exceeded your grace period. Soft Quota Grace Period
> > - (takes a time period) The amount of grace period time you are permitted
> > before soft quotas are enforced. Strict Quotas - (takes yes or no) This
> > changes the behavior of quotas. When in the soft quota you can 'burst' up
> > to the hard limit or grace time (as per filesystem quotas). When strict is
> > off (the default) the client ends up receiving the total quota they
> > bursted up to as their new soft quota. If strict is turned on, then the
> > true soft quota is enforced the next time a backup is attempted to be ran.
>
> Sorry, I'm a bit lost with the "Strict Quotas" keyword, can you explain it
> again with some examples ?
Yes, effectively this is a throwback to something we do as a company. So
it might end up being entirely superfluous and be removed.
A client has a 50G soft quota, 200G hard quota. A 7 day job retention
and a 2 day grace period.
Date Type Size Note
1st F 45G (45 total) In quota
2nd I 10G 55G Warning issued, over softquota. Grace day 1.
3rd I 15G 70G Warning issued, over softquota. Grace day 2.
With strict quotas off..
4th I 15G 0G Backup failed. Their permitted quota is now a
maximum of 70G.
With strict quotas on.
4th I 15G 0G Backup failed. Their permitted quota is now a
maximum of 50G.
So with strict quotas on, just like standard filesystem quotas once
grace has expired you must not exceed your soft quota. With strict
quotas off once grace expires you must not exceed the total quota you
used until your grace expired. Hope this helps. I'm happy to remove this
as an inclusion if it simply doesnt really have any effective benefit. I
agree its scope is quite limited.
Perhaps 'Strict Quotas' is a more relevent term for permitting SD checks
of quotas such as I mentioned above? :)
>
> > In addition, I added a "purge -> quotas" option in bconsole to reset quotas
> > (this effectively resets the grace time so exceeding soft quota starts the
> > grace timer again).
>
> Ok
>
> > Onto the code. Now, I'm certainly open to criticisms and suggestions on how
> > to improve this. There are a few things I consciously did which might need
> > to change.
>
> Your code is rather clean, I see few tweaks that can wait, but this is a very
> good start. A very important point is to have a simple test for this feature.
> You can adapt an existing regress script and ensure that your code does the
> right thing, I can help you on this part. First, take a look to
> regress/tests/prune-test for example, it shows how to add directives and how
> to check results)
I'll have a look. Might have more questions in regards to this at some
point..
>
> > Firstly, the nature of this code means that locking inside of it must be
> > avoided. Thus, I have had to rewrite some functions already used elsewhere
> > in bacula omitting the locks and working around potential indefinite
> > blocks.
>
> This part is associated with the SD pooling code, which is not essential
> (IMHO)
>
> > These have been added to a quota.c file along with the actual
> > quota checking code. For the most part they duplicate already used
> > functions but I felt updating existing functions to support what work I
> > was doing was dangerous. This might not be the best approach and I'm open
> > to suggestions here.
>
> What about group of clients? I can imagine that some users will want to
> allocate quota per "group" of clients (or by Pool), and not client by client
> basis. Is it something that should be considered now? (IMHO, it can wait)
>From a design perspective I think this is a good idea, but I'd prefer
the config options to stay out of the pool or storage resources as I
think quota management is a job related feature not a volume related
one. Perhaps adding it to a Job resource, or JobDef resouce instead?
Theres no relevent SQL support for that currently so something would
have to be produced at a later date.
>
> >From what I can see, the Quota table can be merged with the Client table.
>
Yes, this was me not wanting to fiddle with other tabling arrangements
in bacula. I felt it was obtrusive unless I got a second opinion on
it :). Thanks.
> > The majority of quota checking code is in quota.c, there is some stuff in
> > other header files and some stuff in the cats directory used for database
> > access to quota check with.
> >
> > This patch is running on our own network of 300 backup servers spread
> > amongst 1200 (so far) clients using it. We are gradually cranking up the
> > usage week by week. So far this patch works flawlessly. Other tortures
> > I've done include running every job with quotas on the backup server
> > immediately (about 15) to see if it would cope. So far things have been
> > alright.
> >
> > This does need more testing, particularly on postgres and sqlite3 which I
> > have not done.
> >
> > I also don't know how this code would cope on one director
> > running 1000 concurrent jobs (this is something we intend to be doing in
> > the coming months).
>
> With the SD pooling code, I can imagine that if you have a network problem, it
> can lead to strange things.
I would be greatly appreciative if you could expand here please :).
>
> > OK, enough rambling on. Heres the patch below, I apologize for the size.
> > Feedback is greatly appreciated. Again, Im not a developer so if anyone
> > knows how I can make the patch put quota.c/quota.h in the correct
> > directory (not the one I originally diffed from) I would be much obliged!
>
> Having specific files for quota is ok, no worry. Once we made a decision on
> the pooling part, it will be easier to decide.
If the pooling isnt necessary. Its probably easier to put the functions
somewhere else like in job the files.
>
> To use git, first clone the git repo from sourceforge, create your own branch
> from the Branch-5.1 code, add your changes, and generate diffs (you can also
> publish them on github for example).
>
> git clone git://bacula.git.sourceforge.net/gitroot/bacula/bacula
> cd bacula
> git checkout -b quota origin/Branch-5.1
>
> git add ...
> git commit
>
> git diff origin/Branch-5.1
>
>
> You can find information on our git usage on:
> http://www.bacula.org/5.1.x-manuals/en/developers/developers/Git_Usage.html
>
> Bye
>
Thanks for your input.

Hello Matthew,
Le jeudi 20 janvier 2011 15:23:21, Matthew Ife a écrit :
> The patch below adds native support for quotas in bacula 5.0.3.
Thanks for your interest, this feature is really interesting for big hosting
providers.
> We resell bacula and require a method to prevent resource abuse of the
> service we provide. We have used another method to do this in the past
> (Max Volume Jobs and Max Volume Bytes). However, this solution does not
> scale to our needs, leads to avoidable delays in backups due to waiting
> for volumes to become free and is difficult to administratively manage.
An other quick way to handle this is to have a RunScript that checks for the
quota when the job starts, and prevents or not the job to run. Perhaps more
flexible, but not so well integrated.
> Personally I believe volume management should not be responsible for this
> type of work, and that it belongs in job management. Thus the patch below
> provides this functionality.
>
> Quotas attempt to mostly emulate the functionality of filesystem quotas,
> providing soft and hard limits with grace times.
>
> I've tried to document where I thin necessary. I would like the patch to be
> looked into and criticized - I am not a developer by trade and imagine
> there is some things that could be done better! I am open to suggestions.
No problem, you code is clean (next time, I will appreciate a git diff or
diff -Naur which are easier to read)
> A "Quota" is determined by the sum total of all JobBytes values within the
> JobRetention period. Thus increasing your job retention increases the
> scope for which quota is evaluated. In addition deleting or purging your
> jobs has the effect to modifying the reported quota value.
>
> Quotas are checked at two spots. The first is when the job is started.
> Quotas are checked and if you exceed quota your job is terminated before
> connections to bacula-fd are initialized.
Ok, nice.
> The second place they are
> checked is in the Job Monitor job. Every minute those jobs which have
> quotas enabled are checked again for their quotas against their running
> jobs.
I don't like the idea to spawn a storage daemon connexion every minutes for
every jobs in the watchdog, this is a big overhead, what are the benefits to
check the quota during the job running time?
I can understand that it's more "strict", but the cost associated seems to be
very high for a very small benefit. For example, you can have cases where the
job will be almost completed and you will cancel it for few bytes and have to
restart the whole job later, this job will use resources and won't be usable
for restore (directly).
If you absolutely don't want to use more resources than expected, you can
probably estimate the size of the next job by looking previous job in the
catalog.
> This is done by taking the sum total of all jobs so far, plus the
> bytes value (as reported by the storage daemon for that job) and
> performing the quota check against that.
>
> To enable this facility in the Client resource of a bacula config file the
> following four items have been introduced.
>
> Hard Quota - (takes a byte size) The absolute ceiling limit of size a total
> quota can be. Soft Quota - (takes a byte size) The limit of quota you
> can have once you have exceeded your grace period. Soft Quota Grace Period
> - (takes a time period) The amount of grace period time you are permitted
> before soft quotas are enforced. Strict Quotas - (takes yes or no) This
> changes the behavior of quotas. When in the soft quota you can 'burst' up
> to the hard limit or grace time (as per filesystem quotas). When strict is
> off (the default) the client ends up receiving the total quota they
> bursted up to as their new soft quota. If strict is turned on, then the
> true soft quota is enforced the next time a backup is attempted to be ran.
Sorry, I'm a bit lost with the "Strict Quotas" keyword, can you explain it
again with some examples ?
> In addition, I added a "purge -> quotas" option in bconsole to reset quotas
> (this effectively resets the grace time so exceeding soft quota starts the
> grace timer again).
Ok
> Onto the code. Now, I'm certainly open to criticisms and suggestions on how
> to improve this. There are a few things I consciously did which might need
> to change.
Your code is rather clean, I see few tweaks that can wait, but this is a very
good start. A very important point is to have a simple test for this feature.
You can adapt an existing regress script and ensure that your code does the
right thing, I can help you on this part. First, take a look to
regress/tests/prune-test for example, it shows how to add directives and how
to check results)
> Firstly, the nature of this code means that locking inside of it must be
> avoided. Thus, I have had to rewrite some functions already used elsewhere
> in bacula omitting the locks and working around potential indefinite
> blocks.
This part is associated with the SD pooling code, which is not essential
(IMHO)
> These have been added to a quota.c file along with the actual
> quota checking code. For the most part they duplicate already used
> functions but I felt updating existing functions to support what work I
> was doing was dangerous. This might not be the best approach and I'm open
> to suggestions here.
What about group of clients? I can imagine that some users will want to
allocate quota per "group" of clients (or by Pool), and not client by client
basis. Is it something that should be considered now? (IMHO, it can wait)
From what I can see, the Quota table can be merged with the Client table.
> The majority of quota checking code is in quota.c, there is some stuff in
> other header files and some stuff in the cats directory used for database
> access to quota check with.
>
> This patch is running on our own network of 300 backup servers spread
> amongst 1200 (so far) clients using it. We are gradually cranking up the
> usage week by week. So far this patch works flawlessly. Other tortures
> I've done include running every job with quotas on the backup server
> immediately (about 15) to see if it would cope. So far things have been
> alright.
>
> This does need more testing, particularly on postgres and sqlite3 which I
> have not done.
>
> I also don't know how this code would cope on one director
> running 1000 concurrent jobs (this is something we intend to be doing in
> the coming months).
With the SD pooling code, I can imagine that if you have a network problem, it
can lead to strange things.
> OK, enough rambling on. Heres the patch below, I apologize for the size.
> Feedback is greatly appreciated. Again, Im not a developer so if anyone
> knows how I can make the patch put quota.c/quota.h in the correct
> directory (not the one I originally diffed from) I would be much obliged!
Having specific files for quota is ok, no worry. Once we made a decision on
the pooling part, it will be easier to decide.
To use git, first clone the git repo from sourceforge, create your own branch
from the Branch-5.1 code, add your changes, and generate diffs (you can also
publish them on github for example).
git clone git://bacula.git.sourceforge.net/gitroot/bacula/bacula
cd bacula
git checkout -b quota origin/Branch-5.1
git add ...
git commit
git diff origin/Branch-5.1
You can find information on our git usage on:
http://www.bacula.org/5.1.x-manuals/en/developers/developers/Git_Usage.html
Bye
--
Need professional help and support for Bacula ?
Visit http://www.baculasystems.com