A NOTE has been added to this issue.
======================================================================
http://bugs.bacula.org/view.php?id=1979
======================================================================
Reported By: angeloio
Assigned To:
======================================================================
Project: bacula
Issue ID: 1979
Category: File Daemon
Reproducibility: always
Severity: minor
Priority: normal
Status: feedback
======================================================================
Date Submitted: 2013-01-15 16:24 GMT
Last Modified: 2013-05-10 18:08 BST
======================================================================
Summary: bacula vss windows verify job file count mismatch
Description:
director and storage daemon on debian 6 (5.2.6+dfsg-1~b); file daemon on windows
2008 r2 standard 64 sp1.
if you enable VSS options for backup the verify job fails because the file
number verified by the verify job mismatch respect the numebre of file backed
up.
Steps to Reproduce:
a) enable Windows VSS for backup of e.g. 3 files on Windows daemon. Backup
complete succefully with backup of 4 file! I can see with trace enabled a GUID
added to backup for snapshot VSS pourposes.
b) launch verify on that job:
b1) the verify type volume-to-catalog fail cause number of files is mismatch,
(minus one): file expected 4, file verified 3.
b2) launch verify type disk-to-catalog end with warnings, but the number of file
verified is always zero!
Additional Information:
This issue does not occur with VSS disabled; with Windows 2008 R2 Enterprise.
In the catalog the informations about the file related to the guid are NOT
present.
This issue occur even with bacula director version on 5.0.2
This issue occur even with bacula windows 5.0.2, 5.2.5 and 5.2.5.
======================================================================
----------------------------------------------------------------------
(0006610) kern (administrator) - 2013-02-15 21:29
http://bugs.bacula.org/view.php?id=1979#c6610
----------------------------------------------------------------------
This appears to be a support problem, and nothing you report seems abnormal to
me.
So that I can be sure, please upload your FileSet resource, then after the
backup, run a restore menu item 5 and do "mark *" then "lsmark". I suspect
that you will find that four "files" were backed up. Please include the
complete
output from running those commands.
----------------------------------------------------------------------
(0006688) Steve Lee (reporter) - 2013-05-01 22:30
http://bugs.bacula.org/view.php?id=1979#c6688
----------------------------------------------------------------------
I have the same error
Running Bacula 5.2.6 on Linux version 3.5.0-22
Client running on bacula-win64-5.2.10 on Windows Server 2008 (SP1)
We recently added verify jobs to our schedule and have noticed that one of the
jobs always fails. it shows a mismatch between expected and examined files.
Files Expected: 2
Files Examined: 1
The backup job itself reports 2 files backed up
FD Files Written: 2
SD Files Written: 2
Running queries against the catalog...
Enter SQL query: select jobfiles from job where jobid = 988;
+----------+
| jobfiles |
+----------+
| 2 |
+----------+
Enter SQL query: select count(*) from file where jobid =988;
+-------+
| count |
+-------+
| 1 |
+-------+
The fileset contains a single file
FileSet {
Name = "zmail-delta"
Include {
Options {
signature = MD5
compression = GZIP
aclsupport = yes
}
File = "E:/WindowsImageBackup.delta"
}
}
----------------------------------------------------------------------
(0006689) Steve Lee (reporter) - 2013-05-01 22:34
http://bugs.bacula.org/view.php?id=1979#c6689
----------------------------------------------------------------------
Output from running lsmark
$ ls
E:/
$ mark *
2 files marked.
$ lsmark
*E:/
*WindowsImageBackup.delta
$
----------------------------------------------------------------------
(0006690) martin (reporter) - 2013-05-02 11:50
http://bugs.bacula.org/view.php?id=1979#c6690
----------------------------------------------------------------------
I think it is caused by the job_metadata.xml object, which is not the file
table.
----------------------------------------------------------------------
(0006691) Steve Lee (reporter) - 2013-05-05 08:06
http://bugs.bacula.org/view.php?id=1979#c6691
----------------------------------------------------------------------
I reran with Enable VSS = no and this time the verify job worked
FD Files Written: 1
SD Files Written: 1
Files Expected: 1
Files Examined: 1
Enter SQL query: select jobfiles from job where jobid = 1179;
+----------+
| jobfiles |
+----------+
| 1 |
+----------+
Enter SQL query: select count(*) from file where jobid =1179;
+-------+
| count |
+-------+
| 1 |
+-------+
$ mark *
2 files marked.
$ lsmark
*E:/
*WindowsImageBackup.delta
$
----------------------------------------------------------------------
(0006705) kern (administrator) - 2013-05-10 13:51
http://bugs.bacula.org/view.php?id=1979#c6705
----------------------------------------------------------------------
For Martin: what is the job_metadata.xml object? I never heard of that. The
only xml that I am aware of is what the Bacula Enterprise plugin produces when
it is doing application VSS backups, which are entirely different from what is
happening in the FileSet shown above.
For Steve: Your SQL queries don't mean a lot to me. What is more important is
what the output of the job log is. I don't really understand what you are doing,
that is when you are running with EnableVSS=yes/no and whether those are backup
jobs or verify jobs. Also, it would help if you would show the output of what
you say "it is failing". As I say at the current time, it is not clear exactly
what you are doing, and exactly what is failing or why.
Possibly what is happening is that the entry for the directory E: is being
ignored by the verify job, but then you haven't shown anything concerning it --
neither the FileSet your are using nor the Job resource, nor the output.
----------------------------------------------------------------------
(0006706) martin (reporter) - 2013-05-10 15:27
http://bugs.bacula.org/view.php?id=1979#c6706
----------------------------------------------------------------------
It is all in the source code.
$ grep -Ir 'job_metadata\.xml' src
src/filed/backup.c: ff_pkt->object_name = (char *)"job_metadata.xml";
src/filed/job.c: if (strcmp(rop.object_name, "job_metadata.xml") == 0) {
$
Strangely though, it doesn't seem to be used in the restore code.
----------------------------------------------------------------------
(0006707) kern (administrator) - 2013-05-10 16:45
http://bugs.bacula.org/view.php?id=1979#c6707
----------------------------------------------------------------------
Yes, you are right. The code that writes out the job_metadata.xml really should
do it only when the vss plugin is loaded and tells Bacula to save it. This is an
oversight on my part. The RestoreObjects are only used by the vss plugin during
application specific restores and not by Bacula core code. So saving this
metadata is a "minor" inefficiency that I will fix.
The RestoreObjects don't enter into any counting during restore or verifies so
the user's problem lies elsewhere.
----------------------------------------------------------------------
(0006709) martin (reporter) - 2013-05-10 18:08
http://bugs.bacula.org/view.php?id=1979#c6709
----------------------------------------------------------------------
You are right that RestoreObjects don't enter into any counting during restore
or verifies, but I suspect that they *do* enter into the counting during backup
(assuming encode_and_send_attributes is called and hence JobFiles is
incremented). That causes the mismatch of job.jobfiles (2) v.s. the number of
rows in the file table (1).
Issue History
Date Modified Username Field Change
======================================================================
2013-01-15 16:24 angeloio New Issue
2013-02-15 21:29 kern Note Added: 0006610
2013-02-15 21:29 kern Status new => feedback
2013-05-01 22:30 Steve Lee Note Added: 0006688
2013-05-01 22:34 Steve Lee Note Added: 0006689
2013-05-02 11:50 martin Note Added: 0006690
2013-05-05 08:06 Steve Lee Note Added: 0006691
2013-05-07 15:37 ebollengier Severity major => minor
2013-05-10 13:51 kern Note Added: 0006705
2013-05-10 15:27 martin Note Added: 0006706
2013-05-10 16:45 kern Note Added: 0006707
2013-05-10 18:08 martin Note Added: 0006709
======================================================================

A NOTE has been added to this issue.
======================================================================
http://bugs.bacula.org/view.php?id=1979
======================================================================
Reported By: angeloio
Assigned To:
======================================================================
Project: bacula
Issue ID: 1979
Category: File Daemon
Reproducibility: always
Severity: minor
Priority: normal
Status: feedback
======================================================================
Date Submitted: 2013-01-15 16:24 GMT
Last Modified: 2013-05-11 08:29 BST
======================================================================
Summary: bacula vss windows verify job file count mismatch
Description:
director and storage daemon on debian 6 (5.2.6+dfsg-1~b); file daemon on windows
2008 r2 standard 64 sp1.
if you enable VSS options for backup the verify job fails because the file
number verified by the verify job mismatch respect the numebre of file backed
up.
Steps to Reproduce:
a) enable Windows VSS for backup of e.g. 3 files on Windows daemon. Backup
complete succefully with backup of 4 file! I can see with trace enabled a GUID
added to backup for snapshot VSS pourposes.
b) launch verify on that job:
b1) the verify type volume-to-catalog fail cause number of files is mismatch,
(minus one): file expected 4, file verified 3.
b2) launch verify type disk-to-catalog end with warnings, but the number of file
verified is always zero!
Additional Information:
This issue does not occur with VSS disabled; with Windows 2008 R2 Enterprise.
In the catalog the informations about the file related to the guid are NOT
present.
This issue occur even with bacula director version on 5.0.2
This issue occur even with bacula windows 5.0.2, 5.2.5 and 5.2.5.
======================================================================
----------------------------------------------------------------------
(0006610) kern (administrator) - 2013-02-15 21:29
http://bugs.bacula.org/view.php?id=1979#c6610
----------------------------------------------------------------------
This appears to be a support problem, and nothing you report seems abnormal to
me.
So that I can be sure, please upload your FileSet resource, then after the
backup, run a restore menu item 5 and do "mark *" then "lsmark". I suspect
that you will find that four "files" were backed up. Please include the
complete
output from running those commands.
----------------------------------------------------------------------
(0006688) Steve Lee (reporter) - 2013-05-01 22:30
http://bugs.bacula.org/view.php?id=1979#c6688
----------------------------------------------------------------------
I have the same error
Running Bacula 5.2.6 on Linux version 3.5.0-22
Client running on bacula-win64-5.2.10 on Windows Server 2008 (SP1)
We recently added verify jobs to our schedule and have noticed that one of the
jobs always fails. it shows a mismatch between expected and examined files.
Files Expected: 2
Files Examined: 1
The backup job itself reports 2 files backed up
FD Files Written: 2
SD Files Written: 2
Running queries against the catalog...
Enter SQL query: select jobfiles from job where jobid = 988;
+----------+
| jobfiles |
+----------+
| 2 |
+----------+
Enter SQL query: select count(*) from file where jobid =988;
+-------+
| count |
+-------+
| 1 |
+-------+
The fileset contains a single file
FileSet {
Name = "zmail-delta"
Include {
Options {
signature = MD5
compression = GZIP
aclsupport = yes
}
File = "E:/WindowsImageBackup.delta"
}
}
----------------------------------------------------------------------
(0006689) Steve Lee (reporter) - 2013-05-01 22:34
http://bugs.bacula.org/view.php?id=1979#c6689
----------------------------------------------------------------------
Output from running lsmark
$ ls
E:/
$ mark *
2 files marked.
$ lsmark
*E:/
*WindowsImageBackup.delta
$
----------------------------------------------------------------------
(0006690) martin (reporter) - 2013-05-02 11:50
http://bugs.bacula.org/view.php?id=1979#c6690
----------------------------------------------------------------------
I think it is caused by the job_metadata.xml object, which is not the file
table.
----------------------------------------------------------------------
(0006691) Steve Lee (reporter) - 2013-05-05 08:06
http://bugs.bacula.org/view.php?id=1979#c6691
----------------------------------------------------------------------
I reran with Enable VSS = no and this time the verify job worked
FD Files Written: 1
SD Files Written: 1
Files Expected: 1
Files Examined: 1
Enter SQL query: select jobfiles from job where jobid = 1179;
+----------+
| jobfiles |
+----------+
| 1 |
+----------+
Enter SQL query: select count(*) from file where jobid =1179;
+-------+
| count |
+-------+
| 1 |
+-------+
$ mark *
2 files marked.
$ lsmark
*E:/
*WindowsImageBackup.delta
$
----------------------------------------------------------------------
(0006705) kern (administrator) - 2013-05-10 13:51
http://bugs.bacula.org/view.php?id=1979#c6705
----------------------------------------------------------------------
For Martin: what is the job_metadata.xml object? I never heard of that. The
only xml that I am aware of is what the Bacula Enterprise plugin produces when
it is doing application VSS backups, which are entirely different from what is
happening in the FileSet shown above.
For Steve: Your SQL queries don't mean a lot to me. What is more important is
what the output of the job log is. I don't really understand what you are doing,
that is when you are running with EnableVSS=yes/no and whether those are backup
jobs or verify jobs. Also, it would help if you would show the output of what
you say "it is failing". As I say at the current time, it is not clear exactly
what you are doing, and exactly what is failing or why.
Possibly what is happening is that the entry for the directory E: is being
ignored by the verify job, but then you haven't shown anything concerning it --
neither the FileSet your are using nor the Job resource, nor the output.
----------------------------------------------------------------------
(0006706) martin (reporter) - 2013-05-10 15:27
http://bugs.bacula.org/view.php?id=1979#c6706
----------------------------------------------------------------------
It is all in the source code.
$ grep -Ir 'job_metadata\.xml' src
src/filed/backup.c: ff_pkt->object_name = (char *)"job_metadata.xml";
src/filed/job.c: if (strcmp(rop.object_name, "job_metadata.xml") == 0) {
$
Strangely though, it doesn't seem to be used in the restore code.
----------------------------------------------------------------------
(0006707) kern (administrator) - 2013-05-10 16:45
http://bugs.bacula.org/view.php?id=1979#c6707
----------------------------------------------------------------------
Yes, you are right. The code that writes out the job_metadata.xml really should
do it only when the vss plugin is loaded and tells Bacula to save it. This is an
oversight on my part. The RestoreObjects are only used by the vss plugin during
application specific restores and not by Bacula core code. So saving this
metadata is a "minor" inefficiency that I will fix.
The RestoreObjects don't enter into any counting during restore or verifies so
the user's problem lies elsewhere.
----------------------------------------------------------------------
(0006709) martin (reporter) - 2013-05-10 18:08
http://bugs.bacula.org/view.php?id=1979#c6709
----------------------------------------------------------------------
You are right that RestoreObjects don't enter into any counting during restore
or verifies, but I suspect that they *do* enter into the counting during backup
(assuming encode_and_send_attributes is called and hence JobFiles is
incremented). That causes the mismatch of job.jobfiles (2) v.s. the number of
rows in the file table (1).
----------------------------------------------------------------------
(0006710) kern (administrator) - 2013-05-11 08:29
http://bugs.bacula.org/view.php?id=1979#c6710
----------------------------------------------------------------------
RestoreObject shouldn't be sent off to the SD. I would be surprised if they are
(though I have been surprised before).
They should and will someday be sent off to the SD, but obviously like any other
Bacula metadata they should not be counted as a file. RestoreObjects are just a
form of metadata that (for backup or tape unfriendly designs such as Microsoft
VSS) can be written at the end of a job but restored at the beginning of a
restore job.
Issue History
Date Modified Username Field Change
======================================================================
2013-01-15 16:24 angeloio New Issue
2013-02-15 21:29 kern Note Added: 0006610
2013-02-15 21:29 kern Status new => feedback
2013-05-01 22:30 Steve Lee Note Added: 0006688
2013-05-01 22:34 Steve Lee Note Added: 0006689
2013-05-02 11:50 martin Note Added: 0006690
2013-05-05 08:06 Steve Lee Note Added: 0006691
2013-05-07 15:37 ebollengier Severity major => minor
2013-05-10 13:51 kern Note Added: 0006705
2013-05-10 15:27 martin Note Added: 0006706
2013-05-10 16:45 kern Note Added: 0006707
2013-05-10 18:08 martin Note Added: 0006709
2013-05-11 08:29 kern Note Added: 0006710
======================================================================