Hey there,
Due to the official hubic client being near to dead, I decided recently to switch recently to rclone that seems to be a great piece of software, but I encounter issues. I spend hours and days testing the whole thing.

I first tried without the “-n” (dry-run) flag but I got some inconsistencies so I switched to dry run to perform extended tests.
If I run the same command several times, some large files (> 1Gbyte) are synced over and over.
In the log there is:

I then checked the modification time on the remote and locally with the following commands:

rclone lsl hubic:path_to_the_file_directory

rclone lsl /home/path_to_the_file_directory
Both outputs regarding the file are exactly the same:

124974528 2017-01-09 05:09:17.284681800 DSC_3197.MOV

124974528 2017-01-09 05:09:17.284681800 DSC_3197.MOV
I do not understand why the modtime is detected as different in the first command above and not with the lsd command.
The consequences of modtime being wrongly read for large files is that they are assumed as “dirty” since the md5sum is read to null. The file is synced over and over and at the end of the day it uses a lot of data.

What is more, I have lots of files (< 1 Gbytes) where the modtime is detected as not up to date. What is wrong. These files are not synced again but the modtime is updated.
I have about 34000 files and around 200 are detected as not up to date at each run, and they are not the same each run.
I suspect an issue on reading the modtime, as if randomly, not the modtime but the upload time was considered instead.
I don’t know if it is a bug of rclone or hubic.

Anyhow, I found a command that work well and is superfast. It uses –-update and --use-server-modtime. in dry run mode it takes only a couple of seconds to perform. I copy past it for anyone that would be interested :

If I run the same command several times, some large files (> 1Gbyte) are synced over and over.

Are the troublesome files bigger than 5 GB? I suspect this is something to do with chunked files which are by default chunked at 5GB.

azelic:

I suspect an issue on reading the modtime, as if randomly, not the modtime but the upload time was considered instead

How were these file uploaded? With rclone or a different tool? Rclone uses a special bit of metadata to store the modified time.

azelic:

Anyhow, I found a command that work well and is superfast. It uses –-update and --use-server-modtime. in dry run mode it takes only a couple of seconds to perform. I copy past it for anyone that would be interested :

server mod time is good for use with hubic as it saves an http transaction per file.

You can also use --checksum or --size-only. I sync lots of images regularly to a swift backend and I use --size-only for that.

You can also use --fast-list which will may speed things up at the cost of extra memory.

Are the troublesome files bigger than 5 GB? I suspect this is something to do with chunked files which are by default chunked at 5GB.

No they aren’t. They are mainly between 1 and 2 GB.
Modtime reading errors also occur on small files. But as md5 sum is checked if modtime doesn’t fit, files are not synced but their modtime is updated.

How were these file uploaded? With rclone or a different tool? Rclone uses a special bit of metadata to store the modified time.

Files were mainly uploaded with the hubic client but some were uploaded with previous rclone attempts.

server mod time is good for use with hubic as it saves an http transaction per file.

You can also use --checksum or --size-only. I sync lots of images regularly to a swift backend and I use --size-only for that.

You can also use --fast-list which will may speed things up at the cost of extra memory.

Checksum works fine but takes a while: around 5 hours (dry run with 128 checkers) compared to 50 minutes for the default and 20 seconds for -u --use-server-modtime
–Size-only takes around 25 minutes but is disqualified since I have encrypted containers which size keeps constant even if files inside are changed.
I tried --fast-list but it was slower (I stopped it after 2 hours).

As mentioned earlier hubic is bad. Try to download a large (at least as number of files, if you have just a few encrypted containers the situation might be different) folder you’ve synced previously (and compare with the original). Many files were missing in my case.

Hi,
Here is my experience after having run my rclone sync script daily with cron for over one month.
Before starting the script I distant copied my files to another directory so that all files on the remote shall now have their metadata fixed by rclone.
My script runs the sync with the --backup-dir flag in order to keep a copy of deleted or moved files. But I encounter many errors, mainly on large files (> 1Gb) but not only. What is really problematic is that some files are not synced at all. I have the feeling that if there is an error during the copy of the deleted/moved file, than the sync is cancelled. This is really annoying because some files are no updated.
Here is the main command I run in the script:

If I remove the --backup-dir flag there is no or few errors (far less than with the flag on).
What are the options used for the copy part of the --backup-dir option? Is it also size + modtime and md5sum in case of discrepancy?

What is really problematic is that some files are not synced at all. I have the feeling that if there is an error during the copy of the deleted/moved file, than the sync is cancelled.

Hmm, the swift backend doesn’t do low level retries (which I’ve made an issue about here https://github.com/ncw/rclone/issues/2740 ) given that and that you’ve got --retries 1 failing files will not be retried. You’ll need to increase --retries

azelic:

If I remove the --backup-dir flag there is no or few errors (far less than with the flag on).
What are the options used for the copy part of the --backup-dir option?

Ah, I suspect you are falling into this issue where server side copies fail for large files on the swift backend

Hmm, the swift backend doesn’t do low level retries (which I’ve made an issue about here https://github.com/ncw/rclone/issues/2740 ) given that and that you’ve got --retries 1 failing files will not be retried. You’ll need to increase --retries

I will try again with 3 retries. But from what I have seen until now retries don’t help much in my case. If the sync fails on the first try it will also fail on the following tries. I run the script every day so that in case there is really an error it can be solved the following day. Retries consume a lot of data (my data per month is capped)

ncw:

Ah, I suspect you are falling into this issue where server side copies fail for large files on the swift backend

I will try again with 3 retries. But from what I have seen until now retries don’t help much in my case. If the sync fails on the first try it will also fail on the following tries. I run the script every day so that in case there is really an error it can be solved the following day. Retries consume a lot of data (my data per month is capped)

ncw:

I see. Syncing once per day with --retries 1 is a pretty good solution.

azelic:

No, the largest file is 2 GB

Of course I would like to help to solve the issues. Tell me what to do.

I guess the first thing to do is can you dig up the ERROR logs for the sync so we can see how it is failing. If you aren’t hitting the server side copy bug then fixing the low level retries will probably fix it.

I guess the first thing to do is can you dig up the ERROR logs for the sync so we can see how it is failing. If you aren’t hitting the server side copy bug then fixing the low level retries will probably fix it.

I copy hereunder an extract of the log file.
As far as I understand, here is what happens:
for some unknown (from me) reason, server side copy on hubic triggers a lot of

HTTP Error: 504: 504 Gateway Time-out

I experienced this a few weeks ago when I moved all my files from one directory to another one on the remote. On the first run about 30% of files triggered an error. After trying again and again during a few days some went well but still around 10% of files weren’t copied. I completed the job by doing a sync from my local directory.

In the present case, since the server side copy requested by the --backup-dir flag fails, the file is not synced.

I see 2 ways to solve the problem:
-improve the server side copy robustness but I don’t know if it is possible.
-add a flag to the --backup-dir option to force the sync even if copied/deleted files can’t be backed up.
What do you think about this?

Hi ncw,
Thanks for the new version. I have installed it and get tested for few days. It’s not easy for me to understand how the “pacer” works so I can’t really tell. But it seems it solves some sync errors but not all. For instance my 2 GB files still don’t get synced.
I wondering if the pacer isn’t trying more than 10 retries (like 2 x 10 or 3 x 10) but I’m not sure.
Would you like me to send you my complete logs to have a look? I would send them privately as they are big and I’m not keen to expose them publicly.

The size of files doesn’t appear in the logs. The 2 GB biles are “.data”, “.data.weekly”, “.data.monthly.tar.bz2”

You could try reducing --swift-chunk-size which may help with big files. This is set so that filed don’t normally get chunked (as there are various problems with chunked files) but you could try setting it to 500M say to see if that helps.