›

Missing files

xray
in PhotoBackup for iOS

Hi,

I am new to acrosync - I just tried PhotoBackup for iphone. It is great and easy to use, thanks for the good work!

A little bit of backgroup info - I have iPhone 6 64GB with IOS 9. It is almost full with pictures and long videos. There are about 1000 pciture files, and a few hundred video files. The biggest video file is about 3GB each.

I have Rsync server setup on the NAS in the deamon mode. Version 3.0.9.

After running PhotoBackup, it created three folders:

[base]/~

[base]/100APPLE

[base]/101AAPLE

The sync took about 2 hours, with no errors on the client side or in the rsync deamon server log.

However, I found out that only ~600 files were synced, so a good chunk of files were missing. Repeated sync attempts didn't find or upload any new files.

What's strange is that if I try manual selection of files to sync, I can see the missing pictures/videos in the PhotoBackup app. If I select a few to sync manully, it worked fine, with the files uploaded directly under the [base] directory on the NAS. So it feels that the app can actually find all photos, but the auto sync all somehow skipped a batch of those.

I think a rsync client is really useful for iphone. I'm happy to help iron out the issues if I can.







Comments

  • gchen
    Was that the first time you ran PhotoBackup on that phone?  Auto sync by default skips photo/video files previously uploaded, so what you encountered could happen if you delete some files after the first upload.  If this is the case, then you can turn off the 'Skip photos and videos previously uploaded' option.  Or you can remove the hidden file [base]/.photobackup which keeps track of files having been uploaded.

    If that was your first time running PhotoBackup ever then there is probably a bug in iterating through all photos/videos under the auto sync mode.
  • xray
    It was the first time I ran the app. I did try removing .photobackup several times, it did the index rebuilding again but didn't find any new files to upload, so it just finished quickly. I greped for the missing files names, they didn't show up in the .photobackup file at all. However, I do see those pictures in manual selection mode.

    I remember while I was playing with the uploading original/edited option, there was one time syncing had an error saying something like "(some function) returned nil but no error, could be an IOS bug". Not sure if it was related.

    I can try more logging on the server side that helps. Let me know. Thanks!
  • gchen
    edited September 2015
    Can you check your device log to see if you got any log messages like:

    "Skipped an asset with irregular url"

    "No PHImageFileURLKey found in asset"

    These are two possible cases where a file may be skipped. 



  • xray

    By "device log" you mean the app log on iphone? I am not sure where/how to get that log - could you tell me what the procedure is? I'll check and report.

  • gchen
    Normally device logs can be found in Xcode.  If you don't have Xcode I believe the best method is to install idevicesyslog from https://github.com/benvium/libimobiledevice-macosx/blob/master/README.md.
  • xray
    ok. I got idevicesyslog working.

    After removing .photobackup file and try syncing, I got:

    Sep 30 01:24:15 iPhone6-2014 PhotoBackup[479] <Warning>: FATAL RSYNC_ERROR rsync: link_stat "/.photobackup" (in iphone6) failed: No such file or directory (2)


    Then, it proceeded with index building and finished shortly afterwards. I saw no other messages in the syslog from PhotoBackup.

    I tried to see if I can log rsync transfer details on the rsync daemon server side, so far I couldn't figure out how to do that. My rsyncd.conf has:

    max verbosity = 5
    transfer logging = true

    and I started rsync with --daemon and --verbose. However, it still only logs client connections, with no detailed file transfer list etc. If you know how to do that, let me know and I can try to see if the rsync deamon mentioned any of those ~300 missing files.

    Anything else I can try to find out what the issue is?


  • gchen
    I can get you a special build that would help track down the issue, but that would require you sign up for our beta besting program via Apple.  Would you like to do that?  If so, send an email to support2015@acrosync.com and I'll add you to our beta user list.
  • xray
    Yeah, I'm happy to help test. I'll send u an email.
  • xray
    I got your email reply about beta user - thanks. Let me know when the beta build is ready to download.

    In the mean time, I played with it some more:

    1) If I go to iphone: photos->Albums,  (iOS 9.0.1) I see 1311 photos under 'Camber Roll', 350 videos. 2 Slo-mo, 1 Time-laps. After I remove .photobackup file and hit the 'Backup Camera Roll' button in the app, it says briefly: indexing xxxx of 1311, then indexing xxxx of 352. However, the final .photobackup file has only 553 jpg files and 201 mov files. So it seems the index building process has all the files ( although I have 350+2+1=353 videos, indexing has 352). In the end, 758 photos and 152 videos were missing.  Just to confirm, On iPhone, the reported photo/camera storage usage was 45.5GB. On the NAS, the upload directory has only 20.7GB.

    I managed to get a bit more logging on the NAS rsync daemon side. One thing I noticed is some of the entries have wrong time-stamp, it is off by 4 hours. I suspect it has something to do with the iphone rsync client. Not sure if it is related. Since the serve can control 'max verbosity', it maybe beneficial to always specify most verbose logging on the client side ( for example, if I start a normal linux rsync client with '-avvv', I get tons of useful debugging message in the server log, including the detailed filelist and total number of files etc).

    =======
    2015/10/01 21:57:56 [16363] connect from UNKNOWN (192.168.1.12)
    2015/10/02 01:57:56 [16363] rsync to iphone6/ from UNKNOWN (192.168.1.12)
    2015/10/02 01:57:56 [16363] receiving file list
    2015/10/02 01:57:56 [16363] sent 38 bytes  received 6278 bytes  total size 14891765005
    2015/10/01 21:57:57 [16365] name lookup failed for 192.168.1.12: Name or service not known
    2015/10/01 21:57:57 [16365] connect from UNKNOWN (192.168.1.12)
    2015/10/02 01:57:57 [16365] rsync to iphone6/ from UNKNOWN (192.168.1.12)
    2015/10/02 01:57:57 [16365] receiving file list
    2015/10/02 01:57:57 [16365] recv UNKNOWN [192.168.1.12] iphone6 () .photobackup 19546
    2015/10/02 01:57:57 [16365] sent 54 bytes  received 19644 bytes  total size 19546




  • xray
    Everything has been fixed in the beta version as far as I can tell. Thanks a lot for your help to address this issue!
  • gchen
    The new version 1.4 with a fix is now available in the App Store.

    For others reading this thread,  the root cause of the issue is that one method of the Apple PhotoKit API used for requesting photo and video content, requestImageDataForAsset, may return nil while no errors are being reported.  PhotoBackup can handle this situation by restarting the retrieval from the latest photo/video returned in the first batch.  But unfortunately there is a bug that photos/videos are not requested in date order, therefore those photos/videos earlier than the latest one retrieved will be skipped.

    The new version actually contains two fixes.  One is to request photos/videos sorted by date order.  This other is to call requestAVAssetForVideo instead for video files, which seems to work better and doesn't have undocumented behaviors.

    Kudos to the OP for his persistent efforts in helping us fix these bugs.
Sign In or Register to comment.