On 12/17/2010 5:03 PM, Gregg Tavares (wrk) wrote:
>
>
> On Fri, Dec 17, 2010 at 4:16 PM, Charles Pritchard <chuck@jumis.com
> <mailto:chuck@jumis.com>> wrote:
>
> We're actively developing such functionality.
>
> The limit per directory is for the sake of the os file system. If
> you want to create a data store, use indexedDB or use standard
> file system practices (create a subdirectory tree).
>
>
> I think you're missing the point. If I have a folder with 6000 files
> on some server and I want to mirror that through the FileAPI to the
> user's local machine I'm screwed. I can't **mirror** it if it's not a
> mirror.
I'm not missing the point. I'm actively developing an app that downloads
images from photo sites.
A strict Mirror (lets use a capital M) is something you're not going to
pull off with the File System API at this time.
You can't set meta data, like permissions/flags and
modification/creation dates. Developing a Mirror is not feasible with
the current API.
You can't create a direct Mirror, one which would work seamlessly with
"rsync".
You can "mirror", the data on a remote server, and check that the data
already exists on your file system,
by using a subdirectory system, much like the two level directory
structure that many cache apps use (like Squid).
> Disk space availability (quota) is an issue no matter what
> happens. When downloading 1000 images, you'll still only be doing
> so x at a time.
>
>
> I don't see the point you're trying to make here. I don't know the
> size of the images before hand. Many internet APIs make getting the
> sizes prohibitively expensive. One REST call per file. So before I can
> download a single file I'd have to issue 1000 REST XHRs to find the
> sizes of the files (waiting the several minutes for that to complete)
> before I can ask the user for the space needed. That's not a good
> user experience. If on the other hand the user can give me permission
> for unlimited space then I can just start downloading the files
> without having to find out their ultimate size.
>
> I suppose I can just request 1 terabyte up front. Or query how much
> space is free and ask for all of it.
Yes, that's correct, you'd hope the space is available.
When you run out of space, you can use a limited amount of RAM while
waiting.
There are few resource management APIs available for memory/bandwidth
hungry applications.
My point was that these questions you're bringing up are common to all
cross-platform applications.
Regarding your issue of "a ray tracing program" : You'd also want to
either: create sub directories,
or simply create a large file with its own methods.
>
> This issues are inherent in the design of any application of
> scale. At this point, the file system API does work for the use
> case you're describing.
>
> It'd be nice to see Blob storage better integrated with Web
> Storage Apis. Ian has already spoken to this, but no followers yet
> (afaik).
>
> -Charles
>
>
>
> On Dec 17, 2010, at 3:34 PM, "Gregg Tavares (wrk)"
> <gman@google.com <mailto:gman@google.com>> wrote:
>
> > Sorry if this has been covered before.
> >
> > I've been wanting to write an app to download images from photo
> sites and I'm wondering if this use case has been considered for
> the FileAPI wrt Directories and System.
> >
> > If I understand the current spec it seems like their are some
> possible issues.
> >
> > #1) The spec says there is a 5000 file limit or directory.
> >
> > #2) The spec requires an app to specify how much storage it needs
> >
> > I understand the desire for the limits. What about an app being
> able to request unlimited storage and unlimited files? The UA can
> decide how to present something to the user to grant permission if
> they want.
> >
> > Arguments against leaving it as is:
> >
> > The 5000 file limit seems arbitrary. Any app that hits that
> limit will probably require serious re-engineering to work around
> it. It will not only have to some how describe a mapping between
> files on a server that may not have that limit, it also has the
> issue the user might have something organized that way and will
> require the user to re-organize. I realize that 5000 is a large
> number. I'm sure the author of csh though 1700 entires in a glob
> was a reasonable limit as well. We all know how well that turned
> out :-( It's easy to imagine a video editing app that edits and
> composites still images. If there are a few layers and 1 image per
> layer it could easily add up to more than 5000 files in a single
> folder.
> >
> > The size limit also has issues. For many apps the size limit
> will be no problem but for others.... Example: You make ray
> tracing program, it traces each frame and saves the files to disc.
> When it runs out of room, what is its options? (1) fail. (2)
> request a file system of a larger size. The problem is (2) will
> require user input. Imagine you start your render before you leave
> work expecting it to finish by the time you get in only to find
> that 2 hours after you left the UA popped up a confirmation "this
> app requests more space"
> >
> > The same would be true for an image downloading app. You start
> the app to download a 1000 images, not knowing their sizes before
> hand. You start it and leave. Half way through the UA puts up a
> conformation asking for more space.
> >
> > Have these use cases already come up and been rejected?
> >
> >
> >
>
>