Homec4science

Add a chunking storage engine for files

Authored by epriestley <git@epriestley.com> on Mar 13 2015, 19:30.

Description

Add a chunking storage engine for files

Summary:
Ref T7149. This isn't complete and isn't active yet, but does basically work. I'll shore it up in the next few diffs.

The new workflow goes like this:

Client, file.allocate(): I'd like to upload a file with length L, metadata M, and hash H.

Then the server returns upload (a boolean) and filePHID (a PHID). These mean:

uploadfilePHIDmeans
falsefalseServer can't accept file.
falsetrueFile data already known, file created from hash.
truefalseJust upload normally.
truetrueQuery chunks to start or resume a chunked upload.

All but the last case are uninteresting and work like exising uploads with file.uploadhash (which we can eventually deprecate).

In the last case:

Client, file.querychunks(): Give me a list of chunks that I should upload.

This returns all the chunks for the file. Chunks have a start byte, an end byte, and a "complete" flag to indicate that the server already has the data.

Then, the client fills in chunks by sending them:

Client, file.uploadchunk(): Here is the data for one chunk.

This stuff doesn't work yet or has some caveats:

  • I haven't tested resume much.
  • Files need an "isPartial()" flag for partial uploads, and the UI needs to respect it.
  • The JS client needs to become chunk-aware.
  • Chunk size is set crazy low to make testing easier.
  • Some debugging flags that I'll remove soon-ish.
  • Downloading works, but still streams the whole file into memory.
  • This storage engine is disabled by default (hardcoded as a unit test engine) because it's still sketchy.
  • Need some code to remove the "isParital" flag when the last chunk is uploaded.
  • Maybe do checksumming on chunks.

Test Plan:

  • Hacked up arc upload (see next diff) to be chunk-aware and uploaded a readme in 18 32-byte chunks. Then downloaded it. Got the same file back that I uploaded.
  • File UI now shows some basic chunk info for chunked files:

{F336434}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: joshuaspence, epriestley

Maniphest Tasks: T7149

Differential Revision: https://secure.phabricator.com/D12060

Details

Committed
epriestley <git@epriestley.com>Mar 13 2015, 19:30
Pushed
aubortJan 31 2017, 17:16
Parents
rPH5339b22751da: Conpherence - another tweak to global upload flickering
Branches
Unknown
Tags
Unknown

Event Timeline

epriestley <git@epriestley.com> committed rPH4aed453b06e4: Add a chunking storage engine for files (authored by epriestley <git@epriestley.com>).Mar 13 2015, 19:30