Page MenuHomec4science

Large File Storage (LFS) in GIT
Updated 40 Days AgoPublic

Filesize on c4science is limited to 100MB and push size to 250MB. For special cases we support GIT LFS for large file storage outside of the git index. Only a reference to the file is stored in git tracking the changes, the file itself is stored in phabricator's storage.

WARNING: Do not store huge files (>1GB) in LFS as we have limited capacity, for data you should use your local university storage service.

Installation

Usage

  • In an existing git repository, initialize lfs and start tracking files,
git lfs install
git lfs track bigfile.bin
git lfs track *.png
# etc
  • Add the file to git, commit and push
git add bigfile.bin
git add myimage.png
git add .gitattributes
git commit -m "Added a bigfile using LFS"
git push origin master
  • You can also track a pattern of files, so every new file matching this pattern will be added using lfs:
git lfs track *.png
  • Voila!

Convert already commited files to Git LFS

WARNING: This will rewrite your history and will then impact all people working on your repository
find -type f -exec ls -lh {} \; | awk '{print $5,$9}' | sort -h
  • Go to your git repository directory and use bfg with the extensions you found
java -jar bfg.jar --convert-to-git-lfs '*.{wav,pdf,avi}'
  • Inspect your history to see the changes and clean dirty data to complete the procedure
git reflog expire --expire=now --all && git gc --prune=now --aggressive
  • Activate Dangereous Changes from the repository configuration: ActionsManage RepositoryBasicsActionsAllow Dangerous Changes

  • Push the repository back to c4science
IMPORTANT: it's dangerous and will destroy your old history, have a local backup of this!
git push --force
  • Deactivate Dangerous Changes (same as before)
Last Author
aubort
Last Edited
Jan 7 2019, 16:01

Event Timeline

aubort created this document.Apr 12 2016, 14:29
aubort edited the content of this document. (Show Details)
aubort edited the content of this document. (Show Details)Apr 12 2016, 14:39
aubort edited the content of this document. (Show Details)Apr 14 2016, 11:03
aubort edited the content of this document. (Show Details)Apr 14 2016, 13:52
aubort edited the content of this document. (Show Details)May 17 2016, 14:07
aubort edited the content of this document. (Show Details)May 20 2016, 11:25
aubort changed the visibility from "Public (No Login Required)" to "All Users".Jun 5 2016, 20:53
aubort changed the visibility from "All Users" to "Public (No Login Required)".Jun 7 2016, 12:46
aubort changed the title from Large File Storage in GIT to Large File Storage (LFS) in GIT.Jul 7 2016, 09:25
aubort edited the content of this document. (Show Details)
aubort changed the edit policy from "c4science (Project)" to "Restricted Project (Project)".Dec 16 2016, 16:50
aubort edited the content of this document. (Show Details)Oct 31 2017, 10:37
aubort edited the content of this document. (Show Details)Oct 31 2017, 11:06
aubort edited the content of this document. (Show Details)Nov 23 2017, 10:56
aubort edited the content of this document. (Show Details)
aubort edited the content of this document. (Show Details)Nov 23 2017, 11:01
aubort edited the content of this document. (Show Details)
aubort edited the content of this document. (Show Details)Jul 27 2018, 14:07
aubort added a project: c4science.
aubort edited the content of this document. (Show Details)Aug 2 2018, 12:27
aubort edited the content of this document. (Show Details)
aubort edited the content of this document. (Show Details)
aubort edited the content of this document. (Show Details)Jan 7 2019, 16:01