Homec4science

filedownloadutils: utility for file retrieval

Authored by Jan Aage Lavik <jan.age.lavik@cern.ch> on Sep 4 2013, 09:09.

Description

filedownloadutils: utility for file retrieval

  • Adds a new module named filedownloadutils in miscutils that will bring together a set of functions used for both local and external retrieval (or download) of files, as well as some helper functions when dealing with files.

    Many of the functions have been adapted from or simply taken from the BibDocFile module. The aim is to move the usage of these functions away from BibDocFile and into this new module.

    The main API consists of the download_url() function which will download or copy files from external URLs or locally, respectively. (closes #1076)
  • download_external_url() has been amended to allow clients to more robustly download files given an URL. For example, adding retries and allows clients to specify the expected file-extension to check that the file retrived has the correct format.
  • Adds unit-tests for the new utility.
  • Changes the way PDFs and tarballs are downloaded in plotextractor_getter in order to use the new API available in filedownloadutils. Also checks if the right type is downloaded for the tarball and pdf files.
  • Adds a unit-test for the plotextractor module in using the new utilty.
  • Changes pdfchecker to download files to using filedownloadutils.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>

Details

Event Timeline

Samuele Kaplun <samuele.kaplun@cern.ch> committed R3600:e33c685b3c70: filedownloadutils: utility for file retrieval (authored by Jan Aage Lavik <jan.age.lavik@cern.ch>).Dec 18 2013, 16:21