Diffusion invenio-infoscience (master) History

R3600/modules/bibupload/lib/bibupload.py
master171b1ce5988ce20f63be5e0…

Feb 9 2009

BibUpload: fixed task status updating
171b1ce5988c
·Samuele Kaplun / Tibor Simko on Feb 9 2009, 18:15.

Nov 24 2008

BibUpload FIX-MARC not to erase comments/description/icons.
ac5b7d39980d
·Samuele Kaplun / Tibor Simko on Nov 24 2008, 14:40.

Nov 20 2008

Fixed decompose_bibdocfile_url to support CFG_SITE_URL/record/123 URLs
05d992af1e99
·Samuele Kaplun / Tibor Simko on Nov 20 2008, 15:39.

Nov 5 2008

Merged commit 'sam/get_record'
65e03a4d258e
·Tibor Simko on Nov 5 2008, 15:58.
Improved getfile.py old URL support.
eb46c29edd51
·Samuele Kaplun on Nov 5 2008, 11:47.

Nov 4 2008

New search_engine.get_record function.
9dc22287761b
·Samuele Kaplun on Nov 4 2008, 17:58.
DEMOART SBI to exploit FFT in place of Move_Files_to_Storage.
792b897df0a8
·Samuele Kaplun on Nov 4 2008, 15:57.

Nov 3 2008

Amended and merged commit 'sam/bibupload-protect'
3a15587a90d2
·Tibor Simko on Nov 3 2008, 14:43.

Oct 31 2008

New support for protected tags in bibupload correct.
f231186fbf70
·Samuele Kaplun on Oct 31 2008, 16:27.

Oct 27 2008

Removed $Id$ from file preamble comments.
51bfbcb9fa42
·Tibor Simko on Oct 27 2008, 22:39.

Oct 13 2008

Cleaned implementation of find_records_from_extoaiid WRT extoaisrc.
c3bf13a090f4
·Samuele Kaplun on Oct 13 2008, 14:43.

Sep 10 2008

Added 8564 bibdoc synchronization trigger based on existence of bibdocs.
a0987f79df48
·Samuele Kaplun on Sep 10 2008, 10:57.

Sep 8 2008

Changed prints into write_message(..., verbose=3)
9be906879fa9
·Samuele Kaplun on Sep 8 2008, 11:38.

Aug 29 2008

Holding pen prototype
f34700ca03ee
·Piotr Praczyk on Aug 29 2008, 17:32.

Aug 21 2008

Added logging when harvesting record and later (when inserting into teh…
2bd13ab78c6e
·Piotr Praczyk on Aug 21 2008, 11:52.

Aug 20 2008

Added missing import.
8bad55a7fd5d
·Samuele Kaplun on Aug 20 2008, 14:54.
Improved checking of OAI PROVENANCE subfield (if not specified it behaves like…
8d9a7d7d1219
·Samuele Kaplun on Aug 20 2008, 14:47.

Aug 19 2008

New documentation sections Menu at the top of pages Bug fixes: - Little…
0d119a739a0f
·Piotr Praczyk on Aug 19 2008, 17:31.

Aug 18 2008

New page for displaying daily harvesting logs - displaying the results in…
991b5b87b3fa
·Piotr Praczyk on Aug 18 2008, 19:03.
Added support for CFG_BIBUPLOAD_EXTERNAL_OAIID_PROVENANCE_TAG (i.e. support for…
2fb7275849e8
·Samuele Kaplun on Aug 18 2008, 16:20.

Aug 15 2008

Putting back the original record (if it was deleted as part of the bibupload)…
15cc92778b06
·Samuele Kaplun on Aug 15 2008, 19:03.
Little changes in SQL tables connected with logging
349d2e584685
·Piotr Praczyk on Aug 15 2008, 13:59.

Aug 11 2008

Removed logging when oai source identifier is not provided
1a8b0e6a93f6
·Piotr Praczyk on Aug 11 2008, 16:16.

Aug 7 2008

Logging of harvested records ( new parameter for bibupload indicating the id of…
ea500c914561
·Piotr Praczyk on Aug 7 2008, 17:32.

Jul 30 2008

BibUpload not to fails when FFT is empty.
8a8715496f42
·Samuele Kaplun on Jul 30 2008, 17:44.

Jul 28 2008

New bibupload_pending_recids to automatically discover which recids are…
721c8971094e
·Samuele Kaplun on Jul 28 2008, 14:47.

Jun 30 2008

8564s MARC will be synchronized after bibdocfile tables only if FFT is used or…
8ae591036592
·Samuele Kaplun on Jun 30 2008, 16:39.
Added checks for md5sum in ./configure scripts. bibdoc.touch() to not touch…
1c399a45fe93
·Samuele Kaplun on Jun 30 2008, 12:22.

Jun 26 2008

Plugged 8564 synchronization only when 8564 or FFT exists.
036203538091
·Samuele Kaplun on Jun 26 2008, 19:48.
Calling synchronize 8564 for each upload. Fixed FMT regression test that used a…
45dcbab513ac
·Samuele Kaplun on Jun 26 2008, 19:25.

Jun 23 2008

BibDocFile+FFT to correctly support URL encoding (for spaces etc.)
170f2ea119d9
·Samuele Kaplun on Jun 23 2008, 14:28.

Jun 20 2008

New FIX-MARC (+ FIX as become FIX-ALL) directive to bibdocfilecli and FFT, used…
c7ca1047c5ad
·Samuele Kaplun on Jun 20 2008, 12:32.

Jun 19 2008

Completed support for DELETE-FILE FFT directive.
28752c24a040
·Samuele Kaplun on Jun 19 2008, 08:53.

Jun 18 2008

When using FFT to rename a docname, take care of renaming the corresponding…
01710b6bb530
·Samuele Kaplun on Jun 18 2008, 18:05.

May 15 2008

Fixed fix call.
1e74a99408ee
·Samuele Kaplun on May 15 2008, 10:16.

May 13 2008

Changed socket default timeout from 10s to 40s. This could maybe help in…
2c9671cc8f33
·Samuele Kaplun on May 13 2008, 23:21.

Apr 16 2008

When FFT revising, if restriction/comment/description/icon is not specified, it…
b4895deb2d5b
·Samuele Kaplun on Apr 16 2008, 13:46.

Apr 15 2008

Mega commit that includes: Added support for keeping comments and descriptions…
9840677d2f4a
·Samuele Kaplun on Apr 15 2008, 16:02.
Corrected bad indentation previously committed due to misconfiguration of…
e7f7419ef6b2
·Samuele Kaplun on Apr 15 2008, 11:09.
Deployed fix_argv_paths to fix relative path issues on the command line.
8cc3b43a8385
·Samuele Kaplun on Apr 15 2008, 10:39.

Apr 7 2008

Moved some key functions out to bibdocfile. Implemented failing recid on first…
17044e7b5b7b
·Samuele Kaplun on Apr 7 2008, 17:39.

Apr 3 2008

Added timeout to socket operations (e.g. urlopen) in order not to possibly hang…
614dabc993b4
·Samuele Kaplun on Apr 3 2008, 16:56.
Deployed task_sleep_now_if_required in bibupload.
c7a2cb2becc8
·Samuele Kaplun on Apr 3 2008, 13:45.

Mar 31 2008

Fixed _options & _task_params not declared.
bb8b59146af7
·Samuele Kaplun on Mar 31 2008, 21:15.
New bibsched_low_level_task_submission for enqueing tasks via API. (changes a…
0fb4ae583079
·Samuele Kaplun on Mar 31 2008, 19:08.

Mar 27 2008

Moved globally interesting CFG variables from local foo_config.py files into…
b00299effbda
·Tibor Simko on Mar 27 2008, 00:26.

Mar 20 2008

Strengthened FFT error checking. Improved regression tests.
91d7d8661bc9
·Samuele Kaplun on Mar 20 2008, 12:30.
Moved (-s, --stage) option to (-S --stage) because of clashing with (-s…
f98123ae167b
·Samuele Kaplun on Mar 20 2008, 10:23.

Mar 14 2008

Restored original behaviour of storing current time as last_update time when…
86a7c9bdabe1
·Samuele Kaplun on Mar 14 2008, 19:21.
Completed bibupload support for $d in FMT, plugged check in a test case.
c118ff633251
·Samuele Kaplun on Mar 14 2008, 15:37.
Added d field to FMT tag to provide modification_date to bibupload.
a16b57aed69d
·Samuele Kaplun on Mar 14 2008, 15:10.

Mar 12 2008

Updated codebase to use CFG_SITE_URL instead of weburl everywhere. Updated…
2b99705c3821
·Tibor Simko on Mar 12 2008, 17:47.

Mar 11 2008

Phased out legacy variables 'accessurl', 'urlpath', 'images', 'htdocsurl' used…
fe3f00b1b042
·Tibor Simko on Mar 11 2008, 19:33.

Mar 10 2008

Updated conf files. Replaced old style variable names (e.g. WEBURL) with new…
6dc20e3ca587
·Tibor Simko on Mar 10 2008, 20:32.

Mar 7 2008

Fixed lowering of format bug. Now a file with extension .ps.Z can be happily…
ceb60d4f9bd2
·Samuele Kaplun on Mar 7 2008, 13:36.
Fixed bug that prevented to correctly append documents via FFT.
f5d952728470
·Samuele Kaplun on Mar 7 2008, 11:08.

Feb 14 2008

Formalized __str__() method of BibRecDocs, BibDoc, BibDocFile. Fixed sorting of…
6538e4e25bad
·Samuele Kaplun on Feb 14 2008, 16:01.

Feb 13 2008

Changed behaviour of Bibupload WRT FFT when --replace is used. It now behaves…
013cf6ac8e3f
·Samuele Kaplun on Feb 13 2008, 19:21.

Feb 4 2008

Updated copyright years.
eceb995a1b50
·Tibor Simko on Feb 4 2008, 18:38.

Jan 17 2008

Fixed archive_marcxml_for_history() for cases when the upload is being done not…
ecdbd413672d
·Tibor Simko on Jan 17 2008, 12:28.

Jan 9 2008

Activated writing of MARCXML versions to hstRECORD.
b544aae8d91b
·Tibor Simko on Jan 9 2008, 18:17.

Nov 30 2007

1) Bibupload FFT to support bibdocfile, 2) bibupload FFT to support icon…
3d8dbbfdba21
·Samuele Kaplun on Nov 30 2007, 17:47.

Nov 13 2007

Small bug-fixes for FFT and added verbose mode.
60b52cc20f11
·Samuele Kaplun on Nov 13 2007, 18:49.

Nov 12 2007

Trying to fix some bugs with FFTs and added support for wiping out all linked…
7da63e726510
·Samuele Kaplun on Nov 12 2007, 18:06.

Nov 8 2007

Deleted unused find_record_bibfmt(). Removed unused MySQLdb import. The file…
8d8837afe404
·Tibor Simko on Nov 8 2007, 14:58.

Nov 7 2007

More regression test (some to be fixed) and support for propagatin restrictions…
8f945cded480
·Samuele Kaplun on Nov 7 2007, 19:09.

Nov 6 2007

BibUpload-FFT now in a working state. Previous regression tests works, Demo…
689a847ba674
·Samuele Kaplun on Nov 6 2007, 17:57.

Nov 2 2007

Traditional Big Friday Commit (tm) :-) New FFT BibUpload implementation that…
803e1afe5d8d
·Samuele Kaplun on Nov 2 2007, 19:17.

Aug 7 2007

Correctly supporting -V flag in bibtasks.
602cc38d3271
·Samuele Kaplun on Aug 7 2007, 16:29.

Jun 8 2007

Refactored bibupload, bibindex and bibreformat tasks to new bibtask cleaned…
7b2f8868dcd2
·Samuele Kaplun on Jun 8 2007, 13:31.

Jun 5 2007

Added support for "strong tags" that can resist metadata replace mode ordered…
665a2b881a74
·Tibor Simko on Jun 5 2007, 15:11.

May 23 2007

Change in Fulltext file transfer (FFT): the files are copied instead of moved
6c3a25c4573f
·jkrause on May 23 2007, 15:14.

May 15 2007

Many changes, including: * Implementation of a firewall like role definition…
ceaf067fa2cd
·Samuele Kaplun on May 15 2007, 19:35.

Apr 2 2007

Added support for external OAI ID tag matching.
6212408be663
·Tibor Simko on Apr 2 2007, 00:48.

Mar 30 2007

First provisional implementation of the fulltext file transfer mode by Jan.
1c1bd3cc4ace
·Tibor Simko on Mar 30 2007, 16:16.

Feb 14 2007

Updated copyright years (2007).
df75ba0b1feb
·Tibor Simko on Feb 14 2007, 19:31.

Feb 6 2007

When inserting into bibxxx tables, during the look up of already existing…
04329f860a9e
·Tibor Simko on Feb 6 2007, 19:04.

Jan 26 2007

Use bibformat.format_record() rather than search_engine.print_record() to…
0c7f6ad29216
·Tibor Simko on Jan 26 2007, 10:20.

Jan 25 2007

When matching MARCXML input file records against the database, if a match was…
eb46cb3d4347
·Tibor Simko on Jan 25 2007, 19:58.

Dec 21 2006

Write a short information on the input file and the selected input mode into…
29281729a554
·Tibor Simko on Dec 21 2006, 15:31.
Fixed bad indentation.
85316c5e8c11
·Tibor Simko on Dec 21 2006, 09:52.

Dec 20 2006

When some records could not have been uploaded due to errors such as non…
32b3eb7381d7
·Tibor Simko on Dec 20 2006, 15:03.

Dec 18 2006

Fixed correct mode so that only the field instances matching the complete…
19c6e1c50d9a
·Tibor Simko on Dec 18 2006, 23:30.
Whenever a blank space indicator (or an old-style "empty" indicator) is…
c12e9e381d71
·Tibor Simko on Dec 18 2006, 14:59.

Dec 13 2006

Fixed "reference" upload mode to populate bibxxx tables too. bibupload() now…
0243ff56930f
·Tibor Simko on Dec 13 2006, 16:44.

Dec 7 2006

Changed bibupload behaviour so that it would not modify the "mode" option from…
e790dadaddfa
·Tibor Simko on Dec 7 2006, 16:42.
Do not use table aliases in "DELETE x FROM ... LEFT JOIN" SQL statements…
b87c8d0d9798
·Tibor Simko on Dec 7 2006, 14:23.
Save option insert/replace/both (-r -i) before the upload of each record in…
d99743b653f6
·Jerome Caffaro on Dec 7 2006, 12:32.

Dec 5 2006

Fixed behaviour of CLI options; no options have values except "stage"…
1743992f0cfb
·Tibor Simko on Dec 5 2006, 19:38.

Nov 30 2006

Added support for intelligent upload of records having external SYSNO or OAI ID…
8c20155e33a8
·Tibor Simko on Nov 30 2006, 12:15.

Nov 28 2006

When comparing to None, do not use "== None" or "!= None", but rather "is None"…
5dda83af3d76
·Tibor Simko on Nov 28 2006, 18:46.

Nov 1 2006

When authentifying user-supplied email or nickname, use two SQL queries rather…
7cbd91d4ea9b
·Tibor Simko on Nov 1 2006, 15:38.

Oct 9 2006

Print task stats in the usual output message format.
ff9532b6fbff
·Tibor Simko on Oct 9 2006, 17:27.

Oct 3 2006

(1) Put 001 in front of the newly inserted records, fixing a particular…
b85162d75150
·Tibor Simko on Oct 3 2006, 23:16.

Sep 13 2006

Improved kwalitee of the code.
b91637b2c33b
·Tibor Simko on Sep 13 2006, 14:46.

Aug 29 2006

Fixed the behaviour of the FMT tag, solving the problem occurred during…
c01e7675eac4
·Tibor Simko on Aug 29 2006, 19:39.
Updated every BibSched task to follow the BibTaskEx example of handling task…
536257883e68
·Tibor Simko on Aug 29 2006, 11:26.

Aug 28 2006

Fail properly when the MARCXML input file could not have been parsed.
b72219920258
·Tibor Simko on Aug 28 2006, 17:50.

Aug 24 2006

Update from Eric. bibupload() now returns a tuple (error_code, recID) of the…
99cbaed67840
·Tibor Simko on Aug 24 2006, 19:15.
Committed new version from Eric, and changed behaviour of parse_command() in…
8968b04268ee
·Tibor Simko on Aug 24 2006, 12:04.