Diffusion invenio-infoscience (maint-2.0) History

R3600/modules/bibindex/lib/bibindex_engine.py
maint-2.04f37b87ffdd3e8a5f497c91…

Sep 12 2011

BibIndex: fix double-stemming upon indexing
4f37b87ffdd3
·Samuele Kaplun / Tibor Simko on Sep 12 2011, 21:34.

May 17 2010

BibIndex: index files with unusual extensions
6c9e98b941e2
·Samuele Kaplun / Tibor Simko on May 17 2010, 10:25.

Jun 17 2008

Catch exceptions of lower_index_term() and continue in spite of UTF-8 problems…
6c4510b06391
·Tibor Simko on Jun 17 2008, 16:34.

Jun 4 2008

Cosmetics of output messages for the stemming info.
c3b76be701ba
·Tibor Simko on Jun 4 2008, 19:46.

May 29 2008

Lower phrase-to-be-indexed in a Unicode-friendly manner.
62b87aa91c88
·Tibor Simko on May 29 2008, 23:54.

May 16 2008

First implementation of the intelligent 'journal' index. This is an index…
4dd00adc2995
·Tibor Simko on May 16 2008, 14:53.

May 14 2008

Better closed mkstemp-created files.
e12a89b824b5
·Samuele Kaplun on May 14 2008, 17:49.
Fixed mkstemp calls that leaved open files.
6d17ad8b673e
·Samuele Kaplun on May 14 2008, 16:42.

May 13 2008

Added debugging code WRT encoding, just as for the bibrank word indexer. Now…
a3b4bad725ec
·Tibor Simko on May 13 2008, 17:24.

May 12 2008

Trying to fixing select/update vs select/insert bug.
ffb2b3a00592
·Samuele Kaplun on May 12 2008, 22:00.

May 11 2008

get_word_tables now returns a list of tuples (index_id, index_tags) instead of…
8e489b9e2ffd
·Samuele Kaplun on May 11 2008, 17:35.

Apr 3 2008

Deployed task_sleep_now_if_required to bibindex.
02c67d5a4a19
·Samuele Kaplun on Apr 3 2008, 16:34.

Apr 1 2008

New cleaner signal handling. By using task_sleep_now_if_required(can_stop_too)…
cc5418ab136d
·Samuele Kaplun on Apr 1 2008, 18:59.
Fixed bibindex --reindex to work when no indexes are specified (it reindexes…
3d880f427bbe
·Samuele Kaplun on Apr 1 2008, 10:37.

Mar 31 2008

New bibsched_low_level_task_submission for enqueing tasks via API. (changes a…
0fb4ae583079
·Samuele Kaplun on Mar 31 2008, 19:08.

Mar 26 2008

Removed last spourious printed newline.
c47f7c426b8d
·Samuele Kaplun on Mar 26 2008, 17:45.
Clened fulltext indexing logging: if extraction method does not exist for a…
91c19744a9d8
·Samuele Kaplun on Mar 26 2008, 17:44.
Removed notImplemented get_words_from_local_fulltext: it will be anyway…
caec8e4f6e4e
·Samuele Kaplun on Mar 26 2008, 15:01.

Mar 25 2008

Fixed 175 cases of bad code indentation throughout the codebase. (Please set up…
67226ac65ffd
·Tibor Simko on Mar 25 2008, 21:53.

Mar 12 2008

Updated codebase to use CFG_SITE_URL instead of weburl everywhere. Updated…
2b99705c3821
·Tibor Simko on Mar 12 2008, 17:47.
Hopefully fixed "[must be sys.stdout or sys.stderr] spurious messages.
a0235b21a588
·Samuele Kaplun on Mar 12 2008, 14:32.

Mar 11 2008

Fixed verbosity of some necessary output messages.
8b53ea007afc
·Tibor Simko on Mar 11 2008, 22:37.
Different small fixes after pylint.
b6d3dcd69474
·Samuele Kaplun on Mar 11 2008, 17:48.

Mar 10 2008

Updated conf files. Replaced old style variable names (e.g. WEBURL) with new…
6dc20e3ca587
·Tibor Simko on Mar 10 2008, 20:32.

Feb 28 2008

Fixed bug about last_update_time of an index no more updated.
0035392b57e0
·Samuele Kaplun on Feb 28 2008, 10:25.

Feb 5 2008

Disabled idxPHRASE indexing until websearch will exploit it.
abe7f722fb22
·Samuele Kaplun on Feb 5 2008, 14:53.

Feb 4 2008

Updated copyright years.
eceb995a1b50
·Tibor Simko on Feb 4 2008, 18:38.

Jan 18 2008

Made washing of terms configurable. idxPHRASEs should infact not wash terms (i.
c555c383303e
·Samuele Kaplun on Jan 18 2008, 11:41.

Jan 15 2008

Implemented idxPHRASE indexes that should help when searching for exact phrases…
22223b2f9e3d
·Samuele Kaplun on Jan 15 2008, 18:14.

Jan 10 2008

Moved stemming support from static index config variable structure to column…
0affefbcb29c
·Samuele Kaplun on Jan 10 2008, 17:47.

Dec 7 2007

Fixed stemming applyed to all the indexes, by adding…
229092586df1
·Samuele Kaplun on Dec 7 2007, 18:41.

Dec 4 2007

Ported to bibdocfile, implemented -R,--reindex feature, directly reading of…
17c7f225d6e1
·Samuele Kaplun on Dec 4 2007, 10:36.

Nov 6 2007

Get rid of dbquery's escape_string() and cStringIO factory, escaping SQL…
1115e53b3879
·Tibor Simko on Nov 6 2007, 19:48.
Fixed truncation of 50+ bytes long index terms, respecting strictly UTF-8…
e72ca5f81610
·Tibor Simko on Nov 6 2007, 11:50.

Oct 18 2007

Fixed syntax error in the fulltext indexing branch.
3bbb7e469769
·Tibor Simko on Oct 18 2007, 14:56.

Oct 16 2007

Make HTM and HTML files also fulltext-indexable. This is needed for articles…
6048dd37581e
·Tibor Simko on Oct 16 2007, 16:39.

Aug 9 2007

New intbitset with universe support.
cfdd1ea34eed
·Samuele Kaplun on Aug 9 2007, 16:02.

Aug 7 2007

Correctly supporting -V flag in bibtasks.
602cc38d3271
·Samuele Kaplun on Aug 7 2007, 16:29.

Aug 3 2007

Friday commit: removed dependency to Numeric. Migrated HitSet & Co. to…
7a3572e4228f
·Samuele Kaplun on Aug 3 2007, 19:20.

Jun 8 2007

Refactored bibupload, bibindex and bibreformat tasks to new bibtask cleaned…
7b2f8868dcd2
·Samuele Kaplun on Jun 8 2007, 13:31.

Jun 6 2007

Ported to new BibTask class
c9303324e339
·Samuele Kaplun on Jun 6 2007, 19:53.

May 22 2007

Moved from the obsolete sre module to re.
9657e2ace771
·Samuele Kaplun on May 22 2007, 11:54.

May 15 2007

Clarified emergency output messages in case unrepairable errors were found…
db71f1e58654
·Tibor Simko on May 15 2007, 12:39.

Mar 27 2007

Handle UnicodeEncodeError related problems when printing messages in…
f5dd8d538885
·Tibor Simko on Mar 27 2007, 17:50.

Feb 14 2007

Updated copyright years (2007).
df75ba0b1feb
·Tibor Simko on Feb 14 2007, 19:31.

Nov 1 2006

When authentifying user-supplied email or nickname, use two SQL queries rather…
7cbd91d4ea9b
·Tibor Simko on Nov 1 2006, 15:38.

Sep 20 2006

Fixed config wildcard import everywhere. Fixed some randomly spotted import…
af4a85dac604
·Tibor Simko on Sep 20 2006, 19:27.

Sep 18 2006

Harmonized CFG_PATH_* and CFG_WEBSEARCH_* config variable names.
ec761f4dff4b
·Tibor Simko on Sep 18 2006, 12:15.

Sep 15 2006

Renamed cfg_bibindex_* config variables to follow the uppercase model.
770420bc2854
·Tibor Simko on Sep 15 2006, 11:57.

Sep 13 2006

Fixed the kwalitee problem of re-exposing some of the global config variables…
2311f7764a03
·Tibor Simko on Sep 13 2006, 16:55.
Fixed the kwalitee problem of re-exposure of some of the global config…
0674ca3c5a30
·Tibor Simko on Sep 13 2006, 16:30.

Aug 29 2006

Used urllib2 instead of urllib, mkstemp() instead of mktemp(), and removed some…
dbc8be0c4e5d
·Tibor Simko on Aug 29 2006, 21:38.
Updated every BibSched task to follow the BibTaskEx example of handling task…
536257883e68
·Tibor Simko on Aug 29 2006, 11:26.

Aug 9 2006

Fixed a bug relating to the indexation of fulltexts: When the contents of a PDF…
31f485a1e848
·Nicholas Robinson on Aug 9 2006, 15:26.

Jun 29 2006

BibSched task authentication now accepts nicknames as usernames too. Modified…
5aac7386e47f
·Tibor Simko on Jun 29 2006, 09:51.

Jun 20 2006

Changed database access code in order not to depend on MySQLdb but rather on…
1fe7d8836f40
·Tibor Simko on Jun 20 2006, 11:19.

May 4 2006

Implemented name change CDSware to CDS Invenio. Also, introduced new configure…
18b100fe8575
·Tibor Simko on May 4 2006, 16:34.

May 2 2006

Updated copyright years.
d5b36a574e3c
·Tibor Simko on May 2 2006, 14:54.

Dec 20 2005

Python imports are now done in an absolute way (from cdsware.foo import bar)…
6b1700acdd20
·Tibor Simko on Dec 20 2005, 16:39.

Dec 5 2005

Quick fix in order to be able to fulltext-index Indico URLs that do not contain…
561825a6b79f
·Tibor Simko on Dec 5 2005, 14:49.

May 12 2005

Getting rid of WML.
d41001de533e
·Tibor Simko on May 12 2005, 18:09.

Mar 29 2005

Fixed add_recIDs_by_date() in case user wants to index records based on…
e694f41220b7
·Tibor Simko on Mar 29 2005, 18:10.

Jan 6 2005

Updated copyright years.
84b18f1b27c5
·Tibor Simko on Jan 6 2005, 17:41.
Fixed a bug when calculating index id from tablename for indexes greater than…
b2dc486ac9ea
·Tibor Simko on Jan 6 2005, 10:33.

Dec 16 2004

Changes due to the move of cfg_* variables into the general config file.
89932a344b16
·Tibor Simko on Dec 16 2004, 17:30.

Dec 15 2004

Changed logic when applying stemming (first) and stopword check (second).
5689d701af1c
·Tibor Simko on Dec 15 2004, 19:45.
Fixed regexp for HTML removal.
25b3ea26e6b4
·Tibor Simko on Dec 15 2004, 17:03.
Precompile all major regexps to speed up the execution. (About 10% faster…
5f3193cec995
·Tibor Simko on Dec 15 2004, 12:10.

Nov 19 2004

Added stemming/stopword functionality. Check /lib/bibindex_engine_config.py for…
ceb6867786d0
·Trond Aksel Myklebust on Nov 19 2004, 12:20.

Sep 27 2004

Activate our fancy non-interactive URL opener for fail-safe fulltext-indexing…
bc726da81963
·Tibor Simko on Sep 27 2004, 09:52.

Sep 23 2004

Modified error messages when not authorized
220b909f90a4
·Trond Aksel Myklebust on Sep 23 2004, 14:15.

Sep 8 2004

Eliminated excessive write_message() function. Reduced serialize/deserialize…
e8185fb0dfaf
·Tibor Simko on Sep 8 2004, 17:30.

Jul 7 2004

Added option to fulltext-index local files only, i.e. remote URLs are skipped…
87ea8a9e07df
·Tibor Simko on Jul 7 2004, 11:27.

Jun 15 2004

BibIndex monolithic bin file split into a small bin file and large lib engine…
5df73cfb8570
·Tibor Simko on Jun 15 2004, 18:22.