BibIndex: centralisation of tokenizers
b0a11546ca41
Actions

Authored by Grzegorz Szpura <grzegorz.szpura@cern.ch> on Mar 28 2013, 15:54.

Description

BibIndex: centralisation of tokenizers

Introduces tokenizers for journal, year, fulltext and authorcount indexes. Merges old tokenizing functions for pairs, phrases and words into one 'BibIndexDefaultTokenizer'. Introduces empty tokenizer. Extends admin interface to support tokenizers. (closes #852)

Signed-off-by: Grzegorz Szpura <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>

Committed

Tibor Simko <tibor.simko@cern.ch>

Oct 3 2013, 23:08

Parents

R3600:213e752687e1: BibIndex: new regression tests

Branches

Unknown

Tags

Unknown

Tibor Simko <tibor.simko@cern.ch> committed R3600:b0a11546ca41: BibIndex: centralisation of tokenizers (authored by Grzegorz Szpura <grzegorz.szpura@cern.ch>).Oct 3 2013, 23:08

				Path
	M			modules/bibindex/doc/admin/bibindex-admin-guide.webdoc
	M			modules/bibindex/lib/Makefile.am
	M			modules/bibindex/lib/bibindex_engine.py
	M			modules/bibindex/lib/bibindex_engine_config.py
	M			modules/bibindex/lib/bibindex_engine_tokenizer.py
	M			modules/bibindex/lib/bibindex_engine_tokenizer_unit_tests.py
	M			modules/bibindex/lib/bibindex_engine_unit_tests.py
	A			modules/bibindex/lib/bibindex_engine_utils.py
	M			modules/bibindex/lib/bibindex_engine_washer.py
	M			modules/bibindex/lib/bibindex_regression_tests.py
	M			modules/bibindex/lib/bibindexadmin_regression_tests.py
	M			modules/bibindex/lib/bibindexadminlib.py
	M			modules/bibindex/web/admin/bibindexadmin.py
	M			modules/bibupload/lib/bibupload_regression_tests.py
	A			modules/miscutil/lib/upgrades/invenio_2013_03_28_idxINDEX_tokenizer.py
	M			modules/miscutil/sql/tabcreate.sql
	M			modules/miscutil/sql/tabfill.sql
	M			modules/websearch/lib/search_engine.py
	M			modules/websearch/lib/search_engine_query_parser.py