Diffusion warcbase (master) History

R1473/src/main/java/org/warcbase/pig
masterd58afca206fc0bfc4986376…

Aug 16 2014

Refactoring packge org.warcbase.demo; added Jwat prefix to Hadoop InputFormats…
d58afca206fc
·lintool on Aug 16 2014, 18:23.

Aug 14 2014

Minor tweaks.
b6431bed6f8b
·lintool on Aug 14 2014, 18:09.

Aug 12 2014

Commented out LibmagicJnaWrapper functionality because the jar isn't generally…
a14cda2217fa
·lintool on Aug 12 2014, 23:15.

Dec 19 2013

Added a null pointer check and a more Pig friendly return value from the UDFs
db4ebe9f4826
·pmd on Dec 19 2013, 15:48.
Refactored the configuration of the magic lib into the Pig script.
ba201c27e210
·pmd on Dec 19 2013, 15:23.
Removed warnings
5d312a3f80ec
·pmd on Dec 19 2013, 14:29.
Refactored the DetectMimeType into two seperate methods: one for each detection…
b3fdd488fdcb
·pmd on Dec 19 2013, 14:22.

Dec 11 2013

Added a TODO comment
d6c0cec7efb4
·pmd on Dec 11 2013, 08:45.

Dec 9 2013

Add identification engine as a parameter to the DetectMimeType UDF
259f0057f400
·pmd on Dec 9 2013, 15:15.
Enable the ArcLoader to load all types of files
b69563d53c83
·pmd on Dec 9 2013, 15:10.

Dec 4 2013

First version of a magic lib UDF
1a209cf5b04a
·pmd on Dec 4 2013, 14:43.

Dec 3 2013

use tika for mime type detection
ccb9ea44f90a
·cneud on Dec 3 2013, 15:23.

Dec 2 2013

use tika for language detection
fbf902fcd066
·cneud on Dec 2 2013, 18:05.
Added WarcLoader
eb848477370a
·lintool on Dec 2 2013, 16:32.
Added ExtractRawText UDF, tweaked ExtractLinks.
1c57edb9e225
·lintool on Dec 2 2013, 15:40.
Loader now materializes actual text, added ExtractLinks UDF.
1fd881ddf836
·lintool on Dec 2 2013, 13:26.
Added simple Pig Loader for Arc files, returns (url, time, mime) currently.
02317746e1b2
·lintool on Dec 2 2013, 11:58.