Diffusion warcbase (master) History

R1473/src/main/java/org/warcbase/pig
masterf5d8edf5050628d5b71f48d…

Nov 24 2015

Killed all the Pig stuff.
f5d8edf50506
·lintool on Nov 24 2015, 22:05.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Oct 21 2015

update ArcRecords interface
e5821091ecca
·Alice-Z on Oct 21 2015, 19:56.

Jun 30 2015

Added ExtractBoilerpipeText UDF
11dfe152d4cc
·Jeremy Wiebe on Jun 30 2015, 02:38.

Jun 9 2015

testing fork
51abf7e86696
·ianmilligan1 on Jun 9 2015, 10:20.

Jun 7 2015

Added schema metadata in loader.
caf87fbecf45
·lintool on Jun 7 2015, 16:37.
better trapping of errors.
1f9d853f38a2
·lintool on Jun 7 2015, 16:31.

Jun 6 2015

Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

May 26 2015

Added Stanford NER UDF
1c013a7b064f
·Jeremy Wiebe on May 26 2015, 18:41.

May 24 2015

Cleanup.
eec01a41b18e
·lintool on May 24 2015, 20:31.
Wraps a try block around everything to catch all errors.
1c9b25027e73
·lintool on May 24 2015, 20:27.
Merge branch 'master' into extract-pdf-udf
9acc8ca74c0a
·lintool on May 24 2015, 19:37.

May 13 2015

tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 6 2015

Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.
UDF for extracting top-level domain from URL.
fb95f3ae562e
·lintool on May 6 2015, 01:21.

Dec 18 2014

Reformatting.
5c2540099370
·lintool on Dec 18 2014, 14:20.

Dec 9 2014

ExtractTextFromPDFs updated
c49c58f76ed3
·rwolniak on Dec 9 2014, 00:38.
Updated PIG Tika Parser with new code that should work. Awaiting test on Hadoop
cea1f0cae974
·rwolniak on Dec 9 2014, 00:12.

Nov 12 2014

Still testing ExtractTextFromPDFs
8a3fed9342fc
·rwolniak on Nov 12 2014, 17:07.

Nov 10 2014

Extract text from PDF UDF. Currently still debugging
10b92d78f6ed
·rwolniak on Nov 10 2014, 04:07.

Oct 31 2014

Minor refactoring.
b8915343f5c9
·lintool on Oct 31 2014, 18:28.

Oct 19 2014

Pig ArcLoader exports its own ResourceSchema.
0c52caed50cf
·lintool on Oct 19 2014, 02:02.
Fixed test cases.
3af097ea655d
·lintool on Oct 19 2014, 00:50.
Pig loader materializes the actual content.
4ceefef78cdc
·lintool on Oct 19 2014, 00:47.

Oct 18 2014

Refactored Pig loaded to use WAC API.
58df7da00880
·lintool on Oct 18 2014, 17:05.

Aug 16 2014

Refactoring packge org.warcbase.demo; added Jwat prefix to Hadoop InputFormats…
d58afca206fc
·lintool on Aug 16 2014, 18:23.

Aug 14 2014

Minor tweaks.
b6431bed6f8b
·lintool on Aug 14 2014, 18:09.

Aug 12 2014

Commented out LibmagicJnaWrapper functionality because the jar isn't generally…
a14cda2217fa
·lintool on Aug 12 2014, 23:15.

Dec 19 2013

Added a null pointer check and a more Pig friendly return value from the UDFs
db4ebe9f4826
·pmd on Dec 19 2013, 15:48.
Refactored the configuration of the magic lib into the Pig script.
ba201c27e210
·pmd on Dec 19 2013, 15:23.
Removed warnings
5d312a3f80ec
·pmd on Dec 19 2013, 14:29.
Refactored the DetectMimeType into two seperate methods: one for each detection…
b3fdd488fdcb
·pmd on Dec 19 2013, 14:22.

Dec 11 2013

Added a TODO comment
d6c0cec7efb4
·pmd on Dec 11 2013, 08:45.

Dec 9 2013

Add identification engine as a parameter to the DetectMimeType UDF
259f0057f400
·pmd on Dec 9 2013, 15:15.
Enable the ArcLoader to load all types of files
b69563d53c83
·pmd on Dec 9 2013, 15:10.

Dec 4 2013

First version of a magic lib UDF
1a209cf5b04a
·pmd on Dec 4 2013, 14:43.

Dec 3 2013

use tika for mime type detection
ccb9ea44f90a
·cneud on Dec 3 2013, 15:23.

Dec 2 2013

use tika for language detection
fbf902fcd066
·cneud on Dec 2 2013, 18:05.
Added WarcLoader
eb848477370a
·lintool on Dec 2 2013, 16:32.
Added ExtractRawText UDF, tweaked ExtractLinks.
1c57edb9e225
·lintool on Dec 2 2013, 15:40.
Loader now materializes actual text, added ExtractLinks UDF.
1fd881ddf836
·lintool on Dec 2 2013, 13:26.
Added simple Pig Loader for Arc files, returns (url, time, mime) currently.
02317746e1b2
·lintool on Dec 2 2013, 11:58.