Diffusion warcbase (master) History

R1473/src/test/scala/org/warcbase/spark
master62b8c4e578000035385be98…

Jun 16 2016

Created warcbase-core module.
f972206db516
·lintool on Jun 16 2016, 15:43.

Apr 21 2016

fixed ArcTest
eb4c0e313a67
·ianmilligan1 on Apr 21 2016, 18:13.

Mar 29 2016

renaming ExtractTopLevelDomain to ExtractDomain
fc0d11495cf1
·ianmilligan1 on Mar 29 2016, 03:11.

Mar 27 2016

Add UDF for extracting stuff from tweets, Issue #210
40025b169e93
·lintool on Mar 27 2016, 18:05.
Added test cases.
b80c970f2e50
·lintool on Mar 27 2016, 18:04.
Added ExtractImageLinks test cases.
2da8f0abb983
·lintool on Mar 27 2016, 17:31.

Feb 18 2016

Added tests for generic ARC/WARC classes
f69c7900f12e
·Jeremy Wiebe on Feb 18 2016, 03:00.

Feb 16 2016

Added discardUrlPatterns
8a4c55019413
·Jeremy Wiebe on Feb 16 2016, 16:31.
keepUrlPatterns test
232397a26bf1
·Jeremy Wiebe on Feb 16 2016, 15:58.

Feb 13 2016

Merge branch 'format' for ExtractDate (#154) and TupleFormatter
a7b0e0b07682
·Jeremy Wiebe on Feb 13 2016, 23:01.
Use shapeless to flatten tuples of any arity
4b8f0fb7b482
·Jeremy Wiebe on Feb 13 2016, 06:48.

Dec 25 2015

Add RemovePrefixWWW method
1b6d83c4a7fe
·Alice-Z on Dec 25 2015, 18:11.
add method to filter date by component
4042cac0ad8b
·Alice-Z on Dec 25 2015, 17:49.
Cleanup and add comments
d1c3f5d49783
·Alice-Z on Dec 25 2015, 17:31.
Add Date utils to clarify date extraction
1ec6e05d7540
·Alice-Z on Dec 25 2015, 17:31.
Add Formatter to un-nest tuples and print in tab-delimited format
c773fd242ff5
·Alice-Z on Dec 25 2015, 17:28.

Dec 10 2015

Refactor Record API (#189)
2adce498927d
·Alice-Z on Dec 10 2015, 21:59.

Dec 3 2015

Fix WARecord getContentString
90616dd7b3bd
·Alice-Z on Dec 3 2015, 17:11.

Nov 25 2015

Revision API to be more descriptive
83d4d7232405
·Alice-Z on Nov 25 2015, 20:33.
Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.

Nov 23 2015

Tiny fix: remove unneeded part of test
2dde7b822a5c
·Alice-Z on Nov 23 2015, 02:08.
Port named entities extractor Pig script over to Spark as per issue #158
4a3d31b8babf
·Alice-Z on Nov 23 2015, 02:05.
ExtractEntities takes a classifier file path
f9422c23efae
·Alice-Z on Nov 23 2015, 02:00.
Revert to object version of NER3Classifier
99583f793bc5
·Alice-Z on Nov 23 2015, 01:56.

Nov 21 2015

Turn test off; classifier is too large to be included
533b4152d534
·Alice-Z on Nov 21 2015, 19:50.
Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.
Clean up, fix tests changed by new keepValidPages
14e521794754
·Alice-Z on Nov 21 2015, 16:33.

Nov 19 2015

Fix warcloader bug (issue #166)
758288fb2bd9
·Alice-Z on Nov 19 2015, 23:39.

Nov 12 2015

Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.
Clean up string formatting
e9a3965e2389
·Alice-Z on Nov 12 2015, 15:01.
Working extract entities with correct output string
9f7fa9f26b54
·Alice-Z on Nov 12 2015, 04:18.

Nov 11 2015

Refactor ExtractLinks to be called with the src url
c553ef0d853f
·Alice-Z on Nov 11 2015, 23:08.
tiny formatting change
de66d390f4ae
·Alice-Z on Nov 11 2015, 22:24.
Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.
Rename tests for JUnitTestRunner
e5668e3e3251
·Alice-Z on Nov 11 2015, 22:07.

Nov 9 2015

Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.