Diffusion warcbase (master) History

R1473/src/main/scala/org/warcbase/spark
master70c55839d0c3a88004483ae…

Nov 23 2015

Moved initialization of NER3Classifier into map closure; switched from map() to…
70c55839d0c3
·Jeremy Wiebe on Nov 23 2015, 19:24.
Port named entities extractor Pig script over to Spark as per issue #158
4a3d31b8babf
·Alice-Z on Nov 23 2015, 02:05.
ExtractEntities takes a classifier file path
f9422c23efae
·Alice-Z on Nov 23 2015, 02:00.
Revert to object version of NER3Classifier
99583f793bc5
·Alice-Z on Nov 23 2015, 01:56.

Nov 21 2015

Turn test off; classifier is too large to be included
533b4152d534
·Alice-Z on Nov 21 2015, 19:50.
Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.
Clean up, fix tests changed by new keepValidPages
14e521794754
·Alice-Z on Nov 21 2015, 16:33.

Nov 19 2015

Make keepValidPages smarter as in issue #163
2fd98dbfdb67
·Alice-Z on Nov 19 2015, 23:49.
Fix warcloader bug (issue #166)
758288fb2bd9
·Alice-Z on Nov 19 2015, 23:39.

Nov 12 2015

Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.
Clean up string formatting
e9a3965e2389
·Alice-Z on Nov 12 2015, 15:01.
Merge branch 'ner3classifier' of github.com:lintool/warcbase into ner3classifier
d56d62cf1415
·Alice-Z on Nov 12 2015, 04:22.
Working extract entities with correct output string
9f7fa9f26b54
·Alice-Z on Nov 12 2015, 04:18.
WIP: port NER3 Classifier over to Java with example usage
c9ccf559fafc
·Alice-Z on Nov 12 2015, 02:56.
WIP: port NER3 Classifier over to Java with example usage
31269caa44b6
·Alice-Z on Nov 12 2015, 00:45.

Nov 11 2015

Refactor ExtractLinks to be called with the src url
c553ef0d853f
·Alice-Z on Nov 11 2015, 23:08.
Remove extract method as per referenced in issue #146
229d274d6c13
·Alice-Z on Nov 11 2015, 22:41.
add documentation
a599db1f45b9
·Alice-Z on Nov 11 2015, 22:27.
remove extract methods
1ee0455efdf6
·Alice-Z on Nov 11 2015, 22:24.
Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.

Nov 10 2015

Clean up extracting code, use pattern matching
e1be481cd782
·Alice-Z on Nov 10 2015, 02:58.

Nov 9 2015

Clean up enum to function mapping
3957b2cce7fb
·Alice-Z on Nov 9 2015, 18:07.
Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Oct 23 2015

add extractCrawldateDomainUrlBody
35e39da6e72d
·Alice-Z on Oct 23 2015, 18:21.

Oct 21 2015

update ArcRecords interface
e5821091ecca
·Alice-Z on Oct 21 2015, 19:56.

Oct 15 2015

extend RDD
cfe76f600c7d
·Alice-Z on Oct 15 2015, 21:13.
first commit
e77201892789
·Alice-Z on Oct 15 2015, 20:07.

Jun 4 2015

Better integration of ARC readers in pyspark.
7c1ebde17fb4
·lintool on Jun 4 2015, 22:15.

May 27 2015

Converter to extract all metadata.
b220b0cc175a
·lintool on May 27 2015, 13:16.
Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.