Diffusion warcbase (master) History

matchbox
masterce43a632b44fe35c13c3f9c…

Mar 27 2016

Merge branch 'master' into images
ce43a632b44f
·lintool on Mar 27 2016, 16:49.

Mar 26 2016

Moved TweetJsonFunctions.
c71c4dcc47f8
·lintool on Mar 26 2016, 17:32.

Mar 5 2016

Downgraded json4s, folds in parsing in the loader.
93686da4ff23
·lintool on Mar 5 2016, 16:46.
Initial pass at processing tweets in Warcbase.
f2abb2b5bbc2
·lintool on Mar 5 2016, 16:21.
Added UDF for extracting image links.
b4e49fb40be5
·lintool on Mar 5 2016, 12:54.

Feb 20 2016

Update ExtractLinks.scala
db5c84770adb
·Jeremy Wiebe on Feb 20 2016, 00:03.

Feb 18 2016

Added tests for generic ARC/WARC classes
f69c7900f12e
·Jeremy Wiebe on Feb 18 2016, 03:00.

Feb 17 2016

Added loadArchives and supporting classes (loads WARCs or ARCs).
77891e7d54ad
·Jeremy Wiebe on Feb 17 2016, 08:13.

Feb 13 2016

Merge branch 'format' for ExtractDate (#154) and TupleFormatter
a7b0e0b07682
·Jeremy Wiebe on Feb 13 2016, 23:01.
Use shapeless to flatten tuples of any arity
4b8f0fb7b482
·Jeremy Wiebe on Feb 13 2016, 06:48.

Dec 25 2015

Add RemovePrefixWWW method
1b6d83c4a7fe
·Alice-Z on Dec 25 2015, 18:11.
Cleanup and add comments
d1c3f5d49783
·Alice-Z on Dec 25 2015, 17:31.
Add Date utils to clarify date extraction
1ec6e05d7540
·Alice-Z on Dec 25 2015, 17:31.
Add Formatter to un-nest tuples and print in tab-delimited format
c773fd242ff5
·Alice-Z on Dec 25 2015, 17:28.

Dec 10 2015

Use SerializableWritable wrapper for serialization
042253b579cc
·Alice-Z on Dec 10 2015, 22:03.
Refactor Record API (#189)
2adce498927d
·Alice-Z on Dec 10 2015, 21:59.

Dec 9 2015

Fixed newBufferedWriter() call for JRE < 1.8.
3e9dceac5766
·Jeremy Wiebe on Dec 9 2015, 15:21.
Write directly to local file instead of HDFS
b9f4b0ffa199
·Jeremy Wiebe on Dec 9 2015, 04:50.

Dec 8 2015

Added GDF export function to matchbox
76d82afa6c35
·Jeremy Wiebe on Dec 8 2015, 05:53.

Dec 3 2015

Fix WARecord getContentString
90616dd7b3bd
·Alice-Z on Dec 3 2015, 17:11.

Nov 26 2015

Added matchbox function to NER-classify and generate JSON for visualizer, per…
c44118522844
·Jeremy Wiebe on Nov 26 2015, 16:05.
Modified structure of JSON output
32f3e4ba7d6c
·Jeremy Wiebe on Nov 26 2015, 15:09.
Revision API to be more descriptive as in issue #179
8e8f072aca5f
·Alice-Z on Nov 26 2015, 00:27.
Add Apache license header
58491a2b8a57
·Alice-Z on Nov 26 2015, 00:02.

Nov 25 2015

Merge branch 'refactor-api' of github.com:lintool/warcbase into refactor-api
183b31d74831
·Alice-Z on Nov 25 2015, 20:36.
Revision API to be more descriptive
83d4d7232405
·Alice-Z on Nov 25 2015, 20:33.
Revision API to be more descriptive
15ad4bb50c83
·Alice-Z on Nov 25 2015, 19:06.
Make final output a single file containing single JSON array.
d0facdcd382f
·Jeremy Wiebe on Nov 25 2015, 06:27.
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest
5e316b1bb129
·lintool on Nov 25 2015, 03:49.
Fixed CR/LF issues (i.e., DOS formatting).
0b0839bdbaa9
·lintool on Nov 25 2015, 03:22.
Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.
Added jackson-databind to pom.xml
dd643f9fae5c
·Jeremy Wiebe on Nov 25 2015, 03:17.

Nov 24 2015

Relocated file, converted to class.
3e634fa43920
·Jeremy Wiebe on Nov 24 2015, 23:12.
Killed all the Pig stuff.
f5d8edf50506
·lintool on Nov 24 2015, 22:05.

Nov 23 2015

Fixed emptyString; deleted dupe line
201cfba2d832
·Jeremy Wiebe on Nov 23 2015, 19:25.
Moved initialization of NER3Classifier into map closure; switched from map() to…
70c55839d0c3
·Jeremy Wiebe on Nov 23 2015, 19:24.
Port named entities extractor Pig script over to Spark as per issue #158
4a3d31b8babf
·Alice-Z on Nov 23 2015, 02:05.
ExtractEntities takes a classifier file path
f9422c23efae
·Alice-Z on Nov 23 2015, 02:00.
Revert to object version of NER3Classifier
99583f793bc5
·Alice-Z on Nov 23 2015, 01:56.

Nov 21 2015

Turn test off; classifier is too large to be included
533b4152d534
·Alice-Z on Nov 21 2015, 19:50.
Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.
Clean up, fix tests changed by new keepValidPages
14e521794754
·Alice-Z on Nov 21 2015, 16:33.

Nov 19 2015

Fix warcloader bug (issue #166)
758288fb2bd9
·Alice-Z on Nov 19 2015, 23:39.

Nov 12 2015

Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.
Clean up string formatting
e9a3965e2389
·Alice-Z on Nov 12 2015, 15:01.
Merge branch 'ner3classifier' of github.com:lintool/warcbase into ner3classifier
d56d62cf1415
·Alice-Z on Nov 12 2015, 04:22.
Working extract entities with correct output string
9f7fa9f26b54
·Alice-Z on Nov 12 2015, 04:18.
WIP: port NER3 Classifier over to Java with example usage
c9ccf559fafc
·Alice-Z on Nov 12 2015, 02:56.
WIP: port NER3 Classifier over to Java with example usage
31269caa44b6
·Alice-Z on Nov 12 2015, 00:45.

Nov 11 2015

Refactor ExtractLinks to be called with the src url
c553ef0d853f
·Alice-Z on Nov 11 2015, 23:08.
remove extract methods
1ee0455efdf6
·Alice-Z on Nov 11 2015, 22:24.

Nov 9 2015

Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Oct 23 2015

add extractCrawldateDomainUrlBody
35e39da6e72d
·Alice-Z on Oct 23 2015, 18:21.

Oct 21 2015

update ArcRecords interface
e5821091ecca
·Alice-Z on Oct 21 2015, 19:56.

Oct 15 2015

extend RDD
cfe76f600c7d
·Alice-Z on Oct 15 2015, 21:13.
first commit
e77201892789
·Alice-Z on Oct 15 2015, 20:07.

R1473/src/main/scala/org/warcbase/spark/matchboxmasterce43a632b44fe35c13c3f9c…

Mar 27 2016

Mar 26 2016

Mar 5 2016

Feb 20 2016

Feb 18 2016

Feb 17 2016

Feb 13 2016

Dec 25 2015

Dec 10 2015

Dec 9 2015

Dec 8 2015

Dec 3 2015

Nov 26 2015

Nov 25 2015

Nov 24 2015

Nov 23 2015

Nov 21 2015

Nov 19 2015

Nov 12 2015

Nov 11 2015

Nov 9 2015

Nov 8 2015

Nov 3 2015

Oct 23 2015

Oct 21 2015

Oct 15 2015

R1473/src/main/scala/org/warcbase/spark/matchbox
masterce43a632b44fe35c13c3f9c…