Diffusion warcbase (master) History

History
master1c8cd5c78b1fa439ca49a33…

Mar 5 2016

Added method to extract text.
1c8cd5c78b1f
·lintool on Mar 5 2016, 17:54.
Downgraded json4s, folds in parsing in the loader.
93686da4ff23
·lintool on Mar 5 2016, 16:46.
Initial pass at processing tweets in Warcbase.
f2abb2b5bbc2
·lintool on Mar 5 2016, 16:21.

Feb 20 2016

Update ExtractLinks.scala
db5c84770adb
·Jeremy Wiebe on Feb 20 2016, 00:03.

Feb 18 2016

Added loadArchives(), per request #195
c52fab1a3e7a
·Jeremy Wiebe on Feb 18 2016, 04:06.
Added tests for generic ARC/WARC classes
f69c7900f12e
·Jeremy Wiebe on Feb 18 2016, 03:00.

Feb 17 2016

Added loadArchives and supporting classes (loads WARCs or ARCs).
77891e7d54ad
·Jeremy Wiebe on Feb 17 2016, 08:13.

Feb 16 2016

Added keepUrlPatterns and discardUrlPatterns as per #197.
bf893d3b5da5
·Jeremy Wiebe on Feb 16 2016, 17:08.
Added discardUrlPatterns
8a4c55019413
·Jeremy Wiebe on Feb 16 2016, 16:31.
keepUrlPatterns test
232397a26bf1
·Jeremy Wiebe on Feb 16 2016, 15:58.
Added keepUrlPatterns
bf682d8e8efc
·Jeremy Wiebe on Feb 16 2016, 03:54.

Feb 13 2016

Merge branch 'format' for ExtractDate (#154) and TupleFormatter
a7b0e0b07682
·Jeremy Wiebe on Feb 13 2016, 23:01.
Use shapeless to flatten tuples of any arity
4b8f0fb7b482
·Jeremy Wiebe on Feb 13 2016, 06:48.

Feb 4 2016

Merge pull request #194 from lintool/remove-prefix-www
3323b70e43cb
·Jeremy Wiebe on Feb 4 2016, 21:33.
Added keepLanguages filter to RecordRDD as per Issue #190
16276bc9d3d9
·Jeremy Wiebe on Feb 4 2016, 16:41.
Added keepLanguages RDD filter
30a5e5d8d585
·Jeremy Wiebe on Feb 4 2016, 16:36.

Dec 25 2015

Add RemovePrefixWWW method
1b6d83c4a7fe
·Alice-Z on Dec 25 2015, 18:11.
add method to filter date by component
4042cac0ad8b
·Alice-Z on Dec 25 2015, 17:49.
Cleanup and add comments
d1c3f5d49783
·Alice-Z on Dec 25 2015, 17:31.
Add Date utils to clarify date extraction
1ec6e05d7540
·Alice-Z on Dec 25 2015, 17:31.
Add Formatter to un-nest tuples and print in tab-delimited format
c773fd242ff5
·Alice-Z on Dec 25 2015, 17:28.

Dec 16 2015

URL change to Gephi tutorial
e2697e7df766
·ianmilligan1 on Dec 16 2015, 15:39.

Dec 14 2015

Refactoring API so that I can take individual Spark records in spark-shell and…
ba2b44c52b35
·lintool on Dec 14 2015, 03:24.

Dec 10 2015

Use SerializableWritable wrapper for serialization
0d78b9bb2b5b
·Alice-Z on Dec 10 2015, 22:05.
Use SerializableWritable wrapper for serialization
042253b579cc
·Alice-Z on Dec 10 2015, 22:03.
Refactor Record API (#189)
2adce498927d
·Alice-Z on Dec 10 2015, 21:59.

Dec 9 2015

Fix newBufferedWriter() call for JRE < 1.8.
dc99a69316bc
·Jeremy Wiebe on Dec 9 2015, 15:22.
Fixed newBufferedWriter() call for JRE < 1.8.
3e9dceac5766
·Jeremy Wiebe on Dec 9 2015, 15:21.
Export GDF to local file instead of HDFS
94d974ce3ec7
·Jeremy Wiebe on Dec 9 2015, 04:52.
Write directly to local file instead of HDFS
b9f4b0ffa199
·Jeremy Wiebe on Dec 9 2015, 04:50.

Dec 8 2015

Added GDF export function to matchbox
76d82afa6c35
·Jeremy Wiebe on Dec 8 2015, 05:53.

Dec 3 2015

Fixed links to new documentation: Issues #181
c026cba9e056
·lintool on Dec 3 2015, 22:34.
Fix WarcRecord as per issue #185
db8beaa3d3f5
·Alice-Z on Dec 3 2015, 17:23.
Fix WARecord getContentString
90616dd7b3bd
·Alice-Z on Dec 3 2015, 17:11.

Dec 1 2015

Add example scripts for crawl statistics [Issue #182]
43b9b1f84ef0
·Alice-Z on Dec 1 2015, 02:20.
Add example scripts for crawl statistics and social media links
ba91869cd370
·Alice-Z on Dec 1 2015, 01:18.

Nov 27 2015

Reference to warcbase-resources
e2dae6279b04
·Ian Milligan on Nov 27 2015, 03:14.

Nov 26 2015

Added matchbox function to NER-classify and generate JSON for visualizer, per…
c44118522844
·Jeremy Wiebe on Nov 26 2015, 16:05.
Modified visualizer to read JSON instead of CSV. Also added day view. Removed…
b6e1ba3a1141
·Jeremy Wiebe on Nov 26 2015, 16:04.
Modified structure of JSON output
32f3e4ba7d6c
·Jeremy Wiebe on Nov 26 2015, 15:09.
Revision API to be more descriptive as in issue #179
8e8f072aca5f
·Alice-Z on Nov 26 2015, 00:27.
Add Apache license header
58491a2b8a57
·Alice-Z on Nov 26 2015, 00:02.

Nov 25 2015

Merge branch 'refactor-api' of github.com:lintool/warcbase into refactor-api
183b31d74831
·Alice-Z on Nov 25 2015, 20:36.
Revision API to be more descriptive
83d4d7232405
·Alice-Z on Nov 25 2015, 20:33.
Revision API to be more descriptive
15ad4bb50c83
·Alice-Z on Nov 25 2015, 19:06.
Make final output a single file containing single JSON array.
d0facdcd382f
·Jeremy Wiebe on Nov 25 2015, 06:27.
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest (Issues #172, #173…
26d2ef518fb3
·lintool on Nov 25 2015, 03:53.
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest
5e316b1bb129
·lintool on Nov 25 2015, 03:49.
Fixed CR/LF issues (i.e., DOS formatting).
0b0839bdbaa9
·lintool on Nov 25 2015, 03:22.
Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.
Added jackson-databind to pom.xml
dd643f9fae5c
·Jeremy Wiebe on Nov 25 2015, 03:17.
Add Scaladoc generation info
b7dd047a8dde
·Alice Ran Zhou on Nov 25 2015, 02:46.

Nov 24 2015

Delete -- relocated file.
333776df87c7
·Jeremy Wiebe on Nov 24 2015, 23:18.
Relocated file, converted to class.
3e634fa43920
·Jeremy Wiebe on Nov 24 2015, 23:12.
SQUEAL! Killed the Pig (Issue #159)
73f232c053ba
·lintool on Nov 24 2015, 22:08.
Killed all the Pig stuff.
f5d8edf50506
·lintool on Nov 24 2015, 22:05.
Actually emit JSON.
ff055a163e40
·Jeremy Wiebe on Nov 24 2015, 18:08.
Initial commit: does NER classification, saves to file JSON representation per…
1ea7eed98c41
·Jeremy Wiebe on Nov 24 2015, 17:15.

Nov 23 2015

Make ExtractEntities load a new classifier per partition
924df687ce28
·Alice-Z on Nov 23 2015, 20:44.
Fixed emptyString; deleted dupe line
201cfba2d832
·Jeremy Wiebe on Nov 23 2015, 19:25.
Moved initialization of NER3Classifier into map closure; switched from map() to…
70c55839d0c3
·Jeremy Wiebe on Nov 23 2015, 19:24.
Tiny fix: remove unneeded part of test
2dde7b822a5c
·Alice-Z on Nov 23 2015, 02:08.
Port named entities extractor Pig script over to Spark as per issue #158
4a3d31b8babf
·Alice-Z on Nov 23 2015, 02:05.
ExtractEntities takes a classifier file path
f9422c23efae
·Alice-Z on Nov 23 2015, 02:00.
Revert to object version of NER3Classifier
99583f793bc5
·Alice-Z on Nov 23 2015, 01:56.

Nov 21 2015

Turn test off; classifier is too large to be included
533b4152d534
·Alice-Z on Nov 21 2015, 19:50.
Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.
Make keepValidPages smarter as in issue #163
0483269d7e35
·Alice-Z on Nov 21 2015, 16:34.
Clean up, fix tests changed by new keepValidPages
14e521794754
·Alice-Z on Nov 21 2015, 16:33.

Nov 19 2015

Make keepValidPages smarter as in issue #163
2fd98dbfdb67
·Alice-Z on Nov 19 2015, 23:49.
Merge branch 'master' of github.com:lintool/warcbase
5bf5d4edbcef
·Alice-Z on Nov 19 2015, 23:42.
Fix warcloader bug (issue #166)
43e9bb8fc751
·Alice-Z on Nov 19 2015, 23:42.
Fix warcloader bug (issue #166)
758288fb2bd9
·Alice-Z on Nov 19 2015, 23:39.

Nov 16 2015

Added link to Spark Installation instructions
4988a1a7c6df
·Ian Milligan on Nov 16 2015, 15:23.

Nov 14 2015

Updated README.
374c04a56cbe
·lintool on Nov 14 2015, 14:49.

Nov 12 2015

Copied NER visualization from @jrwiebe/WAHR/nerviz
ab75ed8f3668
·Jeremy Wiebe on Nov 12 2015, 20:28.
Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.
Clean up string formatting
e9a3965e2389
·Alice-Z on Nov 12 2015, 15:01.
Merge branch 'ner3classifier' of github.com:lintool/warcbase into ner3classifier
d56d62cf1415
·Alice-Z on Nov 12 2015, 04:22.
Working extract entities with correct output string
9f7fa9f26b54
·Alice-Z on Nov 12 2015, 04:18.
WIP: port NER3 Classifier over to Java with example usage
c9ccf559fafc
·Alice-Z on Nov 12 2015, 02:56.
Fixed issue #157 - refactor ExtractLinks to include "from url"
c719bd5d2906
·Alice-Z on Nov 12 2015, 02:50.
WIP: port NER3 Classifier over to Java with example usage
31269caa44b6
·Alice-Z on Nov 12 2015, 00:45.

Nov 11 2015

Refactor ExtractLinks to be called with the src url
c553ef0d853f
·Alice-Z on Nov 11 2015, 23:08.
Remove extract method as per referenced in issue #146
229d274d6c13
·Alice-Z on Nov 11 2015, 22:41.
add documentation
a599db1f45b9
·Alice-Z on Nov 11 2015, 22:27.
tiny formatting change
de66d390f4ae
·Alice-Z on Nov 11 2015, 22:24.
remove extract methods
1ee0455efdf6
·Alice-Z on Nov 11 2015, 22:24.
Fixed issues #150, #149 (Pig UDFs), #156 (add keepValidPages) and #155…
80605c859501
·Alice-Z on Nov 11 2015, 22:14.
Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.
Rename tests for JUnitTestRunner
e5668e3e3251
·Alice-Z on Nov 11 2015, 22:07.

Nov 10 2015

Clean up extracting code, use pattern matching
e1be481cd782
·Alice-Z on Nov 10 2015, 02:58.

Nov 9 2015

Merge branch 'matchbox' of github.com:lintool/warcbase into matchbox
ea61e9eea405
·Alice-Z on Nov 9 2015, 18:09.
Clean up enum to function mapping
3957b2cce7fb
·Alice-Z on Nov 9 2015, 18:07.
Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.
Fixed broken build.
5a88901dd9cc
·lintool on Nov 9 2015, 03:55.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Nov 1 2015

Fixed issues #142, #152, #153
f0a2e1d1f94a
·lintool on Nov 1 2015, 17:56.

Historymaster1c8cd5c78b1fa439ca49a33…

Mar 5 2016

Feb 20 2016

Feb 18 2016

Feb 17 2016

Feb 16 2016

Feb 13 2016

Feb 4 2016

Dec 25 2015

Dec 16 2015

Dec 14 2015

Dec 10 2015

Dec 9 2015

Dec 8 2015

Dec 3 2015

Dec 1 2015

Nov 27 2015

Nov 26 2015

Nov 25 2015

Nov 24 2015

Nov 23 2015

Nov 21 2015

Nov 19 2015

Nov 16 2015

Nov 14 2015

Nov 12 2015

Nov 11 2015

Nov 10 2015

Nov 9 2015

Nov 8 2015

Nov 3 2015

Nov 1 2015

History
master1c8cd5c78b1fa439ca49a33…