Diffusion warcbase (master) History

History
masterc2ab2e57d9872c1edc0c20a…

Mar 29 2016

checking in link visualization
c2ab2e57d987
·ianmilligan1 on Mar 29 2016, 17:27.
Issue #208 ExtractTopLevelDomain UDF misnamed
03c2d47d0e7c
·lintool on Mar 29 2016, 05:18.
downgraded guava to fix borked tests, cf https://issues.apache.
52eb2f96304d
·lintool on Mar 29 2016, 05:17.
Fixed compile bug, converted to .removePrefixWWW()
513c69695940
·lintool on Mar 29 2016, 05:10.
Merge branch 'master' into domainlinks
70e6c3a63ff6
·lintool on Mar 29 2016, 04:57.
Merge branch 'graphx'
2ffc60cc178e
·Jeremy Wiebe on Mar 29 2016, 04:09.
Parameterized static vs dynamic PageRank.
44f3548c05cd
·Jeremy Wiebe on Mar 29 2016, 04:08.
renaming ExtractTopLevelDomain to ExtractDomain
fc0d11495cf1
·ianmilligan1 on Mar 29 2016, 03:11.

Mar 28 2016

Use loadArchives
c9e369fef9ef
·Jeremy Wiebe on Mar 28 2016, 16:29.
updated loadArc to loadArchives in README.md
4a4d5e5494bf
·ianmilligan1 on Mar 28 2016, 14:51.

Mar 27 2016

Merging in minor tweaks in NER vis
2b6689ce033e
·lintool on Mar 27 2016, 18:11.
Add UDF for extracting stuff from tweets, Issue #210
40025b169e93
·lintool on Mar 27 2016, 18:05.
Added test cases.
b80c970f2e50
·lintool on Mar 27 2016, 18:04.
Addressed UDF for extracting image links, Issue #203
04d105ded330
·lintool on Mar 27 2016, 17:33.
Added ExtractImageLinks test cases.
2da8f0abb983
·lintool on Mar 27 2016, 17:31.
Merge branch 'master' into images
ce43a632b44f
·lintool on Mar 27 2016, 16:49.
UDFs for extracting at-mentions, hashtags, and URLs from tweets.
f45f4d9fb0d3
·lintool on Mar 27 2016, 01:27.

Mar 26 2016

Merged in tweet analysis functionality, Issue #204
6903b98be69e
·lintool on Mar 26 2016, 18:26.
Moved TweetJsonFunctions.
c71c4dcc47f8
·lintool on Mar 26 2016, 17:32.

Mar 21 2016

Separated graph generation (apply method) from writer, and fixed calculation of…
1d3ddbbd57d1
·Jeremy Wiebe on Mar 21 2016, 22:12.

Mar 8 2016

Revert to old CDH (new version produces errors on cluster)
19eefb5d7f2c
·Jeremy Wiebe on Mar 8 2016, 02:44.

Mar 5 2016

Basic graph analysis, initial check-in
997d2405a84e
·Jeremy Wiebe on Mar 5 2016, 19:24.
Added method to extract text.
1c8cd5c78b1f
·lintool on Mar 5 2016, 17:54.
Downgraded json4s, folds in parsing in the loader.
93686da4ff23
·lintool on Mar 5 2016, 16:46.
Initial pass at processing tweets in Warcbase.
f2abb2b5bbc2
·lintool on Mar 5 2016, 16:21.
Added UDF for extracting image links.
b4e49fb40be5
·lintool on Mar 5 2016, 12:54.

Feb 20 2016

Update ExtractLinks.scala
db5c84770adb
·Jeremy Wiebe on Feb 20 2016, 00:03.

Feb 18 2016

Added loadArchives(), per request #195
c52fab1a3e7a
·Jeremy Wiebe on Feb 18 2016, 04:06.
Added tests for generic ARC/WARC classes
f69c7900f12e
·Jeremy Wiebe on Feb 18 2016, 03:00.

Feb 17 2016

Added loadArchives and supporting classes (loads WARCs or ARCs).
77891e7d54ad
·Jeremy Wiebe on Feb 17 2016, 08:13.

Feb 16 2016

Added keepUrlPatterns and discardUrlPatterns as per #197.
bf893d3b5da5
·Jeremy Wiebe on Feb 16 2016, 17:08.
Added discardUrlPatterns
8a4c55019413
·Jeremy Wiebe on Feb 16 2016, 16:31.
keepUrlPatterns test
232397a26bf1
·Jeremy Wiebe on Feb 16 2016, 15:58.
Added keepUrlPatterns
bf682d8e8efc
·Jeremy Wiebe on Feb 16 2016, 03:54.

Feb 13 2016

Merge branch 'format' for ExtractDate (#154) and TupleFormatter
a7b0e0b07682
·Jeremy Wiebe on Feb 13 2016, 23:01.
Use shapeless to flatten tuples of any arity
4b8f0fb7b482
·Jeremy Wiebe on Feb 13 2016, 06:48.

Feb 4 2016

Removed protocol from URLs pointing to scripts and stylesheets, to avoid mixed…
3fb632ccd905
·Jeremy Wiebe on Feb 4 2016, 21:56.
Merge pull request #194 from lintool/remove-prefix-www
3323b70e43cb
·Jeremy Wiebe on Feb 4 2016, 21:33.
Added keepLanguages filter to RecordRDD as per Issue #190
16276bc9d3d9
·Jeremy Wiebe on Feb 4 2016, 16:41.
Added keepLanguages RDD filter
30a5e5d8d585
·Jeremy Wiebe on Feb 4 2016, 16:36.

Dec 25 2015

Add RemovePrefixWWW method
1b6d83c4a7fe
·Alice-Z on Dec 25 2015, 18:11.
add method to filter date by component
4042cac0ad8b
·Alice-Z on Dec 25 2015, 17:49.
Cleanup and add comments
d1c3f5d49783
·Alice-Z on Dec 25 2015, 17:31.
Add Date utils to clarify date extraction
1ec6e05d7540
·Alice-Z on Dec 25 2015, 17:31.
Add Formatter to un-nest tuples and print in tab-delimited format
c773fd242ff5
·Alice-Z on Dec 25 2015, 17:28.

Dec 16 2015

URL change to Gephi tutorial
e2697e7df766
·ianmilligan1 on Dec 16 2015, 15:39.

Dec 14 2015

Refactoring API so that I can take individual Spark records in spark-shell and…
ba2b44c52b35
·lintool on Dec 14 2015, 03:24.

Dec 10 2015

Use SerializableWritable wrapper for serialization
0d78b9bb2b5b
·Alice-Z on Dec 10 2015, 22:05.
Use SerializableWritable wrapper for serialization
042253b579cc
·Alice-Z on Dec 10 2015, 22:03.
Refactor Record API (#189)
2adce498927d
·Alice-Z on Dec 10 2015, 21:59.

Dec 9 2015

Fix newBufferedWriter() call for JRE < 1.8.
dc99a69316bc
·Jeremy Wiebe on Dec 9 2015, 15:22.
Fixed newBufferedWriter() call for JRE < 1.8.
3e9dceac5766
·Jeremy Wiebe on Dec 9 2015, 15:21.
Export GDF to local file instead of HDFS
94d974ce3ec7
·Jeremy Wiebe on Dec 9 2015, 04:52.
Write directly to local file instead of HDFS
b9f4b0ffa199
·Jeremy Wiebe on Dec 9 2015, 04:50.

Dec 8 2015

Added GDF export function to matchbox
76d82afa6c35
·Jeremy Wiebe on Dec 8 2015, 05:53.

Dec 3 2015

Fixed links to new documentation: Issues #181
c026cba9e056
·lintool on Dec 3 2015, 22:34.
Fix WarcRecord as per issue #185
db8beaa3d3f5
·Alice-Z on Dec 3 2015, 17:23.
Fix WARecord getContentString
90616dd7b3bd
·Alice-Z on Dec 3 2015, 17:11.

Dec 1 2015

Add example scripts for crawl statistics [Issue #182]
43b9b1f84ef0
·Alice-Z on Dec 1 2015, 02:20.
Add example scripts for crawl statistics and social media links
ba91869cd370
·Alice-Z on Dec 1 2015, 01:18.

Nov 27 2015

Reference to warcbase-resources
e2dae6279b04
·Ian Milligan on Nov 27 2015, 03:14.

Nov 26 2015

Added matchbox function to NER-classify and generate JSON for visualizer, per…
c44118522844
·Jeremy Wiebe on Nov 26 2015, 16:05.
Modified visualizer to read JSON instead of CSV. Also added day view. Removed…
b6e1ba3a1141
·Jeremy Wiebe on Nov 26 2015, 16:04.
Modified structure of JSON output
32f3e4ba7d6c
·Jeremy Wiebe on Nov 26 2015, 15:09.
Revision API to be more descriptive as in issue #179
8e8f072aca5f
·Alice-Z on Nov 26 2015, 00:27.
Add Apache license header
58491a2b8a57
·Alice-Z on Nov 26 2015, 00:02.

Nov 25 2015

Merge branch 'refactor-api' of github.com:lintool/warcbase into refactor-api
183b31d74831
·Alice-Z on Nov 25 2015, 20:36.
Revision API to be more descriptive
83d4d7232405
·Alice-Z on Nov 25 2015, 20:33.
Revision API to be more descriptive
15ad4bb50c83
·Alice-Z on Nov 25 2015, 19:06.
Make final output a single file containing single JSON array.
d0facdcd382f
·Jeremy Wiebe on Nov 25 2015, 06:27.
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest (Issues #172, #173…
26d2ef518fb3
·lintool on Nov 25 2015, 03:53.
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest
5e316b1bb129
·lintool on Nov 25 2015, 03:49.
Fixed CR/LF issues (i.e., DOS formatting).
0b0839bdbaa9
·lintool on Nov 25 2015, 03:22.
Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.
Added jackson-databind to pom.xml
dd643f9fae5c
·Jeremy Wiebe on Nov 25 2015, 03:17.
Add Scaladoc generation info
b7dd047a8dde
·Alice Ran Zhou on Nov 25 2015, 02:46.

Nov 24 2015

Delete -- relocated file.
333776df87c7
·Jeremy Wiebe on Nov 24 2015, 23:18.
Relocated file, converted to class.
3e634fa43920
·Jeremy Wiebe on Nov 24 2015, 23:12.
SQUEAL! Killed the Pig (Issue #159)
73f232c053ba
·lintool on Nov 24 2015, 22:08.
Killed all the Pig stuff.
f5d8edf50506
·lintool on Nov 24 2015, 22:05.
Actually emit JSON.
ff055a163e40
·Jeremy Wiebe on Nov 24 2015, 18:08.
Initial commit: does NER classification, saves to file JSON representation per…
1ea7eed98c41
·Jeremy Wiebe on Nov 24 2015, 17:15.

Nov 23 2015

Make ExtractEntities load a new classifier per partition
924df687ce28
·Alice-Z on Nov 23 2015, 20:44.
Fixed emptyString; deleted dupe line
201cfba2d832
·Jeremy Wiebe on Nov 23 2015, 19:25.
Moved initialization of NER3Classifier into map closure; switched from map() to…
70c55839d0c3
·Jeremy Wiebe on Nov 23 2015, 19:24.
Tiny fix: remove unneeded part of test
2dde7b822a5c
·Alice-Z on Nov 23 2015, 02:08.
Port named entities extractor Pig script over to Spark as per issue #158
4a3d31b8babf
·Alice-Z on Nov 23 2015, 02:05.
ExtractEntities takes a classifier file path
f9422c23efae
·Alice-Z on Nov 23 2015, 02:00.
Revert to object version of NER3Classifier
99583f793bc5
·Alice-Z on Nov 23 2015, 01:56.

Nov 21 2015

Turn test off; classifier is too large to be included
533b4152d534
·Alice-Z on Nov 21 2015, 19:50.
Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.
Make keepValidPages smarter as in issue #163
0483269d7e35
·Alice-Z on Nov 21 2015, 16:34.
Clean up, fix tests changed by new keepValidPages
14e521794754
·Alice-Z on Nov 21 2015, 16:33.

Nov 19 2015

Make keepValidPages smarter as in issue #163
2fd98dbfdb67
·Alice-Z on Nov 19 2015, 23:49.
Merge branch 'master' of github.com:lintool/warcbase
5bf5d4edbcef
·Alice-Z on Nov 19 2015, 23:42.
Fix warcloader bug (issue #166)
43e9bb8fc751
·Alice-Z on Nov 19 2015, 23:42.
Fix warcloader bug (issue #166)
758288fb2bd9
·Alice-Z on Nov 19 2015, 23:39.

Nov 16 2015

Added link to Spark Installation instructions
4988a1a7c6df
·Ian Milligan on Nov 16 2015, 15:23.

Nov 14 2015

Updated README.
374c04a56cbe
·lintool on Nov 14 2015, 14:49.

Historymasterc2ab2e57d9872c1edc0c20a…

Mar 29 2016

Mar 28 2016

Mar 27 2016

Mar 26 2016

Mar 21 2016

Mar 8 2016

Mar 5 2016

Feb 20 2016

Feb 18 2016

Feb 17 2016

Feb 16 2016

Feb 13 2016

Feb 4 2016

Dec 25 2015

Dec 16 2015

Dec 14 2015

Dec 10 2015

Dec 9 2015

Dec 8 2015

Dec 3 2015

Dec 1 2015

Nov 27 2015

Nov 26 2015

Nov 25 2015

Nov 24 2015

Nov 23 2015

Nov 21 2015

Nov 19 2015

Nov 16 2015

Nov 14 2015

History
masterc2ab2e57d9872c1edc0c20a…