Diffusion warcbase (master) History

History
masterab72ae4ce23185caafc193b…

Jul 28 2016

Add UDF for computing MD5 checksum. Issue #211
ab72ae4ce231
·lintool on Jul 28 2016, 23:24.

Jul 27 2016

checksum
3b4ebe297a21
·Youngbin Kim on Jul 27 2016, 03:04.

Jun 29 2016

Skips corrupt WARC records with negative lengths. Issue #234.
16db93460896
·lintool on Jun 29 2016, 03:24.
Skips corrupt WARC records with negative lengths.
165d8f38fffe
·lintool on Jun 29 2016, 00:12.

Jun 28 2016

minor tweak, adding build instructions for analytics
46b65a8909e2
·ianmilligan1 on Jun 28 2016, 22:51.
Refactored Warcbase into multiple modules, upgraded to CDH 5.7.1 (w/ Spark 1.6.
62b8c4e57800
·lintool on Jun 28 2016, 18:08.
Updated documentation.
7d2f8a4f2119
·lintool on Jun 28 2016, 18:00.
Fixed Scala matching issue with NER extractor.
6a90426a3d1a
·lintool on Jun 28 2016, 16:50.

Jun 24 2016

Upgraded to CDH 5.7.1.
19be6edfa8cd
·lintool on Jun 24 2016, 23:08.

Jun 22 2016

Added dependencies for boilerpipe.
0dfabaeb484b
·lintool on Jun 22 2016, 17:57.

Jun 17 2016

Cleaned up and simplified dependencies, etc.
e8c43f1b9d24
·lintool on Jun 17 2016, 19:46.

Jun 16 2016

Integrating warcbase-hbase artifact.
6e8740a55414
·lintool on Jun 16 2016, 20:52.
Moved HBase and non-core code out of warcbase-core into warcbase-hbase.
0db46be7ef7f
·lintool on Jun 16 2016, 17:30.
Created warcbase-core module.
f972206db516
·lintool on Jun 16 2016, 15:43.

Jun 15 2016

Fixed issue #234 Error handling for broken ARC/WARC files: empty content in ARC…
63c3d5916d84
·lintool on Jun 15 2016, 16:06.
fixed broken link
e4405a1d8bd1
·ianmilligan1 on Jun 15 2016, 15:45.

May 21 2016

Merge pull request #229 from yb1/pagerank (optimization)
c3f9345bf748
·Jeremy Wiebe on May 21 2016, 22:56.

May 20 2016

default values
c09d35d8a49e
·Youngbin Kim on May 20 2016, 07:43.

May 16 2016

Merge pull request #228 from lintool/filter-content
b0b4d5df2f77
·Ian Milligan on May 16 2016, 19:34.

May 15 2016

Cleaned up README.
e09289e57a88
·lintool on May 15 2016, 15:08.
Fixed issue #225 More robust tweet parsing
0c850642f789
·lintool on May 15 2016, 03:10.

May 13 2016

Added keepContent() and discardContent() methods to RecordRDD
b5fd283dc492
·Jeremy Wiebe on May 13 2016, 06:09.

May 11 2016

Fixed bug in optimization code
9ce41861d5f0
·Youngbin Kim on May 11 2016, 19:32.

May 9 2016

optimize extracting links for Pagerank
2f4f6fce86e8
·Youngbin Kim on May 9 2016, 19:16.

May 5 2016

added OMRI funding acknowlegement; annual report season is here
51c1f57e0a7b
·ianmilligan1 on May 5 2016, 22:27.

May 3 2016

Fix to eliminate build warning, per issue #223.
bf251c0dc714
·Jeremy Wiebe on May 3 2016, 05:25.

May 2 2016

More robust error handling.
03c93f22c014
·lintool on May 2 2016, 02:46.

May 1 2016

More robust JSON parsing.
b7fc45b842f0
·lintool on May 1 2016, 22:10.

Apr 29 2016

removed process.py as per #218 - this is why I should always use branches.
53c7b911efe6
·ianmilligan1 on Apr 29 2016, 01:13.
checking in process.py as per #218, on master
ee783c310c43
·ianmilligan1 on Apr 29 2016, 00:01.

Apr 22 2016

Merge pull request #220 from ianmilligan1/master
72c9ca70de97
·Ian Milligan on Apr 22 2016, 22:18.
Merge pull request #221 from lintool/year-month
71a19fbaaa74
·Jeremy Wiebe on Apr 22 2016, 17:09.

Apr 21 2016

fixed ArcTest
eb4c0e313a67
·ianmilligan1 on Apr 21 2016, 18:13.
more tweaks for case sensitive getCrawlDate
d66f001374b1
·ianmilligan1 on Apr 21 2016, 17:51.
changing case errors in ExtractCrawlDate, checking with TravisCI
b7a81a3afff4
·ianmilligan1 on Apr 21 2016, 17:27.
capitalized function getCrawlDate and getCrawlMonth
58b4e792a2bc
·ianmilligan1 on Apr 21 2016, 16:56.

Apr 20 2016

new getCrawlMonth function
9ee270102a93
·ianmilligan1 on Apr 20 2016, 15:42.

Apr 18 2016

draft of contributing file
0b296eb38067
·ianmilligan1 on Apr 18 2016, 16:18.

Apr 10 2016

updating README; code block, link to NER viz, prominent documentation link
1f6fd06c02a4
·ianmilligan1 on Apr 10 2016, 23:12.

Apr 5 2016

Added funding.
7489b249d557
·lintool on Apr 5 2016, 20:43.
Merge pull request #205 from ruebot/travis
b048f1347b0b
·Jimmy Lin on Apr 5 2016, 20:20.

Apr 3 2016

Setup TravisCI
1f3a4c6166b8
·nruest on Apr 3 2016, 15:40.

Mar 31 2016

Merge pull request #213 from lintool/link-visualization
ee5262c67b04
·Jeremy Wiebe on Mar 31 2016, 17:46.
Deleted unnecessary data processing script
4aab2330ecee
·Jeremy Wiebe on Mar 31 2016, 17:44.

Mar 30 2016

Change pure-min CSS location to CDN with HTTP and HTTPS
666b2572b2f6
·Jeremy Wiebe on Mar 30 2016, 20:53.
minor change to NER URL text in index.html
71e8a03333e1
·ianmilligan1 on Mar 30 2016, 20:39.

Mar 29 2016

checking in link visualization
c2ab2e57d987
·ianmilligan1 on Mar 29 2016, 17:27.
Issue #208 ExtractTopLevelDomain UDF misnamed
03c2d47d0e7c
·lintool on Mar 29 2016, 05:18.
downgraded guava to fix borked tests, cf https://issues.apache.
52eb2f96304d
·lintool on Mar 29 2016, 05:17.
Fixed compile bug, converted to .removePrefixWWW()
513c69695940
·lintool on Mar 29 2016, 05:10.
Merge branch 'master' into domainlinks
70e6c3a63ff6
·lintool on Mar 29 2016, 04:57.
Merge branch 'graphx'
2ffc60cc178e
·Jeremy Wiebe on Mar 29 2016, 04:09.
Parameterized static vs dynamic PageRank.
44f3548c05cd
·Jeremy Wiebe on Mar 29 2016, 04:08.
renaming ExtractTopLevelDomain to ExtractDomain
fc0d11495cf1
·ianmilligan1 on Mar 29 2016, 03:11.

Mar 28 2016

Use loadArchives
c9e369fef9ef
·Jeremy Wiebe on Mar 28 2016, 16:29.
updated loadArc to loadArchives in README.md
4a4d5e5494bf
·ianmilligan1 on Mar 28 2016, 14:51.

Mar 27 2016

Merging in minor tweaks in NER vis
2b6689ce033e
·lintool on Mar 27 2016, 18:11.
Add UDF for extracting stuff from tweets, Issue #210
40025b169e93
·lintool on Mar 27 2016, 18:05.
Added test cases.
b80c970f2e50
·lintool on Mar 27 2016, 18:04.
Addressed UDF for extracting image links, Issue #203
04d105ded330
·lintool on Mar 27 2016, 17:33.
Added ExtractImageLinks test cases.
2da8f0abb983
·lintool on Mar 27 2016, 17:31.
Merge branch 'master' into images
ce43a632b44f
·lintool on Mar 27 2016, 16:49.
UDFs for extracting at-mentions, hashtags, and URLs from tweets.
f45f4d9fb0d3
·lintool on Mar 27 2016, 01:27.

Mar 26 2016

Merged in tweet analysis functionality, Issue #204
6903b98be69e
·lintool on Mar 26 2016, 18:26.
Moved TweetJsonFunctions.
c71c4dcc47f8
·lintool on Mar 26 2016, 17:32.

Mar 21 2016

Separated graph generation (apply method) from writer, and fixed calculation of…
1d3ddbbd57d1
·Jeremy Wiebe on Mar 21 2016, 22:12.

Mar 8 2016

Revert to old CDH (new version produces errors on cluster)
19eefb5d7f2c
·Jeremy Wiebe on Mar 8 2016, 02:44.

Mar 5 2016

Basic graph analysis, initial check-in
997d2405a84e
·Jeremy Wiebe on Mar 5 2016, 19:24.
Added method to extract text.
1c8cd5c78b1f
·lintool on Mar 5 2016, 17:54.
Downgraded json4s, folds in parsing in the loader.
93686da4ff23
·lintool on Mar 5 2016, 16:46.
Initial pass at processing tweets in Warcbase.
f2abb2b5bbc2
·lintool on Mar 5 2016, 16:21.
Added UDF for extracting image links.
b4e49fb40be5
·lintool on Mar 5 2016, 12:54.

Feb 20 2016

Update ExtractLinks.scala
db5c84770adb
·Jeremy Wiebe on Feb 20 2016, 00:03.

Feb 18 2016

Added loadArchives(), per request #195
c52fab1a3e7a
·Jeremy Wiebe on Feb 18 2016, 04:06.
Added tests for generic ARC/WARC classes
f69c7900f12e
·Jeremy Wiebe on Feb 18 2016, 03:00.

Feb 17 2016

Added loadArchives and supporting classes (loads WARCs or ARCs).
77891e7d54ad
·Jeremy Wiebe on Feb 17 2016, 08:13.

Feb 16 2016

Added keepUrlPatterns and discardUrlPatterns as per #197.
bf893d3b5da5
·Jeremy Wiebe on Feb 16 2016, 17:08.
Added discardUrlPatterns
8a4c55019413
·Jeremy Wiebe on Feb 16 2016, 16:31.
keepUrlPatterns test
232397a26bf1
·Jeremy Wiebe on Feb 16 2016, 15:58.
Added keepUrlPatterns
bf682d8e8efc
·Jeremy Wiebe on Feb 16 2016, 03:54.

Feb 13 2016

Merge branch 'format' for ExtractDate (#154) and TupleFormatter
a7b0e0b07682
·Jeremy Wiebe on Feb 13 2016, 23:01.
Use shapeless to flatten tuples of any arity
4b8f0fb7b482
·Jeremy Wiebe on Feb 13 2016, 06:48.

Feb 4 2016

Removed protocol from URLs pointing to scripts and stylesheets, to avoid mixed…
3fb632ccd905
·Jeremy Wiebe on Feb 4 2016, 21:56.
Merge pull request #194 from lintool/remove-prefix-www
3323b70e43cb
·Jeremy Wiebe on Feb 4 2016, 21:33.
Added keepLanguages filter to RecordRDD as per Issue #190
16276bc9d3d9
·Jeremy Wiebe on Feb 4 2016, 16:41.
Added keepLanguages RDD filter
30a5e5d8d585
·Jeremy Wiebe on Feb 4 2016, 16:36.

Dec 25 2015

Add RemovePrefixWWW method
1b6d83c4a7fe
·Alice-Z on Dec 25 2015, 18:11.
add method to filter date by component
4042cac0ad8b
·Alice-Z on Dec 25 2015, 17:49.
Cleanup and add comments
d1c3f5d49783
·Alice-Z on Dec 25 2015, 17:31.
Add Date utils to clarify date extraction
1ec6e05d7540
·Alice-Z on Dec 25 2015, 17:31.
Add Formatter to un-nest tuples and print in tab-delimited format
c773fd242ff5
·Alice-Z on Dec 25 2015, 17:28.

Dec 16 2015

URL change to Gephi tutorial
e2697e7df766
·ianmilligan1 on Dec 16 2015, 15:39.

Dec 14 2015

Refactoring API so that I can take individual Spark records in spark-shell and…
ba2b44c52b35
·lintool on Dec 14 2015, 03:24.

Dec 10 2015

Use SerializableWritable wrapper for serialization
0d78b9bb2b5b
·Alice-Z on Dec 10 2015, 22:05.
Use SerializableWritable wrapper for serialization
042253b579cc
·Alice-Z on Dec 10 2015, 22:03.
Refactor Record API (#189)
2adce498927d
·Alice-Z on Dec 10 2015, 21:59.

Dec 9 2015

Fix newBufferedWriter() call for JRE < 1.8.
dc99a69316bc
·Jeremy Wiebe on Dec 9 2015, 15:22.
Fixed newBufferedWriter() call for JRE < 1.8.
3e9dceac5766
·Jeremy Wiebe on Dec 9 2015, 15:21.
Export GDF to local file instead of HDFS
94d974ce3ec7
·Jeremy Wiebe on Dec 9 2015, 04:52.
Write directly to local file instead of HDFS
b9f4b0ffa199
·Jeremy Wiebe on Dec 9 2015, 04:50.

Historymasterab72ae4ce23185caafc193b…

Jul 28 2016

Jul 27 2016

Jun 29 2016

Jun 28 2016

Jun 24 2016

Jun 22 2016

Jun 17 2016

Jun 16 2016

Jun 15 2016

May 21 2016

May 20 2016

May 16 2016

May 15 2016

May 13 2016

May 11 2016

May 9 2016

May 5 2016

May 3 2016

May 2 2016

May 1 2016

Apr 29 2016

Apr 22 2016

Apr 21 2016

Apr 20 2016

Apr 18 2016

Apr 10 2016

Apr 5 2016

Apr 3 2016

Mar 31 2016

Mar 30 2016

Mar 29 2016

Mar 28 2016

Mar 27 2016

Mar 26 2016

Mar 21 2016

Mar 8 2016

Mar 5 2016

Feb 20 2016

Feb 18 2016

Feb 17 2016

Feb 16 2016

Feb 13 2016

Feb 4 2016

Dec 25 2015

Dec 16 2015

Dec 14 2015

Dec 10 2015

Dec 9 2015

History
masterab72ae4ce23185caafc193b…