Homec4science
Diffusion warcbase (master) Graph

R1473/src
master

History Graph

CommitAuthorDetailsCommitted
eb4c0e313a67ianmilligan1
fixed ArcTest 
Apr 21 2016
d66f001374b1ianmilligan1
more tweaks for case sensitive getCrawlDate 
Apr 21 2016
b7a81a3afff4ianmilligan1
changing case errors in ExtractCrawlDate, checking with TravisCI 
Apr 21 2016
58b4e792a2bcianmilligan1
capitalized function getCrawlDate and getCrawlMonth 
Apr 21 2016
9ee270102a93ianmilligan1
new getCrawlMonth function 
Apr 20 2016
513c69695940lintool
Fixed compile bug, converted to .removePrefixWWW() 
Mar 29 2016
70e6c3a63ff6lintool
Merge branch 'master' into domainlinks 
Mar 29 2016
2ffc60cc178eJeremy Wiebe
Merge branch 'graphx' 
Mar 29 2016
44f3548c05cdJeremy Wiebe
Parameterized static vs dynamic PageRank. 
Mar 29 2016
fc0d11495cf1ianmilligan1
renaming ExtractTopLevelDomain to ExtractDomain 
Mar 29 2016
40025b169e93lintool
Add UDF for extracting stuff from tweets, Issue #210 
Mar 27 2016
b80c970f2e50lintool
Added test cases. 
Mar 27 2016
2da8f0abb983lintool
Added ExtractImageLinks test cases. 
Mar 27 2016
ce43a632b44flintool
Merge branch 'master' into images 
Mar 27 2016
f45f4d9fb0d3lintool
UDFs for extracting at-mentions, hashtags, and URLs from tweets. 
Mar 27 2016
c71c4dcc47f8lintool
Moved TweetJsonFunctions. 
Mar 26 2016
1d3ddbbd57d1Jeremy Wiebe
Separated graph generation (apply method) from writer, and fixed calculation of… 
Mar 21 2016
997d2405a84eJeremy Wiebe
Basic graph analysis, initial check-in 
Mar 5 2016
1c8cd5c78b1flintool
Added method to extract text. 
Mar 5 2016
93686da4ff23lintool
Downgraded json4s, folds in parsing in the loader. 
Mar 5 2016
f2abb2b5bbc2lintool
Initial pass at processing tweets in Warcbase. 
Mar 5 2016
b4e49fb40be5lintool
Added UDF for extracting image links. 
Mar 5 2016
db5c84770adbJeremy Wiebe
Update ExtractLinks.scala 
Feb 20 2016
f69c7900f12eJeremy Wiebe
Added tests for generic ARC/WARC classes 
Feb 18 2016
77891e7d54adJeremy Wiebe
Added loadArchives and supporting classes (loads WARCs or ARCs). 
Feb 17 2016
8a4c55019413Jeremy Wiebe
Added discardUrlPatterns 
Feb 16 2016
232397a26bf1Jeremy Wiebe
keepUrlPatterns test 
Feb 16 2016
bf682d8e8efcJeremy Wiebe
Added keepUrlPatterns 
Feb 16 2016
a7b0e0b07682Jeremy Wiebe
Merge branch 'format' for ExtractDate (#154) and TupleFormatter 
Feb 13 2016
4b8f0fb7b482Jeremy Wiebe
Use shapeless to flatten tuples of any arity 
Feb 13 2016
3323b70e43cbJeremy Wiebe
Merge pull request #194 from lintool/remove-prefix-www 
Feb 4 2016
30a5e5d8d585Jeremy Wiebe
Added keepLanguages RDD filter 
Feb 4 2016
1b6d83c4a7feAlice-Z
Add RemovePrefixWWW method 
Dec 25 2015
4042cac0ad8bAlice-Z
add method to filter date by component 
Dec 25 2015
d1c3f5d49783Alice-Z
Cleanup and add comments 
Dec 25 2015
1ec6e05d7540Alice-Z
Add Date utils to clarify date extraction 
Dec 25 2015
c773fd242ff5Alice-Z
Add Formatter to un-nest tuples and print in tab-delimited format 
Dec 25 2015
0d78b9bb2b5bAlice-Z
Use SerializableWritable wrapper for serialization 
Dec 10 2015
042253b579ccAlice-Z
Use SerializableWritable wrapper for serialization 
Dec 10 2015
2adce498927dAlice-Z
Refactor Record API (#189) 
Dec 10 2015
3e9dceac5766Jeremy Wiebe
Fixed newBufferedWriter() call for JRE < 1.8. 
Dec 9 2015
b9f4b0ffa199Jeremy Wiebe
Write directly to local file instead of HDFS 
Dec 9 2015
76d82afa6c35Jeremy Wiebe
Added GDF export function to matchbox 
Dec 8 2015
90616dd7b3bdAlice-Z
Fix WARecord getContentString 
Dec 3 2015
43b9b1f84ef0Alice-Z
Add example scripts for crawl statistics [Issue #182] 
Dec 1 2015
ba91869cd370Alice-Z
Add example scripts for crawl statistics and social media links 
Dec 1 2015
c44118522844Jeremy Wiebe
Added matchbox function to NER-classify and generate JSON for visualizer, per… 
Nov 26 2015
32f3e4ba7d6cJeremy Wiebe
Modified structure of JSON output 
Nov 26 2015
8e8f072aca5fAlice-Z
Revision API to be more descriptive as in issue #179 
Nov 26 2015
58491a2b8a57Alice-Z
Add Apache license header 
Nov 26 2015
183b31d74831Alice-Z
Merge branch 'refactor-api' of github.com:lintool/warcbase into refactor-api 
Nov 25 2015
83d4d7232405Alice-Z
Revision API to be more descriptive 
Nov 25 2015
15ad4bb50c83Alice-Z
Revision API to be more descriptive 
Nov 25 2015
d0facdcd382fJeremy Wiebe
Make final output a single file containing single JSON array. 
Nov 25 2015
5e316b1bb129lintool
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest 
Nov 25 2015
0b0839bdbaa9lintool
Fixed CR/LF issues (i.e., DOS formatting). 
Nov 25 2015
2e88c1b19afblintool
Slapped Apache License boilerplate -- now we're a *real* open-source project :) 
Nov 25 2015
dd643f9fae5cJeremy Wiebe
Added jackson-databind to pom.xml 
Nov 25 2015
3e634fa43920Jeremy Wiebe
Relocated file, converted to class. 
Nov 24 2015
f5d8edf50506lintool
Killed all the Pig stuff. 
Nov 24 2015
201cfba2d832Jeremy Wiebe
Fixed emptyString; deleted dupe line 
Nov 23 2015
70c55839d0c3Jeremy Wiebe
Moved initialization of NER3Classifier into map closure; switched from map() to… 
Nov 23 2015
2dde7b822a5cAlice-Z
Tiny fix: remove unneeded part of test 
Nov 23 2015
4a3d31b8babfAlice-Z
Port named entities extractor Pig script over to Spark as per issue #158 
Nov 23 2015
f9422c23efaeAlice-Z
ExtractEntities takes a classifier file path 
Nov 23 2015
99583f793bc5Alice-Z
Revert to object version of NER3Classifier 
Nov 23 2015
533b4152d534Alice-Z
Turn test off; classifier is too large to be included 
Nov 21 2015
582b21adefe6Alice-Z
Pass classifier class to ExtractEntities UDF 
Nov 21 2015
86733707e5a1Alice-Z
add test for ner3classifier 
Nov 21 2015
14e521794754Alice-Z
Clean up, fix tests changed by new keepValidPages 
Nov 21 2015
2fd98dbfdb67Alice-Z
Make keepValidPages smarter as in issue #163 
Nov 19 2015
758288fb2bd9Alice-Z
Fix warcloader bug (issue #166) 
Nov 19 2015
cc274ed73b60Alice-Z
Use Jackson JSON serializers to write to String 
Nov 12 2015
e9a3965e2389Alice-Z
Clean up string formatting 
Nov 12 2015
d56d62cf1415Alice-Z
Merge branch 'ner3classifier' of github.com:lintool/warcbase into ner3classifier 
Nov 12 2015
9f7fa9f26b54Alice-Z
Working extract entities with correct output string 
Nov 12 2015
c9ccf559fafcAlice-Z
WIP: port NER3 Classifier over to Java with example usage 
Nov 12 2015
31269caa44b6Alice-Z
WIP: port NER3 Classifier over to Java with example usage 
Nov 12 2015
c553ef0d853fAlice-Z
Refactor ExtractLinks to be called with the src url 
Nov 11 2015
229d274d6c13Alice-Z
Remove extract method as per referenced in issue #146 
Nov 11 2015
a599db1f45b9Alice-Z
add documentation 
Nov 11 2015
de66d390f4aeAlice-Z
tiny formatting change 
Nov 11 2015
1ee0455efdf6Alice-Z
remove extract methods 
Nov 11 2015
2c2607867ef1Alice-Z
Add keepValidPages transformation and layer for counting in Spark 
Nov 11 2015
e5668e3e3251Alice-Z
Rename tests for JUnitTestRunner 
Nov 11 2015
e1be481cd782Alice-Z
Clean up extracting code, use pattern matching 
Nov 10 2015
3957b2cce7fbAlice-Z
Clean up enum to function mapping 
Nov 9 2015
9140519a9d50Alice-Z
Fix imports and enable tests on JUnitRunner 
Nov 9 2015
3eb11b04e499Alice-Z
Commit clean-up 
Nov 8 2015
8057c46945d0Alice-Z
Add Spark support 
Nov 3 2015
35e39da6e72dAlice-Z
add extractCrawldateDomainUrlBody 
Oct 23 2015
e5821091eccaAlice-Z
update ArcRecords interface 
Oct 21 2015
cfe76f600c7dAlice-Z
extend RDD 
Oct 15 2015
e77201892789Alice-Z
first commit 
Oct 15 2015
258a159c796elintool
Fixed issue #137 Integrate with warc-hadoop-indexer for Shine 
Jul 22 2015
17393c207934lintool
Complete revamped cmdline parameter handling. 
Jul 21 2015
0be900c474eflintool
Simplified specification of configs. 
Jul 21 2015
243004be2dedlintool
Config for WARCIndexer. 
Jul 21 2015
b520faa88283lintool
Refactoring to simplify code. 
Jul 21 2015
7936dd172aaelintool
Fixed: https://github.com/ukwa/webarchive-discovery/issues/64 
Jul 14 2015
1
2
Next
c4science · Help