Homec4science
Diffusion warcbase (master) Graph

R1473/src/main/scala/org
master

History Graph

CommitAuthorDetailsCommitted
3e9dceac5766Jeremy Wiebe
Fixed newBufferedWriter() call for JRE < 1.8. 
Dec 9 2015
b9f4b0ffa199Jeremy Wiebe
Write directly to local file instead of HDFS 
Dec 9 2015
76d82afa6c35Jeremy Wiebe
Added GDF export function to matchbox 
Dec 8 2015
90616dd7b3bdAlice-Z
Fix WARecord getContentString 
Dec 3 2015
43b9b1f84ef0Alice-Z
Add example scripts for crawl statistics [Issue #182] 
Dec 1 2015
ba91869cd370Alice-Z
Add example scripts for crawl statistics and social media links 
Dec 1 2015
c44118522844Jeremy Wiebe
Added matchbox function to NER-classify and generate JSON for visualizer, per… 
Nov 26 2015
32f3e4ba7d6cJeremy Wiebe
Modified structure of JSON output 
Nov 26 2015
8e8f072aca5fAlice-Z
Revision API to be more descriptive as in issue #179 
Nov 26 2015
58491a2b8a57Alice-Z
Add Apache license header 
Nov 26 2015
183b31d74831Alice-Z
Merge branch 'refactor-api' of github.com:lintool/warcbase into refactor-api 
Nov 25 2015
83d4d7232405Alice-Z
Revision API to be more descriptive 
Nov 25 2015
15ad4bb50c83Alice-Z
Revision API to be more descriptive 
Nov 25 2015
d0facdcd382fJeremy Wiebe
Make final output a single file containing single JSON array. 
Nov 25 2015
5e316b1bb129lintool
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest 
Nov 25 2015
0b0839bdbaa9lintool
Fixed CR/LF issues (i.e., DOS formatting). 
Nov 25 2015
2e88c1b19afblintool
Slapped Apache License boilerplate -- now we're a *real* open-source project :) 
Nov 25 2015
dd643f9fae5cJeremy Wiebe
Added jackson-databind to pom.xml 
Nov 25 2015
3e634fa43920Jeremy Wiebe
Relocated file, converted to class. 
Nov 24 2015
f5d8edf50506lintool
Killed all the Pig stuff. 
Nov 24 2015
201cfba2d832Jeremy Wiebe
Fixed emptyString; deleted dupe line 
Nov 23 2015
70c55839d0c3Jeremy Wiebe
Moved initialization of NER3Classifier into map closure; switched from map() to… 
Nov 23 2015
4a3d31b8babfAlice-Z
Port named entities extractor Pig script over to Spark as per issue #158 
Nov 23 2015
f9422c23efaeAlice-Z
ExtractEntities takes a classifier file path 
Nov 23 2015
99583f793bc5Alice-Z
Revert to object version of NER3Classifier 
Nov 23 2015
533b4152d534Alice-Z
Turn test off; classifier is too large to be included 
Nov 21 2015
582b21adefe6Alice-Z
Pass classifier class to ExtractEntities UDF 
Nov 21 2015
86733707e5a1Alice-Z
add test for ner3classifier 
Nov 21 2015
14e521794754Alice-Z
Clean up, fix tests changed by new keepValidPages 
Nov 21 2015
2fd98dbfdb67Alice-Z
Make keepValidPages smarter as in issue #163 
Nov 19 2015
758288fb2bd9Alice-Z
Fix warcloader bug (issue #166) 
Nov 19 2015
cc274ed73b60Alice-Z
Use Jackson JSON serializers to write to String 
Nov 12 2015
e9a3965e2389Alice-Z
Clean up string formatting 
Nov 12 2015
d56d62cf1415Alice-Z
Merge branch 'ner3classifier' of github.com:lintool/warcbase into ner3classifier 
Nov 12 2015
9f7fa9f26b54Alice-Z
Working extract entities with correct output string 
Nov 12 2015
c9ccf559fafcAlice-Z
WIP: port NER3 Classifier over to Java with example usage 
Nov 12 2015
31269caa44b6Alice-Z
WIP: port NER3 Classifier over to Java with example usage 
Nov 12 2015
c553ef0d853fAlice-Z
Refactor ExtractLinks to be called with the src url 
Nov 11 2015
229d274d6c13Alice-Z
Remove extract method as per referenced in issue #146 
Nov 11 2015
a599db1f45b9Alice-Z
add documentation 
Nov 11 2015
1ee0455efdf6Alice-Z
remove extract methods 
Nov 11 2015
2c2607867ef1Alice-Z
Add keepValidPages transformation and layer for counting in Spark 
Nov 11 2015
e1be481cd782Alice-Z
Clean up extracting code, use pattern matching 
Nov 10 2015
3957b2cce7fbAlice-Z
Clean up enum to function mapping 
Nov 9 2015
9140519a9d50Alice-Z
Fix imports and enable tests on JUnitRunner 
Nov 9 2015
3eb11b04e499Alice-Z
Commit clean-up 
Nov 8 2015
8057c46945d0Alice-Z
Add Spark support 
Nov 3 2015
35e39da6e72dAlice-Z
add extractCrawldateDomainUrlBody 
Oct 23 2015
e5821091eccaAlice-Z
update ArcRecords interface 
Oct 21 2015
cfe76f600c7dAlice-Z
extend RDD 
Oct 15 2015
e77201892789Alice-Z
first commit 
Oct 15 2015
7c1ebde17fb4lintool
Better integration of ARC readers in pyspark. 
Jun 4 2015
b220b0cc175alintool
Converter to extract all metadata. 
May 27 2015
649f1674352blintool
Initial experiments with Python converters to use Hadoop InputFormat from… 
May 27 2015
c4science · Help