Diffusion warcbase (master) History

History
masterdc99a69316bc0df0933acf4…

Dec 9 2015

Fix newBufferedWriter() call for JRE < 1.8.
dc99a69316bc
·Jeremy Wiebe on Dec 9 2015, 15:22.
Fixed newBufferedWriter() call for JRE < 1.8.
3e9dceac5766
·Jeremy Wiebe on Dec 9 2015, 15:21.
Export GDF to local file instead of HDFS
94d974ce3ec7
·Jeremy Wiebe on Dec 9 2015, 04:52.
Write directly to local file instead of HDFS
b9f4b0ffa199
·Jeremy Wiebe on Dec 9 2015, 04:50.

Dec 8 2015

Added GDF export function to matchbox
76d82afa6c35
·Jeremy Wiebe on Dec 8 2015, 05:53.

Dec 3 2015

Fixed links to new documentation: Issues #181
c026cba9e056
·lintool on Dec 3 2015, 22:34.
Fix WarcRecord as per issue #185
db8beaa3d3f5
·Alice-Z on Dec 3 2015, 17:23.
Fix WARecord getContentString
90616dd7b3bd
·Alice-Z on Dec 3 2015, 17:11.

Dec 1 2015

Add example scripts for crawl statistics [Issue #182]
43b9b1f84ef0
·Alice-Z on Dec 1 2015, 02:20.
Add example scripts for crawl statistics and social media links
ba91869cd370
·Alice-Z on Dec 1 2015, 01:18.

Nov 27 2015

Reference to warcbase-resources
e2dae6279b04
·Ian Milligan on Nov 27 2015, 03:14.

Nov 26 2015

Added matchbox function to NER-classify and generate JSON for visualizer, per…
c44118522844
·Jeremy Wiebe on Nov 26 2015, 16:05.
Modified visualizer to read JSON instead of CSV. Also added day view. Removed…
b6e1ba3a1141
·Jeremy Wiebe on Nov 26 2015, 16:04.
Modified structure of JSON output
32f3e4ba7d6c
·Jeremy Wiebe on Nov 26 2015, 15:09.
Revision API to be more descriptive as in issue #179
8e8f072aca5f
·Alice-Z on Nov 26 2015, 00:27.
Add Apache license header
58491a2b8a57
·Alice-Z on Nov 26 2015, 00:02.

Nov 25 2015

Merge branch 'refactor-api' of github.com:lintool/warcbase into refactor-api
183b31d74831
·Alice-Z on Nov 25 2015, 20:36.
Revision API to be more descriptive
83d4d7232405
·Alice-Z on Nov 25 2015, 20:33.
Revision API to be more descriptive
15ad4bb50c83
·Alice-Z on Nov 25 2015, 19:06.
Make final output a single file containing single JSON array.
d0facdcd382f
·Jeremy Wiebe on Nov 25 2015, 06:27.
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest (Issues #172, #173…
26d2ef518fb3
·lintool on Nov 25 2015, 03:53.
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest
5e316b1bb129
·lintool on Nov 25 2015, 03:49.
Fixed CR/LF issues (i.e., DOS formatting).
0b0839bdbaa9
·lintool on Nov 25 2015, 03:22.
Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.
Added jackson-databind to pom.xml
dd643f9fae5c
·Jeremy Wiebe on Nov 25 2015, 03:17.
Add Scaladoc generation info
b7dd047a8dde
·Alice Ran Zhou on Nov 25 2015, 02:46.

Nov 24 2015

Delete -- relocated file.
333776df87c7
·Jeremy Wiebe on Nov 24 2015, 23:18.
Relocated file, converted to class.
3e634fa43920
·Jeremy Wiebe on Nov 24 2015, 23:12.
SQUEAL! Killed the Pig (Issue #159)
73f232c053ba
·lintool on Nov 24 2015, 22:08.
Killed all the Pig stuff.
f5d8edf50506
·lintool on Nov 24 2015, 22:05.
Actually emit JSON.
ff055a163e40
·Jeremy Wiebe on Nov 24 2015, 18:08.
Initial commit: does NER classification, saves to file JSON representation per…
1ea7eed98c41
·Jeremy Wiebe on Nov 24 2015, 17:15.

Nov 23 2015

Make ExtractEntities load a new classifier per partition
924df687ce28
·Alice-Z on Nov 23 2015, 20:44.
Fixed emptyString; deleted dupe line
201cfba2d832
·Jeremy Wiebe on Nov 23 2015, 19:25.
Moved initialization of NER3Classifier into map closure; switched from map() to…
70c55839d0c3
·Jeremy Wiebe on Nov 23 2015, 19:24.
Tiny fix: remove unneeded part of test
2dde7b822a5c
·Alice-Z on Nov 23 2015, 02:08.
Port named entities extractor Pig script over to Spark as per issue #158
4a3d31b8babf
·Alice-Z on Nov 23 2015, 02:05.
ExtractEntities takes a classifier file path
f9422c23efae
·Alice-Z on Nov 23 2015, 02:00.
Revert to object version of NER3Classifier
99583f793bc5
·Alice-Z on Nov 23 2015, 01:56.

Nov 21 2015

Turn test off; classifier is too large to be included
533b4152d534
·Alice-Z on Nov 21 2015, 19:50.
Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.
Make keepValidPages smarter as in issue #163
0483269d7e35
·Alice-Z on Nov 21 2015, 16:34.
Clean up, fix tests changed by new keepValidPages
14e521794754
·Alice-Z on Nov 21 2015, 16:33.

Nov 19 2015

Make keepValidPages smarter as in issue #163
2fd98dbfdb67
·Alice-Z on Nov 19 2015, 23:49.
Merge branch 'master' of github.com:lintool/warcbase
5bf5d4edbcef
·Alice-Z on Nov 19 2015, 23:42.
Fix warcloader bug (issue #166)
43e9bb8fc751
·Alice-Z on Nov 19 2015, 23:42.
Fix warcloader bug (issue #166)
758288fb2bd9
·Alice-Z on Nov 19 2015, 23:39.

Nov 16 2015

Added link to Spark Installation instructions
4988a1a7c6df
·Ian Milligan on Nov 16 2015, 15:23.

Nov 14 2015

Updated README.
374c04a56cbe
·lintool on Nov 14 2015, 14:49.

Nov 12 2015

Copied NER visualization from @jrwiebe/WAHR/nerviz
ab75ed8f3668
·Jeremy Wiebe on Nov 12 2015, 20:28.
Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.
Clean up string formatting
e9a3965e2389
·Alice-Z on Nov 12 2015, 15:01.
Merge branch 'ner3classifier' of github.com:lintool/warcbase into ner3classifier
d56d62cf1415
·Alice-Z on Nov 12 2015, 04:22.
Working extract entities with correct output string
9f7fa9f26b54
·Alice-Z on Nov 12 2015, 04:18.
WIP: port NER3 Classifier over to Java with example usage
c9ccf559fafc
·Alice-Z on Nov 12 2015, 02:56.
Fixed issue #157 - refactor ExtractLinks to include "from url"
c719bd5d2906
·Alice-Z on Nov 12 2015, 02:50.
WIP: port NER3 Classifier over to Java with example usage
31269caa44b6
·Alice-Z on Nov 12 2015, 00:45.

Nov 11 2015

Refactor ExtractLinks to be called with the src url
c553ef0d853f
·Alice-Z on Nov 11 2015, 23:08.
Remove extract method as per referenced in issue #146
229d274d6c13
·Alice-Z on Nov 11 2015, 22:41.
add documentation
a599db1f45b9
·Alice-Z on Nov 11 2015, 22:27.
tiny formatting change
de66d390f4ae
·Alice-Z on Nov 11 2015, 22:24.
remove extract methods
1ee0455efdf6
·Alice-Z on Nov 11 2015, 22:24.
Fixed issues #150, #149 (Pig UDFs), #156 (add keepValidPages) and #155…
80605c859501
·Alice-Z on Nov 11 2015, 22:14.
Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.
Rename tests for JUnitTestRunner
e5668e3e3251
·Alice-Z on Nov 11 2015, 22:07.

Nov 10 2015

Clean up extracting code, use pattern matching
e1be481cd782
·Alice-Z on Nov 10 2015, 02:58.

Nov 9 2015

Merge branch 'matchbox' of github.com:lintool/warcbase into matchbox
ea61e9eea405
·Alice-Z on Nov 9 2015, 18:09.
Clean up enum to function mapping
3957b2cce7fb
·Alice-Z on Nov 9 2015, 18:07.
Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.
Fixed broken build.
5a88901dd9cc
·lintool on Nov 9 2015, 03:55.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Nov 1 2015

Fixed issues #142, #152, #153
f0a2e1d1f94a
·lintool on Nov 1 2015, 17:56.
Upgraded Spark to 1.3 in CDH 5.4.1
ade10b26bca4
·lintool on Nov 1 2015, 17:53.
Completely revamped documentation.
552813e8ac5b
·lintool on Nov 1 2015, 17:43.

Oct 28 2015

Iterim progress on Issue #146 Prototype fluent Spark API for manipulating…
69c49a0f2989
·lintool on Oct 28 2015, 17:43.

Oct 23 2015

add extractCrawldateDomainUrlBody
35e39da6e72d
·Alice-Z on Oct 23 2015, 18:21.

Oct 21 2015

update ArcRecords interface
e5821091ecca
·Alice-Z on Oct 21 2015, 19:56.

Oct 15 2015

extend RDD
cfe76f600c7d
·Alice-Z on Oct 15 2015, 21:13.
Merge branch 'master' of github.com:lintool/warcbase into alicez-sparkplug
c20eecbe5a3c
·Alice-Z on Oct 15 2015, 20:09.
first commit
e77201892789
·Alice-Z on Oct 15 2015, 20:07.

Oct 7 2015

Fixed clone command per Issue #145
7c74da1b1b00
·lintool on Oct 7 2015, 19:03.

Aug 10 2015

Reference to Wiki in README.md
6308806de2bd
·Ian Milligan on Aug 10 2015, 02:22.

Jul 22 2015

Fixed issue #137 Integrate with warc-hadoop-indexer for Shine
258a159c796e
·lintool on Jul 22 2015, 18:11.
Upgraded to a stable warc-hadoop-indexer artifact (2.2.0-BETA-5)
fac3312c1785
·lintool on Jul 22 2015, 18:03.

Jul 21 2015

Complete revamped cmdline parameter handling.
17393c207934
·lintool on Jul 21 2015, 22:09.
Simplified specification of configs.
0be900c474ef
·lintool on Jul 21 2015, 21:45.
Config for WARCIndexer.
243004be2ded
·lintool on Jul 21 2015, 21:25.
Refactoring to simplify code.
b520faa88283
·lintool on Jul 21 2015, 21:25.

Jul 14 2015

Fixed: https://github.com/ukwa/webarchive-discovery/issues/64
7936dd172aae
·lintool on Jul 14 2015, 00:29.

Jul 13 2015

Copy dependencies into solr home.
34c47c6ba9bc
·lintool on Jul 13 2015, 06:46.
Rename.
38ac5fb5cba5
·lintool on Jul 13 2015, 06:46.
Upgrade CDH; fixed broken tests due classpath conflict issue and Tika upgrade.
151242d2d988
·lintool on Jul 13 2015, 05:59.
More code simplification and refactoring.
a3eb65567484
·lintool on Jul 13 2015, 03:09.
Refactoring to simplify code.
46d7cef43ee3
·lintool on Jul 13 2015, 02:39.
Reformat code.
befb3f9b39fb
·lintool on Jul 13 2015, 01:43.
Getting rid of external dependencies by checking in Solr configs.
04f416e56781
·lintool on Jul 13 2015, 00:59.
Starting to incorporate Hadoop WARC indexing code.
335d751b425c
·lintool on Jul 13 2015, 00:43.

Jun 30 2015

Added ExtractBoilerpipeText UDF
11dfe152d4cc
·Jeremy Wiebe on Jun 30 2015, 02:38.

Historymasterdc99a69316bc0df0933acf4…

Dec 9 2015

Dec 8 2015

Dec 3 2015

Dec 1 2015

Nov 27 2015

Nov 26 2015

Nov 25 2015

Nov 24 2015

Nov 23 2015

Nov 21 2015

Nov 19 2015

Nov 16 2015

Nov 14 2015

Nov 12 2015

Nov 11 2015

Nov 10 2015

Nov 9 2015

Nov 8 2015

Nov 3 2015

Nov 1 2015

Oct 28 2015

Oct 23 2015

Oct 21 2015

Oct 15 2015

Oct 7 2015

Aug 10 2015

Jul 22 2015

Jul 21 2015

Jul 14 2015

Jul 13 2015

Jun 30 2015

History
masterdc99a69316bc0df0933acf4…