Diffusion warcbase (master) History

History
masterdd643f9fae5c93adbb06174…

Nov 25 2015

Added jackson-databind to pom.xml
dd643f9fae5c
·Jeremy Wiebe on Nov 25 2015, 03:17.

Nov 24 2015

Delete -- relocated file.
333776df87c7
·Jeremy Wiebe on Nov 24 2015, 23:18.
Relocated file, converted to class.
3e634fa43920
·Jeremy Wiebe on Nov 24 2015, 23:12.
Actually emit JSON.
ff055a163e40
·Jeremy Wiebe on Nov 24 2015, 18:08.
Initial commit: does NER classification, saves to file JSON representation per…
1ea7eed98c41
·Jeremy Wiebe on Nov 24 2015, 17:15.

Nov 23 2015

Make ExtractEntities load a new classifier per partition
924df687ce28
·Alice-Z on Nov 23 2015, 20:44.
Fixed emptyString; deleted dupe line
201cfba2d832
·Jeremy Wiebe on Nov 23 2015, 19:25.
Moved initialization of NER3Classifier into map closure; switched from map() to…
70c55839d0c3
·Jeremy Wiebe on Nov 23 2015, 19:24.
Tiny fix: remove unneeded part of test
2dde7b822a5c
·Alice-Z on Nov 23 2015, 02:08.
Port named entities extractor Pig script over to Spark as per issue #158
4a3d31b8babf
·Alice-Z on Nov 23 2015, 02:05.
ExtractEntities takes a classifier file path
f9422c23efae
·Alice-Z on Nov 23 2015, 02:00.
Revert to object version of NER3Classifier
99583f793bc5
·Alice-Z on Nov 23 2015, 01:56.

Nov 21 2015

Turn test off; classifier is too large to be included
533b4152d534
·Alice-Z on Nov 21 2015, 19:50.
Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.
Make keepValidPages smarter as in issue #163
0483269d7e35
·Alice-Z on Nov 21 2015, 16:34.
Clean up, fix tests changed by new keepValidPages
14e521794754
·Alice-Z on Nov 21 2015, 16:33.

Nov 19 2015

Make keepValidPages smarter as in issue #163
2fd98dbfdb67
·Alice-Z on Nov 19 2015, 23:49.
Merge branch 'master' of github.com:lintool/warcbase
5bf5d4edbcef
·Alice-Z on Nov 19 2015, 23:42.
Fix warcloader bug (issue #166)
43e9bb8fc751
·Alice-Z on Nov 19 2015, 23:42.
Fix warcloader bug (issue #166)
758288fb2bd9
·Alice-Z on Nov 19 2015, 23:39.

Nov 16 2015

Added link to Spark Installation instructions
4988a1a7c6df
·Ian Milligan on Nov 16 2015, 15:23.

Nov 14 2015

Updated README.
374c04a56cbe
·lintool on Nov 14 2015, 14:49.

Nov 12 2015

Copied NER visualization from @jrwiebe/WAHR/nerviz
ab75ed8f3668
·Jeremy Wiebe on Nov 12 2015, 20:28.
Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.
Clean up string formatting
e9a3965e2389
·Alice-Z on Nov 12 2015, 15:01.
Merge branch 'ner3classifier' of github.com:lintool/warcbase into ner3classifier
d56d62cf1415
·Alice-Z on Nov 12 2015, 04:22.
Working extract entities with correct output string
9f7fa9f26b54
·Alice-Z on Nov 12 2015, 04:18.
WIP: port NER3 Classifier over to Java with example usage
c9ccf559fafc
·Alice-Z on Nov 12 2015, 02:56.
Fixed issue #157 - refactor ExtractLinks to include "from url"
c719bd5d2906
·Alice-Z on Nov 12 2015, 02:50.
WIP: port NER3 Classifier over to Java with example usage
31269caa44b6
·Alice-Z on Nov 12 2015, 00:45.

Nov 11 2015

Refactor ExtractLinks to be called with the src url
c553ef0d853f
·Alice-Z on Nov 11 2015, 23:08.
Remove extract method as per referenced in issue #146
229d274d6c13
·Alice-Z on Nov 11 2015, 22:41.
add documentation
a599db1f45b9
·Alice-Z on Nov 11 2015, 22:27.
tiny formatting change
de66d390f4ae
·Alice-Z on Nov 11 2015, 22:24.
remove extract methods
1ee0455efdf6
·Alice-Z on Nov 11 2015, 22:24.
Fixed issues #150, #149 (Pig UDFs), #156 (add keepValidPages) and #155…
80605c859501
·Alice-Z on Nov 11 2015, 22:14.
Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.
Rename tests for JUnitTestRunner
e5668e3e3251
·Alice-Z on Nov 11 2015, 22:07.

Nov 10 2015

Clean up extracting code, use pattern matching
e1be481cd782
·Alice-Z on Nov 10 2015, 02:58.

Nov 9 2015

Merge branch 'matchbox' of github.com:lintool/warcbase into matchbox
ea61e9eea405
·Alice-Z on Nov 9 2015, 18:09.
Clean up enum to function mapping
3957b2cce7fb
·Alice-Z on Nov 9 2015, 18:07.
Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.
Fixed broken build.
5a88901dd9cc
·lintool on Nov 9 2015, 03:55.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Nov 1 2015

Fixed issues #142, #152, #153
f0a2e1d1f94a
·lintool on Nov 1 2015, 17:56.
Upgraded Spark to 1.3 in CDH 5.4.1
ade10b26bca4
·lintool on Nov 1 2015, 17:53.
Completely revamped documentation.
552813e8ac5b
·lintool on Nov 1 2015, 17:43.

Oct 28 2015

Iterim progress on Issue #146 Prototype fluent Spark API for manipulating…
69c49a0f2989
·lintool on Oct 28 2015, 17:43.

Oct 23 2015

add extractCrawldateDomainUrlBody
35e39da6e72d
·Alice-Z on Oct 23 2015, 18:21.

Oct 21 2015

update ArcRecords interface
e5821091ecca
·Alice-Z on Oct 21 2015, 19:56.

Oct 15 2015

extend RDD
cfe76f600c7d
·Alice-Z on Oct 15 2015, 21:13.
Merge branch 'master' of github.com:lintool/warcbase into alicez-sparkplug
c20eecbe5a3c
·Alice-Z on Oct 15 2015, 20:09.
first commit
e77201892789
·Alice-Z on Oct 15 2015, 20:07.

Oct 7 2015

Fixed clone command per Issue #145
7c74da1b1b00
·lintool on Oct 7 2015, 19:03.

Aug 10 2015

Reference to Wiki in README.md
6308806de2bd
·Ian Milligan on Aug 10 2015, 02:22.

Jul 22 2015

Fixed issue #137 Integrate with warc-hadoop-indexer for Shine
258a159c796e
·lintool on Jul 22 2015, 18:11.
Upgraded to a stable warc-hadoop-indexer artifact (2.2.0-BETA-5)
fac3312c1785
·lintool on Jul 22 2015, 18:03.

Jul 21 2015

Complete revamped cmdline parameter handling.
17393c207934
·lintool on Jul 21 2015, 22:09.
Simplified specification of configs.
0be900c474ef
·lintool on Jul 21 2015, 21:45.
Config for WARCIndexer.
243004be2ded
·lintool on Jul 21 2015, 21:25.
Refactoring to simplify code.
b520faa88283
·lintool on Jul 21 2015, 21:25.

Jul 14 2015

Fixed: https://github.com/ukwa/webarchive-discovery/issues/64
7936dd172aae
·lintool on Jul 14 2015, 00:29.

Jul 13 2015

Copy dependencies into solr home.
34c47c6ba9bc
·lintool on Jul 13 2015, 06:46.
Rename.
38ac5fb5cba5
·lintool on Jul 13 2015, 06:46.
Upgrade CDH; fixed broken tests due classpath conflict issue and Tika upgrade.
151242d2d988
·lintool on Jul 13 2015, 05:59.
More code simplification and refactoring.
a3eb65567484
·lintool on Jul 13 2015, 03:09.
Refactoring to simplify code.
46d7cef43ee3
·lintool on Jul 13 2015, 02:39.
Reformat code.
befb3f9b39fb
·lintool on Jul 13 2015, 01:43.
Getting rid of external dependencies by checking in Solr configs.
04f416e56781
·lintool on Jul 13 2015, 00:59.
Starting to incorporate Hadoop WARC indexing code.
335d751b425c
·lintool on Jul 13 2015, 00:43.

Jun 30 2015

Added ExtractBoilerpipeText UDF
11dfe152d4cc
·Jeremy Wiebe on Jun 30 2015, 02:38.

Jun 16 2015

Fixed issue #136 Incorporate Crawl Size Visualization
04b962ac89fc
·lintool on Jun 16 2015, 08:40.

Jun 15 2015

Code cleanup, added description of vis.
12f91d394e25
·lintool on Jun 15 2015, 23:34.
Initial check-in of visualization of crawl sites.
9b5bc63a1ccf
·lintool on Jun 15 2015, 22:59.

Jun 14 2015

Fixed issue #125 Better pyspark integration (use Python dict instead of string)
96dbe38a778d
·lintool on Jun 14 2015, 22:12.
Merge branch 'master' into pyspark
c9f27489b093
·lintool on Jun 14 2015, 18:46.

Jun 9 2015

Fixed issue #135
40d0d3d8a9fc
·lintool on Jun 9 2015, 20:10.
Fixed merge conflict that got committed to master accidentally.
988a2354d84d
·lintool on Jun 9 2015, 20:10.
Fixed issues #119, #122, #130, #131
86d5ec617e52
·lintool on Jun 9 2015, 14:05.
Merge branch 'jrwiebe-master'
bbe29b4ad65d
·ianmilligan1 on Jun 9 2015, 10:22.
testing fork
51abf7e86696
·ianmilligan1 on Jun 9 2015, 10:20.
added Python scripts dir
7cb0c2e8d2cc
·Jeremy Wiebe on Jun 9 2015, 04:55.

Jun 7 2015

Fixed issues #126, #127, #128
61ca00e10cb2
·lintool on Jun 7 2015, 18:22.
Removed JWAT jars.
2f462979da0d
·lintool on Jun 7 2015, 18:18.
removed JWAT testcase.
9b8e85db50e3
·lintool on Jun 7 2015, 17:11.
Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…
d9b553c76ced
·lintool on Jun 7 2015, 17:10.
Added schema metadata in loader.
caf87fbecf45
·lintool on Jun 7 2015, 16:37.
Merge branch 'pig-warc' of github.com:lintool/warcbase into pig-warc
49fc639ae5b6
·lintool on Jun 7 2015, 16:33.
revamped tests.
195f4af7474e
·lintool on Jun 7 2015, 16:33.
better trapping of errors.
1f9d853f38a2
·lintool on Jun 7 2015, 16:31.

Jun 6 2015

Added test case for PigWarcLoader.
84ebe4f68bbd
·lintool on Jun 6 2015, 23:10.
Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

Jun 4 2015

Better integration of ARC readers in pyspark.
7c1ebde17fb4
·lintool on Jun 4 2015, 22:15.

May 30 2015

Fixed issue #120
6b17e9f5e6f1
·lintool on May 30 2015, 04:43.
whitespace
a75d58d5b0ee
·lintool on May 30 2015, 04:25.

May 27 2015

Converter to extract all metadata.
b220b0cc175a
·lintool on May 27 2015, 13:16.
Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.

May 26 2015

Added Stanford NER UDF
1c013a7b064f
·Jeremy Wiebe on May 26 2015, 18:41.

Historymasterdd643f9fae5c93adbb06174…

Nov 25 2015

Nov 24 2015

Nov 23 2015

Nov 21 2015

Nov 19 2015

Nov 16 2015

Nov 14 2015

Nov 12 2015

Nov 11 2015

Nov 10 2015

Nov 9 2015

Nov 8 2015

Nov 3 2015

Nov 1 2015

Oct 28 2015

Oct 23 2015

Oct 21 2015

Oct 15 2015

Oct 7 2015

Aug 10 2015

Jul 22 2015

Jul 21 2015

Jul 14 2015

Jul 13 2015

Jun 30 2015

Jun 16 2015

Jun 15 2015

Jun 14 2015

Jun 9 2015

Jun 7 2015

Jun 6 2015

Jun 4 2015

May 30 2015

May 27 2015

May 26 2015

History
masterdd643f9fae5c93adbb06174…