Diffusion warcbase (master) History

History
master582b21adefe6e802f2a15fe…

Nov 21 2015

Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.

Nov 12 2015

Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.
Clean up string formatting
e9a3965e2389
·Alice-Z on Nov 12 2015, 15:01.
Merge branch 'ner3classifier' of github.com:lintool/warcbase into ner3classifier
d56d62cf1415
·Alice-Z on Nov 12 2015, 04:22.
Working extract entities with correct output string
9f7fa9f26b54
·Alice-Z on Nov 12 2015, 04:18.
WIP: port NER3 Classifier over to Java with example usage
c9ccf559fafc
·Alice-Z on Nov 12 2015, 02:56.
Fixed issue #157 - refactor ExtractLinks to include "from url"
c719bd5d2906
·Alice-Z on Nov 12 2015, 02:50.
WIP: port NER3 Classifier over to Java with example usage
31269caa44b6
·Alice-Z on Nov 12 2015, 00:45.

Nov 11 2015

Refactor ExtractLinks to be called with the src url
c553ef0d853f
·Alice-Z on Nov 11 2015, 23:08.
Remove extract method as per referenced in issue #146
229d274d6c13
·Alice-Z on Nov 11 2015, 22:41.
add documentation
a599db1f45b9
·Alice-Z on Nov 11 2015, 22:27.
tiny formatting change
de66d390f4ae
·Alice-Z on Nov 11 2015, 22:24.
remove extract methods
1ee0455efdf6
·Alice-Z on Nov 11 2015, 22:24.
Fixed issues #150, #149 (Pig UDFs), #156 (add keepValidPages) and #155…
80605c859501
·Alice-Z on Nov 11 2015, 22:14.
Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.
Rename tests for JUnitTestRunner
e5668e3e3251
·Alice-Z on Nov 11 2015, 22:07.

Nov 10 2015

Clean up extracting code, use pattern matching
e1be481cd782
·Alice-Z on Nov 10 2015, 02:58.

Nov 9 2015

Merge branch 'matchbox' of github.com:lintool/warcbase into matchbox
ea61e9eea405
·Alice-Z on Nov 9 2015, 18:09.
Clean up enum to function mapping
3957b2cce7fb
·Alice-Z on Nov 9 2015, 18:07.
Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.
Fixed broken build.
5a88901dd9cc
·lintool on Nov 9 2015, 03:55.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Nov 1 2015

Fixed issues #142, #152, #153
f0a2e1d1f94a
·lintool on Nov 1 2015, 17:56.
Upgraded Spark to 1.3 in CDH 5.4.1
ade10b26bca4
·lintool on Nov 1 2015, 17:53.
Completely revamped documentation.
552813e8ac5b
·lintool on Nov 1 2015, 17:43.

Oct 28 2015

Iterim progress on Issue #146 Prototype fluent Spark API for manipulating…
69c49a0f2989
·lintool on Oct 28 2015, 17:43.

Oct 23 2015

add extractCrawldateDomainUrlBody
35e39da6e72d
·Alice-Z on Oct 23 2015, 18:21.

Oct 21 2015

update ArcRecords interface
e5821091ecca
·Alice-Z on Oct 21 2015, 19:56.

Oct 15 2015

extend RDD
cfe76f600c7d
·Alice-Z on Oct 15 2015, 21:13.
Merge branch 'master' of github.com:lintool/warcbase into alicez-sparkplug
c20eecbe5a3c
·Alice-Z on Oct 15 2015, 20:09.
first commit
e77201892789
·Alice-Z on Oct 15 2015, 20:07.

Oct 7 2015

Fixed clone command per Issue #145
7c74da1b1b00
·lintool on Oct 7 2015, 19:03.

Aug 10 2015

Reference to Wiki in README.md
6308806de2bd
·Ian Milligan on Aug 10 2015, 02:22.

Jul 22 2015

Fixed issue #137 Integrate with warc-hadoop-indexer for Shine
258a159c796e
·lintool on Jul 22 2015, 18:11.
Upgraded to a stable warc-hadoop-indexer artifact (2.2.0-BETA-5)
fac3312c1785
·lintool on Jul 22 2015, 18:03.

Jul 21 2015

Complete revamped cmdline parameter handling.
17393c207934
·lintool on Jul 21 2015, 22:09.
Simplified specification of configs.
0be900c474ef
·lintool on Jul 21 2015, 21:45.
Config for WARCIndexer.
243004be2ded
·lintool on Jul 21 2015, 21:25.
Refactoring to simplify code.
b520faa88283
·lintool on Jul 21 2015, 21:25.

Jul 14 2015

Fixed: https://github.com/ukwa/webarchive-discovery/issues/64
7936dd172aae
·lintool on Jul 14 2015, 00:29.

Jul 13 2015

Copy dependencies into solr home.
34c47c6ba9bc
·lintool on Jul 13 2015, 06:46.
Rename.
38ac5fb5cba5
·lintool on Jul 13 2015, 06:46.
Upgrade CDH; fixed broken tests due classpath conflict issue and Tika upgrade.
151242d2d988
·lintool on Jul 13 2015, 05:59.
More code simplification and refactoring.
a3eb65567484
·lintool on Jul 13 2015, 03:09.
Refactoring to simplify code.
46d7cef43ee3
·lintool on Jul 13 2015, 02:39.
Reformat code.
befb3f9b39fb
·lintool on Jul 13 2015, 01:43.
Getting rid of external dependencies by checking in Solr configs.
04f416e56781
·lintool on Jul 13 2015, 00:59.
Starting to incorporate Hadoop WARC indexing code.
335d751b425c
·lintool on Jul 13 2015, 00:43.

Jun 30 2015

Added ExtractBoilerpipeText UDF
11dfe152d4cc
·Jeremy Wiebe on Jun 30 2015, 02:38.

Jun 16 2015

Fixed issue #136 Incorporate Crawl Size Visualization
04b962ac89fc
·lintool on Jun 16 2015, 08:40.

Jun 15 2015

Code cleanup, added description of vis.
12f91d394e25
·lintool on Jun 15 2015, 23:34.
Initial check-in of visualization of crawl sites.
9b5bc63a1ccf
·lintool on Jun 15 2015, 22:59.

Jun 14 2015

Fixed issue #125 Better pyspark integration (use Python dict instead of string)
96dbe38a778d
·lintool on Jun 14 2015, 22:12.
Merge branch 'master' into pyspark
c9f27489b093
·lintool on Jun 14 2015, 18:46.

Jun 9 2015

Fixed issue #135
40d0d3d8a9fc
·lintool on Jun 9 2015, 20:10.
Fixed merge conflict that got committed to master accidentally.
988a2354d84d
·lintool on Jun 9 2015, 20:10.
Fixed issues #119, #122, #130, #131
86d5ec617e52
·lintool on Jun 9 2015, 14:05.
Merge branch 'jrwiebe-master'
bbe29b4ad65d
·ianmilligan1 on Jun 9 2015, 10:22.
testing fork
51abf7e86696
·ianmilligan1 on Jun 9 2015, 10:20.
added Python scripts dir
7cb0c2e8d2cc
·Jeremy Wiebe on Jun 9 2015, 04:55.

Jun 7 2015

Fixed issues #126, #127, #128
61ca00e10cb2
·lintool on Jun 7 2015, 18:22.
Removed JWAT jars.
2f462979da0d
·lintool on Jun 7 2015, 18:18.
removed JWAT testcase.
9b8e85db50e3
·lintool on Jun 7 2015, 17:11.
Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…
d9b553c76ced
·lintool on Jun 7 2015, 17:10.
Added schema metadata in loader.
caf87fbecf45
·lintool on Jun 7 2015, 16:37.
Merge branch 'pig-warc' of github.com:lintool/warcbase into pig-warc
49fc639ae5b6
·lintool on Jun 7 2015, 16:33.
revamped tests.
195f4af7474e
·lintool on Jun 7 2015, 16:33.
better trapping of errors.
1f9d853f38a2
·lintool on Jun 7 2015, 16:31.

Jun 6 2015

Added test case for PigWarcLoader.
84ebe4f68bbd
·lintool on Jun 6 2015, 23:10.
Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

Jun 4 2015

Better integration of ARC readers in pyspark.
7c1ebde17fb4
·lintool on Jun 4 2015, 22:15.

May 30 2015

Fixed issue #120
6b17e9f5e6f1
·lintool on May 30 2015, 04:43.
whitespace
a75d58d5b0ee
·lintool on May 30 2015, 04:25.

May 27 2015

Converter to extract all metadata.
b220b0cc175a
·lintool on May 27 2015, 13:16.
Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.

May 26 2015

Added Stanford NER UDF
1c013a7b064f
·Jeremy Wiebe on May 26 2015, 18:41.
Merge remote-tracking branch 'upstream/master'
bb394dbc81cf
·Jeremy Wiebe on May 26 2015, 18:09.

May 24 2015

Fixed issues #101, #103, #115
46a6069f9b07
·lintool on May 24 2015, 20:56.
Merge branch 'patch-1' of github.com:machawk1/warcbase into feature-integration
04dffbc5db7d
·lintool on May 24 2015, 20:52.
Merge branch 'spark-integration-notes' into feature-integration
4b0b8f61c298
·lintool on May 24 2015, 20:51.
Updated README about Spark integration.
1d7b3a0e0451
·lintool on May 24 2015, 20:42.
Cleanup.
eec01a41b18e
·lintool on May 24 2015, 20:31.
Wraps a try block around everything to catch all errors.
1c9b25027e73
·lintool on May 24 2015, 20:27.
Merge branch 'master' into extract-pdf-udf
9acc8ca74c0a
·lintool on May 24 2015, 19:37.

May 21 2015

Updated README removing text about "only ARC, not WARC" limits per #64
5ea3fbd756fb
·Mat Kelly on May 21 2015, 15:59.

May 13 2015

Fixed issue #118 ExtractLinks UDF doesn't properly handle relative URLs
0a4cae2e79d7
·lintool on May 13 2015, 14:33.
tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 7 2015

Fixed issues #104, #117
063d7a4b2593
·lintool on May 7 2015, 01:43.

May 6 2015

Added test case for ExtractLinks UDF.
0a781f364480
·lintool on May 6 2015, 18:20.
Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.
UDF for extracting top-level domain from URL.
fb95f3ae562e
·lintool on May 6 2015, 01:21.

Jan 27 2015

Update README.OSX.md
451322ac2cf8
·Jeremy Wiebe on Jan 27 2015, 16:27.

Jan 21 2015

Create README.OSX.md
4dc290c46b7a
·jrwiebe on Jan 21 2015, 23:28.

Dec 25 2014

Fixed issues #88, #109, #110, #111, #112
6ef497988df1
·lintool on Dec 25 2014, 20:36.
Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112
f263769c86b7
·lintool on Dec 25 2014, 01:42.

Dec 24 2014

Fixed typo.
52b9696fba87
·lintool on Dec 24 2014, 21:41.
Add ingestion option to select either Snappy or GZ compression, Issue #110
5d471280e7db
·lintool on Dec 24 2014, 21:31.

Dec 23 2014

Renaming.
fb3edb4e40c7
·lintool on Dec 23 2014, 22:03.

Historymaster582b21adefe6e802f2a15fe…

Nov 21 2015

Nov 12 2015

Nov 11 2015

Nov 10 2015

Nov 9 2015

Nov 8 2015

Nov 3 2015

Nov 1 2015

Oct 28 2015

Oct 23 2015

Oct 21 2015

Oct 15 2015

Oct 7 2015

Aug 10 2015

Jul 22 2015

Jul 21 2015

Jul 14 2015

Jul 13 2015

Jun 30 2015

Jun 16 2015

Jun 15 2015

Jun 14 2015

Jun 9 2015

Jun 7 2015

Jun 6 2015

Jun 4 2015

May 30 2015

May 27 2015

May 26 2015

May 24 2015

May 21 2015

May 13 2015

May 7 2015

May 6 2015

Jan 27 2015

Jan 21 2015

Dec 25 2014

Dec 24 2014

Dec 23 2014

History
master582b21adefe6e802f2a15fe…