Diffusion warcbase (master) History

test
master2c2607867ef1ef9ffe76ddc…

Nov 11 2015

Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.
Rename tests for JUnitTestRunner
e5668e3e3251
·Alice-Z on Nov 11 2015, 22:07.

Nov 9 2015

Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Jul 13 2015

Upgrade CDH; fixed broken tests due classpath conflict issue and Tika upgrade.
151242d2d988
·lintool on Jul 13 2015, 05:59.

Jun 7 2015

removed JWAT testcase.
9b8e85db50e3
·lintool on Jun 7 2015, 17:11.
revamped tests.
195f4af7474e
·lintool on Jun 7 2015, 16:33.

Jun 6 2015

Added test case for PigWarcLoader.
84ebe4f68bbd
·lintool on Jun 6 2015, 23:10.
Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

May 13 2015

tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 6 2015

Added test case for ExtractLinks UDF.
0a781f364480
·lintool on May 6 2015, 18:20.
Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.

Oct 22 2014

Refactoring to create method that extracts MIME from WARC response records.
0145d74ed35e
·lintool on Oct 22 2014, 18:42.

Oct 19 2014

Merge branch 'pig' into warc
d491020c8287
·lintool on Oct 19 2014, 19:01.
WARC-related Hadoop bindings.
410cfd81a069
·lintool on Oct 19 2014, 18:59.
Minor refactoring.
f4a249469cbc
·lintool on Oct 19 2014, 16:58.
Pig ArcLoader exports its own ResourceSchema.
0c52caed50cf
·lintool on Oct 19 2014, 02:02.
Fixed test cases.
3af097ea655d
·lintool on Oct 19 2014, 00:50.

Aug 30 2014

Figured out how to extract MIME type and date from WARC.
4798a4314b6e
·lintool on Aug 30 2014, 03:10.

Aug 29 2014

Added test cases to try loading WARC records from a stream; back-ported same…
f3516c7fd7f0
·lintool on Aug 29 2014, 04:15.

Aug 28 2014

Added simple test case.
28c5c007f4fd
·lintool on Aug 28 2014, 05:48.
WARC sample from https://archive.org/details/ExampleArcAndWarcFiles
8e67d49d44b3
·lintool on Aug 28 2014, 05:19.

Aug 23 2014

Implemented ArcRecordWritable.
ccee8fd2204a
·lintool on Aug 23 2014, 05:25.
Minor refactoring.
846ebb216100
·lintool on Aug 23 2014, 03:46.

Aug 22 2014

Added Hadoop bindings for webarchive-commons ARC readers, demo, test cases.
47f3c46c099d
·lintool on Aug 22 2014, 14:28.
Added/refactored test cases for JWAT.
b07376aba3fe
·lintool on Aug 22 2014, 13:56.

Aug 14 2014

Refactoring UrlUtil and related classes.
bf606c07d453
·lintool on Aug 14 2014, 23:31.

Aug 12 2014

Fixed compile and broken test issues.
93f6d42f4ff4
·lintool on Aug 12 2014, 21:47.
Merge branch 'master' of github.com:cneud/warcbase into cneud-integration
accd1978862d
·lintool on Aug 12 2014, 19:38.

Jun 18 2014

issue #50
f3015cd7ba4b
·lintool on Jun 18 2014, 19:57.

Jun 17 2014

Started working on issue #46 cleanup of org.warcbase.data.Util
02c26d6d8ea4
·lintool on Jun 17 2014, 17:20.

Jun 13 2014

ExtractLinks using HBase appears to be working.
a6be4375e0c7
·lintool on Jun 13 2014, 03:08.

Jun 5 2014

Refactored getIdRange method signature, add more test cases to UriMapping.
60b827d1c174
·lintool on Jun 5 2014, 16:16.
fix issue 40 that UriMapping prefix search should return empty result when no…
7fc1b92d60a7
·Jeffyrao on Jun 5 2014, 03:37.

Jun 4 2014

Refactoring, added test case (currently broken).
a8953616248c
·lintool on Jun 4 2014, 21:12.

Mar 31 2014

Light refactoring, fixed a few errors.
758948b463b9
·lintool on Mar 31 2014, 17:56.
add prefix search feature to UriMapping class; add test class for UriMapping…
71c68ecae0fa
·Jeffyrao on Mar 31 2014, 05:32.

Dec 19 2013

Added a null pointer check and a more Pig friendly return value from the UDFs
db4ebe9f4826
·pmd on Dec 19 2013, 15:48.
Refactored the configuration of the magic lib into the Pig script.
ba201c27e210
·pmd on Dec 19 2013, 15:23.
Improved the DetectMimeTypeTika Pig script.
71b90c81859e
·pmd on Dec 19 2013, 15:14.
Corrected an error in the DetectMimeTypeMagic Pig script and the corresponding…
bedac9288080
·pmd on Dec 19 2013, 15:07.
Removed warnings
5d312a3f80ec
·pmd on Dec 19 2013, 14:29.
Refactored the DetectMimeType into two seperate methods: one for each detection…
b3fdd488fdcb
·pmd on Dec 19 2013, 14:22.

Dec 10 2013

Corrected a comment
842012e0de5f
·pmd on Dec 10 2013, 11:51.
Changed unit test to match the change in ArcLoader that removed the filter for…
e71f20141a36
·pmd on Dec 10 2013, 11:47.
Changed unit test to match the change in ArcLoader that removed the filter for…
10b02ec31780
·pmd on Dec 10 2013, 11:25.

Dec 9 2013

Use the provided ARC file for the unit test.
b8f63da4d685
·pmd on Dec 9 2013, 15:18.
Improving the unit test of the DetectMimeType by using the two identification…
25ca4df65a50
·pmd on Dec 9 2013, 15:17.
Add the magic lib UDF to the Pig script
49f50dd95873
·pmd on Dec 9 2013, 15:13.

Dec 4 2013

First version of a magic lib UDF
1a209cf5b04a
·pmd on Dec 4 2013, 14:43.

Dec 3 2013

Added unit test for the language detection UDF.
c0e996ec2e61
·pmd on Dec 3 2013, 15:03.

Dec 2 2013

Cleaned up Pig test cases, added JWAT test case.
a4282d81d6e9
·lintool on Dec 2 2013, 17:33.
added a pig unit test
17bc9616a180
·graemon on Dec 2 2013, 16:28.
added a pig unit test
8885f5db4ded
·graemon on Dec 2 2013, 14:55.

R1473/src/testmaster2c2607867ef1ef9ffe76ddc…

Nov 11 2015

Nov 9 2015

Nov 8 2015

Nov 3 2015

Jul 13 2015

Jun 7 2015

Jun 6 2015

May 13 2015

May 6 2015

Oct 22 2014

Oct 19 2014

Aug 30 2014

Aug 29 2014

Aug 28 2014

Aug 23 2014

Aug 22 2014

Aug 14 2014

Aug 12 2014

Jun 18 2014

Jun 17 2014

Jun 13 2014

Jun 5 2014

Jun 4 2014

Mar 31 2014

Dec 19 2013

Dec 10 2013

Dec 9 2013

Dec 4 2013

Dec 3 2013

Dec 2 2013

R1473/src/test
master2c2607867ef1ef9ffe76ddc…