Diffusion warcbase (master) History

R1473/src/test/java/org
masterf972206db5168eb9c45ee1a…

Jun 16 2016

Created warcbase-core module.
f972206db516
·lintool on Jun 16 2016, 15:43.

Feb 18 2016

Added tests for generic ARC/WARC classes
f69c7900f12e
·Jeremy Wiebe on Feb 18 2016, 03:00.

Nov 25 2015

Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest
5e316b1bb129
·lintool on Nov 25 2015, 03:49.
Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.

Nov 24 2015

Killed all the Pig stuff.
f5d8edf50506
·lintool on Nov 24 2015, 22:05.

Jul 13 2015

Upgrade CDH; fixed broken tests due classpath conflict issue and Tika upgrade.
151242d2d988
·lintool on Jul 13 2015, 05:59.

Jun 7 2015

removed JWAT testcase.
9b8e85db50e3
·lintool on Jun 7 2015, 17:11.
revamped tests.
195f4af7474e
·lintool on Jun 7 2015, 16:33.

Jun 6 2015

Added test case for PigWarcLoader.
84ebe4f68bbd
·lintool on Jun 6 2015, 23:10.
Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

May 13 2015

tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 6 2015

Added test case for ExtractLinks UDF.
0a781f364480
·lintool on May 6 2015, 18:20.
Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.

Oct 22 2014

Refactoring to create method that extracts MIME from WARC response records.
0145d74ed35e
·lintool on Oct 22 2014, 18:42.

Oct 19 2014

WARC-related Hadoop bindings.
410cfd81a069
·lintool on Oct 19 2014, 18:59.
Minor refactoring.
f4a249469cbc
·lintool on Oct 19 2014, 16:58.
Fixed test cases.
3af097ea655d
·lintool on Oct 19 2014, 00:50.

Aug 30 2014

Figured out how to extract MIME type and date from WARC.
4798a4314b6e
·lintool on Aug 30 2014, 03:10.

Aug 29 2014

Added test cases to try loading WARC records from a stream; back-ported same…
f3516c7fd7f0
·lintool on Aug 29 2014, 04:15.

Aug 28 2014

Added simple test case.
28c5c007f4fd
·lintool on Aug 28 2014, 05:48.

Aug 23 2014

Implemented ArcRecordWritable.
ccee8fd2204a
·lintool on Aug 23 2014, 05:25.
Minor refactoring.
846ebb216100
·lintool on Aug 23 2014, 03:46.

Aug 22 2014

Added Hadoop bindings for webarchive-commons ARC readers, demo, test cases.
47f3c46c099d
·lintool on Aug 22 2014, 14:28.
Added/refactored test cases for JWAT.
b07376aba3fe
·lintool on Aug 22 2014, 13:56.

Aug 14 2014

Refactoring UrlUtil and related classes.
bf606c07d453
·lintool on Aug 14 2014, 23:31.

Aug 12 2014

Fixed compile and broken test issues.
93f6d42f4ff4
·lintool on Aug 12 2014, 21:47.
Merge branch 'master' of github.com:cneud/warcbase into cneud-integration
accd1978862d
·lintool on Aug 12 2014, 19:38.

Jun 18 2014

issue #50
f3015cd7ba4b
·lintool on Jun 18 2014, 19:57.

Jun 17 2014

Started working on issue #46 cleanup of org.warcbase.data.Util
02c26d6d8ea4
·lintool on Jun 17 2014, 17:20.

Jun 13 2014

ExtractLinks using HBase appears to be working.
a6be4375e0c7
·lintool on Jun 13 2014, 03:08.

Jun 5 2014

Refactored getIdRange method signature, add more test cases to UriMapping.
60b827d1c174
·lintool on Jun 5 2014, 16:16.
fix issue 40 that UriMapping prefix search should return empty result when no…
7fc1b92d60a7
·Jeffyrao on Jun 5 2014, 03:37.

Jun 4 2014

Refactoring, added test case (currently broken).
a8953616248c
·lintool on Jun 4 2014, 21:12.

Mar 31 2014

Light refactoring, fixed a few errors.
758948b463b9
·lintool on Mar 31 2014, 17:56.
add prefix search feature to UriMapping class; add test class for UriMapping…
71c68ecae0fa
·Jeffyrao on Mar 31 2014, 05:32.

Dec 19 2013

Added a null pointer check and a more Pig friendly return value from the UDFs
db4ebe9f4826
·pmd on Dec 19 2013, 15:48.
Corrected an error in the DetectMimeTypeMagic Pig script and the corresponding…
bedac9288080
·pmd on Dec 19 2013, 15:07.
Removed warnings
5d312a3f80ec
·pmd on Dec 19 2013, 14:29.
Refactored the DetectMimeType into two seperate methods: one for each detection…
b3fdd488fdcb
·pmd on Dec 19 2013, 14:22.

Dec 10 2013

Corrected a comment
842012e0de5f
·pmd on Dec 10 2013, 11:51.
Changed unit test to match the change in ArcLoader that removed the filter for…
e71f20141a36
·pmd on Dec 10 2013, 11:47.
Changed unit test to match the change in ArcLoader that removed the filter for…
10b02ec31780
·pmd on Dec 10 2013, 11:25.

Dec 9 2013

Use the provided ARC file for the unit test.
b8f63da4d685
·pmd on Dec 9 2013, 15:18.
Improving the unit test of the DetectMimeType by using the two identification…
25ca4df65a50
·pmd on Dec 9 2013, 15:17.

Dec 4 2013

First version of a magic lib UDF
1a209cf5b04a
·pmd on Dec 4 2013, 14:43.

Dec 3 2013

Added unit test for the language detection UDF.
c0e996ec2e61
·pmd on Dec 3 2013, 15:03.

Dec 2 2013

Cleaned up Pig test cases, added JWAT test case.
a4282d81d6e9
·lintool on Dec 2 2013, 17:33.
added a pig unit test
17bc9616a180
·graemon on Dec 2 2013, 16:28.
added a pig unit test
8885f5db4ded
·graemon on Dec 2 2013, 14:55.