Diffusion warcbase (master) History

test
masterbf893d3b5da5936c5fdc155…

Feb 16 2016

Added discardUrlPatterns
8a4c55019413
·Jeremy Wiebe on Feb 16 2016, 16:31.
keepUrlPatterns test
232397a26bf1
·Jeremy Wiebe on Feb 16 2016, 15:58.

Feb 13 2016

Merge branch 'format' for ExtractDate (#154) and TupleFormatter
a7b0e0b07682
·Jeremy Wiebe on Feb 13 2016, 23:01.
Use shapeless to flatten tuples of any arity
4b8f0fb7b482
·Jeremy Wiebe on Feb 13 2016, 06:48.

Dec 25 2015

Add RemovePrefixWWW method
1b6d83c4a7fe
·Alice-Z on Dec 25 2015, 18:11.
add method to filter date by component
4042cac0ad8b
·Alice-Z on Dec 25 2015, 17:49.
Cleanup and add comments
d1c3f5d49783
·Alice-Z on Dec 25 2015, 17:31.
Add Date utils to clarify date extraction
1ec6e05d7540
·Alice-Z on Dec 25 2015, 17:31.
Add Formatter to un-nest tuples and print in tab-delimited format
c773fd242ff5
·Alice-Z on Dec 25 2015, 17:28.

Dec 10 2015

Refactor Record API (#189)
2adce498927d
·Alice-Z on Dec 10 2015, 21:59.

Dec 3 2015

Fix WARecord getContentString
90616dd7b3bd
·Alice-Z on Dec 3 2015, 17:11.

Nov 25 2015

Revision API to be more descriptive
83d4d7232405
·Alice-Z on Nov 25 2015, 20:33.
Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest
5e316b1bb129
·lintool on Nov 25 2015, 03:49.
Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.

Nov 24 2015

Killed all the Pig stuff.
f5d8edf50506
·lintool on Nov 24 2015, 22:05.

Nov 23 2015

Tiny fix: remove unneeded part of test
2dde7b822a5c
·Alice-Z on Nov 23 2015, 02:08.
Port named entities extractor Pig script over to Spark as per issue #158
4a3d31b8babf
·Alice-Z on Nov 23 2015, 02:05.
ExtractEntities takes a classifier file path
f9422c23efae
·Alice-Z on Nov 23 2015, 02:00.
Revert to object version of NER3Classifier
99583f793bc5
·Alice-Z on Nov 23 2015, 01:56.

Nov 21 2015

Turn test off; classifier is too large to be included
533b4152d534
·Alice-Z on Nov 21 2015, 19:50.
Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.
Clean up, fix tests changed by new keepValidPages
14e521794754
·Alice-Z on Nov 21 2015, 16:33.

Nov 19 2015

Fix warcloader bug (issue #166)
758288fb2bd9
·Alice-Z on Nov 19 2015, 23:39.

Nov 12 2015

Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.
Clean up string formatting
e9a3965e2389
·Alice-Z on Nov 12 2015, 15:01.
Working extract entities with correct output string
9f7fa9f26b54
·Alice-Z on Nov 12 2015, 04:18.

Nov 11 2015

Refactor ExtractLinks to be called with the src url
c553ef0d853f
·Alice-Z on Nov 11 2015, 23:08.
tiny formatting change
de66d390f4ae
·Alice-Z on Nov 11 2015, 22:24.
Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.
Rename tests for JUnitTestRunner
e5668e3e3251
·Alice-Z on Nov 11 2015, 22:07.

Nov 9 2015

Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Jul 13 2015

Upgrade CDH; fixed broken tests due classpath conflict issue and Tika upgrade.
151242d2d988
·lintool on Jul 13 2015, 05:59.

Jun 7 2015

removed JWAT testcase.
9b8e85db50e3
·lintool on Jun 7 2015, 17:11.
revamped tests.
195f4af7474e
·lintool on Jun 7 2015, 16:33.

Jun 6 2015

Added test case for PigWarcLoader.
84ebe4f68bbd
·lintool on Jun 6 2015, 23:10.
Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

May 13 2015

tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 6 2015

Added test case for ExtractLinks UDF.
0a781f364480
·lintool on May 6 2015, 18:20.
Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.

Oct 22 2014

Refactoring to create method that extracts MIME from WARC response records.
0145d74ed35e
·lintool on Oct 22 2014, 18:42.

Oct 19 2014

Merge branch 'pig' into warc
d491020c8287
·lintool on Oct 19 2014, 19:01.
WARC-related Hadoop bindings.
410cfd81a069
·lintool on Oct 19 2014, 18:59.
Minor refactoring.
f4a249469cbc
·lintool on Oct 19 2014, 16:58.
Pig ArcLoader exports its own ResourceSchema.
0c52caed50cf
·lintool on Oct 19 2014, 02:02.
Fixed test cases.
3af097ea655d
·lintool on Oct 19 2014, 00:50.

Aug 30 2014

Figured out how to extract MIME type and date from WARC.
4798a4314b6e
·lintool on Aug 30 2014, 03:10.

Aug 29 2014

Added test cases to try loading WARC records from a stream; back-ported same…
f3516c7fd7f0
·lintool on Aug 29 2014, 04:15.

Aug 28 2014

Added simple test case.
28c5c007f4fd
·lintool on Aug 28 2014, 05:48.
WARC sample from https://archive.org/details/ExampleArcAndWarcFiles
8e67d49d44b3
·lintool on Aug 28 2014, 05:19.

Aug 23 2014

Implemented ArcRecordWritable.
ccee8fd2204a
·lintool on Aug 23 2014, 05:25.
Minor refactoring.
846ebb216100
·lintool on Aug 23 2014, 03:46.

Aug 22 2014

Added Hadoop bindings for webarchive-commons ARC readers, demo, test cases.
47f3c46c099d
·lintool on Aug 22 2014, 14:28.
Added/refactored test cases for JWAT.
b07376aba3fe
·lintool on Aug 22 2014, 13:56.

Aug 14 2014

Refactoring UrlUtil and related classes.
bf606c07d453
·lintool on Aug 14 2014, 23:31.

Aug 12 2014

Fixed compile and broken test issues.
93f6d42f4ff4
·lintool on Aug 12 2014, 21:47.
Merge branch 'master' of github.com:cneud/warcbase into cneud-integration
accd1978862d
·lintool on Aug 12 2014, 19:38.

Jun 18 2014

issue #50
f3015cd7ba4b
·lintool on Jun 18 2014, 19:57.

Jun 17 2014

Started working on issue #46 cleanup of org.warcbase.data.Util
02c26d6d8ea4
·lintool on Jun 17 2014, 17:20.

Jun 13 2014

ExtractLinks using HBase appears to be working.
a6be4375e0c7
·lintool on Jun 13 2014, 03:08.

Jun 5 2014

Refactored getIdRange method signature, add more test cases to UriMapping.
60b827d1c174
·lintool on Jun 5 2014, 16:16.
fix issue 40 that UriMapping prefix search should return empty result when no…
7fc1b92d60a7
·Jeffyrao on Jun 5 2014, 03:37.

Jun 4 2014

Refactoring, added test case (currently broken).
a8953616248c
·lintool on Jun 4 2014, 21:12.

Mar 31 2014

Light refactoring, fixed a few errors.
758948b463b9
·lintool on Mar 31 2014, 17:56.
add prefix search feature to UriMapping class; add test class for UriMapping…
71c68ecae0fa
·Jeffyrao on Mar 31 2014, 05:32.

Dec 19 2013

Added a null pointer check and a more Pig friendly return value from the UDFs
db4ebe9f4826
·pmd on Dec 19 2013, 15:48.
Refactored the configuration of the magic lib into the Pig script.
ba201c27e210
·pmd on Dec 19 2013, 15:23.
Improved the DetectMimeTypeTika Pig script.
71b90c81859e
·pmd on Dec 19 2013, 15:14.
Corrected an error in the DetectMimeTypeMagic Pig script and the corresponding…
bedac9288080
·pmd on Dec 19 2013, 15:07.
Removed warnings
5d312a3f80ec
·pmd on Dec 19 2013, 14:29.
Refactored the DetectMimeType into two seperate methods: one for each detection…
b3fdd488fdcb
·pmd on Dec 19 2013, 14:22.

Dec 10 2013

Corrected a comment
842012e0de5f
·pmd on Dec 10 2013, 11:51.
Changed unit test to match the change in ArcLoader that removed the filter for…
e71f20141a36
·pmd on Dec 10 2013, 11:47.
Changed unit test to match the change in ArcLoader that removed the filter for…
10b02ec31780
·pmd on Dec 10 2013, 11:25.

Dec 9 2013

Use the provided ARC file for the unit test.
b8f63da4d685
·pmd on Dec 9 2013, 15:18.
Improving the unit test of the DetectMimeType by using the two identification…
25ca4df65a50
·pmd on Dec 9 2013, 15:17.
Add the magic lib UDF to the Pig script
49f50dd95873
·pmd on Dec 9 2013, 15:13.

Dec 4 2013

First version of a magic lib UDF
1a209cf5b04a
·pmd on Dec 4 2013, 14:43.

Dec 3 2013

Added unit test for the language detection UDF.
c0e996ec2e61
·pmd on Dec 3 2013, 15:03.

Dec 2 2013

Cleaned up Pig test cases, added JWAT test case.
a4282d81d6e9
·lintool on Dec 2 2013, 17:33.
added a pig unit test
17bc9616a180
·graemon on Dec 2 2013, 16:28.
added a pig unit test
8885f5db4ded
·graemon on Dec 2 2013, 14:55.

R1473/src/testmasterbf893d3b5da5936c5fdc155…

Feb 16 2016

Feb 13 2016

Dec 25 2015

Dec 10 2015

Dec 3 2015

Nov 25 2015

Nov 24 2015

Nov 23 2015

Nov 21 2015

Nov 19 2015

Nov 12 2015

Nov 11 2015

Nov 9 2015

Nov 8 2015

Nov 3 2015

Jul 13 2015

Jun 7 2015

Jun 6 2015

May 13 2015

May 6 2015

Oct 22 2014

Oct 19 2014

Aug 30 2014

Aug 29 2014

Aug 28 2014

Aug 23 2014

Aug 22 2014

Aug 14 2014

Aug 12 2014

Jun 18 2014

Jun 17 2014

Jun 13 2014

Jun 5 2014

Jun 4 2014

Mar 31 2014

Dec 19 2013

Dec 10 2013

Dec 9 2013

Dec 4 2013

Dec 3 2013

Dec 2 2013

R1473/src/test
masterbf893d3b5da5936c5fdc155…