Diffusion warcbase (master) History

History
master6b17e9f5e6f12a0770ea3ac…

May 30 2015

Fixed issue #120
6b17e9f5e6f1
·lintool on May 30 2015, 04:43.
whitespace
a75d58d5b0ee
·lintool on May 30 2015, 04:25.

May 27 2015

Converter to extract all metadata.
b220b0cc175a
·lintool on May 27 2015, 13:16.
Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.

May 24 2015

Fixed issues #101, #103, #115
46a6069f9b07
·lintool on May 24 2015, 20:56.
Merge branch 'patch-1' of github.com:machawk1/warcbase into feature-integration
04dffbc5db7d
·lintool on May 24 2015, 20:52.
Merge branch 'spark-integration-notes' into feature-integration
4b0b8f61c298
·lintool on May 24 2015, 20:51.
Updated README about Spark integration.
1d7b3a0e0451
·lintool on May 24 2015, 20:42.
Cleanup.
eec01a41b18e
·lintool on May 24 2015, 20:31.
Wraps a try block around everything to catch all errors.
1c9b25027e73
·lintool on May 24 2015, 20:27.
Merge branch 'master' into extract-pdf-udf
9acc8ca74c0a
·lintool on May 24 2015, 19:37.

May 21 2015

Updated README removing text about "only ARC, not WARC" limits per #64
5ea3fbd756fb
·Mat Kelly on May 21 2015, 15:59.

May 13 2015

Fixed issue #118 ExtractLinks UDF doesn't properly handle relative URLs
0a4cae2e79d7
·lintool on May 13 2015, 14:33.
tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 7 2015

Fixed issues #104, #117
063d7a4b2593
·lintool on May 7 2015, 01:43.

May 6 2015

Added test case for ExtractLinks UDF.
0a781f364480
·lintool on May 6 2015, 18:20.
Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.
UDF for extracting top-level domain from URL.
fb95f3ae562e
·lintool on May 6 2015, 01:21.

Dec 25 2014

Fixed issues #88, #109, #110, #111, #112
6ef497988df1
·lintool on Dec 25 2014, 20:36.
Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112
f263769c86b7
·lintool on Dec 25 2014, 01:42.

Dec 24 2014

Fixed typo.
52b9696fba87
·lintool on Dec 24 2014, 21:41.
Add ingestion option to select either Snappy or GZ compression, Issue #110
5d471280e7db
·lintool on Dec 24 2014, 21:31.

Dec 23 2014

Renaming.
fb3edb4e40c7
·lintool on Dec 23 2014, 22:03.
Fixed OOM issues.
f6caf0f6f226
·lintool on Dec 23 2014, 22:00.
Fixed warnings.
b1c6995aa034
·lintool on Dec 23 2014, 21:21.
Fixed OOM errors.
59949b3d7af4
·lintool on Dec 23 2014, 04:59.
Fixed Issue #109: OOM when running UrlMappingBuilder
e6b21cc557d6
·lintool on Dec 23 2014, 04:01.
Updated versions of some artifacts.
0fd8f3f72822
·lintool on Dec 23 2014, 02:44.

Dec 22 2014

Merge branch 'hbase-api-refactoring' into dev
6acf52f320cf
·lintool on Dec 22 2014, 21:34.
Fixed issues #64, #105, #108
e03a08388734
·lintool on Dec 22 2014, 21:30.

Dec 21 2014

Changed compression back from GZ to Snappy.
7f6d571424a9
·lintool on Dec 21 2014, 04:01.
Fixed minor code formatting issues.
12a00be93209
·lintool on Dec 21 2014, 03:58.

Dec 18 2014

Reformatting.
5c2540099370
·lintool on Dec 18 2014, 14:20.

Dec 11 2014

Updated regex for "Content-Type" (RFC2616 sec. 4.2 says HTTP header field names…
dbaa917eb8b2
·Jeremy Wiebe on Dec 11 2014, 22:26.
Use WarcRecordUtils.getWarcResponseMimeType() for ingest
329ddf9ce247
·Jeremy Wiebe on Dec 11 2014, 22:21.
Error in README
025a0a63ec02
·Jeremy Wiebe on Dec 11 2014, 20:23.
Added WARC support to analysis/graph classes (except ExtractLinksJwat -- since…
8b1e82f16828
·Jeremy Wiebe on Dec 11 2014, 20:17.
Added getBodyContent(WARCRecord r) method to WarcRecordUtils
eb28dc02a8d2
·Jeremy Wiebe on Dec 11 2014, 20:15.

Dec 10 2014

Merge branch 'master' into warc-support
c2b059444750
·Jeremy Wiebe on Dec 10 2014, 16:23.
Fixed typos in README.md
cdbada044f9d
·Jeremy Wiebe on Dec 10 2014, 16:22.
Added WARC support to analysis:
ec4285807f39
·Jeremy Wiebe on Dec 10 2014, 15:54.

Dec 9 2014

Added WARC support to UrlMappingMapReduceBuilder. It can now accept a path…
65837e09aa3a
·Jeremy Wiebe on Dec 9 2014, 23:01.
ExtractTextFromPDFs updated
c49c58f76ed3
·rwolniak on Dec 9 2014, 00:38.
Updated PIG Tika Parser with new code that should work. Awaiting test on Hadoop
cea1f0cae974
·rwolniak on Dec 9 2014, 00:12.

Dec 8 2014

Added WARC support to WarcbaseResoureStore. OpenWayback replay of WARC files…
724eed27b771
·Jeremy Wiebe on Dec 8 2014, 20:26.
Added WARC file support to IngestFiles.
4239f1f9ae87
·Jeremy Wiebe on Dec 8 2014, 20:25.
Fixed bug in ExtractLinksWac where fst.loadMapping() would choke on filename…
3d8672c631db
·Jeremy Wiebe on Dec 8 2014, 20:18.
For building on OS X, changed HBase compression from Snappy to GZ.
5590850ab2f0
·Jeremy Wiebe on Dec 8 2014, 20:16.
Ooops. *Now* synced with lintool's master…
97550c3727c5
·Jeremy Wiebe on Dec 8 2014, 20:14.
Brought fork back into sync with lintool's master branch…
5ce9524aab80
·Jeremy Wiebe on Dec 8 2014, 20:02.
Merge branch 'master' of https://github.com/lintool/warcbase
0379553cc55c
·Jeremy Wiebe on Dec 8 2014, 19:57.

Nov 12 2014

Still testing ExtractTextFromPDFs
8a3fed9342fc
·rwolniak on Nov 12 2014, 17:07.

Nov 10 2014

Extract text from PDF UDF. Currently still debugging
10b92d78f6ed
·rwolniak on Nov 10 2014, 04:07.
Merge pull request #1 from lintool/master
7e659c0eb678
·Ryan Wolniak on Nov 10 2014, 02:27.

Oct 31 2014

Fixed issues #96, #98, #99
df7f8711c64f
·lintool on Oct 31 2014, 18:32.
Minor refactoring.
b8915343f5c9
·lintool on Oct 31 2014, 18:28.

Oct 22 2014

Refactoring to create method that extracts MIME from WARC response records.
0145d74ed35e
·lintool on Oct 22 2014, 18:42.

Oct 19 2014

Merge branch 'pig' into warc
d491020c8287
·lintool on Oct 19 2014, 19:01.
WARC-related Hadoop bindings.
410cfd81a069
·lintool on Oct 19 2014, 18:59.
Minor refactoring.
f4a249469cbc
·lintool on Oct 19 2014, 16:58.
Pig ArcLoader exports its own ResourceSchema.
0c52caed50cf
·lintool on Oct 19 2014, 02:02.
Merge branch 'master' into hbase-api-refactoring
f97523f440ff
·lintool on Oct 19 2014, 01:21.
Fixed issues #93, #94, #95
e791bec61020
·lintool on Oct 19 2014, 01:17.
Added simple Pig script.
da7a94be2c70
·lintool on Oct 19 2014, 01:14.
Fixed test cases.
3af097ea655d
·lintool on Oct 19 2014, 00:50.
Pig loader materializes the actual content.
4ceefef78cdc
·lintool on Oct 19 2014, 00:47.

Oct 18 2014

Refactored Pig loaded to use WAC API.
58df7da00880
·lintool on Oct 18 2014, 17:05.
Fixed issues #90, #91, #92
63e8352c664b
·lintool on Oct 18 2014, 14:36.
Updates.
722b6402b98b
·lintool on Oct 18 2014, 14:34.
Merge branch 'master' of github.com:rwolniak/warcbase into warc
a48bb9781feb
·lintool on Oct 18 2014, 14:32.
upgraded to webarchive-commons 1.1.4.
21397e4e4ff3
·lintool on Oct 18 2014, 14:30.

Oct 13 2014

Updated Building the URL Mapping Instructions to include instruction to move…
da890acb0be1
·rwolniak on Oct 13 2014, 17:57.
Updated documentation.
de4267bf28f1
·lintool on Oct 13 2014, 14:21.

Oct 8 2014

Testing commit and push + README update
27b0038b631f
·rwolniak on Oct 8 2014, 06:01.

Sep 15 2014

Merge branch 'master' into warc
457a71345d2b
·lintool on Sep 15 2014, 01:57.

Sep 14 2014

Refactored to eliminate deprecated HBase APIs.
2b8e721063fa
·lintool on Sep 14 2014, 20:42.
Fixed issue #87
8fae0c067d68
·lintool on Sep 14 2014, 20:14.
Added timing info.
05db518ccd83
·lintool on Sep 14 2014, 20:13.
Cleanup.
157df31c0f15
·lintool on Sep 14 2014, 19:48.
Added debug output; Fixed deprecated HBase APIs.
85c5b4a2ecc5
·lintool on Sep 14 2014, 19:40.

Sep 13 2014

Fixed build issues in upgrade to CDH 5.1.2.
159596e9b378
·lintool on Sep 13 2014, 03:14.

Sep 10 2014

Fixed dependency issues and JVM memory allocation, allowing for successful…
dcffdc5f21c2
·Jeremy Wiebe on Sep 10 2014, 22:05.

Sep 3 2014

Updated pom.xml and code to use current stable releases of Hadoop and HBase, et…
e05a7b2f92ee
·Jeremy Wiebe on Sep 3 2014, 22:53.

Aug 30 2014

Figured out how to extract MIME type and date from WARC.
4798a4314b6e
·lintool on Aug 30 2014, 03:10.

Aug 29 2014

Added test cases to try loading WARC records from a stream; back-ported same…
f3516c7fd7f0
·lintool on Aug 29 2014, 04:15.

Aug 28 2014

Added simple test case.
28c5c007f4fd
·lintool on Aug 28 2014, 05:48.
WARC sample from https://archive.org/details/ExampleArcAndWarcFiles
8e67d49d44b3
·lintool on Aug 28 2014, 05:19.

Aug 23 2014

Fixed issues #86, #85, #74
fb07114090f5
·lintool on Aug 23 2014, 19:31.
Refactored WacMapReduceHBaseWrapperDemo, now takes advantage of…
9faed3817dcf
·lintool on Aug 23 2014, 18:58.
Merge branch 'working' into table-wrapper
999fa0af0280
·lintool on Aug 23 2014, 18:25.
WacArcInputFormat now generates ArcRecordWritables.
08530c1cbe52
·lintool on Aug 23 2014, 18:24.
Switched ExtractSiteLinks and InvertAnchorText over to WacArcInputFormat; link…
98d7150f5eec
·lintool on Aug 23 2014, 17:53.
Minor refactoring, revised counters.
369ba2731f5b
·lintool on Aug 23 2014, 15:40.
Both the Jwat and Wac versions of ExtractLinks gives the same exact output.
74b023ed8b16
·lintool on Aug 23 2014, 06:46.
Implemented ArcRecordWritable.
ccee8fd2204a
·lintool on Aug 23 2014, 05:25.
Fixed issues #84, #72
1e1c2dcde9e2
·lintool on Aug 23 2014, 03:58.
Minor refactoring.
846ebb216100
·lintool on Aug 23 2014, 03:46.
Very rough prototype of wrapper that allows interoperability between HBase…
c5f4b9efa212
·lintool on Aug 23 2014, 03:38.

Aug 22 2014

Added Hadoop bindings for webarchive-commons ARC readers, demo, test cases.
47f3c46c099d
·lintool on Aug 22 2014, 14:28.
Added/refactored test cases for JWAT.
b07376aba3fe
·lintool on Aug 22 2014, 13:56.

Historymaster6b17e9f5e6f12a0770ea3ac…

May 30 2015

May 27 2015

May 24 2015

May 21 2015

May 13 2015

May 7 2015

May 6 2015

Dec 25 2014

Dec 24 2014

Dec 23 2014

Dec 22 2014

Dec 21 2014

Dec 18 2014

Dec 11 2014

Dec 10 2014

Dec 9 2014

Dec 8 2014

Nov 12 2014

Nov 10 2014

Oct 31 2014

Oct 22 2014

Oct 19 2014

Oct 18 2014

Oct 13 2014

Oct 8 2014

Sep 15 2014

Sep 14 2014

Sep 13 2014

Sep 10 2014

Sep 3 2014

Aug 30 2014

Aug 29 2014

Aug 28 2014

Aug 23 2014

Aug 22 2014

History
master6b17e9f5e6f12a0770ea3ac…