Diffusion warcbase (master) History

R1473/src
master51abf7e866968a9934b04e4…

Jun 9 2015

testing fork
51abf7e86696
·ianmilligan1 on Jun 9 2015, 10:20.
added Python scripts dir
7cb0c2e8d2cc
·Jeremy Wiebe on Jun 9 2015, 04:55.

Jun 7 2015

removed JWAT testcase.
9b8e85db50e3
·lintool on Jun 7 2015, 17:11.
Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…
d9b553c76ced
·lintool on Jun 7 2015, 17:10.
Added schema metadata in loader.
caf87fbecf45
·lintool on Jun 7 2015, 16:37.
Merge branch 'pig-warc' of github.com:lintool/warcbase into pig-warc
49fc639ae5b6
·lintool on Jun 7 2015, 16:33.
revamped tests.
195f4af7474e
·lintool on Jun 7 2015, 16:33.
better trapping of errors.
1f9d853f38a2
·lintool on Jun 7 2015, 16:31.

Jun 6 2015

Added test case for PigWarcLoader.
84ebe4f68bbd
·lintool on Jun 6 2015, 23:10.
Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

May 27 2015

Converter to extract all metadata.
b220b0cc175a
·lintool on May 27 2015, 13:16.
Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.

May 26 2015

Added Stanford NER UDF
1c013a7b064f
·Jeremy Wiebe on May 26 2015, 18:41.

May 24 2015

Cleanup.
eec01a41b18e
·lintool on May 24 2015, 20:31.
Wraps a try block around everything to catch all errors.
1c9b25027e73
·lintool on May 24 2015, 20:27.
Merge branch 'master' into extract-pdf-udf
9acc8ca74c0a
·lintool on May 24 2015, 19:37.

May 13 2015

tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 6 2015

Added test case for ExtractLinks UDF.
0a781f364480
·lintool on May 6 2015, 18:20.
Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.
UDF for extracting top-level domain from URL.
fb95f3ae562e
·lintool on May 6 2015, 01:21.

Dec 25 2014

Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112
f263769c86b7
·lintool on Dec 25 2014, 01:42.

Dec 24 2014

Fixed typo.
52b9696fba87
·lintool on Dec 24 2014, 21:41.
Add ingestion option to select either Snappy or GZ compression, Issue #110
5d471280e7db
·lintool on Dec 24 2014, 21:31.

Dec 23 2014

Renaming.
fb3edb4e40c7
·lintool on Dec 23 2014, 22:03.
Fixed OOM issues.
f6caf0f6f226
·lintool on Dec 23 2014, 22:00.
Fixed warnings.
b1c6995aa034
·lintool on Dec 23 2014, 21:21.
Fixed OOM errors.
59949b3d7af4
·lintool on Dec 23 2014, 04:59.
Fixed Issue #109: OOM when running UrlMappingBuilder
e6b21cc557d6
·lintool on Dec 23 2014, 04:01.

Dec 22 2014

Merge branch 'hbase-api-refactoring' into dev
6acf52f320cf
·lintool on Dec 22 2014, 21:34.

Dec 21 2014

Changed compression back from GZ to Snappy.
7f6d571424a9
·lintool on Dec 21 2014, 04:01.
Fixed minor code formatting issues.
12a00be93209
·lintool on Dec 21 2014, 03:58.

Dec 18 2014

Reformatting.
5c2540099370
·lintool on Dec 18 2014, 14:20.

Dec 11 2014

Updated regex for "Content-Type" (RFC2616 sec. 4.2 says HTTP header field names…
dbaa917eb8b2
·Jeremy Wiebe on Dec 11 2014, 22:26.
Use WarcRecordUtils.getWarcResponseMimeType() for ingest
329ddf9ce247
·Jeremy Wiebe on Dec 11 2014, 22:21.
Added WARC support to analysis/graph classes (except ExtractLinksJwat -- since…
8b1e82f16828
·Jeremy Wiebe on Dec 11 2014, 20:17.
Added getBodyContent(WARCRecord r) method to WarcRecordUtils
eb28dc02a8d2
·Jeremy Wiebe on Dec 11 2014, 20:15.

Dec 10 2014

Added WARC support to analysis:
ec4285807f39
·Jeremy Wiebe on Dec 10 2014, 15:54.

Dec 9 2014

Added WARC support to UrlMappingMapReduceBuilder. It can now accept a path…
65837e09aa3a
·Jeremy Wiebe on Dec 9 2014, 23:01.
ExtractTextFromPDFs updated
c49c58f76ed3
·rwolniak on Dec 9 2014, 00:38.
Updated PIG Tika Parser with new code that should work. Awaiting test on Hadoop
cea1f0cae974
·rwolniak on Dec 9 2014, 00:12.

Dec 8 2014

Added WARC support to WarcbaseResoureStore. OpenWayback replay of WARC files…
724eed27b771
·Jeremy Wiebe on Dec 8 2014, 20:26.
Added WARC file support to IngestFiles.
4239f1f9ae87
·Jeremy Wiebe on Dec 8 2014, 20:25.
Fixed bug in ExtractLinksWac where fst.loadMapping() would choke on filename…
3d8672c631db
·Jeremy Wiebe on Dec 8 2014, 20:18.
For building on OS X, changed HBase compression from Snappy to GZ.
5590850ab2f0
·Jeremy Wiebe on Dec 8 2014, 20:16.
Ooops. *Now* synced with lintool's master…
97550c3727c5
·Jeremy Wiebe on Dec 8 2014, 20:14.
Merge branch 'master' of https://github.com/lintool/warcbase
0379553cc55c
·Jeremy Wiebe on Dec 8 2014, 19:57.

Nov 12 2014

Still testing ExtractTextFromPDFs
8a3fed9342fc
·rwolniak on Nov 12 2014, 17:07.

Nov 10 2014

Extract text from PDF UDF. Currently still debugging
10b92d78f6ed
·rwolniak on Nov 10 2014, 04:07.

Oct 31 2014

Minor refactoring.
b8915343f5c9
·lintool on Oct 31 2014, 18:28.

Oct 22 2014

Refactoring to create method that extracts MIME from WARC response records.
0145d74ed35e
·lintool on Oct 22 2014, 18:42.

Oct 19 2014

Merge branch 'pig' into warc
d491020c8287
·lintool on Oct 19 2014, 19:01.
WARC-related Hadoop bindings.
410cfd81a069
·lintool on Oct 19 2014, 18:59.
Minor refactoring.
f4a249469cbc
·lintool on Oct 19 2014, 16:58.
Pig ArcLoader exports its own ResourceSchema.
0c52caed50cf
·lintool on Oct 19 2014, 02:02.
Merge branch 'master' into hbase-api-refactoring
f97523f440ff
·lintool on Oct 19 2014, 01:21.
Fixed test cases.
3af097ea655d
·lintool on Oct 19 2014, 00:50.
Pig loader materializes the actual content.
4ceefef78cdc
·lintool on Oct 19 2014, 00:47.

Oct 18 2014

Refactored Pig loaded to use WAC API.
58df7da00880
·lintool on Oct 18 2014, 17:05.

Oct 13 2014

Updated documentation.
de4267bf28f1
·lintool on Oct 13 2014, 14:21.

Sep 15 2014

Merge branch 'master' into warc
457a71345d2b
·lintool on Sep 15 2014, 01:57.

Sep 14 2014

Refactored to eliminate deprecated HBase APIs.
2b8e721063fa
·lintool on Sep 14 2014, 20:42.
Added timing info.
05db518ccd83
·lintool on Sep 14 2014, 20:13.
Added debug output; Fixed deprecated HBase APIs.
85c5b4a2ecc5
·lintool on Sep 14 2014, 19:40.

Sep 13 2014

Fixed build issues in upgrade to CDH 5.1.2.
159596e9b378
·lintool on Sep 13 2014, 03:14.

Sep 3 2014

Updated pom.xml and code to use current stable releases of Hadoop and HBase, et…
e05a7b2f92ee
·Jeremy Wiebe on Sep 3 2014, 22:53.

Aug 30 2014

Figured out how to extract MIME type and date from WARC.
4798a4314b6e
·lintool on Aug 30 2014, 03:10.

Aug 29 2014

Added test cases to try loading WARC records from a stream; back-ported same…
f3516c7fd7f0
·lintool on Aug 29 2014, 04:15.

Aug 28 2014

Added simple test case.
28c5c007f4fd
·lintool on Aug 28 2014, 05:48.
WARC sample from https://archive.org/details/ExampleArcAndWarcFiles
8e67d49d44b3
·lintool on Aug 28 2014, 05:19.

Aug 23 2014

Refactored WacMapReduceHBaseWrapperDemo, now takes advantage of…
9faed3817dcf
·lintool on Aug 23 2014, 18:58.
Merge branch 'working' into table-wrapper
999fa0af0280
·lintool on Aug 23 2014, 18:25.
WacArcInputFormat now generates ArcRecordWritables.
08530c1cbe52
·lintool on Aug 23 2014, 18:24.
Switched ExtractSiteLinks and InvertAnchorText over to WacArcInputFormat; link…
98d7150f5eec
·lintool on Aug 23 2014, 17:53.
Minor refactoring, revised counters.
369ba2731f5b
·lintool on Aug 23 2014, 15:40.
Both the Jwat and Wac versions of ExtractLinks gives the same exact output.
74b023ed8b16
·lintool on Aug 23 2014, 06:46.
Implemented ArcRecordWritable.
ccee8fd2204a
·lintool on Aug 23 2014, 05:25.
Minor refactoring.
846ebb216100
·lintool on Aug 23 2014, 03:46.
Very rough prototype of wrapper that allows interoperability between HBase…
c5f4b9efa212
·lintool on Aug 23 2014, 03:38.

Aug 22 2014

Added Hadoop bindings for webarchive-commons ARC readers, demo, test cases.
47f3c46c099d
·lintool on Aug 22 2014, 14:28.
Added/refactored test cases for JWAT.
b07376aba3fe
·lintool on Aug 22 2014, 13:56.
Added @Override annotations to appropriate methods.
4663558b1d1a
·lintool on Aug 22 2014, 02:41.

Aug 19 2014

Minor tweaks.
9cbd26b2e3bb
·lintool on Aug 19 2014, 22:59.
Merge branch 'selenium' into working
43e952bfdd99
·lintool on Aug 19 2014, 22:30.
Refactoring of Wayback/Warcbase integration points.
3c7da3b094fd
·lintool on Aug 19 2014, 22:23.

Aug 17 2014

Merge branch 'master' into selenium
c5b4973aba96
·lintool on Aug 17 2014, 15:10.
Cleaned up scripts.
4090318d2f99
·lintool on Aug 17 2014, 03:37.
Cute Selenium browser to conduct a random walk through the archive.
b02fd402870f
·lintool on Aug 17 2014, 03:32.
Cleaned up programs for manipulating graphs; removed scanning HBase option for…
52cf7bb90940
·lintool on Aug 17 2014, 01:05.

Aug 16 2014

Removed outdated code; cleaned up WarcBrowserServlet.
f7eefd5da751
·lintool on Aug 16 2014, 21:27.
Refactoring: created ArcRecordUtils.
a57c6b83eaf2
·lintool on Aug 16 2014, 18:56.
Refactoring packge org.warcbase.demo; added Jwat prefix to Hadoop InputFormats…
d58afca206fc
·lintool on Aug 16 2014, 18:23.

Aug 15 2014

Better handling of ARC parse errors.
377a3e3a429f
·lintool on Aug 15 2014, 01:31.

Aug 14 2014

Bump max value size up to 10 MB, tweak WAL settings.
2d16fd032060
·lintool on Aug 14 2014, 23:33.
Refactoring UrlUtil and related classes.
bf606c07d453
·lintool on Aug 14 2014, 23:31.
Minor tweaks.
b6431bed6f8b
·lintool on Aug 14 2014, 18:09.
Merge branch 'master' into cneud-integration
72c1afbca77c
·lintool on Aug 14 2014, 17:35.
Added command-line options.
f29edf052e19
·lintool on Aug 14 2014, 17:19.
More refactoring.
14fa0f329072
·lintool on Aug 14 2014, 17:05.
Refactoring on local UrlMappingBuilder.
6eeb0ea431f1
·lintool on Aug 14 2014, 16:24.
Lightweight refactoring.
5022874b83e5
·lintool on Aug 14 2014, 14:30.

R1473/srcmaster51abf7e866968a9934b04e4…

Jun 9 2015

Jun 7 2015

Jun 6 2015

May 27 2015

May 26 2015

May 24 2015

May 13 2015

May 6 2015

Dec 25 2014

Dec 24 2014

Dec 23 2014

Dec 22 2014

Dec 21 2014

Dec 18 2014

Dec 11 2014

Dec 10 2014

Dec 9 2014

Dec 8 2014

Nov 12 2014

Nov 10 2014

Oct 31 2014

Oct 22 2014

Oct 19 2014

Oct 18 2014

Oct 13 2014

Sep 15 2014

Sep 14 2014

Sep 13 2014

Sep 3 2014

Aug 30 2014

Aug 29 2014

Aug 28 2014

Aug 23 2014

Aug 22 2014

Aug 19 2014

Aug 17 2014

Aug 16 2014

Aug 15 2014

Aug 14 2014

R1473/src
master51abf7e866968a9934b04e4…