Diffusion warcbase (master) History

main
master1ee0455efdf6d3cc11ecad9…

Nov 11 2015

remove extract methods
1ee0455efdf6
·Alice-Z on Nov 11 2015, 22:24.
Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.

Nov 10 2015

Clean up extracting code, use pattern matching
e1be481cd782
·Alice-Z on Nov 10 2015, 02:58.

Nov 9 2015

Clean up enum to function mapping
3957b2cce7fb
·Alice-Z on Nov 9 2015, 18:07.
Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Oct 23 2015

add extractCrawldateDomainUrlBody
35e39da6e72d
·Alice-Z on Oct 23 2015, 18:21.

Oct 21 2015

update ArcRecords interface
e5821091ecca
·Alice-Z on Oct 21 2015, 19:56.

Oct 15 2015

extend RDD
cfe76f600c7d
·Alice-Z on Oct 15 2015, 21:13.
first commit
e77201892789
·Alice-Z on Oct 15 2015, 20:07.

Jul 22 2015

Fixed issue #137 Integrate with warc-hadoop-indexer for Shine
258a159c796e
·lintool on Jul 22 2015, 18:11.

Jul 21 2015

Complete revamped cmdline parameter handling.
17393c207934
·lintool on Jul 21 2015, 22:09.
Simplified specification of configs.
0be900c474ef
·lintool on Jul 21 2015, 21:45.
Config for WARCIndexer.
243004be2ded
·lintool on Jul 21 2015, 21:25.
Refactoring to simplify code.
b520faa88283
·lintool on Jul 21 2015, 21:25.

Jul 14 2015

Fixed: https://github.com/ukwa/webarchive-discovery/issues/64
7936dd172aae
·lintool on Jul 14 2015, 00:29.

Jul 13 2015

Rename.
38ac5fb5cba5
·lintool on Jul 13 2015, 06:46.
More code simplification and refactoring.
a3eb65567484
·lintool on Jul 13 2015, 03:09.
Refactoring to simplify code.
46d7cef43ee3
·lintool on Jul 13 2015, 02:39.
Reformat code.
befb3f9b39fb
·lintool on Jul 13 2015, 01:43.
Getting rid of external dependencies by checking in Solr configs.
04f416e56781
·lintool on Jul 13 2015, 00:59.
Starting to incorporate Hadoop WARC indexing code.
335d751b425c
·lintool on Jul 13 2015, 00:43.

Jun 30 2015

Added ExtractBoilerpipeText UDF
11dfe152d4cc
·Jeremy Wiebe on Jun 30 2015, 02:38.

Jun 14 2015

Merge branch 'master' into pyspark
c9f27489b093
·lintool on Jun 14 2015, 18:46.

Jun 9 2015

testing fork
51abf7e86696
·ianmilligan1 on Jun 9 2015, 10:20.
added Python scripts dir
7cb0c2e8d2cc
·Jeremy Wiebe on Jun 9 2015, 04:55.

Jun 7 2015

Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…
d9b553c76ced
·lintool on Jun 7 2015, 17:10.
Added schema metadata in loader.
caf87fbecf45
·lintool on Jun 7 2015, 16:37.
better trapping of errors.
1f9d853f38a2
·lintool on Jun 7 2015, 16:31.

Jun 6 2015

Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

Jun 4 2015

Better integration of ARC readers in pyspark.
7c1ebde17fb4
·lintool on Jun 4 2015, 22:15.

May 27 2015

Converter to extract all metadata.
b220b0cc175a
·lintool on May 27 2015, 13:16.
Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.

May 26 2015

Added Stanford NER UDF
1c013a7b064f
·Jeremy Wiebe on May 26 2015, 18:41.

May 24 2015

Cleanup.
eec01a41b18e
·lintool on May 24 2015, 20:31.
Wraps a try block around everything to catch all errors.
1c9b25027e73
·lintool on May 24 2015, 20:27.
Merge branch 'master' into extract-pdf-udf
9acc8ca74c0a
·lintool on May 24 2015, 19:37.

May 13 2015

tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 6 2015

Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.
UDF for extracting top-level domain from URL.
fb95f3ae562e
·lintool on May 6 2015, 01:21.

Dec 25 2014

Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112
f263769c86b7
·lintool on Dec 25 2014, 01:42.

Dec 24 2014

Fixed typo.
52b9696fba87
·lintool on Dec 24 2014, 21:41.
Add ingestion option to select either Snappy or GZ compression, Issue #110
5d471280e7db
·lintool on Dec 24 2014, 21:31.

Dec 23 2014

Renaming.
fb3edb4e40c7
·lintool on Dec 23 2014, 22:03.
Fixed OOM issues.
f6caf0f6f226
·lintool on Dec 23 2014, 22:00.
Fixed warnings.
b1c6995aa034
·lintool on Dec 23 2014, 21:21.
Fixed OOM errors.
59949b3d7af4
·lintool on Dec 23 2014, 04:59.
Fixed Issue #109: OOM when running UrlMappingBuilder
e6b21cc557d6
·lintool on Dec 23 2014, 04:01.

Dec 22 2014

Merge branch 'hbase-api-refactoring' into dev
6acf52f320cf
·lintool on Dec 22 2014, 21:34.

Dec 21 2014

Changed compression back from GZ to Snappy.
7f6d571424a9
·lintool on Dec 21 2014, 04:01.
Fixed minor code formatting issues.
12a00be93209
·lintool on Dec 21 2014, 03:58.

Dec 18 2014

Reformatting.
5c2540099370
·lintool on Dec 18 2014, 14:20.

Dec 11 2014

Updated regex for "Content-Type" (RFC2616 sec. 4.2 says HTTP header field names…
dbaa917eb8b2
·Jeremy Wiebe on Dec 11 2014, 22:26.
Use WarcRecordUtils.getWarcResponseMimeType() for ingest
329ddf9ce247
·Jeremy Wiebe on Dec 11 2014, 22:21.
Added WARC support to analysis/graph classes (except ExtractLinksJwat -- since…
8b1e82f16828
·Jeremy Wiebe on Dec 11 2014, 20:17.
Added getBodyContent(WARCRecord r) method to WarcRecordUtils
eb28dc02a8d2
·Jeremy Wiebe on Dec 11 2014, 20:15.

Dec 10 2014

Added WARC support to analysis:
ec4285807f39
·Jeremy Wiebe on Dec 10 2014, 15:54.

Dec 9 2014

Added WARC support to UrlMappingMapReduceBuilder. It can now accept a path…
65837e09aa3a
·Jeremy Wiebe on Dec 9 2014, 23:01.
ExtractTextFromPDFs updated
c49c58f76ed3
·rwolniak on Dec 9 2014, 00:38.
Updated PIG Tika Parser with new code that should work. Awaiting test on Hadoop
cea1f0cae974
·rwolniak on Dec 9 2014, 00:12.

Dec 8 2014

Added WARC support to WarcbaseResoureStore. OpenWayback replay of WARC files…
724eed27b771
·Jeremy Wiebe on Dec 8 2014, 20:26.
Added WARC file support to IngestFiles.
4239f1f9ae87
·Jeremy Wiebe on Dec 8 2014, 20:25.
Fixed bug in ExtractLinksWac where fst.loadMapping() would choke on filename…
3d8672c631db
·Jeremy Wiebe on Dec 8 2014, 20:18.
For building on OS X, changed HBase compression from Snappy to GZ.
5590850ab2f0
·Jeremy Wiebe on Dec 8 2014, 20:16.
Ooops. *Now* synced with lintool's master…
97550c3727c5
·Jeremy Wiebe on Dec 8 2014, 20:14.
Merge branch 'master' of https://github.com/lintool/warcbase
0379553cc55c
·Jeremy Wiebe on Dec 8 2014, 19:57.

Nov 12 2014

Still testing ExtractTextFromPDFs
8a3fed9342fc
·rwolniak on Nov 12 2014, 17:07.

Nov 10 2014

Extract text from PDF UDF. Currently still debugging
10b92d78f6ed
·rwolniak on Nov 10 2014, 04:07.

Oct 31 2014

Minor refactoring.
b8915343f5c9
·lintool on Oct 31 2014, 18:28.

Oct 22 2014

Refactoring to create method that extracts MIME from WARC response records.
0145d74ed35e
·lintool on Oct 22 2014, 18:42.

Oct 19 2014

Merge branch 'pig' into warc
d491020c8287
·lintool on Oct 19 2014, 19:01.
WARC-related Hadoop bindings.
410cfd81a069
·lintool on Oct 19 2014, 18:59.
Minor refactoring.
f4a249469cbc
·lintool on Oct 19 2014, 16:58.
Pig ArcLoader exports its own ResourceSchema.
0c52caed50cf
·lintool on Oct 19 2014, 02:02.
Merge branch 'master' into hbase-api-refactoring
f97523f440ff
·lintool on Oct 19 2014, 01:21.
Fixed test cases.
3af097ea655d
·lintool on Oct 19 2014, 00:50.
Pig loader materializes the actual content.
4ceefef78cdc
·lintool on Oct 19 2014, 00:47.

Oct 18 2014

Refactored Pig loaded to use WAC API.
58df7da00880
·lintool on Oct 18 2014, 17:05.

Oct 13 2014

Updated documentation.
de4267bf28f1
·lintool on Oct 13 2014, 14:21.

Sep 15 2014

Merge branch 'master' into warc
457a71345d2b
·lintool on Sep 15 2014, 01:57.

Sep 14 2014

Refactored to eliminate deprecated HBase APIs.
2b8e721063fa
·lintool on Sep 14 2014, 20:42.
Added timing info.
05db518ccd83
·lintool on Sep 14 2014, 20:13.
Added debug output; Fixed deprecated HBase APIs.
85c5b4a2ecc5
·lintool on Sep 14 2014, 19:40.

Sep 13 2014

Fixed build issues in upgrade to CDH 5.1.2.
159596e9b378
·lintool on Sep 13 2014, 03:14.

Sep 3 2014

Updated pom.xml and code to use current stable releases of Hadoop and HBase, et…
e05a7b2f92ee
·Jeremy Wiebe on Sep 3 2014, 22:53.

Aug 29 2014

Added test cases to try loading WARC records from a stream; back-ported same…
f3516c7fd7f0
·lintool on Aug 29 2014, 04:15.

Aug 28 2014

Added simple test case.
28c5c007f4fd
·lintool on Aug 28 2014, 05:48.

Aug 23 2014

Refactored WacMapReduceHBaseWrapperDemo, now takes advantage of…
9faed3817dcf
·lintool on Aug 23 2014, 18:58.
Merge branch 'working' into table-wrapper
999fa0af0280
·lintool on Aug 23 2014, 18:25.
WacArcInputFormat now generates ArcRecordWritables.
08530c1cbe52
·lintool on Aug 23 2014, 18:24.
Switched ExtractSiteLinks and InvertAnchorText over to WacArcInputFormat; link…
98d7150f5eec
·lintool on Aug 23 2014, 17:53.
Minor refactoring, revised counters.
369ba2731f5b
·lintool on Aug 23 2014, 15:40.
Both the Jwat and Wac versions of ExtractLinks gives the same exact output.
74b023ed8b16
·lintool on Aug 23 2014, 06:46.
Implemented ArcRecordWritable.
ccee8fd2204a
·lintool on Aug 23 2014, 05:25.
Minor refactoring.
846ebb216100
·lintool on Aug 23 2014, 03:46.
Very rough prototype of wrapper that allows interoperability between HBase…
c5f4b9efa212
·lintool on Aug 23 2014, 03:38.

Aug 22 2014

Added Hadoop bindings for webarchive-commons ARC readers, demo, test cases.
47f3c46c099d
·lintool on Aug 22 2014, 14:28.
Added @Override annotations to appropriate methods.
4663558b1d1a
·lintool on Aug 22 2014, 02:41.

Aug 19 2014

Minor tweaks.
9cbd26b2e3bb
·lintool on Aug 19 2014, 22:59.

R1473/src/mainmaster1ee0455efdf6d3cc11ecad9…

Nov 11 2015

Nov 10 2015

Nov 9 2015

Nov 8 2015

Nov 3 2015

Oct 23 2015

Oct 21 2015

Oct 15 2015

Jul 22 2015

Jul 21 2015

Jul 14 2015

Jul 13 2015

Jun 30 2015

Jun 14 2015

Jun 9 2015

Jun 7 2015

Jun 6 2015

Jun 4 2015

May 27 2015

May 26 2015

May 24 2015

May 13 2015

May 6 2015

Dec 25 2014

Dec 24 2014

Dec 23 2014

Dec 22 2014

Dec 21 2014

Dec 18 2014

Dec 11 2014

Dec 10 2014

Dec 9 2014

Dec 8 2014

Nov 12 2014

Nov 10 2014

Oct 31 2014

Oct 22 2014

Oct 19 2014

Oct 18 2014

Oct 13 2014

Sep 15 2014

Sep 14 2014

Sep 13 2014

Sep 3 2014

Aug 29 2014

Aug 28 2014

Aug 23 2014

Aug 22 2014

Aug 19 2014

R1473/src/main
master1ee0455efdf6d3cc11ecad9…