Diffusion warcbase (master) History

History
master552813e8ac5baf9c7f19675…

Nov 1 2015

Completely revamped documentation.
552813e8ac5b
·lintool on Nov 1 2015, 17:43.

Oct 28 2015

Iterim progress on Issue #146 Prototype fluent Spark API for manipulating…
69c49a0f2989
·lintool on Oct 28 2015, 17:43.

Oct 23 2015

add extractCrawldateDomainUrlBody
35e39da6e72d
·Alice-Z on Oct 23 2015, 18:21.

Oct 21 2015

update ArcRecords interface
e5821091ecca
·Alice-Z on Oct 21 2015, 19:56.

Oct 15 2015

extend RDD
cfe76f600c7d
·Alice-Z on Oct 15 2015, 21:13.
Merge branch 'master' of github.com:lintool/warcbase into alicez-sparkplug
c20eecbe5a3c
·Alice-Z on Oct 15 2015, 20:09.
first commit
e77201892789
·Alice-Z on Oct 15 2015, 20:07.

Oct 7 2015

Fixed clone command per Issue #145
7c74da1b1b00
·lintool on Oct 7 2015, 19:03.

Aug 10 2015

Reference to Wiki in README.md
6308806de2bd
·Ian Milligan on Aug 10 2015, 02:22.

Jul 22 2015

Fixed issue #137 Integrate with warc-hadoop-indexer for Shine
258a159c796e
·lintool on Jul 22 2015, 18:11.
Upgraded to a stable warc-hadoop-indexer artifact (2.2.0-BETA-5)
fac3312c1785
·lintool on Jul 22 2015, 18:03.

Jul 21 2015

Complete revamped cmdline parameter handling.
17393c207934
·lintool on Jul 21 2015, 22:09.
Simplified specification of configs.
0be900c474ef
·lintool on Jul 21 2015, 21:45.
Config for WARCIndexer.
243004be2ded
·lintool on Jul 21 2015, 21:25.
Refactoring to simplify code.
b520faa88283
·lintool on Jul 21 2015, 21:25.

Jul 14 2015

Fixed: https://github.com/ukwa/webarchive-discovery/issues/64
7936dd172aae
·lintool on Jul 14 2015, 00:29.

Jul 13 2015

Copy dependencies into solr home.
34c47c6ba9bc
·lintool on Jul 13 2015, 06:46.
Rename.
38ac5fb5cba5
·lintool on Jul 13 2015, 06:46.
Upgrade CDH; fixed broken tests due classpath conflict issue and Tika upgrade.
151242d2d988
·lintool on Jul 13 2015, 05:59.
More code simplification and refactoring.
a3eb65567484
·lintool on Jul 13 2015, 03:09.
Refactoring to simplify code.
46d7cef43ee3
·lintool on Jul 13 2015, 02:39.
Reformat code.
befb3f9b39fb
·lintool on Jul 13 2015, 01:43.
Getting rid of external dependencies by checking in Solr configs.
04f416e56781
·lintool on Jul 13 2015, 00:59.
Starting to incorporate Hadoop WARC indexing code.
335d751b425c
·lintool on Jul 13 2015, 00:43.

Jun 30 2015

Added ExtractBoilerpipeText UDF
11dfe152d4cc
·Jeremy Wiebe on Jun 30 2015, 02:38.

Jun 16 2015

Fixed issue #136 Incorporate Crawl Size Visualization
04b962ac89fc
·lintool on Jun 16 2015, 08:40.

Jun 15 2015

Code cleanup, added description of vis.
12f91d394e25
·lintool on Jun 15 2015, 23:34.
Initial check-in of visualization of crawl sites.
9b5bc63a1ccf
·lintool on Jun 15 2015, 22:59.

Jun 14 2015

Fixed issue #125 Better pyspark integration (use Python dict instead of string)
96dbe38a778d
·lintool on Jun 14 2015, 22:12.
Merge branch 'master' into pyspark
c9f27489b093
·lintool on Jun 14 2015, 18:46.

Jun 9 2015

Fixed issue #135
40d0d3d8a9fc
·lintool on Jun 9 2015, 20:10.
Fixed merge conflict that got committed to master accidentally.
988a2354d84d
·lintool on Jun 9 2015, 20:10.
Fixed issues #119, #122, #130, #131
86d5ec617e52
·lintool on Jun 9 2015, 14:05.
Merge branch 'jrwiebe-master'
bbe29b4ad65d
·ianmilligan1 on Jun 9 2015, 10:22.
testing fork
51abf7e86696
·ianmilligan1 on Jun 9 2015, 10:20.
added Python scripts dir
7cb0c2e8d2cc
·Jeremy Wiebe on Jun 9 2015, 04:55.

Jun 7 2015

Fixed issues #126, #127, #128
61ca00e10cb2
·lintool on Jun 7 2015, 18:22.
Removed JWAT jars.
2f462979da0d
·lintool on Jun 7 2015, 18:18.
removed JWAT testcase.
9b8e85db50e3
·lintool on Jun 7 2015, 17:11.
Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…
d9b553c76ced
·lintool on Jun 7 2015, 17:10.
Added schema metadata in loader.
caf87fbecf45
·lintool on Jun 7 2015, 16:37.
Merge branch 'pig-warc' of github.com:lintool/warcbase into pig-warc
49fc639ae5b6
·lintool on Jun 7 2015, 16:33.
revamped tests.
195f4af7474e
·lintool on Jun 7 2015, 16:33.
better trapping of errors.
1f9d853f38a2
·lintool on Jun 7 2015, 16:31.

Jun 6 2015

Added test case for PigWarcLoader.
84ebe4f68bbd
·lintool on Jun 6 2015, 23:10.
Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

Jun 4 2015

Better integration of ARC readers in pyspark.
7c1ebde17fb4
·lintool on Jun 4 2015, 22:15.

May 30 2015

Fixed issue #120
6b17e9f5e6f1
·lintool on May 30 2015, 04:43.
whitespace
a75d58d5b0ee
·lintool on May 30 2015, 04:25.

May 27 2015

Converter to extract all metadata.
b220b0cc175a
·lintool on May 27 2015, 13:16.
Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.

May 26 2015

Added Stanford NER UDF
1c013a7b064f
·Jeremy Wiebe on May 26 2015, 18:41.
Merge remote-tracking branch 'upstream/master'
bb394dbc81cf
·Jeremy Wiebe on May 26 2015, 18:09.

May 24 2015

Fixed issues #101, #103, #115
46a6069f9b07
·lintool on May 24 2015, 20:56.
Merge branch 'patch-1' of github.com:machawk1/warcbase into feature-integration
04dffbc5db7d
·lintool on May 24 2015, 20:52.
Merge branch 'spark-integration-notes' into feature-integration
4b0b8f61c298
·lintool on May 24 2015, 20:51.
Updated README about Spark integration.
1d7b3a0e0451
·lintool on May 24 2015, 20:42.
Cleanup.
eec01a41b18e
·lintool on May 24 2015, 20:31.
Wraps a try block around everything to catch all errors.
1c9b25027e73
·lintool on May 24 2015, 20:27.
Merge branch 'master' into extract-pdf-udf
9acc8ca74c0a
·lintool on May 24 2015, 19:37.

May 21 2015

Updated README removing text about "only ARC, not WARC" limits per #64
5ea3fbd756fb
·Mat Kelly on May 21 2015, 15:59.

May 13 2015

Fixed issue #118 ExtractLinks UDF doesn't properly handle relative URLs
0a4cae2e79d7
·lintool on May 13 2015, 14:33.
tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 7 2015

Fixed issues #104, #117
063d7a4b2593
·lintool on May 7 2015, 01:43.

May 6 2015

Added test case for ExtractLinks UDF.
0a781f364480
·lintool on May 6 2015, 18:20.
Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.
UDF for extracting top-level domain from URL.
fb95f3ae562e
·lintool on May 6 2015, 01:21.

Jan 27 2015

Update README.OSX.md
451322ac2cf8
·Jeremy Wiebe on Jan 27 2015, 16:27.

Jan 21 2015

Create README.OSX.md
4dc290c46b7a
·jrwiebe on Jan 21 2015, 23:28.

Dec 25 2014

Fixed issues #88, #109, #110, #111, #112
6ef497988df1
·lintool on Dec 25 2014, 20:36.
Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112
f263769c86b7
·lintool on Dec 25 2014, 01:42.

Dec 24 2014

Fixed typo.
52b9696fba87
·lintool on Dec 24 2014, 21:41.
Add ingestion option to select either Snappy or GZ compression, Issue #110
5d471280e7db
·lintool on Dec 24 2014, 21:31.

Dec 23 2014

Renaming.
fb3edb4e40c7
·lintool on Dec 23 2014, 22:03.
Fixed OOM issues.
f6caf0f6f226
·lintool on Dec 23 2014, 22:00.
Fixed warnings.
b1c6995aa034
·lintool on Dec 23 2014, 21:21.
Fixed OOM errors.
59949b3d7af4
·lintool on Dec 23 2014, 04:59.
Fixed Issue #109: OOM when running UrlMappingBuilder
e6b21cc557d6
·lintool on Dec 23 2014, 04:01.
Updated versions of some artifacts.
0fd8f3f72822
·lintool on Dec 23 2014, 02:44.

Dec 22 2014

Merge branch 'hbase-api-refactoring' into dev
6acf52f320cf
·lintool on Dec 22 2014, 21:34.
Fixed issues #64, #105, #108
e03a08388734
·lintool on Dec 22 2014, 21:30.

Dec 21 2014

Changed compression back from GZ to Snappy.
7f6d571424a9
·lintool on Dec 21 2014, 04:01.
Fixed minor code formatting issues.
12a00be93209
·lintool on Dec 21 2014, 03:58.

Dec 18 2014

Reformatting.
5c2540099370
·lintool on Dec 18 2014, 14:20.

Dec 11 2014

Updated regex for "Content-Type" (RFC2616 sec. 4.2 says HTTP header field names…
dbaa917eb8b2
·Jeremy Wiebe on Dec 11 2014, 22:26.
Use WarcRecordUtils.getWarcResponseMimeType() for ingest
329ddf9ce247
·Jeremy Wiebe on Dec 11 2014, 22:21.
Error in README
025a0a63ec02
·Jeremy Wiebe on Dec 11 2014, 20:23.
Added WARC support to analysis/graph classes (except ExtractLinksJwat -- since…
8b1e82f16828
·Jeremy Wiebe on Dec 11 2014, 20:17.
Added getBodyContent(WARCRecord r) method to WarcRecordUtils
eb28dc02a8d2
·Jeremy Wiebe on Dec 11 2014, 20:15.

Dec 10 2014

Merge branch 'master' into warc-support
c2b059444750
·Jeremy Wiebe on Dec 10 2014, 16:23.
Fixed typos in README.md
cdbada044f9d
·Jeremy Wiebe on Dec 10 2014, 16:22.
Added WARC support to analysis:
ec4285807f39
·Jeremy Wiebe on Dec 10 2014, 15:54.

Dec 9 2014

Added WARC support to UrlMappingMapReduceBuilder. It can now accept a path…
65837e09aa3a
·Jeremy Wiebe on Dec 9 2014, 23:01.
ExtractTextFromPDFs updated
c49c58f76ed3
·rwolniak on Dec 9 2014, 00:38.
Updated PIG Tika Parser with new code that should work. Awaiting test on Hadoop
cea1f0cae974
·rwolniak on Dec 9 2014, 00:12.

Dec 8 2014

Added WARC support to WarcbaseResoureStore. OpenWayback replay of WARC files…
724eed27b771
·Jeremy Wiebe on Dec 8 2014, 20:26.
Added WARC file support to IngestFiles.
4239f1f9ae87
·Jeremy Wiebe on Dec 8 2014, 20:25.
Fixed bug in ExtractLinksWac where fst.loadMapping() would choke on filename…
3d8672c631db
·Jeremy Wiebe on Dec 8 2014, 20:18.
For building on OS X, changed HBase compression from Snappy to GZ.
5590850ab2f0
·Jeremy Wiebe on Dec 8 2014, 20:16.
Ooops. *Now* synced with lintool's master…
97550c3727c5
·Jeremy Wiebe on Dec 8 2014, 20:14.

Historymaster552813e8ac5baf9c7f19675…

Nov 1 2015

Oct 28 2015

Oct 23 2015

Oct 21 2015

Oct 15 2015

Oct 7 2015

Aug 10 2015

Jul 22 2015

Jul 21 2015

Jul 14 2015

Jul 13 2015

Jun 30 2015

Jun 16 2015

Jun 15 2015

Jun 14 2015

Jun 9 2015

Jun 7 2015

Jun 6 2015

Jun 4 2015

May 30 2015

May 27 2015

May 26 2015

May 24 2015

May 21 2015

May 13 2015

May 7 2015

May 6 2015

Jan 27 2015

Jan 21 2015

Dec 25 2014

Dec 24 2014

Dec 23 2014

Dec 22 2014

Dec 21 2014

Dec 18 2014

Dec 11 2014

Dec 10 2014

Dec 9 2014

Dec 8 2014

History
master552813e8ac5baf9c7f19675…