Diffusion warcbase (master) History

History
master9b5bc63a1ccf3b36e8fea3b…

Jun 15 2015

Initial check-in of visualization of crawl sites.
9b5bc63a1ccf
·lintool on Jun 15 2015, 22:59.

Jun 14 2015

Fixed issue #125 Better pyspark integration (use Python dict instead of string)
96dbe38a778d
·lintool on Jun 14 2015, 22:12.
Merge branch 'master' into pyspark
c9f27489b093
·lintool on Jun 14 2015, 18:46.

Jun 9 2015

Fixed issue #135
40d0d3d8a9fc
·lintool on Jun 9 2015, 20:10.
Fixed merge conflict that got committed to master accidentally.
988a2354d84d
·lintool on Jun 9 2015, 20:10.
Fixed issues #119, #122, #130, #131
86d5ec617e52
·lintool on Jun 9 2015, 14:05.
Merge branch 'jrwiebe-master'
bbe29b4ad65d
·ianmilligan1 on Jun 9 2015, 10:22.
testing fork
51abf7e86696
·ianmilligan1 on Jun 9 2015, 10:20.
added Python scripts dir
7cb0c2e8d2cc
·Jeremy Wiebe on Jun 9 2015, 04:55.

Jun 7 2015

Fixed issues #126, #127, #128
61ca00e10cb2
·lintool on Jun 7 2015, 18:22.
Removed JWAT jars.
2f462979da0d
·lintool on Jun 7 2015, 18:18.
removed JWAT testcase.
9b8e85db50e3
·lintool on Jun 7 2015, 17:11.
Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…
d9b553c76ced
·lintool on Jun 7 2015, 17:10.
Added schema metadata in loader.
caf87fbecf45
·lintool on Jun 7 2015, 16:37.
Merge branch 'pig-warc' of github.com:lintool/warcbase into pig-warc
49fc639ae5b6
·lintool on Jun 7 2015, 16:33.
revamped tests.
195f4af7474e
·lintool on Jun 7 2015, 16:33.
better trapping of errors.
1f9d853f38a2
·lintool on Jun 7 2015, 16:31.

Jun 6 2015

Added test case for PigWarcLoader.
84ebe4f68bbd
·lintool on Jun 6 2015, 23:10.
Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

Jun 4 2015

Better integration of ARC readers in pyspark.
7c1ebde17fb4
·lintool on Jun 4 2015, 22:15.

May 30 2015

Fixed issue #120
6b17e9f5e6f1
·lintool on May 30 2015, 04:43.
whitespace
a75d58d5b0ee
·lintool on May 30 2015, 04:25.

May 27 2015

Converter to extract all metadata.
b220b0cc175a
·lintool on May 27 2015, 13:16.
Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.

May 26 2015

Added Stanford NER UDF
1c013a7b064f
·Jeremy Wiebe on May 26 2015, 18:41.
Merge remote-tracking branch 'upstream/master'
bb394dbc81cf
·Jeremy Wiebe on May 26 2015, 18:09.

May 24 2015

Fixed issues #101, #103, #115
46a6069f9b07
·lintool on May 24 2015, 20:56.
Merge branch 'patch-1' of github.com:machawk1/warcbase into feature-integration
04dffbc5db7d
·lintool on May 24 2015, 20:52.
Merge branch 'spark-integration-notes' into feature-integration
4b0b8f61c298
·lintool on May 24 2015, 20:51.
Updated README about Spark integration.
1d7b3a0e0451
·lintool on May 24 2015, 20:42.
Cleanup.
eec01a41b18e
·lintool on May 24 2015, 20:31.
Wraps a try block around everything to catch all errors.
1c9b25027e73
·lintool on May 24 2015, 20:27.
Merge branch 'master' into extract-pdf-udf
9acc8ca74c0a
·lintool on May 24 2015, 19:37.

May 21 2015

Updated README removing text about "only ARC, not WARC" limits per #64
5ea3fbd756fb
·Mat Kelly on May 21 2015, 15:59.

May 13 2015

Fixed issue #118 ExtractLinks UDF doesn't properly handle relative URLs
0a4cae2e79d7
·lintool on May 13 2015, 14:33.
tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 7 2015

Fixed issues #104, #117
063d7a4b2593
·lintool on May 7 2015, 01:43.

May 6 2015

Added test case for ExtractLinks UDF.
0a781f364480
·lintool on May 6 2015, 18:20.
Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.
UDF for extracting top-level domain from URL.
fb95f3ae562e
·lintool on May 6 2015, 01:21.

Jan 27 2015

Update README.OSX.md
451322ac2cf8
·Jeremy Wiebe on Jan 27 2015, 16:27.

Jan 21 2015

Create README.OSX.md
4dc290c46b7a
·jrwiebe on Jan 21 2015, 23:28.

Dec 25 2014

Fixed issues #88, #109, #110, #111, #112
6ef497988df1
·lintool on Dec 25 2014, 20:36.
Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112
f263769c86b7
·lintool on Dec 25 2014, 01:42.

Dec 24 2014

Fixed typo.
52b9696fba87
·lintool on Dec 24 2014, 21:41.
Add ingestion option to select either Snappy or GZ compression, Issue #110
5d471280e7db
·lintool on Dec 24 2014, 21:31.

Dec 23 2014

Renaming.
fb3edb4e40c7
·lintool on Dec 23 2014, 22:03.
Fixed OOM issues.
f6caf0f6f226
·lintool on Dec 23 2014, 22:00.
Fixed warnings.
b1c6995aa034
·lintool on Dec 23 2014, 21:21.
Fixed OOM errors.
59949b3d7af4
·lintool on Dec 23 2014, 04:59.
Fixed Issue #109: OOM when running UrlMappingBuilder
e6b21cc557d6
·lintool on Dec 23 2014, 04:01.
Updated versions of some artifacts.
0fd8f3f72822
·lintool on Dec 23 2014, 02:44.

Dec 22 2014

Merge branch 'hbase-api-refactoring' into dev
6acf52f320cf
·lintool on Dec 22 2014, 21:34.
Fixed issues #64, #105, #108
e03a08388734
·lintool on Dec 22 2014, 21:30.

Dec 21 2014

Changed compression back from GZ to Snappy.
7f6d571424a9
·lintool on Dec 21 2014, 04:01.
Fixed minor code formatting issues.
12a00be93209
·lintool on Dec 21 2014, 03:58.

Dec 18 2014

Reformatting.
5c2540099370
·lintool on Dec 18 2014, 14:20.

Dec 11 2014

Updated regex for "Content-Type" (RFC2616 sec. 4.2 says HTTP header field names…
dbaa917eb8b2
·Jeremy Wiebe on Dec 11 2014, 22:26.
Use WarcRecordUtils.getWarcResponseMimeType() for ingest
329ddf9ce247
·Jeremy Wiebe on Dec 11 2014, 22:21.
Error in README
025a0a63ec02
·Jeremy Wiebe on Dec 11 2014, 20:23.
Added WARC support to analysis/graph classes (except ExtractLinksJwat -- since…
8b1e82f16828
·Jeremy Wiebe on Dec 11 2014, 20:17.
Added getBodyContent(WARCRecord r) method to WarcRecordUtils
eb28dc02a8d2
·Jeremy Wiebe on Dec 11 2014, 20:15.

Dec 10 2014

Merge branch 'master' into warc-support
c2b059444750
·Jeremy Wiebe on Dec 10 2014, 16:23.
Fixed typos in README.md
cdbada044f9d
·Jeremy Wiebe on Dec 10 2014, 16:22.
Added WARC support to analysis:
ec4285807f39
·Jeremy Wiebe on Dec 10 2014, 15:54.

Dec 9 2014

Added WARC support to UrlMappingMapReduceBuilder. It can now accept a path…
65837e09aa3a
·Jeremy Wiebe on Dec 9 2014, 23:01.
ExtractTextFromPDFs updated
c49c58f76ed3
·rwolniak on Dec 9 2014, 00:38.
Updated PIG Tika Parser with new code that should work. Awaiting test on Hadoop
cea1f0cae974
·rwolniak on Dec 9 2014, 00:12.

Dec 8 2014

Added WARC support to WarcbaseResoureStore. OpenWayback replay of WARC files…
724eed27b771
·Jeremy Wiebe on Dec 8 2014, 20:26.
Added WARC file support to IngestFiles.
4239f1f9ae87
·Jeremy Wiebe on Dec 8 2014, 20:25.
Fixed bug in ExtractLinksWac where fst.loadMapping() would choke on filename…
3d8672c631db
·Jeremy Wiebe on Dec 8 2014, 20:18.
For building on OS X, changed HBase compression from Snappy to GZ.
5590850ab2f0
·Jeremy Wiebe on Dec 8 2014, 20:16.
Ooops. *Now* synced with lintool's master…
97550c3727c5
·Jeremy Wiebe on Dec 8 2014, 20:14.
Brought fork back into sync with lintool's master branch…
5ce9524aab80
·Jeremy Wiebe on Dec 8 2014, 20:02.
Merge branch 'master' of https://github.com/lintool/warcbase
0379553cc55c
·Jeremy Wiebe on Dec 8 2014, 19:57.

Nov 12 2014

Still testing ExtractTextFromPDFs
8a3fed9342fc
·rwolniak on Nov 12 2014, 17:07.

Nov 10 2014

Extract text from PDF UDF. Currently still debugging
10b92d78f6ed
·rwolniak on Nov 10 2014, 04:07.
Merge pull request #1 from lintool/master
7e659c0eb678
·Ryan Wolniak on Nov 10 2014, 02:27.

Oct 31 2014

Fixed issues #96, #98, #99
df7f8711c64f
·lintool on Oct 31 2014, 18:32.
Minor refactoring.
b8915343f5c9
·lintool on Oct 31 2014, 18:28.

Oct 22 2014

Refactoring to create method that extracts MIME from WARC response records.
0145d74ed35e
·lintool on Oct 22 2014, 18:42.

Oct 19 2014

Merge branch 'pig' into warc
d491020c8287
·lintool on Oct 19 2014, 19:01.
WARC-related Hadoop bindings.
410cfd81a069
·lintool on Oct 19 2014, 18:59.
Minor refactoring.
f4a249469cbc
·lintool on Oct 19 2014, 16:58.
Pig ArcLoader exports its own ResourceSchema.
0c52caed50cf
·lintool on Oct 19 2014, 02:02.
Merge branch 'master' into hbase-api-refactoring
f97523f440ff
·lintool on Oct 19 2014, 01:21.
Fixed issues #93, #94, #95
e791bec61020
·lintool on Oct 19 2014, 01:17.
Added simple Pig script.
da7a94be2c70
·lintool on Oct 19 2014, 01:14.
Fixed test cases.
3af097ea655d
·lintool on Oct 19 2014, 00:50.
Pig loader materializes the actual content.
4ceefef78cdc
·lintool on Oct 19 2014, 00:47.

Oct 18 2014

Refactored Pig loaded to use WAC API.
58df7da00880
·lintool on Oct 18 2014, 17:05.
Fixed issues #90, #91, #92
63e8352c664b
·lintool on Oct 18 2014, 14:36.
Updates.
722b6402b98b
·lintool on Oct 18 2014, 14:34.
Merge branch 'master' of github.com:rwolniak/warcbase into warc
a48bb9781feb
·lintool on Oct 18 2014, 14:32.
upgraded to webarchive-commons 1.1.4.
21397e4e4ff3
·lintool on Oct 18 2014, 14:30.

Oct 13 2014

Updated Building the URL Mapping Instructions to include instruction to move…
da890acb0be1
·rwolniak on Oct 13 2014, 17:57.
Updated documentation.
de4267bf28f1
·lintool on Oct 13 2014, 14:21.

Oct 8 2014

Testing commit and push + README update
27b0038b631f
·rwolniak on Oct 8 2014, 06:01.

Sep 15 2014

Merge branch 'master' into warc
457a71345d2b
·lintool on Sep 15 2014, 01:57.

Sep 14 2014

Refactored to eliminate deprecated HBase APIs.
2b8e721063fa
·lintool on Sep 14 2014, 20:42.

Historymaster9b5bc63a1ccf3b36e8fea3b…

Jun 15 2015

Jun 14 2015

Jun 9 2015

Jun 7 2015

Jun 6 2015

Jun 4 2015

May 30 2015

May 27 2015

May 26 2015

May 24 2015

May 21 2015

May 13 2015

May 7 2015

May 6 2015

Jan 27 2015

Jan 21 2015

Dec 25 2014

Dec 24 2014

Dec 23 2014

Dec 22 2014

Dec 21 2014

Dec 18 2014

Dec 11 2014

Dec 10 2014

Dec 9 2014

Dec 8 2014

Nov 12 2014

Nov 10 2014

Oct 31 2014

Oct 22 2014

Oct 19 2014

Oct 18 2014

Oct 13 2014

Oct 8 2014

Sep 15 2014

Sep 14 2014

History
master9b5bc63a1ccf3b36e8fea3b…