Diffusion warcbase (master) History

R1473/src/main/java/org/warcbase/ingest/IngestFiles.java
master2e88c1b19afb2d546af976e…

Nov 25 2015

Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.

Dec 25 2014

Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112
f263769c86b7
·lintool on Dec 25 2014, 01:42.

Dec 24 2014

Fixed typo.
52b9696fba87
·lintool on Dec 24 2014, 21:41.
Add ingestion option to select either Snappy or GZ compression, Issue #110
5d471280e7db
·lintool on Dec 24 2014, 21:31.

Dec 11 2014

Use WarcRecordUtils.getWarcResponseMimeType() for ingest
329ddf9ce247
·Jeremy Wiebe on Dec 11 2014, 22:21.

Dec 8 2014

Added WARC file support to IngestFiles.
4239f1f9ae87
·Jeremy Wiebe on Dec 8 2014, 20:25.

Aug 15 2014

Better handling of ARC parse errors.
377a3e3a429f
·lintool on Aug 15 2014, 01:31.

Aug 14 2014

Bump max value size up to 10 MB, tweak WAL settings.
2d16fd032060
·lintool on Aug 14 2014, 23:33.
Refactoring UrlUtil and related classes.
bf606c07d453
·lintool on Aug 14 2014, 23:31.

Aug 11 2014

Code cleanup for browser code; removed unneeded files and associated web files.
515bab098cff
·lintool on Aug 11 2014, 14:22.

Aug 10 2014

Fixed a few minor ingestion issues.
bc6aab1ffd21
·lintool on Aug 10 2014, 16:11.
Refactoring; removing WARC ingestion for now.
b20ef84e5df9
·lintool on Aug 10 2014, 14:29.
Tweaks to ingest code.
cfe508a831f0
·lintool on Aug 10 2014, 14:05.
Janky, but seems to work: ingesting and serving up raw ARC records.
180b57fa5dc1
·lintool on Aug 10 2014, 08:28.
Quick and dirty switch over to webarchive-commons API; stores raw ARC records.
8b5a6be9db61
·lintool on Aug 10 2014, 04:16.

Jun 17 2014

Started working on issue #46 cleanup of org.warcbase.data.Util
02c26d6d8ea4
·lintool on Jun 17 2014, 17:20.

May 23 2014

Fixed robustness and OOM issues when ingesting corrupt ARC files.
7c21e2df9267
·lintool on May 23 2014, 23:09.

Mar 17 2014

Reformatting source.
303860c9ae3b
·lintool on Mar 17 2014, 14:11.
Added try/catch block in ingestion code.
a33f92f340e1
·lintool on Mar 17 2014, 13:28.

Mar 16 2014

Ignores all files that aren't WARC/ARC files in ingest.
987f9518ebb8
·lintool on Mar 16 2014, 23:26.
Tried disabling WAL to see impact on performance.
dfb63a169b43
·lintool on Mar 16 2014, 21:22.

Nov 18 2013

Code refactoring after pair coding.
bf51f6b75bd8
·milad621 on Nov 18 2013, 13:22.

Nov 8 2013

Fixed ava.lang.NegativeArraySizeException at org.apache.commons.io.output.
43d12364a70f
·milad621 on Nov 8 2013, 22:35.

Nov 7 2013

updated servlet.
bf1b3616e4c0
·milad621 on Nov 7 2013, 05:23.

Nov 6 2013

Added a seperate class to manage HBase connection and addRecord
76949eacabd5
·milad621 on Nov 6 2013, 23:27.

Nov 5 2013

One runnable to process both arc and warc files in a folder.
01dff2a1f2a5
·milad621 on Nov 5 2013, 23:30.

Oct 30 2013

IngestWarcFiles fixed. Now it uses jwat-warc to ingest warcfiles to hbase.
e9e3d1c9d940
·milad621 on Oct 30 2013, 01:56.