Fixed issues #88, #109, #110, #111, #112
6ef497988df1
Actions

Authored by lintool <jimmylin@umd.edu> on Dec 25 2014, 20:36.

Description

Fixed issues #88, #109, #110, #111, #112

Issue #88 Refactor deprecated HBase API
Issue #109 OOM when running UrlMappingBuilder
Issue #110 Add ingestion option to select either Snappy or GZ compression
Issue #111 OOM errors caused by corrupt records: ExtractLinksWac and InvertAnchorText
Issue #112 Add option to change MAX_CONTENT_SIZE in IngestFiles

Details

Committed

lintool <jimmylin@umd.edu>

Dec 25 2014, 20:36

Pushed

dportabella

Oct 19 2016, 16:29

Parents

R1473:f263769c86b7: Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112
R1473:e03a08388734: Fixed issues #64, #105, #108

Branches

Unknown

Tags

Unknown

Event Timeline

lintool <jimmylin@umd.edu> committed R1473:6ef497988df1: Fixed issues #88, #109, #110, #111, #112 (authored by lintool <jimmylin@umd.edu>).Dec 25 2014, 20:36

Merged Changes

Commit	Author	Details	Committed
f263769c86b7	lintool	Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112	Dec 25 2014
52b9696fba87	lintool	Fixed typo.	Dec 24 2014
5d471280e7db	lintool	Add ingestion option to select either Snappy or GZ compression, Issue #110	Dec 24 2014
fb3edb4e40c7	lintool	Renaming.	Dec 23 2014
f6caf0f6f226	lintool	Fixed OOM issues.	Dec 23 2014
b1c6995aa034	lintool	Fixed warnings.	Dec 23 2014
59949b3d7af4	lintool	Fixed OOM errors.	Dec 23 2014
e6b21cc557d6	lintool	Fixed Issue #109: OOM when running UrlMappingBuilder	Dec 23 2014
0fd8f3f72822	lintool	Updated versions of some artifacts.	Dec 23 2014
6acf52f320cf	lintool	Merge branch 'hbase-api-refactoring' into dev	Dec 22 2014
f97523f440ff	lintool	Merge branch 'master' into hbase-api-refactoring	Oct 19 2014
2b8e721063fa	lintool	Refactored to eliminate deprecated HBase APIs.	Sep 14 2014

Changes (18)

				Path
	M			pom.xml
	V			src/main/java/org/warcbase/analysis/{ArcCounter.java → GenericArcRecordCounter.java}
	M			src/main/java/org/warcbase/analysis/CountArcContentTypes.java
	M			src/main/java/org/warcbase/analysis/CountArcCrawlDates.java
	M			src/main/java/org/warcbase/analysis/CountArcUrls.java
	M			src/main/java/org/warcbase/analysis/CountWarcContentTypes.java
	M			src/main/java/org/warcbase/analysis/CountWarcCrawlDates.java
	M			src/main/java/org/warcbase/analysis/CountWarcUrls.java
	V			src/main/java/org/warcbase/analysis/{GenericArcRecordCounter.java ← ArcCounter.java}
	V			src/main/java/org/warcbase/analysis/{GenericWarcRecordCounter.java ← WarcCounter.java}
	V			src/main/java/org/warcbase/analysis/{WarcCounter.java → GenericWarcRecordCounter.java}
	M			src/main/java/org/warcbase/analysis/graph/ExtractLinksWac.java
	M			src/main/java/org/warcbase/analysis/graph/InvertAnchorText.java
	M			src/main/java/org/warcbase/browser/WarcBrowserServlet.java
	M			src/main/java/org/warcbase/data/HBaseTableManager.java
	M			src/main/java/org/warcbase/data/UrlMappingBuilder.java
	M			src/main/java/org/warcbase/ingest/IngestFiles.java
	M			src/main/java/org/warcbase/wayback/WarcbaseResourceStore.java

Fixed issues #88, #109, #110, #111, #1126ef497988df1Actions

Description

Details

Event Timeline

Merged Changes

Changes (18)

R1473:6ef497988df1

pom.xml

src/main/java/org/warcbase/analysis/ArcCounter.java

src/main/java/org/warcbase/analysis/CountArcContentTypes.java

src/main/java/org/warcbase/analysis/CountArcCrawlDates.java

src/main/java/org/warcbase/analysis/CountArcUrls.java

src/main/java/org/warcbase/analysis/CountWarcContentTypes.java

src/main/java/org/warcbase/analysis/CountWarcCrawlDates.java

src/main/java/org/warcbase/analysis/CountWarcUrls.java

src/main/java/org/warcbase/analysis/GenericArcRecordCounter.java

src/main/java/org/warcbase/analysis/GenericWarcRecordCounter.java

src/main/java/org/warcbase/analysis/WarcCounter.java

src/main/java/org/warcbase/analysis/graph/ExtractLinksWac.java

src/main/java/org/warcbase/analysis/graph/InvertAnchorText.java

src/main/java/org/warcbase/browser/WarcBrowserServlet.java

src/main/java/org/warcbase/data/HBaseTableManager.java

src/main/java/org/warcbase/data/UrlMappingBuilder.java

src/main/java/org/warcbase/ingest/IngestFiles.java

src/main/java/org/warcbase/wayback/WarcbaseResourceStore.java

Fixed issues #88, #109, #110, #111, #112
6ef497988df1
Actions