Fixed issues #126, #127, #128
61ca00e10cb2
Actions

Authored by lintool <jimmylin@umd.edu> on Jun 7 2015, 18:22.

Description

Fixed issues #126, #127, #128

Issue #128 Clean up org.warcbase.analysis
Issue #127 Remove dependence on JWAT
Issue #126 Pig WarcLoader to use WacWarcInputFormat

Details

Committed

lintool <jimmylin@umd.edu>

Jun 7 2015, 18:22

Pushed

dportabella

Oct 19 2016, 16:29

Parents

R1473:2f462979da0d: Removed JWAT jars.
R1473:6b17e9f5e6f1: Fixed issue #120

Branches

Unknown

Tags

Unknown

Event Timeline

lintool <jimmylin@umd.edu> committed R1473:61ca00e10cb2: Fixed issues #126, #127, #128 (authored by lintool <jimmylin@umd.edu>).Jun 7 2015, 18:22

Merged Changes

Commit	Author	Details	Committed
2f462979da0d	lintool	Removed JWAT jars.	Jun 7 2015
9b8e85db50e3	lintool	removed JWAT testcase.	Jun 7 2015
d9b553c76ced	lintool	Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…	Jun 7 2015
caf87fbecf45	lintool	Added schema metadata in loader.	Jun 7 2015
49fc639ae5b6	lintool	Merge branch 'pig-warc' of github.com:lintool/warcbase into pig-warc	Jun 7 2015
195f4af7474e	lintool	revamped tests.	Jun 7 2015
1f9d853f38a2	lintool	better trapping of errors.	Jun 7 2015
84ebe4f68bbd	lintool	Added test case for PigWarcLoader.	Jun 6 2015
13619567667f	lintool	Changed over to WacWarcInputFormatTest as the underlying InputFormat.	Jun 6 2015

Changes (28)

				Path
	M			pom.xml
	D			src/main/java/org/warcbase/analysis/CountArcContentTypes.java
	D			src/main/java/org/warcbase/analysis/CountArcCrawlDates.java
	D			src/main/java/org/warcbase/analysis/CountArcUrls.java
	D			src/main/java/org/warcbase/analysis/CountWarcContentTypes.java
	D			src/main/java/org/warcbase/analysis/CountWarcCrawlDates.java
	D			src/main/java/org/warcbase/analysis/CountWarcUrls.java
	D			src/main/java/org/warcbase/analysis/DetectDuplicates.java
	D			src/main/java/org/warcbase/analysis/ExtractUniqueUrls.java
	M			src/main/java/org/warcbase/analysis/FindArcUrls.java
	M			src/main/java/org/warcbase/analysis/FindWarcUrls.java
	D			src/main/java/org/warcbase/analysis/GenericArcRecordCounter.java
	D			src/main/java/org/warcbase/analysis/GenericWarcRecordCounter.java
	D			src/main/java/org/warcbase/analysis/graph/ExtractLinksJwat.java
	M			src/main/java/org/warcbase/data/UrlMappingMapReduceBuilder.java
	D			src/main/java/org/warcbase/demo/JwatMapReduceArcDemo.java
	D			src/main/java/org/warcbase/demo/JwatMapReduceWarcDemo.java
	D			src/main/java/org/warcbase/mapreduce/JwatArcInputFormat.java
	D			src/main/java/org/warcbase/mapreduce/JwatWarcInputFormat.java
	P			src/main/java/org/warcbase/pig/WarcLoader.java Copied from src/main/java/org/warcbase/pig/ArcLoader.java
	D			src/test/java/org/warcbase/mapreduce/JwatArcInputFormatTest.java
	M			src/test/java/org/warcbase/mapreduce/WacWarcInputFormatTest.java
	M			src/test/java/org/warcbase/pig/PigArcLoaderTest.java
	A			src/test/java/org/warcbase/pig/PigWarcLoaderTest.java
	V			src/test/resources/scripts/{TestArcCountLinks.pig ← TestCountLinks.pig}
	A			src/test/resources/scripts/TestArcLoaderCount.pig
	V			src/test/resources/scripts/{TestCountLinks.pig → TestArcCountLinks.pig}
	A			src/test/resources/scripts/TestWarcLoaderCount.pig