Fixed issues #86, #85, #74
fb07114090f5
Actions

Authored by lintool <jimmylin@umd.edu> on Aug 23 2014, 19:31.

Description

Fixed issues #86, #85, #74

Issue #86: Move link extraction code over to using new WacArcInputFormat
Issue #85: Writable container for ARCRecord
Issue #74: TableMapper wrapper for interoperability between HBase vs. processing (W)ARC data

Details

Committed

lintool <jimmylin@umd.edu>

Aug 23 2014, 19:31

Pushed

dportabella

Oct 19 2016, 16:29

Parents

R1473:9faed3817dcf: Refactored WacMapReduceHBaseWrapperDemo, now takes advantage of…
R1473:1e1c2dcde9e2: Fixed issues #84, #72

Branches

Unknown

Tags

Unknown

Event Timeline

lintool <jimmylin@umd.edu> committed R1473:fb07114090f5: Fixed issues #86, #85, #74 (authored by lintool <jimmylin@umd.edu>).Aug 23 2014, 19:31

Merged Changes

Commit	Author	Details	Committed
9faed3817dcf	lintool	Refactored WacMapReduceHBaseWrapperDemo, now takes advantage of…	Aug 23 2014
999fa0af0280	lintool	Merge branch 'working' into table-wrapper	Aug 23 2014
08530c1cbe52	lintool	WacArcInputFormat now generates ArcRecordWritables.	Aug 23 2014
98d7150f5eec	lintool	Switched ExtractSiteLinks and InvertAnchorText over to WacArcInputFormat; link…	Aug 23 2014
369ba2731f5b	lintool	Minor refactoring, revised counters.	Aug 23 2014
74b023ed8b16	lintool	Both the Jwat and Wac versions of ExtractLinks gives the same exact output.	Aug 23 2014
ccee8fd2204a	lintool	Implemented ArcRecordWritable.	Aug 23 2014
c5f4b9efa212	lintool	Very rough prototype of wrapper that allows interoperability between HBase…	Aug 23 2014

Changes (20)

		Path
P		src/main/java/org/warcbase/analysis/graph/ExtractLinks.java Deleted after being copied to multiple locations: src/main/java/org/warcbase/analysis/graph/ExtractLinksJwat.java src/main/java/org/warcbase/analysis/graph/ExtractLinksWac.java
P		src/main/java/org/warcbase/analysis/graph/ExtractLinksJwat.java Copied from src/main/java/org/warcbase/analysis/graph/ExtractLinks.java
V		src/main/java/org/warcbase/analysis/graph/{ExtractLinksWac.java ← ExtractLinks.java}
M		src/main/java/org/warcbase/analysis/graph/ExtractSiteLinks.java
M		src/main/java/org/warcbase/analysis/graph/InvertAnchorText.java
M		src/main/java/org/warcbase/data/ArcRecordUtils.java
M		src/main/java/org/warcbase/demo/WacMapReduceArcDemo.java
P		src/main/java/org/warcbase/demo/WacMapReduceHBaseDemo.java Copied to src/main/java/org/warcbase/demo/WacMapReduceHBaseWrapperDemo.java
P		src/main/java/org/warcbase/demo/WacMapReduceHBaseWrapperDemo.java Copied from src/main/java/org/warcbase/demo/WacMapReduceHBaseDemo.java
A	(dir)	src/main/java/org/warcbase/io/
A		src/main/java/org/warcbase/io/ArcRecordWritable.java
M		src/main/java/org/warcbase/mapreduce/WacArcInputFormat.java
A	(dir)	src/main/java/org/warcbase/mapreduce/lib/
A		src/main/java/org/warcbase/mapreduce/lib/Chain.java
A		src/main/java/org/warcbase/mapreduce/lib/ChainMapContextImpl.java
A		src/main/java/org/warcbase/mapreduce/lib/HBaseRowToArcRecordWritableMapper.java
A		src/main/java/org/warcbase/mapreduce/lib/TableChainMapper.java
A	(dir)	src/test/java/org/warcbase/io/
P		src/test/java/org/warcbase/io/ArcRecordWritableTest.java Copied from src/test/java/org/warcbase/mapreduce/WacArcInputFormatTest.java
P		src/test/java/org/warcbase/mapreduce/WacArcInputFormatTest.java Copied to src/test/java/org/warcbase/io/ArcRecordWritableTest.java