Add Spark support
8057c46945d0
Actions

Authored by Alice-Z <alice.zhou@gmail.com> on Oct 23 2015, 18:12.

Description

Add Spark support

Add support for extract("...") methods
Add support for WARC records and ARC records via a common API over RDD
Translate Pig UDFs to Spark

Committed

Alice-Z <alice.zhou@gmail.com>

Nov 3 2015, 02:58

Pushed

dportabella

Oct 19 2016, 16:29

Parents

R1473:f0a2e1d1f94a: Fixed issues #142, #152, #153

Branches

Unknown

Tags

Unknown

Alice-Z <alice.zhou@gmail.com> committed R1473:8057c46945d0: Add Spark support (authored by Alice-Z <alice.zhou@gmail.com>).Nov 3 2015, 02:58

		Path
M		pom.xml
A		src/main/java/org/warcbase/data/RecordUtils.java
M		src/main/java/org/warcbase/data/WarcRecordUtils.java
M		src/main/java/org/warcbase/pig/WarcLoader.java
M		src/main/java/org/warcbase/pig/piggybank/DetectMimeTypeMagic.java
D		src/main/scala/org/warcbase/spark/matchbox/ArcRecords.scala
A		src/main/scala/org/warcbase/spark/matchbox/DetectLanguage.scala
A		src/main/scala/org/warcbase/spark/matchbox/DetectMimeTypeTika.scala
A		src/main/scala/org/warcbase/spark/matchbox/ExtractBoilerpipeText.scala
A		src/main/scala/org/warcbase/spark/matchbox/ExtractLinks.scala
A		src/main/scala/org/warcbase/spark/matchbox/ExtractLinksAndText.scala
A		src/main/scala/org/warcbase/spark/matchbox/ExtractTextFromPDFs.scala
M		src/main/scala/org/warcbase/spark/matchbox/ExtractTopLevelDomain.scala
A		src/main/scala/org/warcbase/spark/matchbox/RecordLoader.scala
A		src/main/scala/org/warcbase/spark/matchbox/RecordTransformers.scala
A	(dir)	src/main/scala/org/warcbase/spark/rdd/
A		src/main/scala/org/warcbase/spark/rdd/RecordRDD.scala
M		src/main/scala/org/warcbase/spark/scripts/Filter.scala
A	(dir)	src/test/scala/
A	(dir)	src/test/scala/org/
A	(dir)	src/test/scala/org/warcbase/
A	(dir)	src/test/scala/org/warcbase/spark/
A	(dir)	src/test/scala/org/warcbase/spark/matchbox/
A		src/test/scala/org/warcbase/spark/matchbox/Spec.scala