Refactored Warcbase into multiple modules, upgraded to CDH 5.7.1 (w/ Spark 1.6.
62b8c4e57800
Actions

Authored by lintool <jimmylin@uwaterloo.ca> on Jun 28 2016, 18:05.

Description

Refactored Warcbase into multiple modules, upgraded to CDH 5.7.1 (w/ Spark 1.6.0).

Closed following issues:
Issue #236 Trantor upgraded to CDH 5.7.1
Issue #235 Break Warcbase up into sub-artifacts
Issue #231 Upgrade to Spark 1.6.1?

Details

Committed

lintool <jimmylin@uwaterloo.ca>

Jun 28 2016, 18:08

Pushed

dportabella

Oct 19 2016, 16:29

Parents

R1473:7d2f8a4f2119: Updated documentation.
R1473:63c3d5916d84: Fixed issue #234 Error handling for broken ARC/WARC files: empty content in ARC…

Branches

Unknown

Tags

Unknown

Event Timeline

lintool <jimmylin@uwaterloo.ca> committed R1473:62b8c4e57800: Refactored Warcbase into multiple modules, upgraded to CDH 5.7.1 (w/ Spark 1.6. (authored by lintool <jimmylin@uwaterloo.ca>).Jun 28 2016, 18:08

Merged Changes

Commit	Author	Details	Committed
7d2f8a4f2119	lintool	Updated documentation.	Jun 28 2016
6a90426a3d1a	lintool	Fixed Scala matching issue with NER extractor.	Jun 28 2016
19be6edfa8cd	lintool	Upgraded to CDH 5.7.1.	Jun 24 2016
0dfabaeb484b	lintool	Added dependencies for boilerpipe.	Jun 22 2016
e8c43f1b9d24	lintool	Cleaned up and simplified dependencies, etc.	Jun 17 2016
6e8740a55414	lintool	Integrating warcbase-hbase artifact.	Jun 16 2016
0db46be7ef7f	lintool	Moved HBase and non-core code out of warcbase-core into warcbase-hbase.	Jun 16 2016
f972206db516	lintool	Created warcbase-core module.	Jun 16 2016

Changes (442)
Show All Changes

Very Large Commit

This commit is very large. Load each file individually.

		Path
D		.settings
D		.settings/org.eclipse.core.resources.prefs
D		.settings/org.eclipse.jdt.core.prefs
D		.settings/org.eclipse.jdt.ui.prefs
D		.settings/org.eclipse.m2e.core.prefs
M		.travis.yml
M		README.md
M		pom.xml
D		src
D		src/main
D		src/main/java
D		src/main/java/org
D		src/main/java/org/warcbase
V		{ → warcbase-hbase}/src/main/java/org/warcbase/WarcbaseAdmin.java
V	(dir)	{src/main/java/org/warcbase/analysis/ → warcbase-hbase/src/main/java/org/warcbase/analysis}
V		{ → warcbase-hbase}/src/main/java/org/warcbase/analysis/FindArcUrls.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/analysis/FindWarcUrls.java
V	(dir)	{src/main/java/org/warcbase/analysis/graph/ → warcbase-hbase/src/main/java/org/warcbase/analysis/graph}
V		{ → warcbase-hbase}/src/main/java/org/warcbase/analysis/graph/ExtractLinksWac.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/analysis/graph/ExtractSiteLinks.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/analysis/graph/InvertAnchorText.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/analysis/graph/PrefixMapping.java
V	(dir)	{src/main/java/org/warcbase/browser/ → warcbase-hbase/src/main/java/org/warcbase/browser}
V		{ → warcbase-hbase}/src/main/java/org/warcbase/browser/SeleniumBrowser.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/browser/WarcBrowser.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/browser/WarcBrowserServlet.java
D		src/main/java/org/warcbase/data
V		{ → warcbase-core}/src/main/java/org/warcbase/data/ArcRecordUtils.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/data/HBaseTableManager.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/data/UrlMapping.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/data/UrlMappingBuilder.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/data/UrlMappingMapReduceBuilder.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/data/UrlUtils.java
V		{ → warcbase-core}/src/main/java/org/warcbase/data/WarcRecordUtils.java
D		src/main/java/org/warcbase/demo
V		{ → warcbase-core}/src/main/java/org/warcbase/demo/WacMapReduceArcDemo.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/demo/WacMapReduceHBaseDemo.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/demo/WacMapReduceHBaseWrapperDemo.java
V	(dir)	{src/main/java/org/warcbase/index/ → warcbase-hbase/src/main/java/org/warcbase/index}
V		{ → warcbase-hbase}/src/main/java/org/warcbase/index/IndexerMapper.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/index/IndexerReducer.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/index/IndexerRunner.java
V	(dir)	{src/main/java/org/warcbase/ingest/ → warcbase-hbase/src/main/java/org/warcbase/ingest}
V		{ → warcbase-hbase}/src/main/java/org/warcbase/ingest/IngestFiles.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/ingest/SearchForUrl.java
V	(dir)	{src/main/java/org/warcbase/io/ → warcbase-core/src/main/java/org/warcbase/io}
V		{ → warcbase-core}/src/main/java/org/warcbase/io/ArcRecordWritable.java
V		{ → warcbase-core}/src/main/java/org/warcbase/io/GenericArchiveRecordWritable.java
V		{ → warcbase-core}/src/main/java/org/warcbase/io/WarcRecordWritable.java
D		src/main/java/org/warcbase/mapreduce
V		{ → warcbase-core}/src/main/java/org/warcbase/mapreduce/WacArcInputFormat.java
V		{ → warcbase-core}/src/main/java/org/warcbase/mapreduce/WacGenericInputFormat.java
V		{ → warcbase-core}/src/main/java/org/warcbase/mapreduce/WacWarcInputFormat.java
V	(dir)	{src/main/java/org/warcbase/mapreduce/lib/ → warcbase-hbase/src/main/java/org/warcbase/mapreduce/lib}
V		{ → warcbase-hbase}/src/main/java/org/warcbase/mapreduce/lib/Chain.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/mapreduce/lib/ChainMapContextImpl.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/mapreduce/lib/HBaseRowToArcRecordWritableMapper.java
V		{ → warcbase-hbase}/src/main/java/org/warcbase/mapreduce/lib/TableChainMapper.java
V	(dir)	{src/main/java/org/warcbase/wayback/ → warcbase-core/src/main/java/org/warcbase/wayback}
V		{ → warcbase-core}/src/main/java/org/warcbase/wayback/WarcbaseResourceIndex.java
V		{ → warcbase-core}/src/main/java/org/warcbase/wayback/WarcbaseResourceStore.java
V	(dir)	{src/main/python/ → warcbase-core/src/main/python}
V		{ → warcbase-core}/src/main/python/break-into-date-scrapes.py
V		{ → warcbase-core}/src/main/python/combine-entity-results-split-by-date.py
V		{ → warcbase-core}/src/main/python/combine-entity-results.py
V		{ → warcbase-core}/src/main/python/pig2gdf.py
V	(dir)	{src/main/resources/ → warcbase-core/src/main/resources}
V		{ → warcbase-core}/src/main/resources/BDBCollection.xml
V		{ → warcbase-core}/src/main/resources/log4j.properties
D		src/main/scala
D		src/main/scala/org
D		src/main/scala/org/warcbase
D		src/main/scala/org/warcbase/spark
V	(dir)	{src/main/scala/org/warcbase/spark/archive/ → warcbase-core/src/main/scala/org/warcbase/spark/archive}
V	(dir)	{src/main/scala/org/warcbase/spark/archive/io/ → warcbase-core/src/main/scala/org/warcbase/spark/archive/io}
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/archive/io/ArcRecord.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/archive/io/ArchiveRecord.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/archive/io/GenericArchiveRecord.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/archive/io/WarcRecord.scala
D		src/main/scala/org/warcbase/spark/matchbox
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/DetectLanguage.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/DetectMimeTypeTika.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractAtMentions.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractBoilerpipeText.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractDate.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractDomain.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractEntities.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractGraph.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractHashtags.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractImageLinks.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractLinks.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractTextFromPDFs.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/ExtractUrls.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/NER3Classifier.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/NERCombinedJson.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/RecordLoader.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/RemoveHTML.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/StringUtils.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/TupleFormatter.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/TweetUtils.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/matchbox/WriteGDF.scala
V	(dir)	{src/main/scala/org/warcbase/spark/pythonconverters/ → warcbase-core/src/main/scala/org/warcbase/spark/pythonconverters}
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/pythonconverters/ArcRecordConverter.scala
V	(dir)	{src/main/scala/org/warcbase/spark/rdd/ → warcbase-core/src/main/scala/org/warcbase/spark/rdd}
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/rdd/RecordRDD.scala
V	(dir)	{src/main/scala/org/warcbase/spark/scripts/ → warcbase-core/src/main/scala/org/warcbase/spark/scripts}
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/scripts/CrawlStatistics.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/scripts/Filter.scala
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/scripts/SocialMediaLinks.scala
V	(dir)	{src/main/scala/org/warcbase/spark/utils/ → warcbase-core/src/main/scala/org/warcbase/spark/utils}
V		{ → warcbase-core}/src/main/scala/org/warcbase/spark/utils/JsonUtil.scala
V	(dir)	{src/main/solr/ → warcbase-hbase/src/main/solr}
V		{ → warcbase-hbase}/src/main/solr/README.txt
V		{ → warcbase-hbase}/src/main/solr/WARCIndexer.conf
V	(dir)	{src/main/solr/discovery/ → warcbase-hbase/src/main/solr/discovery}
V	(dir)	{src/main/solr/discovery/conf/ → warcbase-hbase/src/main/solr/discovery/conf}
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/currency.xml
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/elevate.xml
V	(dir)	{src/main/solr/discovery/conf/lang/ → warcbase-hbase/src/main/solr/discovery/conf/lang}
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/contractions_ca.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/contractions_fr.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/contractions_ga.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/contractions_it.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/hyphenations_ga.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stemdict_nl.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stoptags_ja.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_ar.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_bg.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_ca.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_cz.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_da.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_de.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_el.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_en.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_es.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_eu.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_fa.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_fi.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_fr.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_ga.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_gl.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_hi.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_hu.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_hy.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_id.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_it.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_ja.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_lv.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_nl.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_no.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_pt.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_ro.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_ru.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_sv.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_th.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/stopwords_tr.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/lang/userdict_ja.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/protwords.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/schema.xml
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/solrconfig-production.xml
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/solrconfig-server-4.10.4.xml
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/solrconfig.xml
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/solrcore.properties
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/solrcore.properties-production
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/stopwords.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/conf/synonyms.txt
V		{ → warcbase-hbase}/src/main/solr/discovery/core.properties
V		{ → warcbase-hbase}/src/main/solr/solr.xml
V		{ → warcbase-hbase}/src/main/solr/zoo.cfg
V	(dir)	{src/main/webapp/ → warcbase-core/src/main/webapp}
V	(dir)	{src/main/webapp/WEB-INF/ → warcbase-core/src/main/webapp/WEB-INF}
V		{ → warcbase-core}/src/main/webapp/WEB-INF/web.xml
D		src/test
D		src/test/java
D		src/test/java/org
D		src/test/java/org/warcbase
V	(dir)	{src/test/java/org/warcbase/data/ → warcbase-hbase/src/test/java/org/warcbase/data}
V		{ → warcbase-hbase}/src/test/java/org/warcbase/data/UrlMappingTest.java
V		{ → warcbase-hbase}/src/test/java/org/warcbase/data/UrlUtilsTest.java
V	(dir)	{src/test/java/org/warcbase/ingest/ → warcbase-core/src/test/java/org/warcbase/ingest}
V		{ → warcbase-core}/src/test/java/org/warcbase/ingest/WacArcLoaderTest.java
V		{ → warcbase-core}/src/test/java/org/warcbase/ingest/WacWarcLoaderTest.java
V	(dir)	{src/test/java/org/warcbase/io/ → warcbase-core/src/test/java/org/warcbase/io}
V		{ → warcbase-core}/src/test/java/org/warcbase/io/ArcRecordWritableTest.java
V		{ → warcbase-core}/src/test/java/org/warcbase/io/GenericArchiveRecordWritableTest.java
V		{ → warcbase-core}/src/test/java/org/warcbase/io/WarcRecordWritableTest.java
V	(dir)	{src/test/java/org/warcbase/mapreduce/ → warcbase-core/src/test/java/org/warcbase/mapreduce}
V		{ → warcbase-core}/src/test/java/org/warcbase/mapreduce/WacArcInputFormatTest.java
V		{ → warcbase-core}/src/test/java/org/warcbase/mapreduce/WacGenericInputFormatTest.java
V		{ → warcbase-core}/src/test/java/org/warcbase/mapreduce/WacWarcInputFormatTest.java
V	(dir)	{src/test/resources/ → warcbase-core/src/test/resources}
V	(dir)	{src/test/resources/arc/ → warcbase-core/src/test/resources/arc}
V		{ → warcbase-core}/src/test/resources/arc/example.arc.gz
V	(dir)	{src/test/resources/ner/ → warcbase-core/src/test/resources/ner}
V		{ → warcbase-core}/src/test/resources/ner/example.txt
V	(dir)	{src/test/resources/warc/ → warcbase-core/src/test/resources/warc}
V		{ → warcbase-core}/src/test/resources/warc/example.warc.gz
V	(dir)	{src/test/scala/ → warcbase-core/src/test/scala}
V	(dir)	{src/test/scala/org/ → warcbase-core/src/test/scala/org}
V	(dir)	{src/test/scala/org/warcbase/ → warcbase-core/src/test/scala/org/warcbase}
V	(dir)	{src/test/scala/org/warcbase/spark/ → warcbase-core/src/test/scala/org/warcbase/spark}
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/ArcTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/GenericArchiveRecordTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/WarcTest.scala
V	(dir)	{src/test/scala/org/warcbase/spark/matchbox/ → warcbase-core/src/test/scala/org/warcbase/spark/matchbox}
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/ExtractAtMentionsTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/ExtractDateTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/ExtractDomainTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/ExtractEntitiesTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/ExtractHashtagsTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/ExtractImageLinksTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/ExtractLinksTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/ExtractUrlsTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/StringUtilsTest.scala
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/matchbox/TupleFormatterTest.scala
V	(dir)	{src/test/scala/org/warcbase/spark/rdd/ → warcbase-core/src/test/scala/org/warcbase/spark/rdd}
V		{ → warcbase-core}/src/test/scala/org/warcbase/spark/rdd/CountableRDDTest.scala
A	(dir)	warcbase-core/
A		warcbase-core/pom.xml
A	(dir)	warcbase-core/src/
A	(dir)	warcbase-core/src/main/
A	(dir)	warcbase-core/src/main/java/
A	(dir)	warcbase-core/src/main/java/org/
A	(dir)	warcbase-core/src/main/java/org/warcbase/
A	(dir)	warcbase-core/src/main/java/org/warcbase/data/
V		{warcbase-core ← }/src/main/java/org/warcbase/data/ArcRecordUtils.java
V		{warcbase-core ← }/src/main/java/org/warcbase/data/WarcRecordUtils.java
A	(dir)	warcbase-core/src/main/java/org/warcbase/demo/
V		{warcbase-core ← }/src/main/java/org/warcbase/demo/WacMapReduceArcDemo.java
V	(dir)	{warcbase-core/src/main/java/org/warcbase/io/ ← src/main/java/org/warcbase/io}
V		{warcbase-core ← }/src/main/java/org/warcbase/io/ArcRecordWritable.java
V		{warcbase-core ← }/src/main/java/org/warcbase/io/GenericArchiveRecordWritable.java
V		{warcbase-core ← }/src/main/java/org/warcbase/io/WarcRecordWritable.java
A	(dir)	warcbase-core/src/main/java/org/warcbase/mapreduce/
V		{warcbase-core ← }/src/main/java/org/warcbase/mapreduce/WacArcInputFormat.java
V		{warcbase-core ← }/src/main/java/org/warcbase/mapreduce/WacGenericInputFormat.java
V		{warcbase-core ← }/src/main/java/org/warcbase/mapreduce/WacWarcInputFormat.java
V	(dir)	{warcbase-core/src/main/java/org/warcbase/wayback/ ← src/main/java/org/warcbase/wayback}
V		{warcbase-core ← }/src/main/java/org/warcbase/wayback/WarcbaseResourceIndex.java
V		{warcbase-core ← }/src/main/java/org/warcbase/wayback/WarcbaseResourceStore.java
V	(dir)	{warcbase-core/src/main/python/ ← src/main/python}
V		{warcbase-core ← }/src/main/python/break-into-date-scrapes.py
V		{warcbase-core ← }/src/main/python/combine-entity-results-split-by-date.py
V		{warcbase-core ← }/src/main/python/combine-entity-results.py
V		{warcbase-core ← }/src/main/python/pig2gdf.py
V	(dir)	{warcbase-core/src/main/resources/ ← src/main/resources}
V		{warcbase-core ← }/src/main/resources/BDBCollection.xml
V		{warcbase-core ← }/src/main/resources/log4j.properties
A	(dir)	warcbase-core/src/main/scala/
A	(dir)	warcbase-core/src/main/scala/org/
A	(dir)	warcbase-core/src/main/scala/org/warcbase/
A	(dir)	warcbase-core/src/main/scala/org/warcbase/spark/
V	(dir)	{warcbase-core/src/main/scala/org/warcbase/spark/archive/ ← src/main/scala/org/warcbase/spark/archive}
V	(dir)	{warcbase-core/src/main/scala/org/warcbase/spark/archive/io/ ← src/main/scala/org/warcbase/spark/archive/io}
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/archive/io/ArcRecord.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/archive/io/ArchiveRecord.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/archive/io/GenericArchiveRecord.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/archive/io/WarcRecord.scala
A	(dir)	warcbase-core/src/main/scala/org/warcbase/spark/matchbox/
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/DetectLanguage.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/DetectMimeTypeTika.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractAtMentions.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractBoilerpipeText.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractDate.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractDomain.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractEntities.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractGraph.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractHashtags.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractImageLinks.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractLinks.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractTextFromPDFs.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/ExtractUrls.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/NER3Classifier.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/NERCombinedJson.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/RecordLoader.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/RemoveHTML.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/StringUtils.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/TupleFormatter.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/TweetUtils.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/matchbox/WriteGDF.scala
V	(dir)	{warcbase-core/src/main/scala/org/warcbase/spark/pythonconverters/ ← src/main/scala/org/warcbase/spark/pythonconverters}
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/pythonconverters/ArcRecordConverter.scala
V	(dir)	{warcbase-core/src/main/scala/org/warcbase/spark/rdd/ ← src/main/scala/org/warcbase/spark/rdd}
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/rdd/RecordRDD.scala
V	(dir)	{warcbase-core/src/main/scala/org/warcbase/spark/scripts/ ← src/main/scala/org/warcbase/spark/scripts}
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/scripts/CrawlStatistics.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/scripts/Filter.scala
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/scripts/SocialMediaLinks.scala
V	(dir)	{warcbase-core/src/main/scala/org/warcbase/spark/utils/ ← src/main/scala/org/warcbase/spark/utils}
V		{warcbase-core ← }/src/main/scala/org/warcbase/spark/utils/JsonUtil.scala
V	(dir)	{warcbase-core/src/main/webapp/ ← src/main/webapp}
V	(dir)	{warcbase-core/src/main/webapp/WEB-INF/ ← src/main/webapp/WEB-INF}
V		{warcbase-core ← }/src/main/webapp/WEB-INF/web.xml
A	(dir)	warcbase-core/src/test/
A	(dir)	warcbase-core/src/test/java/
A	(dir)	warcbase-core/src/test/java/org/
A	(dir)	warcbase-core/src/test/java/org/warcbase/
V	(dir)	{warcbase-core/src/test/java/org/warcbase/ingest/ ← src/test/java/org/warcbase/ingest}
V		{warcbase-core ← }/src/test/java/org/warcbase/ingest/WacArcLoaderTest.java
V		{warcbase-core ← }/src/test/java/org/warcbase/ingest/WacWarcLoaderTest.java
V	(dir)	{warcbase-core/src/test/java/org/warcbase/io/ ← src/test/java/org/warcbase/io}
V		{warcbase-core ← }/src/test/java/org/warcbase/io/ArcRecordWritableTest.java
V		{warcbase-core ← }/src/test/java/org/warcbase/io/GenericArchiveRecordWritableTest.java
V		{warcbase-core ← }/src/test/java/org/warcbase/io/WarcRecordWritableTest.java
V	(dir)	{warcbase-core/src/test/java/org/warcbase/mapreduce/ ← src/test/java/org/warcbase/mapreduce}
V		{warcbase-core ← }/src/test/java/org/warcbase/mapreduce/WacArcInputFormatTest.java
V		{warcbase-core ← }/src/test/java/org/warcbase/mapreduce/WacGenericInputFormatTest.java
V		{warcbase-core ← }/src/test/java/org/warcbase/mapreduce/WacWarcInputFormatTest.java
V	(dir)	{warcbase-core/src/test/resources/ ← src/test/resources}
V	(dir)	{warcbase-core/src/test/resources/arc/ ← src/test/resources/arc}
V		{warcbase-core ← }/src/test/resources/arc/example.arc.gz
V	(dir)	{warcbase-core/src/test/resources/ner/ ← src/test/resources/ner}
V		{warcbase-core ← }/src/test/resources/ner/example.txt
V	(dir)	{warcbase-core/src/test/resources/warc/ ← src/test/resources/warc}
V		{warcbase-core ← }/src/test/resources/warc/example.warc.gz
V	(dir)	{warcbase-core/src/test/scala/ ← src/test/scala}
V	(dir)	{warcbase-core/src/test/scala/org/ ← src/test/scala/org}
V	(dir)	{warcbase-core/src/test/scala/org/warcbase/ ← src/test/scala/org/warcbase}
V	(dir)	{warcbase-core/src/test/scala/org/warcbase/spark/ ← src/test/scala/org/warcbase/spark}
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/ArcTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/GenericArchiveRecordTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/WarcTest.scala
V	(dir)	{warcbase-core/src/test/scala/org/warcbase/spark/matchbox/ ← src/test/scala/org/warcbase/spark/matchbox}
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/ExtractAtMentionsTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/ExtractDateTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/ExtractDomainTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/ExtractEntitiesTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/ExtractHashtagsTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/ExtractImageLinksTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/ExtractLinksTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/ExtractUrlsTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/StringUtilsTest.scala
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/matchbox/TupleFormatterTest.scala
V	(dir)	{warcbase-core/src/test/scala/org/warcbase/spark/rdd/ ← src/test/scala/org/warcbase/spark/rdd}
V		{warcbase-core ← }/src/test/scala/org/warcbase/spark/rdd/CountableRDDTest.scala
A	(dir)	warcbase-hbase/
A		warcbase-hbase/pom.xml
A	(dir)	warcbase-hbase/src/
A	(dir)	warcbase-hbase/src/main/
A	(dir)	warcbase-hbase/src/main/java/
A	(dir)	warcbase-hbase/src/main/java/org/
A	(dir)	warcbase-hbase/src/main/java/org/warcbase/
V		{warcbase-hbase ← }/src/main/java/org/warcbase/WarcbaseAdmin.java
V	(dir)	{warcbase-hbase/src/main/java/org/warcbase/analysis/ ← src/main/java/org/warcbase/analysis}
V		{warcbase-hbase ← }/src/main/java/org/warcbase/analysis/FindArcUrls.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/analysis/FindWarcUrls.java
V	(dir)	{warcbase-hbase/src/main/java/org/warcbase/analysis/graph/ ← src/main/java/org/warcbase/analysis/graph}
V		{warcbase-hbase ← }/src/main/java/org/warcbase/analysis/graph/ExtractLinksWac.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/analysis/graph/ExtractSiteLinks.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/analysis/graph/InvertAnchorText.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/analysis/graph/PrefixMapping.java
V	(dir)	{warcbase-hbase/src/main/java/org/warcbase/browser/ ← src/main/java/org/warcbase/browser}
V		{warcbase-hbase ← }/src/main/java/org/warcbase/browser/SeleniumBrowser.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/browser/WarcBrowser.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/browser/WarcBrowserServlet.java
A	(dir)	warcbase-hbase/src/main/java/org/warcbase/data/
V		{warcbase-hbase ← }/src/main/java/org/warcbase/data/HBaseTableManager.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/data/UrlMapping.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/data/UrlMappingBuilder.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/data/UrlMappingMapReduceBuilder.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/data/UrlUtils.java
A	(dir)	warcbase-hbase/src/main/java/org/warcbase/demo/
V		{warcbase-hbase ← }/src/main/java/org/warcbase/demo/WacMapReduceHBaseDemo.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/demo/WacMapReduceHBaseWrapperDemo.java
V	(dir)	{warcbase-hbase/src/main/java/org/warcbase/index/ ← src/main/java/org/warcbase/index}
V		{warcbase-hbase ← }/src/main/java/org/warcbase/index/IndexerMapper.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/index/IndexerReducer.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/index/IndexerRunner.java
V	(dir)	{warcbase-hbase/src/main/java/org/warcbase/ingest/ ← src/main/java/org/warcbase/ingest}
V		{warcbase-hbase ← }/src/main/java/org/warcbase/ingest/IngestFiles.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/ingest/SearchForUrl.java
A	(dir)	warcbase-hbase/src/main/java/org/warcbase/mapreduce/
V	(dir)	{warcbase-hbase/src/main/java/org/warcbase/mapreduce/lib/ ← src/main/java/org/warcbase/mapreduce/lib}
V		{warcbase-hbase ← }/src/main/java/org/warcbase/mapreduce/lib/Chain.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/mapreduce/lib/ChainMapContextImpl.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/mapreduce/lib/HBaseRowToArcRecordWritableMapper.java
V		{warcbase-hbase ← }/src/main/java/org/warcbase/mapreduce/lib/TableChainMapper.java
V	(dir)	{warcbase-hbase/src/main/solr/ ← src/main/solr}
V		{warcbase-hbase ← }/src/main/solr/README.txt
V		{warcbase-hbase ← }/src/main/solr/WARCIndexer.conf
V	(dir)	{warcbase-hbase/src/main/solr/discovery/ ← src/main/solr/discovery}
V	(dir)	{warcbase-hbase/src/main/solr/discovery/conf/ ← src/main/solr/discovery/conf}
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/currency.xml
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/elevate.xml
V	(dir)	{warcbase-hbase/src/main/solr/discovery/conf/lang/ ← src/main/solr/discovery/conf/lang}
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/contractions_ca.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/contractions_fr.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/contractions_ga.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/contractions_it.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/hyphenations_ga.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stemdict_nl.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stoptags_ja.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_ar.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_bg.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_ca.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_cz.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_da.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_de.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_el.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_en.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_es.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_eu.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_fa.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_fi.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_fr.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_ga.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_gl.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_hi.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_hu.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_hy.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_id.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_it.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_ja.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_lv.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_nl.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_no.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_pt.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_ro.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_ru.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_sv.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_th.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/stopwords_tr.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/lang/userdict_ja.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/protwords.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/schema.xml
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/solrconfig-production.xml
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/solrconfig-server-4.10.4.xml
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/solrconfig.xml
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/solrcore.properties
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/solrcore.properties-production
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/stopwords.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/conf/synonyms.txt
V		{warcbase-hbase ← }/src/main/solr/discovery/core.properties
V		{warcbase-hbase ← }/src/main/solr/solr.xml
V		{warcbase-hbase ← }/src/main/solr/zoo.cfg
A	(dir)	warcbase-hbase/src/test/
A	(dir)	warcbase-hbase/src/test/java/
A	(dir)	warcbase-hbase/src/test/java/org/
A	(dir)	warcbase-hbase/src/test/java/org/warcbase/
V	(dir)	{warcbase-hbase/src/test/java/org/warcbase/data/ ← src/test/java/org/warcbase/data}
V		{warcbase-hbase ← }/src/test/java/org/warcbase/data/UrlMappingTest.java
V		{warcbase-hbase ← }/src/test/java/org/warcbase/data/UrlUtilsTest.java