Homec4science
Diffusion warcbase (master) Graph

R1473/src
master

History Graph

CommitAuthorDetailsCommitted
8057c46945d0Alice-Z
Add Spark support 
Nov 3 2015
35e39da6e72dAlice-Z
add extractCrawldateDomainUrlBody 
Oct 23 2015
e5821091eccaAlice-Z
update ArcRecords interface 
Oct 21 2015
cfe76f600c7dAlice-Z
extend RDD 
Oct 15 2015
e77201892789Alice-Z
first commit 
Oct 15 2015
258a159c796elintool
Fixed issue #137 Integrate with warc-hadoop-indexer for Shine 
Jul 22 2015
17393c207934lintool
Complete revamped cmdline parameter handling. 
Jul 21 2015
0be900c474eflintool
Simplified specification of configs. 
Jul 21 2015
243004be2dedlintool
Config for WARCIndexer. 
Jul 21 2015
b520faa88283lintool
Refactoring to simplify code. 
Jul 21 2015
7936dd172aaelintool
Fixed: https://github.com/ukwa/webarchive-discovery/issues/64 
Jul 14 2015
38ac5fb5cba5lintool
Rename. 
Jul 13 2015
151242d2d988lintool
Upgrade CDH; fixed broken tests due classpath conflict issue and Tika upgrade. 
Jul 13 2015
a3eb65567484lintool
More code simplification and refactoring. 
Jul 13 2015
46d7cef43ee3lintool
Refactoring to simplify code. 
Jul 13 2015
befb3f9b39fblintool
Reformat code. 
Jul 13 2015
04f416e56781lintool
Getting rid of external dependencies by checking in Solr configs. 
Jul 13 2015
335d751b425clintool
Starting to incorporate Hadoop WARC indexing code. 
Jul 13 2015
11dfe152d4ccJeremy Wiebe
Added ExtractBoilerpipeText UDF 
Jun 30 2015
c9f27489b093lintool
Merge branch 'master' into pyspark 
Jun 14 2015
51abf7e86696ianmilligan1
testing fork 
Jun 9 2015
7cb0c2e8d2ccJeremy Wiebe
added Python scripts dir 
Jun 9 2015
9b8e85db50e3lintool
removed JWAT testcase. 
Jun 7 2015
d9b553c76cedlintool
Removed JWAT; removed unneeded files in org.warcbase.analysis; switched… 
Jun 7 2015
caf87fbecf45lintool
Added schema metadata in loader. 
Jun 7 2015
49fc639ae5b6lintool
Merge branch 'pig-warc' of github.com:lintool/warcbase into pig-warc 
Jun 7 2015
195f4af7474elintool
revamped tests. 
Jun 7 2015
1f9d853f38a2lintool
better trapping of errors. 
Jun 7 2015
84ebe4f68bbdlintool
Added test case for PigWarcLoader. 
Jun 6 2015
13619567667flintool
Changed over to WacWarcInputFormatTest as the underlying InputFormat. 
Jun 6 2015
7c1ebde17fb4lintool
Better integration of ARC readers in pyspark. 
Jun 4 2015
b220b0cc175alintool
Converter to extract all metadata. 
May 27 2015
649f1674352blintool
Initial experiments with Python converters to use Hadoop InputFormat from… 
May 27 2015
1c013a7b064fJeremy Wiebe
Added Stanford NER UDF 
May 26 2015
eec01a41b18elintool
Cleanup. 
May 24 2015
1c9b25027e73lintool
Wraps a try block around everything to catch all errors. 
May 24 2015
9acc8ca74c0alintool
Merge branch 'master' into extract-pdf-udf 
May 24 2015
a3a2cd7853c8lintool
tries to fix issues with relative links 
May 13 2015
0a781f364480lintool
Added test case for ExtractLinks UDF. 
May 6 2015
5b10497c2d53lintool
Updated UDF to handle relative paths (with source page). Added test case. 
May 6 2015
fb95f3ae562elintool
UDF for extracting top-level domain from URL. 
May 6 2015
f263769c86b7lintool
Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112 
Dec 25 2014
52b9696fba87lintool
Fixed typo. 
Dec 24 2014
5d471280e7dblintool
Add ingestion option to select either Snappy or GZ compression, Issue #110 
Dec 24 2014
fb3edb4e40c7lintool
Renaming. 
Dec 23 2014
f6caf0f6f226lintool
Fixed OOM issues. 
Dec 23 2014
b1c6995aa034lintool
Fixed warnings. 
Dec 23 2014
59949b3d7af4lintool
Fixed OOM errors. 
Dec 23 2014
e6b21cc557d6lintool
Fixed Issue #109: OOM when running UrlMappingBuilder 
Dec 23 2014
6acf52f320cflintool
Merge branch 'hbase-api-refactoring' into dev 
Dec 22 2014
7f6d571424a9lintool
Changed compression back from GZ to Snappy. 
Dec 21 2014
12a00be93209lintool
Fixed minor code formatting issues. 
Dec 21 2014
5c2540099370lintool
Reformatting. 
Dec 18 2014
dbaa917eb8b2Jeremy Wiebe
Updated regex for "Content-Type" (RFC2616 sec. 4.2 says HTTP header field names… 
Dec 11 2014
329ddf9ce247Jeremy Wiebe
Use WarcRecordUtils.getWarcResponseMimeType() for ingest 
Dec 11 2014
8b1e82f16828Jeremy Wiebe
Added WARC support to analysis/graph classes (except ExtractLinksJwat -- since… 
Dec 11 2014
eb28dc02a8d2Jeremy Wiebe
Added getBodyContent(WARCRecord r) method to WarcRecordUtils 
Dec 11 2014
ec4285807f39Jeremy Wiebe
Added WARC support to analysis: 
Dec 10 2014
65837e09aa3aJeremy Wiebe
Added WARC support to UrlMappingMapReduceBuilder. It can now accept a path… 
Dec 9 2014
c49c58f76ed3rwolniak
ExtractTextFromPDFs updated 
Dec 9 2014
cea1f0cae974rwolniak
Updated PIG Tika Parser with new code that should work. Awaiting test on Hadoop 
Dec 9 2014
724eed27b771Jeremy Wiebe
Added WARC support to WarcbaseResoureStore. OpenWayback replay of WARC files… 
Dec 8 2014
4239f1f9ae87Jeremy Wiebe
Added WARC file support to IngestFiles. 
Dec 8 2014
3d8672c631dbJeremy Wiebe
Fixed bug in ExtractLinksWac where fst.loadMapping() would choke on filename… 
Dec 8 2014
5590850ab2f0Jeremy Wiebe
For building on OS X, changed HBase compression from Snappy to GZ. 
Dec 8 2014
97550c3727c5Jeremy Wiebe
Ooops. *Now* synced with lintool's master… 
Dec 8 2014
0379553cc55cJeremy Wiebe
Merge branch 'master' of https://github.com/lintool/warcbase 
Dec 8 2014
8a3fed9342fcrwolniak
Still testing ExtractTextFromPDFs 
Nov 12 2014
10b92d78f6edrwolniak
Extract text from PDF UDF. Currently still debugging 
Nov 10 2014
b8915343f5c9lintool
Minor refactoring. 
Oct 31 2014
0145d74ed35elintool
Refactoring to create method that extracts MIME from WARC response records. 
Oct 22 2014
d491020c8287lintool
Merge branch 'pig' into warc 
Oct 19 2014
410cfd81a069lintool
WARC-related Hadoop bindings. 
Oct 19 2014
f4a249469cbclintool
Minor refactoring. 
Oct 19 2014
0c52caed50cflintool
Pig ArcLoader exports its own ResourceSchema. 
Oct 19 2014
f97523f440fflintool
Merge branch 'master' into hbase-api-refactoring 
Oct 19 2014
3af097ea655dlintool
Fixed test cases. 
Oct 19 2014
4ceefef78cdclintool
Pig loader materializes the actual content. 
Oct 19 2014
58df7da00880lintool
Refactored Pig loaded to use WAC API. 
Oct 18 2014
de4267bf28f1lintool
Updated documentation. 
Oct 13 2014
457a71345d2blintool
Merge branch 'master' into warc 
Sep 15 2014
2b8e721063falintool
Refactored to eliminate deprecated HBase APIs. 
Sep 14 2014
05db518ccd83lintool
Added timing info. 
Sep 14 2014
85c5b4a2ecc5lintool
Added debug output; Fixed deprecated HBase APIs. 
Sep 14 2014
159596e9b378lintool
Fixed build issues in upgrade to CDH 5.1.2. 
Sep 13 2014
e05a7b2f92eeJeremy Wiebe
Updated pom.xml and code to use current stable releases of Hadoop and HBase, et… 
Sep 3 2014
4798a4314b6elintool
Figured out how to extract MIME type and date from WARC. 
Aug 30 2014
f3516c7fd7f0lintool
Added test cases to try loading WARC records from a stream; back-ported same… 
Aug 29 2014
28c5c007f4fdlintool
Added simple test case. 
Aug 28 2014
8e67d49d44b3lintool
WARC sample from https://archive.org/details/ExampleArcAndWarcFiles 
Aug 28 2014
9faed3817dcflintool
Refactored WacMapReduceHBaseWrapperDemo, now takes advantage of… 
Aug 23 2014
999fa0af0280lintool
Merge branch 'working' into table-wrapper 
Aug 23 2014
08530c1cbe52lintool
WacArcInputFormat now generates ArcRecordWritables. 
Aug 23 2014
98d7150f5eeclintool
Switched ExtractSiteLinks and InvertAnchorText over to WacArcInputFormat; link… 
Aug 23 2014
369ba2731f5blintool
Minor refactoring, revised counters. 
Aug 23 2014
74b023ed8b16lintool
Both the Jwat and Wac versions of ExtractLinks gives the same exact output. 
Aug 23 2014
ccee8fd2204alintool
Implemented ArcRecordWritable. 
Aug 23 2014
846ebb216100lintool
Minor refactoring. 
Aug 23 2014
c5f4b9efa212lintool
Very rough prototype of wrapper that allows interoperability between HBase… 
Aug 23 2014
47f3c46c099dlintool
Added Hadoop bindings for webarchive-commons ARC readers, demo, test cases. 
Aug 22 2014
1
2
Next
c4science · Help