Homec4science
Diffusion warcbase (master) Graph

R1473/src
master

History Graph

CommitAuthorDetailsCommitted
0a781f364480lintool
Added test case for ExtractLinks UDF. 
May 6 2015
5b10497c2d53lintool
Updated UDF to handle relative paths (with source page). Added test case. 
May 6 2015
fb95f3ae562elintool
UDF for extracting top-level domain from URL. 
May 6 2015
f263769c86b7lintool
Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112 
Dec 25 2014
52b9696fba87lintool
Fixed typo. 
Dec 24 2014
5d471280e7dblintool
Add ingestion option to select either Snappy or GZ compression, Issue #110 
Dec 24 2014
fb3edb4e40c7lintool
Renaming. 
Dec 23 2014
f6caf0f6f226lintool
Fixed OOM issues. 
Dec 23 2014
b1c6995aa034lintool
Fixed warnings. 
Dec 23 2014
59949b3d7af4lintool
Fixed OOM errors. 
Dec 23 2014
e6b21cc557d6lintool
Fixed Issue #109: OOM when running UrlMappingBuilder 
Dec 23 2014
6acf52f320cflintool
Merge branch 'hbase-api-refactoring' into dev 
Dec 22 2014
7f6d571424a9lintool
Changed compression back from GZ to Snappy. 
Dec 21 2014
12a00be93209lintool
Fixed minor code formatting issues. 
Dec 21 2014
dbaa917eb8b2Jeremy Wiebe
Updated regex for "Content-Type" (RFC2616 sec. 4.2 says HTTP header field names… 
Dec 11 2014
329ddf9ce247Jeremy Wiebe
Use WarcRecordUtils.getWarcResponseMimeType() for ingest 
Dec 11 2014
8b1e82f16828Jeremy Wiebe
Added WARC support to analysis/graph classes (except ExtractLinksJwat -- since… 
Dec 11 2014
eb28dc02a8d2Jeremy Wiebe
Added getBodyContent(WARCRecord r) method to WarcRecordUtils 
Dec 11 2014
ec4285807f39Jeremy Wiebe
Added WARC support to analysis: 
Dec 10 2014
65837e09aa3aJeremy Wiebe
Added WARC support to UrlMappingMapReduceBuilder. It can now accept a path… 
Dec 9 2014
724eed27b771Jeremy Wiebe
Added WARC support to WarcbaseResoureStore. OpenWayback replay of WARC files… 
Dec 8 2014
4239f1f9ae87Jeremy Wiebe
Added WARC file support to IngestFiles. 
Dec 8 2014
3d8672c631dbJeremy Wiebe
Fixed bug in ExtractLinksWac where fst.loadMapping() would choke on filename… 
Dec 8 2014
5590850ab2f0Jeremy Wiebe
For building on OS X, changed HBase compression from Snappy to GZ. 
Dec 8 2014
97550c3727c5Jeremy Wiebe
Ooops. *Now* synced with lintool's master… 
Dec 8 2014
0379553cc55cJeremy Wiebe
Merge branch 'master' of https://github.com/lintool/warcbase 
Dec 8 2014
b8915343f5c9lintool
Minor refactoring. 
Oct 31 2014
0145d74ed35elintool
Refactoring to create method that extracts MIME from WARC response records. 
Oct 22 2014
d491020c8287lintool
Merge branch 'pig' into warc 
Oct 19 2014
410cfd81a069lintool
WARC-related Hadoop bindings. 
Oct 19 2014
f4a249469cbclintool
Minor refactoring. 
Oct 19 2014
0c52caed50cflintool
Pig ArcLoader exports its own ResourceSchema. 
Oct 19 2014
f97523f440fflintool
Merge branch 'master' into hbase-api-refactoring 
Oct 19 2014
3af097ea655dlintool
Fixed test cases. 
Oct 19 2014
4ceefef78cdclintool
Pig loader materializes the actual content. 
Oct 19 2014
58df7da00880lintool
Refactored Pig loaded to use WAC API. 
Oct 18 2014
de4267bf28f1lintool
Updated documentation. 
Oct 13 2014
457a71345d2blintool
Merge branch 'master' into warc 
Sep 15 2014
2b8e721063falintool
Refactored to eliminate deprecated HBase APIs. 
Sep 14 2014
05db518ccd83lintool
Added timing info. 
Sep 14 2014
85c5b4a2ecc5lintool
Added debug output; Fixed deprecated HBase APIs. 
Sep 14 2014
159596e9b378lintool
Fixed build issues in upgrade to CDH 5.1.2. 
Sep 13 2014
e05a7b2f92eeJeremy Wiebe
Updated pom.xml and code to use current stable releases of Hadoop and HBase, et… 
Sep 3 2014
4798a4314b6elintool
Figured out how to extract MIME type and date from WARC. 
Aug 30 2014
f3516c7fd7f0lintool
Added test cases to try loading WARC records from a stream; back-ported same… 
Aug 29 2014
28c5c007f4fdlintool
Added simple test case. 
Aug 28 2014
8e67d49d44b3lintool
WARC sample from https://archive.org/details/ExampleArcAndWarcFiles 
Aug 28 2014
9faed3817dcflintool
Refactored WacMapReduceHBaseWrapperDemo, now takes advantage of… 
Aug 23 2014
999fa0af0280lintool
Merge branch 'working' into table-wrapper 
Aug 23 2014
08530c1cbe52lintool
WacArcInputFormat now generates ArcRecordWritables. 
Aug 23 2014
98d7150f5eeclintool
Switched ExtractSiteLinks and InvertAnchorText over to WacArcInputFormat; link… 
Aug 23 2014
369ba2731f5blintool
Minor refactoring, revised counters. 
Aug 23 2014
74b023ed8b16lintool
Both the Jwat and Wac versions of ExtractLinks gives the same exact output. 
Aug 23 2014
ccee8fd2204alintool
Implemented ArcRecordWritable. 
Aug 23 2014
846ebb216100lintool
Minor refactoring. 
Aug 23 2014
c5f4b9efa212lintool
Very rough prototype of wrapper that allows interoperability between HBase… 
Aug 23 2014
47f3c46c099dlintool
Added Hadoop bindings for webarchive-commons ARC readers, demo, test cases. 
Aug 22 2014
b07376aba3felintool
Added/refactored test cases for JWAT. 
Aug 22 2014
4663558b1d1alintool
Added @Override annotations to appropriate methods. 
Aug 22 2014
9cbd26b2e3bblintool
Minor tweaks. 
Aug 19 2014
43e952bfdd99lintool
Merge branch 'selenium' into working 
Aug 19 2014
3c7da3b094fdlintool
Refactoring of Wayback/Warcbase integration points. 
Aug 19 2014
c5b4973aba96lintool
Merge branch 'master' into selenium 
Aug 17 2014
4090318d2f99lintool
Cleaned up scripts. 
Aug 17 2014
b02fd402870flintool
Cute Selenium browser to conduct a random walk through the archive. 
Aug 17 2014
52cf7bb90940lintool
Cleaned up programs for manipulating graphs; removed scanning HBase option for… 
Aug 17 2014
f7eefd5da751lintool
Removed outdated code; cleaned up WarcBrowserServlet. 
Aug 16 2014
a57c6b83eaf2lintool
Refactoring: created ArcRecordUtils. 
Aug 16 2014
d58afca206fclintool
Refactoring packge org.warcbase.demo; added Jwat prefix to Hadoop InputFormats… 
Aug 16 2014
377a3e3a429flintool
Better handling of ARC parse errors. 
Aug 15 2014
2d16fd032060lintool
Bump max value size up to 10 MB, tweak WAL settings. 
Aug 14 2014
bf606c07d453lintool
Refactoring UrlUtil and related classes. 
Aug 14 2014
b6431bed6f8blintool
Minor tweaks. 
Aug 14 2014
72c1afbca77clintool
Merge branch 'master' into cneud-integration 
Aug 14 2014
f29edf052e19lintool
Added command-line options. 
Aug 14 2014
14fa0f329072lintool
More refactoring. 
Aug 14 2014
6eeb0ea431f1lintool
Refactoring on local UrlMappingBuilder. 
Aug 14 2014
5022874b83e5lintool
Lightweight refactoring. 
Aug 14 2014
0bc5d4876199lintool
Uri -> Url classes renaming. 
Aug 14 2014
ab64384ff204lintool
Light refactoring 
Aug 14 2014
bc47be0fea19Jeffyrao
resolve conflicts 
Aug 13 2014
0623b52cd05cJeffyrao
reformat code 
Aug 13 2014
83414e51b351lintool
Configuration for Wayback/Warcbase integration. 
Aug 13 2014
910c2f03e6b1lintool
Removed extra command-line argument. 
Aug 13 2014
a14cda2217falintool
Commented out LibmagicJnaWrapper functionality because the jar isn't generally… 
Aug 12 2014
93f6d42f4ff4lintool
Fixed compile and broken test issues. 
Aug 12 2014
accd1978862dlintool
Merge branch 'master' of github.com:cneud/warcbase into cneud-integration 
Aug 12 2014
eb4893e7201clintool
Merge branch 'cleanup' into wayback-integration 
Aug 12 2014
c665a23bf1f3lintool
Fixes issue #58: Wayback reads directly from REST API instead of writing and… 
Aug 12 2014
ffec5b2caa3dlintool
Fixed issue #59: Unable to fetch URLs from archive with '?' in them 
Aug 12 2014
0f4f541c24d7lintool
Fixed issues with fetching URLs with spaces in them. 
Aug 12 2014
b5ccb86d330elintool
Better handling of errors: when REST API is unavailable, when URL isn't found… 
Aug 11 2014
515bab098cfflintool
Code cleanup for browser code; removed unneeded files and associated web files. 
Aug 11 2014
74f3dece6c62lintool
Merge branch 'rest-api-bug-fix' of github.com:lintool/warcbase into wayback… 
Aug 11 2014
3075ca19d76dlintool
Converted host/port/table information to bean settings. 
Aug 11 2014
37e97073d57clintool
Simplified code. 
Aug 11 2014
bca5b6944ac7lintool
Refactoring; mostly reformatting. 
Aug 11 2014
9e8765b4e3f0lintool
Minor fix for NPE when capture isn't in HBase. 
Aug 11 2014
4f387e8952a7lintool
Initial check-in of Warcbase integration points with Open Wayback. 
Aug 11 2014
03d5c5365481lintool
/*/ query returns MIME type. 
Aug 11 2014
1
2
Next
c4science · Help