Homec4science
Diffusion warcbase (master) Graph

R1473/src/main/java/org/warcbase
master

History Graph

CommitAuthorDetailsCommitted
627743751578milad621
juniversalchardet removed. 
Apr 19 2014
0d34860367d9milad621
For issue17 it now shows a page with three frames, each has a seperate search… 
Apr 19 2014
8d9ef6d2ac6bJeffyrao
add extract site-level links 
Apr 16 2014
fd511b6eaf7bmilad621
Fixed Issue 24. 
Apr 6 2014
997a8e84e816lintool
Exposed -getPrefix command-line option. 
Mar 31 2014
758948b463b9lintool
Light refactoring, fixed a few errors. 
Mar 31 2014
5227ba65232clintool
Merge branch 'extract-links' of github.com:Jeffyrao/warcbase into fst 
Mar 31 2014
e3ccb37b2e1dJeffyrao
remove TestFSTPrefix class 
Mar 31 2014
71c68ecae0faJeffyrao
add prefix search feature to UriMapping class; add test class for UriMapping… 
Mar 31 2014
2f1574575438lintool
More work on Issue #22: prev/next buttons turns on and off appropriately. 
Mar 29 2014
a48c65e498abJeffyrao
add Lucene FST prefix search 
Mar 28 2014
6d1747bd14afmilad621
Fixed issue 18. 
Mar 28 2014
88e5d97286e4milad621
Fixed issues 21 and 22. 
Mar 28 2014
dac5827fa1e5milad621
Merge branch 'master' of https://github.com/lintool/warcbase 
Mar 28 2014
f2f1e34837f3milad621
merged with master 
Mar 28 2014
3da8131e077blintool
Minor code cleanup. 
Mar 27 2014
d965c015a914milad621
Bug with getting close dates fixed. 
Mar 26 2014
be5a4d366786lintool
Fixed bug with MAX_VERSION blowing up array length. 
Mar 25 2014
72b9558b3ca5lintool
Merge branch 'issue12' of github.com:milad621/warcbase into multi_version_cell 
Mar 25 2014
ef6650cb7bc4milad621
fixing branches 
Mar 25 2014
0d43c98ee177milad621
Fixed issue12 and some issues with dates in browser. 
Mar 25 2014
6c48ce81ee80milad621
Issue12 fixed 
Mar 24 2014
e0d6be1055f4milad621
test push 
Mar 24 2014
252eb87988b1milad621
issue12 fixed 
Mar 24 2014
24370c4f051fJeffyrao
update README file, add steps for getting URLs and extracting links 
Mar 21 2014
cb9e68d4f94dJeffyrao
add ExtractLinks and PrefixQuery(ExtractSiteLinks) 
Mar 20 2014
374b6f2c2173lintool
Adds Snappy compression. 
Mar 19 2014
232a07ac56c9cneud
Merge remote-tracking branch 'lintool/master' 
Mar 18 2014
303860c9ae3blintool
Reformatting source. 
Mar 17 2014
c52474128b38lintool
Merge branch 'disable_wal' of github.com:lintool/warcbase into disable_wal 
Mar 17 2014
d543a4957916lintool
Merge branch 'master' into disable_wal 
Mar 17 2014
b430dc7c94f6lintool
Merge branch 'jar_upgrade' into disable_wal 
Mar 17 2014
8c649e5336aflintool
Fixed Issue #7 
Mar 17 2014
a33f92f340e1lintool
Added try/catch block in ingestion code. 
Mar 17 2014
53aacb335403Jeffyrao
code reformat via eclipse imported project 
Mar 17 2014
987f9518ebb8lintool
Ignores all files that aren't WARC/ARC files in ingest. 
Mar 16 2014
dfb63a169b43lintool
Tried disabling WAL to see impact on performance. 
Mar 16 2014
1ad985045d9aJeffyrao
code reformat 
Mar 16 2014
073fd3206185lintool
Simplified Maven dependencies; Moved from IA artifacts to openwayback artifacts. 
Mar 16 2014
228ca87cd0c3lintool
Light refactoring. 
Mar 16 2014
19453fc7ac8flintool
whitespace 
Mar 16 2014
1e0783ff8a6clintool
Merge branch 'new_hbase_structure' of https://github.com/milad621/warcbase into… 
Mar 16 2014
fdb75decbdf4lintool
Whitespace. 
Mar 15 2014
d86dcb73c854lintool
Merge branch 'extract-links' of https://github.com/Jeffyrao/warcbase into… 
Mar 15 2014
75a0820c382emilad621
servlet updated with the new hbase structure. 
Dec 28 2013
db4ebe9f4826pmd
Added a null pointer check and a more Pig friendly return value from the UDFs 
Dec 19 2013
ba201c27e210pmd
Refactored the configuration of the magic lib into the Pig script. 
Dec 19 2013
5d312a3f80ecpmd
Removed warnings 
Dec 19 2013
b3fdd488fdcbpmd
Refactored the DetectMimeType into two seperate methods: one for each detection… 
Dec 19 2013
a80e87ca6f74milad621
new hbase structure for ingest files 
Dec 11 2013
d6c0cec7efb4pmd
Added a TODO comment 
Dec 11 2013
259f0057f400pmd
Add identification engine as a parameter to the DetectMimeType UDF 
Dec 9 2013
b69563d53c83pmd
Enable the ArcLoader to load all types of files 
Dec 9 2013
2b49ec425597Jeffyrao
check text/html type and modify Jsoup.parse charset as ISO-8859-1 
Dec 8 2013
2e7e0cb81775Jeffyrao
modify UriMappingBuilder to read all files under given directory 
Dec 7 2013
c4248d61b8b7lintool
Simple MapReduce program to count number of unique URLs. 
Dec 7 2013
8e1cadb7e70eJeffyrao
Extract links and Lucene FST for URLs. 
Dec 6 2013
bbee2e38f787milad621
removed dead code 
Dec 5 2013
1a209cf5b04apmd
First version of a magic lib UDF 
Dec 4 2013
ccb9ea44f90acneud
use tika for mime type detection 
Dec 3 2013
fbf902fcd066cneud
use tika for language detection 
Dec 2 2013
eb848477370alintool
Added WarcLoader 
Dec 2 2013
1c57edb9e225lintool
Added ExtractRawText UDF, tweaked ExtractLinks. 
Dec 2 2013
1fd881ddf836lintool
Loader now materializes actual text, added ExtractLinks UDF. 
Dec 2 2013
02317746e1b2lintool
Added simple Pig Loader for Arc files, returns (url, time, mime) currently. 
Dec 2 2013
19735bbbbd86milad621
Started a new branch to extract text from warcbase data and organize the data… 
Dec 1 2013
944b138ee0d2milad621
started a new branch to extract text from warcbase data and organize the data… 
Dec 1 2013
7ee8dbb88c3clintool
Improved error checking for dates. 
Nov 25 2013
7e76e19f2547milad621
PrintAllUris add to appassembler which will output a urls.html file with all… 
Nov 23 2013
22dd0757c01elintool
Tweaked browser; added MR programs for simple content analysis. 
Nov 23 2013
b5b4e211ea05lintool
Hadoop InputFormats for ARC and WARC files + simple demos. 
Nov 22 2013
bf51f6b75bd8milad621
Code refactoring after pair coding. 
Nov 18 2013
43d12364a70fmilad621
Fixed ava.lang.NegativeArraySizeException at org.apache.commons.io.output. 
Nov 8 2013
c3e37d50e856milad621
Created a new runnable to find a uri inside warc/arc files. 
Nov 7 2013
33966c708354milad621
Uses HTablePool instead of creating a new connection each time. 
Nov 7 2013
64ef175db2fdmilad621
Some code cleanup in servlet. DetectDuplicates fixed with new hbase table… 
Nov 7 2013
bf1b3616e4c0milad621
updated servlet. 
Nov 7 2013
76949eacabd5milad621
Added a seperate class to manage HBase connection and addRecord 
Nov 6 2013
01dff2a1f2a5milad621
One runnable to process both arc and warc files in a folder. 
Nov 5 2013
1354c68684d9milad621
fixed some issues with the servlet. content and types might not follow eachother 
Oct 30 2013
e9e3d1c9d940milad621
IngestWarcFiles fixed. Now it uses jwat-warc to ingest warcfiles to hbase. 
Oct 30 2013
70c00addd99emilad621
Updated WarcBrowser to work with the new structure of hbase table (supports… 
Oct 24 2013
e8f7af4176f8milad621
IngestArc updated. 
Oct 23 2013
735b77efcad4milad621
IngestWarcFiles updated. Uses jwat and stores content type in htable 
Oct 23 2013
b1bee7dc3a8emilad621
Arc Processing tools added. Not working with hbase yet. 
Oct 17 2013
803daecd43efmilad621
No need for name arg 
Oct 9 2013
b321544c8a1fmilad621
url style fixed. Can capture table names from url and home page changed to http… 
Oct 9 2013
4b7ec6fab139milad621
URL style changed 
Oct 2 2013
3770fab2cafemilad621
close button fixed 
Sep 27 2013
127a533cf731milad621
fixed a baseurl problem 
Sep 24 2013
c27fb0fb72d9milad621
A bug with swf files fixed 
Sep 20 2013
dc20e4fc805fmilad621
header (navigation bar?) added 
Sep 17 2013
24cf7be01beamilad621
Some url bugs with senate data fixed. A few code refactoring done 
Aug 17 2013
392b5d139973milad621
Some urls fixed. House is OK now 
Aug 16 2013
e7a88b5211bbmilad621
urls 
Aug 15 2013
55b82f3dcb61milad621
urls 
Aug 15 2013
142aa0c8a34fmilad621
urls 
Aug 15 2013
1331620ce670milad621
urls 
Aug 15 2013
ecb3d1dd7e6dmilad621
urls 
Aug 15 2013
bb3985aec41amilad621
urls 
Aug 15 2013
Prev
1
2
3
Next
c4science · Help