History Graph
History Graph
Commit | Author | Details | Committed | |||
---|---|---|---|---|---|---|
67888b472702 | milad621 | Fixed issue29. | May 23 2014 | |||
fcc4acf98a6f | lintool | Simple program to find URL patterns in archives. | May 22 2014 | |||
52b39cfecc82 | lintool | Minor refactoring. | May 6 2014 | |||
0856f3faddd1 | lintool | Merge branch 'master' of github.com:milad621/warcbase into display_issues | May 6 2014 | |||
a4247934797c | milad621 | changed type of nobanner to boolean. | May 5 2014 | |||
c5432ac2468c | milad621 | Working on issue17. Added /noframes to the url but still needs to add warcbase… | May 4 2014 | |||
8799707e9856 | milad621 | Fixed issue 25 | Apr 24 2014 | |||
24d7d8e08807 | Jeffyrao | modified ExtractSiteLinks.java, changed to read csv format of prefix input | Apr 20 2014 | |||
627743751578 | milad621 | juniversalchardet removed. | Apr 19 2014 | |||
0d34860367d9 | milad621 | For issue17 it now shows a page with three frames, each has a seperate search… | Apr 19 2014 | |||
8d9ef6d2ac6b | Jeffyrao | add extract site-level links | Apr 16 2014 | |||
fd511b6eaf7b | milad621 | Fixed Issue 24. | Apr 6 2014 | |||
997a8e84e816 | lintool | Exposed -getPrefix command-line option. | Mar 31 2014 | |||
758948b463b9 | lintool | Light refactoring, fixed a few errors. | Mar 31 2014 | |||
5227ba65232c | lintool | Merge branch 'extract-links' of github.com:Jeffyrao/warcbase into fst | Mar 31 2014 | |||
e3ccb37b2e1d | Jeffyrao | remove TestFSTPrefix class | Mar 31 2014 | |||
71c68ecae0fa | Jeffyrao | add prefix search feature to UriMapping class; add test class for UriMapping… | Mar 31 2014 | |||
2f1574575438 | lintool | More work on Issue #22: prev/next buttons turns on and off appropriately. | Mar 29 2014 | |||
a48c65e498ab | Jeffyrao | add Lucene FST prefix search | Mar 28 2014 | |||
6d1747bd14af | milad621 | Fixed issue 18. | Mar 28 2014 | |||
88e5d97286e4 | milad621 | Fixed issues 21 and 22. | Mar 28 2014 | |||
dac5827fa1e5 | milad621 | Merge branch 'master' of https://github.com/lintool/warcbase | Mar 28 2014 | |||
f2f1e34837f3 | milad621 | merged with master | Mar 28 2014 | |||
3da8131e077b | lintool | Minor code cleanup. | Mar 27 2014 | |||
d965c015a914 | milad621 | Bug with getting close dates fixed. | Mar 26 2014 | |||
be5a4d366786 | lintool | Fixed bug with MAX_VERSION blowing up array length. | Mar 25 2014 | |||
72b9558b3ca5 | lintool | Merge branch 'issue12' of github.com:milad621/warcbase into multi_version_cell | Mar 25 2014 | |||
ef6650cb7bc4 | milad621 | fixing branches | Mar 25 2014 | |||
0d43c98ee177 | milad621 | Fixed issue12 and some issues with dates in browser. | Mar 25 2014 | |||
6c48ce81ee80 | milad621 | Issue12 fixed | Mar 24 2014 | |||
e0d6be1055f4 | milad621 | test push | Mar 24 2014 | |||
252eb87988b1 | milad621 | issue12 fixed | Mar 24 2014 | |||
24370c4f051f | Jeffyrao | update README file, add steps for getting URLs and extracting links | Mar 21 2014 | |||
cb9e68d4f94d | Jeffyrao | add ExtractLinks and PrefixQuery(ExtractSiteLinks) | Mar 20 2014 | |||
374b6f2c2173 | lintool | Adds Snappy compression. | Mar 19 2014 | |||
232a07ac56c9 | cneud | Merge remote-tracking branch 'lintool/master' | Mar 18 2014 | |||
303860c9ae3b | lintool | Reformatting source. | Mar 17 2014 | |||
c52474128b38 | lintool | Merge branch 'disable_wal' of github.com:lintool/warcbase into disable_wal | Mar 17 2014 | |||
d543a4957916 | lintool | Merge branch 'master' into disable_wal | Mar 17 2014 | |||
b430dc7c94f6 | lintool | Merge branch 'jar_upgrade' into disable_wal | Mar 17 2014 | |||
8c649e5336af | lintool | Fixed Issue #7 | Mar 17 2014 | |||
a33f92f340e1 | lintool | Added try/catch block in ingestion code. | Mar 17 2014 | |||
53aacb335403 | Jeffyrao | code reformat via eclipse imported project | Mar 17 2014 | |||
987f9518ebb8 | lintool | Ignores all files that aren't WARC/ARC files in ingest. | Mar 16 2014 | |||
dfb63a169b43 | lintool | Tried disabling WAL to see impact on performance. | Mar 16 2014 | |||
1ad985045d9a | Jeffyrao | code reformat | Mar 16 2014 | |||
073fd3206185 | lintool | Simplified Maven dependencies; Moved from IA artifacts to openwayback artifacts. | Mar 16 2014 | |||
228ca87cd0c3 | lintool | Light refactoring. | Mar 16 2014 | |||
19453fc7ac8f | lintool | whitespace | Mar 16 2014 | |||
1e0783ff8a6c | lintool | Merge branch 'new_hbase_structure' of https://github.com/milad621/warcbase into… | Mar 16 2014 | |||
fdb75decbdf4 | lintool | Whitespace. | Mar 15 2014 | |||
d86dcb73c854 | lintool | Merge branch 'extract-links' of https://github.com/Jeffyrao/warcbase into… | Mar 15 2014 | |||
75a0820c382e | milad621 | servlet updated with the new hbase structure. | Dec 28 2013 | |||
db4ebe9f4826 | pmd | Added a null pointer check and a more Pig friendly return value from the UDFs | Dec 19 2013 | |||
ba201c27e210 | pmd | Refactored the configuration of the magic lib into the Pig script. | Dec 19 2013 | |||
5d312a3f80ec | pmd | Removed warnings | Dec 19 2013 | |||
b3fdd488fdcb | pmd | Refactored the DetectMimeType into two seperate methods: one for each detection… | Dec 19 2013 | |||
a80e87ca6f74 | milad621 | new hbase structure for ingest files | Dec 11 2013 | |||
d6c0cec7efb4 | pmd | Added a TODO comment | Dec 11 2013 | |||
259f0057f400 | pmd | Add identification engine as a parameter to the DetectMimeType UDF | Dec 9 2013 | |||
b69563d53c83 | pmd | Enable the ArcLoader to load all types of files | Dec 9 2013 | |||
2b49ec425597 | Jeffyrao | check text/html type and modify Jsoup.parse charset as ISO-8859-1 | Dec 8 2013 | |||
2e7e0cb81775 | Jeffyrao | modify UriMappingBuilder to read all files under given directory | Dec 7 2013 | |||
c4248d61b8b7 | lintool | Simple MapReduce program to count number of unique URLs. | Dec 7 2013 | |||
8e1cadb7e70e | Jeffyrao | Extract links and Lucene FST for URLs. | Dec 6 2013 | |||
bbee2e38f787 | milad621 | removed dead code | Dec 5 2013 | |||
1a209cf5b04a | pmd | First version of a magic lib UDF | Dec 4 2013 | |||
ccb9ea44f90a | cneud | use tika for mime type detection | Dec 3 2013 | |||
fbf902fcd066 | cneud | use tika for language detection | Dec 2 2013 | |||
eb848477370a | lintool | Added WarcLoader | Dec 2 2013 | |||
1c57edb9e225 | lintool | Added ExtractRawText UDF, tweaked ExtractLinks. | Dec 2 2013 | |||
1fd881ddf836 | lintool | Loader now materializes actual text, added ExtractLinks UDF. | Dec 2 2013 | |||
02317746e1b2 | lintool | Added simple Pig Loader for Arc files, returns (url, time, mime) currently. | Dec 2 2013 | |||
19735bbbbd86 | milad621 | Started a new branch to extract text from warcbase data and organize the data… | Dec 1 2013 | |||
944b138ee0d2 | milad621 | started a new branch to extract text from warcbase data and organize the data… | Dec 1 2013 | |||
7ee8dbb88c3c | lintool | Improved error checking for dates. | Nov 25 2013 | |||
7e76e19f2547 | milad621 | PrintAllUris add to appassembler which will output a urls.html file with all… | Nov 23 2013 | |||
22dd0757c01e | lintool | Tweaked browser; added MR programs for simple content analysis. | Nov 23 2013 | |||
b5b4e211ea05 | lintool | Hadoop InputFormats for ARC and WARC files + simple demos. | Nov 22 2013 | |||
bf51f6b75bd8 | milad621 | Code refactoring after pair coding. | Nov 18 2013 | |||
43d12364a70f | milad621 | Fixed ava.lang.NegativeArraySizeException at org.apache.commons.io.output. | Nov 8 2013 | |||
c3e37d50e856 | milad621 | Created a new runnable to find a uri inside warc/arc files. | Nov 7 2013 | |||
33966c708354 | milad621 | Uses HTablePool instead of creating a new connection each time. | Nov 7 2013 | |||
64ef175db2fd | milad621 | Some code cleanup in servlet. DetectDuplicates fixed with new hbase table… | Nov 7 2013 | |||
bf1b3616e4c0 | milad621 | updated servlet. | Nov 7 2013 | |||
76949eacabd5 | milad621 | Added a seperate class to manage HBase connection and addRecord | Nov 6 2013 | |||
01dff2a1f2a5 | milad621 | One runnable to process both arc and warc files in a folder. | Nov 5 2013 | |||
1354c68684d9 | milad621 | fixed some issues with the servlet. content and types might not follow eachother | Oct 30 2013 | |||
e9e3d1c9d940 | milad621 | IngestWarcFiles fixed. Now it uses jwat-warc to ingest warcfiles to hbase. | Oct 30 2013 | |||
70c00addd99e | milad621 | Updated WarcBrowser to work with the new structure of hbase table (supports… | Oct 24 2013 | |||
e8f7af4176f8 | milad621 | IngestArc updated. | Oct 23 2013 | |||
735b77efcad4 | milad621 | IngestWarcFiles updated. Uses jwat and stores content type in htable | Oct 23 2013 | |||
b1bee7dc3a8e | milad621 | Arc Processing tools added. Not working with hbase yet. | Oct 17 2013 | |||
803daecd43ef | milad621 | No need for name arg | Oct 9 2013 | |||
b321544c8a1f | milad621 | url style fixed. Can capture table names from url and home page changed to http… | Oct 9 2013 | |||
4b7ec6fab139 | milad621 | URL style changed | Oct 2 2013 | |||
3770fab2cafe | milad621 | close button fixed | Sep 27 2013 | |||
127a533cf731 | milad621 | fixed a baseurl problem | Sep 24 2013 | |||
c27fb0fb72d9 | milad621 | A bug with swf files fixed | Sep 20 2013 | |||
dc20e4fc805f | milad621 | header (navigation bar?) added | Sep 17 2013 |
c4science · Help