Diffusion warcbase (master) History

History
master939473b8d15786f72ea8382…

Dec 4 2013

Added .idea
939473b8d157
·pmd on Dec 4 2013, 13:19.
force maven to use Java 1.7
a1d8f3eed43e
·cneud on Dec 4 2013, 09:28.

Dec 3 2013

Merge pull request #1 from perdalum/pig-integration
030ffa9f9449
·Clemens Neudecker on Dec 3 2013, 15:29.
use tika for mime type detection
ccb9ea44f90a
·cneud on Dec 3 2013, 15:23.
Added unit test for the language detection UDF.
c0e996ec2e61
·pmd on Dec 3 2013, 15:03.

Dec 2 2013

use tika for language detection
fbf902fcd066
·cneud on Dec 2 2013, 18:05.
Cleaned up Pig test cases, added JWAT test case.
a4282d81d6e9
·lintool on Dec 2 2013, 17:33.
Merge branch 'pig' of https://github.com/graemon/warcbase into pig-integration
64daa7dbb395
·lintool on Dec 2 2013, 16:34.
Added WarcLoader
eb848477370a
·lintool on Dec 2 2013, 16:32.
added a pig unit test
17bc9616a180
·graemon on Dec 2 2013, 16:28.
Added ExtractRawText UDF, tweaked ExtractLinks.
1c57edb9e225
·lintool on Dec 2 2013, 15:40.
added a pig unit test
8885f5db4ded
·graemon on Dec 2 2013, 14:55.
Loader now materializes actual text, added ExtractLinks UDF.
1fd881ddf836
·lintool on Dec 2 2013, 13:26.
Added simple Pig Loader for Arc files, returns (url, time, mime) currently.
02317746e1b2
·lintool on Dec 2 2013, 11:58.

Nov 25 2013

Improved error checking for dates.
7ee8dbb88c3c
·lintool on Nov 25 2013, 00:44.

Nov 23 2013

Tweaked browser; added MR programs for simple content analysis.
22dd0757c01e
·lintool on Nov 23 2013, 18:11.

Nov 22 2013

Hadoop InputFormats for ARC and WARC files + simple demos.
b5b4e211ea05
·lintool on Nov 22 2013, 17:17.

Nov 20 2013

Merge branch 'working'
0e4d1592c244
·lintool on Nov 20 2013, 15:47.
Tweaked README.
0aeaeb3a754b
·lintool on Nov 20 2013, 15:46.

Nov 18 2013

Code refactoring after pair coding.
bf51f6b75bd8
·milad621 on Nov 18 2013, 13:22.

Nov 8 2013

Fixed ava.lang.NegativeArraySizeException at org.apache.commons.io.output.
43d12364a70f
·milad621 on Nov 8 2013, 22:35.

Nov 7 2013

Created a new runnable to find a uri inside warc/arc files.
c3e37d50e856
·milad621 on Nov 7 2013, 19:32.
Uses HTablePool instead of creating a new connection each time.
33966c708354
·milad621 on Nov 7 2013, 06:28.
Some code cleanup in servlet. DetectDuplicates fixed with new hbase table…
64ef175db2fd
·milad621 on Nov 7 2013, 05:43.
updated servlet.
bf1b3616e4c0
·milad621 on Nov 7 2013, 05:23.

Nov 6 2013

Added a seperate class to manage HBase connection and addRecord
76949eacabd5
·milad621 on Nov 6 2013, 23:27.

Nov 5 2013

One runnable to process both arc and warc files in a folder.
01dff2a1f2a5
·milad621 on Nov 5 2013, 23:30.

Oct 30 2013

fixed some issues with the servlet. content and types might not follow eachother
1354c68684d9
·milad621 on Oct 30 2013, 13:40.
IngestWarcFiles fixed. Now it uses jwat-warc to ingest warcfiles to hbase.
e9e3d1c9d940
·milad621 on Oct 30 2013, 01:56.

Oct 24 2013

Updated WarcBrowser to work with the new structure of hbase table (supports…
70c00addd99e
·milad621 on Oct 24 2013, 05:02.

Oct 23 2013

IngestArc updated.
e8f7af4176f8
·milad621 on Oct 23 2013, 21:00.
IngestWarcFiles updated. Uses jwat and stores content type in htable
735b77efcad4
·milad621 on Oct 23 2013, 20:22.

Oct 17 2013

Arc Processing tools added. Not working with hbase yet.
b1bee7dc3a8e
·milad621 on Oct 17 2013, 19:33.

Oct 9 2013

No need for name arg
803daecd43ef
·milad621 on Oct 9 2013, 09:44.
url style fixed. Can capture table names from url and home page changed to http…
b321544c8a1f
·milad621 on Oct 9 2013, 09:39.

Oct 2 2013

URL style changed
c860aef8368c
·milad621 on Oct 2 2013, 09:24.
URL style changed
4b7ec6fab139
·milad621 on Oct 2 2013, 09:23.

Oct 1 2013

Merge branch 'milad-master'
79dee598b2df
·lintool on Oct 1 2013, 18:27.

Sep 27 2013

close button fixed
3770fab2cafe
·milad621 on Sep 27 2013, 22:18.

Sep 24 2013

fixed a baseurl problem
127a533cf731
·milad621 on Sep 24 2013, 14:58.

Sep 20 2013

A bug with swf files fixed
c27fb0fb72d9
·milad621 on Sep 20 2013, 21:31.

Sep 17 2013

header (navigation bar?) added. Pom update
8706a55f8e36
·milad621 on Sep 17 2013, 19:16.
header (navigation bar?) added
dc20e4fc805f
·milad621 on Sep 17 2013, 19:15.

Aug 17 2013

Some url bugs with senate data fixed. A few code refactoring done
24cf7be01bea
·milad621 on Aug 17 2013, 20:18.

Aug 16 2013

Some urls fixed. House is OK now
392b5d139973
·milad621 on Aug 16 2013, 17:23.

Aug 15 2013

urls
e7a88b5211bb
·milad621 on Aug 15 2013, 08:04.
urls
55b82f3dcb61
·milad621 on Aug 15 2013, 06:59.
urls
142aa0c8a34f
·milad621 on Aug 15 2013, 06:48.
urls
1331620ce670
·milad621 on Aug 15 2013, 06:41.
urls
ecb3d1dd7e6d
·milad621 on Aug 15 2013, 06:23.
urls
bb3985aec41a
·milad621 on Aug 15 2013, 06:17.
urls
9e729d1fc54f
·milad621 on Aug 15 2013, 06:14.
urls
4f7ad0cbd161
·milad621 on Aug 15 2013, 06:10.
urls
45b62820e8ca
·milad621 on Aug 15 2013, 06:02.
urls
d0d442b95fad
·milad621 on Aug 15 2013, 05:56.
urls
b08e35a91dc0
·milad621 on Aug 15 2013, 05:39.
urls
24d59470fd5e
·milad621 on Aug 15 2013, 05:32.
urls
8c2223aca070
·milad621 on Aug 15 2013, 05:21.
urls
211d4536f4c2
·milad621 on Aug 15 2013, 05:05.

Aug 14 2013

urls for house fixed
b293db10c799
·milad621 on Aug 14 2013, 21:30.

Aug 13 2013

Refactored dependencies.
19ab27133471
·lintool on Aug 13 2013, 15:17.
Merge branch 'ingest-refactoring'
480651f0384b
·lintool on Aug 13 2013, 11:08.
Cleaned up analysis code. Removed dead code. Added README.
3a2aec9c5698
·lintool on Aug 13 2013, 10:54.

Aug 12 2013

Refactoring of browser; each archive is now stored in its own separate table…
4fffd3905e9e
·lintool on Aug 12 2013, 23:11.
Interim check-in, refactoring of ingestion program.
0f580edb6186
·lintool on Aug 12 2013, 14:38.

Aug 8 2013

duplicate count
17e566d63370
·milad621 on Aug 8 2013, 20:34.

Aug 7 2013

duplicate count
52dec81548a6
·milad621 on Aug 7 2013, 09:54.
duplicate count
e5f69f95e7ea
·milad621 on Aug 7 2013, 09:45.
duplicate count
ae13087a3ffc
·milad621 on Aug 7 2013, 09:35.
duplicate count
efb7cebae1b6
·milad621 on Aug 7 2013, 09:31.
duplicate count
992379b830c3
·milad621 on Aug 7 2013, 09:28.
duplicate count
020c97770fe2
·milad621 on Aug 7 2013, 09:26.
duplicate count
42dbb7faf5bb
·milad621 on Aug 7 2013, 09:24.
duplicate count
f33edee2d50f
·milad621 on Aug 7 2013, 09:21.
duplicate count
94d4c056456e
·milad621 on Aug 7 2013, 09:18.
duplicate count
44061ad84f9f
·milad621 on Aug 7 2013, 09:16.
duplicate count
06cefdf9191f
·milad621 on Aug 7 2013, 09:15.
duplicate count
a61852dfcea3
·milad621 on Aug 7 2013, 09:14.
duplicate count
c9ae5b9ff356
·milad621 on Aug 7 2013, 09:10.
duplicate count
c24dba8cb858
·milad621 on Aug 7 2013, 09:09.
duplicate count
5bb4160f39a6
·milad621 on Aug 7 2013, 09:07.
duplicate count
cd1c983fd0c9
·milad621 on Aug 7 2013, 09:02.
duplicate count
176c9170ee3f
·milad621 on Aug 7 2013, 08:24.
duplicate count
ca2f7c1f0995
·milad621 on Aug 7 2013, 08:21.
duplicate count
5c4aad52d667
·milad621 on Aug 7 2013, 05:06.
duplicate count
b679deabebf7
·milad621 on Aug 7 2013, 05:03.
duplicate count
5adb7197f665
·milad621 on Aug 7 2013, 05:01.
duplicate count
36da127d3c9f
·milad621 on Aug 7 2013, 04:57.
duplicate count
c8906c4bb619
·milad621 on Aug 7 2013, 04:53.
duplicate count
fcdd21ba1de4
·milad621 on Aug 7 2013, 04:47.
duplicate count
9f74d062a1f0
·milad621 on Aug 7 2013, 04:35.

Aug 6 2013

pom.xml updated
8c61d21d5167
·milad621 on Aug 6 2013, 21:00.
Duplicates
f67d4cd2b850
·milad621 on Aug 6 2013, 20:29.

Jul 30 2013

Dashboard
e9e904fbd89f
·milad621 on Jul 30 2013, 23:23.
Dashboard
117d39b7fee0
·milad621 on Jul 30 2013, 01:04.

Jul 29 2013

Dashboard
33f60647635e
·milad621 on Jul 29 2013, 20:51.
Dashboard
4ccea04e83ab
·milad621 on Jul 29 2013, 20:50.
Dashboard
5981632ecfbb
·milad621 on Jul 29 2013, 05:52.
Dashboard
fcbee4abf787
·milad621 on Jul 29 2013, 05:50.
Dashboard
499af6b86454
·milad621 on Jul 29 2013, 05:36.

Historymaster939473b8d15786f72ea8382…

Dec 4 2013

Dec 3 2013

Dec 2 2013

Nov 25 2013

Nov 23 2013

Nov 22 2013

Nov 20 2013

Nov 18 2013

Nov 8 2013

Nov 7 2013

Nov 6 2013

Nov 5 2013

Oct 30 2013

Oct 24 2013

Oct 23 2013

Oct 17 2013

Oct 9 2013

Oct 2 2013

Oct 1 2013

Sep 27 2013

Sep 24 2013

Sep 20 2013

Sep 17 2013

Aug 17 2013

Aug 16 2013

Aug 15 2013

Aug 14 2013

Aug 13 2013

Aug 12 2013

Aug 8 2013

Aug 7 2013

Aug 6 2013

Jul 30 2013

Jul 29 2013

History
master939473b8d15786f72ea8382…