Diffusion warcbase (master) History

main
master0a781f3644802205ba097d2…

Aug 10 2014

Refactoring; removing WARC ingestion for now.
b20ef84e5df9
·lintool on Aug 10 2014, 14:29.
Tweaks to ingest code.
cfe508a831f0
·lintool on Aug 10 2014, 14:05.
Janky, but seems to work: ingesting and serving up raw ARC records.
180b57fa5dc1
·lintool on Aug 10 2014, 08:28.
Quick and dirty switch over to webarchive-commons API; stores raw ARC records.
8b5a6be9db61
·lintool on Aug 10 2014, 04:16.

Aug 4 2014

add UriMappingBuilder Mapreduce version
65c6e54a948e
·Jeffyrao on Aug 4 2014, 01:48.

Jul 25 2014

fix the bug of selecting webpage by date ineffective
59aa95aab857
·Jeffyrao on Jul 25 2014, 18:36.

Jul 22 2014

add Hadoop/HBase input choice for ExtractLinks and ExtractSiteLinks classes
7175239e0751
·Jeffyrao on Jul 22 2014, 23:43.

Jun 26 2014

Merging with master.
60a9b96beebe
·Milad Gholami on Jun 26 2014, 16:16.
Fixing git history.
149ce6cf969f
·Milad Gholami on Jun 26 2014, 16:03.

Jun 18 2014

More work on the admin interface.
3b8484a944a8
·lintool on Jun 18 2014, 21:48.
issue #50
f3015cd7ba4b
·lintool on Jun 18 2014, 19:57.

Jun 17 2014

Fixed broken merge.
10d60bd28c75
·lintool on Jun 17 2014, 22:48.
Merge branch 'master' into admin
6c452cbb6b5d
·lintool on Jun 17 2014, 22:47.
Fixed issue #48
781fe4247b31
·lintool on Jun 17 2014, 22:41.
Added comment.
5f62c2f6fb9d
·lintool on Jun 17 2014, 22:15.
Minor refactoring.
d17dddca8deb
·lintool on Jun 17 2014, 21:44.
Initial working version of anchor text inversion program: issue #43
ee7f7749a30a
·lintool on Jun 17 2014, 20:06.
Started working on issue #46 cleanup of org.warcbase.data.Util
02c26d6d8ea4
·lintool on Jun 17 2014, 17:20.
Appears to have fixed issue #49, starting work on admin tool, issue #45.
c7a7247d5fa2
·lintool on Jun 17 2014, 04:18.

Jun 13 2014

ExtractLinks using HBase appears to be working.
a6be4375e0c7
·lintool on Jun 13 2014, 03:08.
Refactored HDFS extractor; HBase extractor still broken.
21a07efb350e
·lintool on Jun 13 2014, 00:01.

Jun 12 2014

Merge branch 'master' into refactoring
bbc73ab64808
·lintool on Jun 12 2014, 19:54.
Light refactoring, pushed column family filter into scan.
273e5969e943
·lintool on Jun 12 2014, 19:51.
Alternative implementation based on iterating over maps... slightly slower.
3283bb8512ef
·lintool on Jun 12 2014, 16:18.
More light refactoring.
dbfbcb0b3c7e
·lintool on Jun 12 2014, 15:05.
add hbase option for ExtractLinks
34273fdb935a
·Jeffyrao on Jun 12 2014, 06:34.
reformat Jinfeng's code
cfb89d2d3379
·Jeffyrao on Jun 12 2014, 05:10.

Jun 11 2014

Refactoring, code cleanup.
b02f33c9b43a
·lintool on Jun 11 2014, 22:36.
Fixed issue #39
d4c29085ee12
·lintool on Jun 11 2014, 04:24.
Debugged HBase scan parameters so that they don't knock over region servers…
c3a4348e1250
·lintool on Jun 11 2014, 04:05.

Jun 6 2014

Issue 38 fixed. Still need to add other URL Encoding characters.
0e61a3094550
·milad621 on Jun 6 2014, 19:29.

Jun 5 2014

Moving ExtractLinks and ExtractSiteLinks into analysis.graph package, per Issue…
20ef503dbfe5
·lintool on Jun 5 2014, 22:41.
Extracts values at different timestamps.
dc58365a9579
·lintool on Jun 5 2014, 22:27.
Refactored getIdRange method signature, add more test cases to UriMapping.
60b827d1c174
·lintool on Jun 5 2014, 16:16.
Merge branch 'extract-links' of github.com:Jeffyrao/warcbase into refactoring
a6b17787ccd1
·lintool on Jun 5 2014, 16:04.
fix issue 40 that UriMapping prefix search should return empty result when no…
7fc1b92d60a7
·Jeffyrao on Jun 5 2014, 03:37.

Jun 4 2014

Refactoring, added test case (currently broken).
a8953616248c
·lintool on Jun 4 2014, 21:12.

Jun 3 2014

fix issue 32, update ExtractSiteLinks code
6518836b7ea0
·Jeffyrao on Jun 3 2014, 20:27.

May 29 2014

fix issue 30, add ExtractSiteLinks code
a17e3c74954a
·Jeffyrao on May 29 2014, 20:13.

May 27 2014

fix issue 31
3f36eff4f429
·Jeffyrao on May 27 2014, 04:39.

May 26 2014

Initial MapReduce over HBase demo.
b4ab2ea0d499
·lintool on May 26 2014, 15:30.

May 25 2014

more edits
0598dcd91073
·Jeffyrao on May 25 2014, 21:20.
remove javacsv dependency, add opencsv dependency
cad5eddceb19
·Jeffyrao on May 25 2014, 21:12.
Merge remote-tracking branch 'upstream/master' into extract-links
f0e36b46b872
·Jinfeng Rao on May 25 2014, 20:31.

May 23 2014

Fixed robustness and OOM issues when ingesting corrupt ARC files.
7c21e2df9267
·lintool on May 23 2014, 23:09.
Fixed issue29.
67888b472702
·milad621 on May 23 2014, 16:01.

May 22 2014

Simple program to find URL patterns in archives.
fcc4acf98a6f
·lintool on May 22 2014, 20:14.

May 6 2014

Minor refactoring.
52b39cfecc82
·lintool on May 6 2014, 03:40.
Merge branch 'master' of github.com:milad621/warcbase into display_issues
0856f3faddd1
·lintool on May 6 2014, 03:33.

May 5 2014

changed type of nobanner to boolean.
a4247934797c
·milad621 on May 5 2014, 20:33.

May 4 2014

Working on issue17. Added /noframes to the url but still needs to add warcbase…
c5432ac2468c
·milad621 on May 4 2014, 09:20.

Apr 24 2014

Fixed issue 25
8799707e9856
·milad621 on Apr 24 2014, 06:27.

Apr 20 2014

modified ExtractSiteLinks.java, changed to read csv format of prefix input
24d7d8e08807
·Jeffyrao on Apr 20 2014, 19:01.

Apr 19 2014

juniversalchardet removed.
627743751578
·milad621 on Apr 19 2014, 07:08.
For issue17 it now shows a page with three frames, each has a seperate search…
0d34860367d9
·milad621 on Apr 19 2014, 06:18.

Apr 16 2014

add extract site-level links
8d9ef6d2ac6b
·Jeffyrao on Apr 16 2014, 16:58.

Apr 6 2014

Fixed Issue 24.
fd511b6eaf7b
·milad621 on Apr 6 2014, 09:50.

Mar 31 2014

Exposed -getPrefix command-line option.
997a8e84e816
·lintool on Mar 31 2014, 18:46.
Light refactoring, fixed a few errors.
758948b463b9
·lintool on Mar 31 2014, 17:56.
Merge branch 'extract-links' of github.com:Jeffyrao/warcbase into fst
5227ba65232c
·lintool on Mar 31 2014, 17:40.
remove TestFSTPrefix class
e3ccb37b2e1d
·Jeffyrao on Mar 31 2014, 05:33.
add prefix search feature to UriMapping class; add test class for UriMapping…
71c68ecae0fa
·Jeffyrao on Mar 31 2014, 05:32.

Mar 29 2014

More work on Issue #22: prev/next buttons turns on and off appropriately.
2f1574575438
·lintool on Mar 29 2014, 01:28.

Mar 28 2014

add Lucene FST prefix search
a48c65e498ab
·Jeffyrao on Mar 28 2014, 21:23.
Fixed issue 18.
6d1747bd14af
·milad621 on Mar 28 2014, 08:08.
Fixed issues 21 and 22.
88e5d97286e4
·milad621 on Mar 28 2014, 07:52.
Merge branch 'master' of https://github.com/lintool/warcbase
dac5827fa1e5
·milad621 on Mar 28 2014, 06:48.
merged with master
f2f1e34837f3
·milad621 on Mar 28 2014, 06:46.

Mar 27 2014

Minor code cleanup.
3da8131e077b
·lintool on Mar 27 2014, 13:47.

Mar 26 2014

Bug with getting close dates fixed.
d965c015a914
·milad621 on Mar 26 2014, 21:45.

Mar 25 2014

Fixed bug with MAX_VERSION blowing up array length.
be5a4d366786
·lintool on Mar 25 2014, 14:05.
Merge branch 'issue12' of github.com:milad621/warcbase into multi_version_cell
72b9558b3ca5
·lintool on Mar 25 2014, 13:37.
fixing branches
ef6650cb7bc4
·milad621 on Mar 25 2014, 06:47.
Fixed issue12 and some issues with dates in browser.
0d43c98ee177
·milad621 on Mar 25 2014, 06:39.

Mar 24 2014

Issue12 fixed
6c48ce81ee80
·milad621 on Mar 24 2014, 02:48.
test push
e0d6be1055f4
·milad621 on Mar 24 2014, 02:41.
issue12 fixed
252eb87988b1
·milad621 on Mar 24 2014, 02:30.

Mar 21 2014

update README file, add steps for getting URLs and extracting links
24370c4f051f
·Jeffyrao on Mar 21 2014, 00:32.

Mar 20 2014

add ExtractLinks and PrefixQuery(ExtractSiteLinks)
cb9e68d4f94d
·Jeffyrao on Mar 20 2014, 22:35.

Mar 19 2014

Adds Snappy compression.
374b6f2c2173
·lintool on Mar 19 2014, 02:09.

Mar 18 2014

Merge remote-tracking branch 'lintool/master'
232a07ac56c9
·cneud on Mar 18 2014, 22:36.

Mar 17 2014

Reformatting source.
303860c9ae3b
·lintool on Mar 17 2014, 14:11.
Merge branch 'disable_wal' of github.com:lintool/warcbase into disable_wal
c52474128b38
·lintool on Mar 17 2014, 14:04.
Merge branch 'master' into disable_wal
d543a4957916
·lintool on Mar 17 2014, 14:04.
Merge branch 'jar_upgrade' into disable_wal
b430dc7c94f6
·lintool on Mar 17 2014, 13:57.
Fixed Issue #7
8c649e5336af
·lintool on Mar 17 2014, 13:53.
Added log4j properties.
48ac86cd8479
·lintool on Mar 17 2014, 13:51.
Added try/catch block in ingestion code.
a33f92f340e1
·lintool on Mar 17 2014, 13:28.
code reformat via eclipse imported project
53aacb335403
·Jeffyrao on Mar 17 2014, 03:04.

Mar 16 2014

Ignores all files that aren't WARC/ARC files in ingest.
987f9518ebb8
·lintool on Mar 16 2014, 23:26.
Tried disabling WAL to see impact on performance.
dfb63a169b43
·lintool on Mar 16 2014, 21:22.
code reformat
1ad985045d9a
·Jeffyrao on Mar 16 2014, 20:58.
Simplified Maven dependencies; Moved from IA artifacts to openwayback artifacts.
073fd3206185
·lintool on Mar 16 2014, 13:50.
Light refactoring.
228ca87cd0c3
·lintool on Mar 16 2014, 01:04.
whitespace
19453fc7ac8f
·lintool on Mar 16 2014, 00:39.
Merge branch 'new_hbase_structure' of https://github.com/milad621/warcbase into…
1e0783ff8a6c
·lintool on Mar 16 2014, 00:32.

Mar 15 2014

Whitespace.
fdb75decbdf4
·lintool on Mar 15 2014, 20:58.
Merge branch 'extract-links' of https://github.com/Jeffyrao/warcbase into…
d86dcb73c854
·lintool on Mar 15 2014, 20:51.

Dec 28 2013

servlet updated with the new hbase structure.
75a0820c382e
·milad621 on Dec 28 2013, 13:08.

Dec 19 2013

Added a null pointer check and a more Pig friendly return value from the UDFs
db4ebe9f4826
·pmd on Dec 19 2013, 15:48.

R1473/src/mainmaster0a781f3644802205ba097d2…

Aug 10 2014

Aug 4 2014

Jul 25 2014

Jul 22 2014

Jun 26 2014

Jun 18 2014

Jun 17 2014

Jun 13 2014

Jun 12 2014

Jun 11 2014

Jun 6 2014

Jun 5 2014

Jun 4 2014

Jun 3 2014

May 29 2014

May 27 2014

May 26 2014

May 25 2014

May 23 2014

May 22 2014

May 6 2014

May 5 2014

May 4 2014

Apr 24 2014

Apr 20 2014

Apr 19 2014

Apr 16 2014

Apr 6 2014

Mar 31 2014

Mar 29 2014

Mar 28 2014

Mar 27 2014

Mar 26 2014

Mar 25 2014

Mar 24 2014

Mar 21 2014

Mar 20 2014

Mar 19 2014

Mar 18 2014

Mar 17 2014

Mar 16 2014

Mar 15 2014

Dec 28 2013

Dec 19 2013

R1473/src/main
master0a781f3644802205ba097d2…