Diffusion warcbase (master) History

History
masterb02fd402870f9950b31f2dc…

Aug 17 2014

Cute Selenium browser to conduct a random walk through the archive.
b02fd402870f
·lintool on Aug 17 2014, 03:32.

Aug 16 2014

Fixed issues #75, #73, #67
45895b659265
·lintool on Aug 16 2014, 21:35.
Removed outdated code; cleaned up WarcBrowserServlet.
f7eefd5da751
·lintool on Aug 16 2014, 21:27.
Refactoring: created ArcRecordUtils.
a57c6b83eaf2
·lintool on Aug 16 2014, 18:56.
Refactoring packge org.warcbase.demo; added Jwat prefix to Hadoop InputFormats…
d58afca206fc
·lintool on Aug 16 2014, 18:23.
Fixed issues #71, #70, #66, #63 and #62 (partially)
a00e413edff4
·lintool on Aug 16 2014, 15:43.

Aug 15 2014

Better handling of ARC parse errors.
377a3e3a429f
·lintool on Aug 15 2014, 01:31.

Aug 14 2014

Bump max value size up to 10 MB, tweak WAL settings.
2d16fd032060
·lintool on Aug 14 2014, 23:33.
Refactoring UrlUtil and related classes.
bf606c07d453
·lintool on Aug 14 2014, 23:31.
Restored some sanity to versions and transitive dependencies.
ff34480981dd
·lintool on Aug 14 2014, 23:10.
Fixed issue #60: Merge in Clemens et al. contributions to Warcbase
55a7334ae0bb
·lintool on Aug 14 2014, 18:10.
Minor tweaks.
b6431bed6f8b
·lintool on Aug 14 2014, 18:09.
Tweaked settings.
37cc53a7fd66
·lintool on Aug 14 2014, 18:01.
Merge branch 'master' into cneud-integration
72c1afbca77c
·lintool on Aug 14 2014, 17:35.
Fixed:
5a8dc8ab24e3
·lintool on Aug 14 2014, 17:29.
Updated documentation.
a77bdc3fac02
·lintool on Aug 14 2014, 17:28.
Added command-line options.
f29edf052e19
·lintool on Aug 14 2014, 17:19.
More refactoring.
14fa0f329072
·lintool on Aug 14 2014, 17:05.
Refactoring on local UrlMappingBuilder.
6eeb0ea431f1
·lintool on Aug 14 2014, 16:24.
Lightweight refactoring.
5022874b83e5
·lintool on Aug 14 2014, 14:30.
Bumped up memory, fixed class renaming.
ae5581b740fc
·lintool on Aug 14 2014, 14:30.
Uri -> Url classes renaming.
0bc5d4876199
·lintool on Aug 14 2014, 14:04.
Light refactoring
ab64384ff204
·lintool on Aug 14 2014, 13:28.

Aug 13 2014

resolve conflicts
bc47be0fea19
·Jeffyrao on Aug 13 2014, 06:26.
reformat code
0623b52cd05c
·Jeffyrao on Aug 13 2014, 05:36.
Updated documentation, issue #61
607a42335808
·lintool on Aug 13 2014, 01:36.
Updated documentation.
7f13ec11d968
·lintool on Aug 13 2014, 01:33.
Configuration for Wayback/Warcbase integration.
83414e51b351
·lintool on Aug 13 2014, 01:20.
Removed extra command-line argument.
910c2f03e6b1
·lintool on Aug 13 2014, 00:59.

Aug 12 2014

Commented out LibmagicJnaWrapper functionality because the jar isn't generally…
a14cda2217fa
·lintool on Aug 12 2014, 23:15.
Fixed compile and broken test issues.
93f6d42f4ff4
·lintool on Aug 12 2014, 21:47.
Merge branch 'master' of github.com:cneud/warcbase into cneud-integration
accd1978862d
·lintool on Aug 12 2014, 19:38.
Prototype integration of Wayback/Warcbase via REST API on HBase.
a0a594f92b94
·lintool on Aug 12 2014, 02:42.
Merge branch 'cleanup' into wayback-integration
eb4893e7201c
·lintool on Aug 12 2014, 02:41.
Fixes issue #58: Wayback reads directly from REST API instead of writing and…
c665a23bf1f3
·lintool on Aug 12 2014, 02:07.
Fixed issue #59: Unable to fetch URLs from archive with '?' in them
ffec5b2caa3d
·lintool on Aug 12 2014, 01:59.
Fixed issues with fetching URLs with spaces in them.
0f4f541c24d7
·lintool on Aug 12 2014, 01:19.

Aug 11 2014

Better handling of errors: when REST API is unavailable, when URL isn't found…
b5ccb86d330e
·lintool on Aug 11 2014, 15:33.
Code cleanup for browser code; removed unneeded files and associated web files.
515bab098cff
·lintool on Aug 11 2014, 14:22.
Merge branch 'rest-api-bug-fix' of github.com:lintool/warcbase into wayback…
74f3dece6c62
·lintool on Aug 11 2014, 14:02.
Converted host/port/table information to bean settings.
3075ca19d76d
·lintool on Aug 11 2014, 13:53.
Simplified code.
37e97073d57c
·lintool on Aug 11 2014, 04:48.
Refactoring; mostly reformatting.
bca5b6944ac7
·lintool on Aug 11 2014, 04:30.
Minor fix for NPE when capture isn't in HBase.
9e8765b4e3f0
·lintool on Aug 11 2014, 04:28.
Initial check-in of Warcbase integration points with Open Wayback.
4f387e8952a7
·lintool on Aug 11 2014, 04:01.
/*/ query returns MIME type.
03d5c5365481
·lintool on Aug 11 2014, 03:44.

Aug 10 2014

Refactor to confirm to /*/ of Wayback to fetch list of available versions.
65b6dc5b7138
·lintool on Aug 10 2014, 19:49.
Switched over to 14 digit dates for URLs to align with Wayback. Further…
096878f5f932
·lintool on Aug 10 2014, 19:18.
Cleaned up servlet fetch code.
7f9764b1a793
·lintool on Aug 10 2014, 18:52.
Fixed 14 digit date parsing issue (now uses ArchiveUtils); was an issue with…
b685d65eba7b
·lintool on Aug 10 2014, 18:28.
Fixed a few minor ingestion issues.
bc6aab1ffd21
·lintool on Aug 10 2014, 16:11.
Refactoring; removing WARC ingestion for now.
b20ef84e5df9
·lintool on Aug 10 2014, 14:29.
Tweaks to ingest code.
cfe508a831f0
·lintool on Aug 10 2014, 14:05.
Janky, but seems to work: ingesting and serving up raw ARC records.
180b57fa5dc1
·lintool on Aug 10 2014, 08:28.
Quick and dirty switch over to webarchive-commons API; stores raw ARC records.
8b5a6be9db61
·lintool on Aug 10 2014, 04:16.

Aug 4 2014

add UriMappingBuilder Mapreduce version
65c6e54a948e
·Jeffyrao on Aug 4 2014, 01:48.

Jul 25 2014

fix the bug of selecting webpage by date ineffective
59aa95aab857
·Jeffyrao on Jul 25 2014, 18:36.

Jul 22 2014

add Hadoop/HBase input choice for ExtractLinks and ExtractSiteLinks classes
7175239e0751
·Jeffyrao on Jul 22 2014, 23:43.

Jun 27 2014

Fixed issues #17, #25, #30, #35
17f5787120af
·lintool on Jun 27 2014, 17:37.

Jun 26 2014

Merging with master.
60a9b96beebe
·Milad Gholami on Jun 26 2014, 16:16.
Fixing git history.
149ce6cf969f
·Milad Gholami on Jun 26 2014, 16:03.

Jun 18 2014

Fixed issues #45, #46, #49, #50
113332758f06
·lintool on Jun 18 2014, 21:50.
More work on the admin interface.
3b8484a944a8
·lintool on Jun 18 2014, 21:48.
issue #50
f3015cd7ba4b
·lintool on Jun 18 2014, 19:57.

Jun 17 2014

Fixed broken merge.
10d60bd28c75
·lintool on Jun 17 2014, 22:48.
Merge branch 'master' into admin
6c452cbb6b5d
·lintool on Jun 17 2014, 22:47.
fixed issue #43 and issue #48
71d5a4e0803e
·lintool on Jun 17 2014, 22:42.
Fixed issue #48
781fe4247b31
·lintool on Jun 17 2014, 22:41.
Added comment.
5f62c2f6fb9d
·lintool on Jun 17 2014, 22:15.
Minor refactoring.
d17dddca8deb
·lintool on Jun 17 2014, 21:44.
Initial working version of anchor text inversion program: issue #43
ee7f7749a30a
·lintool on Jun 17 2014, 20:06.
Started working on issue #46 cleanup of org.warcbase.data.Util
02c26d6d8ea4
·lintool on Jun 17 2014, 17:20.
Appears to have fixed issue #49, starting work on admin tool, issue #45.
c7a7247d5fa2
·lintool on Jun 17 2014, 04:18.

Jun 16 2014

openwayback upgraded to 2.0.0.BETA.2
37f9b1fce90f
·milad621 on Jun 16 2014, 22:49.
Fixed issue #42 and issue #44
59c50bb33254
·lintool on Jun 16 2014, 00:08.

Jun 13 2014

ExtractLinks using HBase appears to be working.
a6be4375e0c7
·lintool on Jun 13 2014, 03:08.
Refactored HDFS extractor; HBase extractor still broken.
21a07efb350e
·lintool on Jun 13 2014, 00:01.

Jun 12 2014

Merge branch 'master' into refactoring
bbc73ab64808
·lintool on Jun 12 2014, 19:54.
Merge branch 'hbase_experiments'
6d50f37d6ada
·lintool on Jun 12 2014, 19:52.
Light refactoring, pushed column family filter into scan.
273e5969e943
·lintool on Jun 12 2014, 19:51.
Alternative implementation based on iterating over maps... slightly slower.
3283bb8512ef
·lintool on Jun 12 2014, 16:18.
More light refactoring.
dbfbcb0b3c7e
·lintool on Jun 12 2014, 15:05.
add hbase option for ExtractLinks
34273fdb935a
·Jeffyrao on Jun 12 2014, 06:34.
reformat Jinfeng's code
cfb89d2d3379
·Jeffyrao on Jun 12 2014, 05:10.

Jun 11 2014

Refactoring, code cleanup.
b02f33c9b43a
·lintool on Jun 11 2014, 22:36.
Fixed issue #39
d4c29085ee12
·lintool on Jun 11 2014, 04:24.
Debugged HBase scan parameters so that they don't knock over region servers…
c3a4348e1250
·lintool on Jun 11 2014, 04:05.

Jun 6 2014

Issue 38 fixed. Still need to add other URL Encoding characters.
0e61a3094550
·milad621 on Jun 6 2014, 19:29.

Jun 5 2014

Fixed Issues #31, #32, #40, #41
5425313990da
·lintool on Jun 5 2014, 22:43.
Moving ExtractLinks and ExtractSiteLinks into analysis.graph package, per Issue…
20ef503dbfe5
·lintool on Jun 5 2014, 22:41.
Extracts values at different timestamps.
dc58365a9579
·lintool on Jun 5 2014, 22:27.
Refactored getIdRange method signature, add more test cases to UriMapping.
60b827d1c174
·lintool on Jun 5 2014, 16:16.
Merge branch 'extract-links' of github.com:Jeffyrao/warcbase into refactoring
a6b17787ccd1
·lintool on Jun 5 2014, 16:04.
fix issue 40 that UriMapping prefix search should return empty result when no…
7fc1b92d60a7
·Jeffyrao on Jun 5 2014, 03:37.

Jun 4 2014

Refactoring, added test case (currently broken).
a8953616248c
·lintool on Jun 4 2014, 21:12.

Jun 3 2014

fix issue 32, update ExtractSiteLinks code
6518836b7ea0
·Jeffyrao on Jun 3 2014, 20:27.

May 29 2014

fix issue 30, add ExtractSiteLinks code
a17e3c74954a
·Jeffyrao on May 29 2014, 20:13.

May 27 2014

fix issue 31
3f36eff4f429
·Jeffyrao on May 27 2014, 04:39.

May 26 2014

Initial MapReduce over HBase demo.
b4ab2ea0d499
·lintool on May 26 2014, 15:30.

May 25 2014

more edits
0598dcd91073
·Jeffyrao on May 25 2014, 21:20.

Historymasterb02fd402870f9950b31f2dc…

Aug 17 2014

Aug 16 2014

Aug 15 2014

Aug 14 2014

Aug 13 2014

Aug 12 2014

Aug 11 2014

Aug 10 2014

Aug 4 2014

Jul 25 2014

Jul 22 2014

Jun 27 2014

Jun 26 2014

Jun 18 2014

Jun 17 2014

Jun 16 2014

Jun 13 2014

Jun 12 2014

Jun 11 2014

Jun 6 2014

Jun 5 2014

Jun 4 2014

Jun 3 2014

May 29 2014

May 27 2014

May 26 2014

May 25 2014

History
masterb02fd402870f9950b31f2dc…