Diffusion warcbase (master) History

History
master4ceefef78cdc934768cc4c6…

Oct 19 2014

Pig loader materializes the actual content.
4ceefef78cdc
·lintool on Oct 19 2014, 00:47.

Oct 18 2014

Refactored Pig loaded to use WAC API.
58df7da00880
·lintool on Oct 18 2014, 17:05.
Fixed issues #90, #91, #92
63e8352c664b
·lintool on Oct 18 2014, 14:36.
Updates.
722b6402b98b
·lintool on Oct 18 2014, 14:34.
Merge branch 'master' of github.com:rwolniak/warcbase into warc
a48bb9781feb
·lintool on Oct 18 2014, 14:32.
upgraded to webarchive-commons 1.1.4.
21397e4e4ff3
·lintool on Oct 18 2014, 14:30.

Oct 13 2014

Updated Building the URL Mapping Instructions to include instruction to move…
da890acb0be1
·rwolniak on Oct 13 2014, 17:57.
Updated documentation.
de4267bf28f1
·lintool on Oct 13 2014, 14:21.

Oct 8 2014

Testing commit and push + README update
27b0038b631f
·rwolniak on Oct 8 2014, 06:01.

Sep 15 2014

Merge branch 'master' into warc
457a71345d2b
·lintool on Sep 15 2014, 01:57.

Sep 14 2014

Fixed issue #87
8fae0c067d68
·lintool on Sep 14 2014, 20:14.
Added timing info.
05db518ccd83
·lintool on Sep 14 2014, 20:13.
Cleanup.
157df31c0f15
·lintool on Sep 14 2014, 19:48.
Added debug output; Fixed deprecated HBase APIs.
85c5b4a2ecc5
·lintool on Sep 14 2014, 19:40.

Sep 13 2014

Fixed build issues in upgrade to CDH 5.1.2.
159596e9b378
·lintool on Sep 13 2014, 03:14.

Aug 30 2014

Figured out how to extract MIME type and date from WARC.
4798a4314b6e
·lintool on Aug 30 2014, 03:10.

Aug 29 2014

Added test cases to try loading WARC records from a stream; back-ported same…
f3516c7fd7f0
·lintool on Aug 29 2014, 04:15.

Aug 28 2014

Added simple test case.
28c5c007f4fd
·lintool on Aug 28 2014, 05:48.
WARC sample from https://archive.org/details/ExampleArcAndWarcFiles
8e67d49d44b3
·lintool on Aug 28 2014, 05:19.

Aug 23 2014

Fixed issues #86, #85, #74
fb07114090f5
·lintool on Aug 23 2014, 19:31.
Refactored WacMapReduceHBaseWrapperDemo, now takes advantage of…
9faed3817dcf
·lintool on Aug 23 2014, 18:58.
Merge branch 'working' into table-wrapper
999fa0af0280
·lintool on Aug 23 2014, 18:25.
WacArcInputFormat now generates ArcRecordWritables.
08530c1cbe52
·lintool on Aug 23 2014, 18:24.
Switched ExtractSiteLinks and InvertAnchorText over to WacArcInputFormat; link…
98d7150f5eec
·lintool on Aug 23 2014, 17:53.
Minor refactoring, revised counters.
369ba2731f5b
·lintool on Aug 23 2014, 15:40.
Both the Jwat and Wac versions of ExtractLinks gives the same exact output.
74b023ed8b16
·lintool on Aug 23 2014, 06:46.
Implemented ArcRecordWritable.
ccee8fd2204a
·lintool on Aug 23 2014, 05:25.
Fixed issues #84, #72
1e1c2dcde9e2
·lintool on Aug 23 2014, 03:58.
Minor refactoring.
846ebb216100
·lintool on Aug 23 2014, 03:46.
Very rough prototype of wrapper that allows interoperability between HBase…
c5f4b9efa212
·lintool on Aug 23 2014, 03:38.

Aug 22 2014

Added Hadoop bindings for webarchive-commons ARC readers, demo, test cases.
47f3c46c099d
·lintool on Aug 22 2014, 14:28.
Added/refactored test cases for JWAT.
b07376aba3fe
·lintool on Aug 22 2014, 13:56.
Added @Override annotations to appropriate methods.
4663558b1d1a
·lintool on Aug 22 2014, 02:41.

Aug 19 2014

Fixed issues #80 and #78
6e2e6628ea08
·lintool on Aug 19 2014, 23:00.
Minor tweaks.
9cbd26b2e3bb
·lintool on Aug 19 2014, 22:59.
Merge branch 'selenium' into working
43e952bfdd99
·lintool on Aug 19 2014, 22:30.
Refactoring of Wayback/Warcbase integration points.
3c7da3b094fd
·lintool on Aug 19 2014, 22:23.

Aug 17 2014

Minor tweaks.
afa516d702c9
·lintool on Aug 17 2014, 16:17.
Fixed Issue #79: ExtractSiteLinks documentation is out of date in README.md
1622603fcfc8
·lintool on Aug 17 2014, 16:08.
Merge branch 'master' into selenium
c5b4973aba96
·lintool on Aug 17 2014, 15:10.
Fixed issue #77: Make sure ExtractLinks and related classes still work
f9671b6db516
·lintool on Aug 17 2014, 15:06.
Cleaned up scripts.
4090318d2f99
·lintool on Aug 17 2014, 03:37.
Cute Selenium browser to conduct a random walk through the archive.
b02fd402870f
·lintool on Aug 17 2014, 03:32.
Cleaned up programs for manipulating graphs; removed scanning HBase option for…
52cf7bb90940
·lintool on Aug 17 2014, 01:05.

Aug 16 2014

Fixed issues #75, #73, #67
45895b659265
·lintool on Aug 16 2014, 21:35.
Removed outdated code; cleaned up WarcBrowserServlet.
f7eefd5da751
·lintool on Aug 16 2014, 21:27.
Refactoring: created ArcRecordUtils.
a57c6b83eaf2
·lintool on Aug 16 2014, 18:56.
Refactoring packge org.warcbase.demo; added Jwat prefix to Hadoop InputFormats…
d58afca206fc
·lintool on Aug 16 2014, 18:23.
Fixed issues #71, #70, #66, #63 and #62 (partially)
a00e413edff4
·lintool on Aug 16 2014, 15:43.

Aug 15 2014

Better handling of ARC parse errors.
377a3e3a429f
·lintool on Aug 15 2014, 01:31.

Aug 14 2014

Bump max value size up to 10 MB, tweak WAL settings.
2d16fd032060
·lintool on Aug 14 2014, 23:33.
Refactoring UrlUtil and related classes.
bf606c07d453
·lintool on Aug 14 2014, 23:31.
Restored some sanity to versions and transitive dependencies.
ff34480981dd
·lintool on Aug 14 2014, 23:10.
Fixed issue #60: Merge in Clemens et al. contributions to Warcbase
55a7334ae0bb
·lintool on Aug 14 2014, 18:10.
Minor tweaks.
b6431bed6f8b
·lintool on Aug 14 2014, 18:09.
Tweaked settings.
37cc53a7fd66
·lintool on Aug 14 2014, 18:01.
Merge branch 'master' into cneud-integration
72c1afbca77c
·lintool on Aug 14 2014, 17:35.
Fixed:
5a8dc8ab24e3
·lintool on Aug 14 2014, 17:29.
Updated documentation.
a77bdc3fac02
·lintool on Aug 14 2014, 17:28.
Added command-line options.
f29edf052e19
·lintool on Aug 14 2014, 17:19.
More refactoring.
14fa0f329072
·lintool on Aug 14 2014, 17:05.
Refactoring on local UrlMappingBuilder.
6eeb0ea431f1
·lintool on Aug 14 2014, 16:24.
Lightweight refactoring.
5022874b83e5
·lintool on Aug 14 2014, 14:30.
Bumped up memory, fixed class renaming.
ae5581b740fc
·lintool on Aug 14 2014, 14:30.
Uri -> Url classes renaming.
0bc5d4876199
·lintool on Aug 14 2014, 14:04.
Light refactoring
ab64384ff204
·lintool on Aug 14 2014, 13:28.

Aug 13 2014

resolve conflicts
bc47be0fea19
·Jeffyrao on Aug 13 2014, 06:26.
reformat code
0623b52cd05c
·Jeffyrao on Aug 13 2014, 05:36.
Updated documentation, issue #61
607a42335808
·lintool on Aug 13 2014, 01:36.
Updated documentation.
7f13ec11d968
·lintool on Aug 13 2014, 01:33.
Configuration for Wayback/Warcbase integration.
83414e51b351
·lintool on Aug 13 2014, 01:20.
Removed extra command-line argument.
910c2f03e6b1
·lintool on Aug 13 2014, 00:59.

Aug 12 2014

Commented out LibmagicJnaWrapper functionality because the jar isn't generally…
a14cda2217fa
·lintool on Aug 12 2014, 23:15.
Fixed compile and broken test issues.
93f6d42f4ff4
·lintool on Aug 12 2014, 21:47.
Merge branch 'master' of github.com:cneud/warcbase into cneud-integration
accd1978862d
·lintool on Aug 12 2014, 19:38.
Prototype integration of Wayback/Warcbase via REST API on HBase.
a0a594f92b94
·lintool on Aug 12 2014, 02:42.
Merge branch 'cleanup' into wayback-integration
eb4893e7201c
·lintool on Aug 12 2014, 02:41.
Fixes issue #58: Wayback reads directly from REST API instead of writing and…
c665a23bf1f3
·lintool on Aug 12 2014, 02:07.
Fixed issue #59: Unable to fetch URLs from archive with '?' in them
ffec5b2caa3d
·lintool on Aug 12 2014, 01:59.
Fixed issues with fetching URLs with spaces in them.
0f4f541c24d7
·lintool on Aug 12 2014, 01:19.

Aug 11 2014

Better handling of errors: when REST API is unavailable, when URL isn't found…
b5ccb86d330e
·lintool on Aug 11 2014, 15:33.
Code cleanup for browser code; removed unneeded files and associated web files.
515bab098cff
·lintool on Aug 11 2014, 14:22.
Merge branch 'rest-api-bug-fix' of github.com:lintool/warcbase into wayback…
74f3dece6c62
·lintool on Aug 11 2014, 14:02.
Converted host/port/table information to bean settings.
3075ca19d76d
·lintool on Aug 11 2014, 13:53.
Simplified code.
37e97073d57c
·lintool on Aug 11 2014, 04:48.
Refactoring; mostly reformatting.
bca5b6944ac7
·lintool on Aug 11 2014, 04:30.
Minor fix for NPE when capture isn't in HBase.
9e8765b4e3f0
·lintool on Aug 11 2014, 04:28.
Initial check-in of Warcbase integration points with Open Wayback.
4f387e8952a7
·lintool on Aug 11 2014, 04:01.
/*/ query returns MIME type.
03d5c5365481
·lintool on Aug 11 2014, 03:44.

Aug 10 2014

Refactor to confirm to /*/ of Wayback to fetch list of available versions.
65b6dc5b7138
·lintool on Aug 10 2014, 19:49.
Switched over to 14 digit dates for URLs to align with Wayback. Further…
096878f5f932
·lintool on Aug 10 2014, 19:18.
Cleaned up servlet fetch code.
7f9764b1a793
·lintool on Aug 10 2014, 18:52.
Fixed 14 digit date parsing issue (now uses ArchiveUtils); was an issue with…
b685d65eba7b
·lintool on Aug 10 2014, 18:28.
Fixed a few minor ingestion issues.
bc6aab1ffd21
·lintool on Aug 10 2014, 16:11.
Refactoring; removing WARC ingestion for now.
b20ef84e5df9
·lintool on Aug 10 2014, 14:29.
Tweaks to ingest code.
cfe508a831f0
·lintool on Aug 10 2014, 14:05.
Janky, but seems to work: ingesting and serving up raw ARC records.
180b57fa5dc1
·lintool on Aug 10 2014, 08:28.
Quick and dirty switch over to webarchive-commons API; stores raw ARC records.
8b5a6be9db61
·lintool on Aug 10 2014, 04:16.

Aug 4 2014

add UriMappingBuilder Mapreduce version
65c6e54a948e
·Jeffyrao on Aug 4 2014, 01:48.

Jul 25 2014

fix the bug of selecting webpage by date ineffective
59aa95aab857
·Jeffyrao on Jul 25 2014, 18:36.

Historymaster4ceefef78cdc934768cc4c6…

Oct 19 2014

Oct 18 2014

Oct 13 2014

Oct 8 2014

Sep 15 2014

Sep 14 2014

Sep 13 2014

Aug 30 2014

Aug 29 2014

Aug 28 2014

Aug 23 2014

Aug 22 2014

Aug 19 2014

Aug 17 2014

Aug 16 2014

Aug 15 2014

Aug 14 2014

Aug 13 2014

Aug 12 2014

Aug 11 2014

Aug 10 2014

Aug 4 2014

Jul 25 2014

History
master4ceefef78cdc934768cc4c6…