Diffusion warcbase (master) History

R1473/src/main/java/org/warcbase/data
master63c3d5916d847e0e5d808b9…

Jun 15 2016

Fixed issue #234 Error handling for broken ARC/WARC files: empty content in ARC…
63c3d5916d84
·lintool on Jun 15 2016, 16:06.

Nov 25 2015

Killed RecordUtils, ExtractLinksAndText, JwatArcLoaderTest
5e316b1bb129
·lintool on Nov 25 2015, 03:49.
Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Jun 7 2015

Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…
d9b553c76ced
·lintool on Jun 7 2015, 17:10.

Dec 24 2014

Add ingestion option to select either Snappy or GZ compression, Issue #110
5d471280e7db
·lintool on Dec 24 2014, 21:31.

Dec 23 2014

Fixed Issue #109: OOM when running UrlMappingBuilder
e6b21cc557d6
·lintool on Dec 23 2014, 04:01.

Dec 22 2014

Merge branch 'hbase-api-refactoring' into dev
6acf52f320cf
·lintool on Dec 22 2014, 21:34.

Dec 21 2014

Changed compression back from GZ to Snappy.
7f6d571424a9
·lintool on Dec 21 2014, 04:01.

Dec 11 2014

Updated regex for "Content-Type" (RFC2616 sec. 4.2 says HTTP header field names…
dbaa917eb8b2
·Jeremy Wiebe on Dec 11 2014, 22:26.
Added getBodyContent(WARCRecord r) method to WarcRecordUtils
eb28dc02a8d2
·Jeremy Wiebe on Dec 11 2014, 20:15.

Dec 9 2014

Added WARC support to UrlMappingMapReduceBuilder. It can now accept a path…
65837e09aa3a
·Jeremy Wiebe on Dec 9 2014, 23:01.

Dec 8 2014

For building on OS X, changed HBase compression from Snappy to GZ.
5590850ab2f0
·Jeremy Wiebe on Dec 8 2014, 20:16.

Oct 22 2014

Refactoring to create method that extracts MIME from WARC response records.
0145d74ed35e
·lintool on Oct 22 2014, 18:42.

Oct 19 2014

WARC-related Hadoop bindings.
410cfd81a069
·lintool on Oct 19 2014, 18:59.
Merge branch 'master' into hbase-api-refactoring
f97523f440ff
·lintool on Oct 19 2014, 01:21.

Oct 13 2014

Updated documentation.
de4267bf28f1
·lintool on Oct 13 2014, 14:21.

Sep 15 2014

Merge branch 'master' into warc
457a71345d2b
·lintool on Sep 15 2014, 01:57.

Sep 14 2014

Refactored to eliminate deprecated HBase APIs.
2b8e721063fa
·lintool on Sep 14 2014, 20:42.

Sep 13 2014

Fixed build issues in upgrade to CDH 5.1.2.
159596e9b378
·lintool on Sep 13 2014, 03:14.

Aug 29 2014

Added test cases to try loading WARC records from a stream; back-ported same…
f3516c7fd7f0
·lintool on Aug 29 2014, 04:15.

Aug 28 2014

Added simple test case.
28c5c007f4fd
·lintool on Aug 28 2014, 05:48.

Aug 23 2014

Implemented ArcRecordWritable.
ccee8fd2204a
·lintool on Aug 23 2014, 05:25.

Aug 16 2014

Refactoring: created ArcRecordUtils.
a57c6b83eaf2
·lintool on Aug 16 2014, 18:56.
Refactoring packge org.warcbase.demo; added Jwat prefix to Hadoop InputFormats…
d58afca206fc
·lintool on Aug 16 2014, 18:23.

Aug 14 2014

Bump max value size up to 10 MB, tweak WAL settings.
2d16fd032060
·lintool on Aug 14 2014, 23:33.
Refactoring UrlUtil and related classes.
bf606c07d453
·lintool on Aug 14 2014, 23:31.
Added command-line options.
f29edf052e19
·lintool on Aug 14 2014, 17:19.
More refactoring.
14fa0f329072
·lintool on Aug 14 2014, 17:05.
Refactoring on local UrlMappingBuilder.
6eeb0ea431f1
·lintool on Aug 14 2014, 16:24.
Lightweight refactoring.
5022874b83e5
·lintool on Aug 14 2014, 14:30.
Uri -> Url classes renaming.
0bc5d4876199
·lintool on Aug 14 2014, 14:04.
Light refactoring
ab64384ff204
·lintool on Aug 14 2014, 13:28.

Aug 13 2014

resolve conflicts
bc47be0fea19
·Jeffyrao on Aug 13 2014, 06:26.

Aug 11 2014

Code cleanup for browser code; removed unneeded files and associated web files.
515bab098cff
·lintool on Aug 11 2014, 14:22.

Aug 10 2014

Fixed 14 digit date parsing issue (now uses ArchiveUtils); was an issue with…
b685d65eba7b
·lintool on Aug 10 2014, 18:28.

Aug 4 2014

add UriMappingBuilder Mapreduce version
65c6e54a948e
·Jeffyrao on Aug 4 2014, 01:48.

Jun 26 2014

Merging with master.
60a9b96beebe
·Milad Gholami on Jun 26 2014, 16:16.
Fixing git history.
149ce6cf969f
·Milad Gholami on Jun 26 2014, 16:03.

Jun 18 2014

issue #50
f3015cd7ba4b
·lintool on Jun 18 2014, 19:57.

Jun 17 2014

Started working on issue #46 cleanup of org.warcbase.data.Util
02c26d6d8ea4
·lintool on Jun 17 2014, 17:20.

Jun 6 2014

Issue 38 fixed. Still need to add other URL Encoding characters.
0e61a3094550
·milad621 on Jun 6 2014, 19:29.

Jun 5 2014

Moving ExtractLinks and ExtractSiteLinks into analysis.graph package, per Issue…
20ef503dbfe5
·lintool on Jun 5 2014, 22:41.
Refactored getIdRange method signature, add more test cases to UriMapping.
60b827d1c174
·lintool on Jun 5 2014, 16:16.
Merge branch 'extract-links' of github.com:Jeffyrao/warcbase into refactoring
a6b17787ccd1
·lintool on Jun 5 2014, 16:04.
fix issue 40 that UriMapping prefix search should return empty result when no…
7fc1b92d60a7
·Jeffyrao on Jun 5 2014, 03:37.

Jun 4 2014

Refactoring, added test case (currently broken).
a8953616248c
·lintool on Jun 4 2014, 21:12.

Jun 3 2014

fix issue 32, update ExtractSiteLinks code
6518836b7ea0
·Jeffyrao on Jun 3 2014, 20:27.

May 29 2014

fix issue 30, add ExtractSiteLinks code
a17e3c74954a
·Jeffyrao on May 29 2014, 20:13.

May 27 2014

fix issue 31
3f36eff4f429
·Jeffyrao on May 27 2014, 04:39.

May 25 2014

more edits
0598dcd91073
·Jeffyrao on May 25 2014, 21:20.
remove javacsv dependency, add opencsv dependency
cad5eddceb19
·Jeffyrao on May 25 2014, 21:12.

May 23 2014

Fixed issue29.
67888b472702
·milad621 on May 23 2014, 16:01.

May 6 2014

Merge branch 'master' of github.com:milad621/warcbase into display_issues
0856f3faddd1
·lintool on May 6 2014, 03:33.

May 4 2014

Working on issue17. Added /noframes to the url but still needs to add warcbase…
c5432ac2468c
·milad621 on May 4 2014, 09:20.

Apr 20 2014

modified ExtractSiteLinks.java, changed to read csv format of prefix input
24d7d8e08807
·Jeffyrao on Apr 20 2014, 19:01.

Apr 16 2014

add extract site-level links
8d9ef6d2ac6b
·Jeffyrao on Apr 16 2014, 16:58.

Mar 31 2014

Exposed -getPrefix command-line option.
997a8e84e816
·lintool on Mar 31 2014, 18:46.
Light refactoring, fixed a few errors.
758948b463b9
·lintool on Mar 31 2014, 17:56.
Merge branch 'extract-links' of github.com:Jeffyrao/warcbase into fst
5227ba65232c
·lintool on Mar 31 2014, 17:40.
remove TestFSTPrefix class
e3ccb37b2e1d
·Jeffyrao on Mar 31 2014, 05:33.
add prefix search feature to UriMapping class; add test class for UriMapping…
71c68ecae0fa
·Jeffyrao on Mar 31 2014, 05:32.

Mar 28 2014

add Lucene FST prefix search
a48c65e498ab
·Jeffyrao on Mar 28 2014, 21:23.

Mar 25 2014

Fixed bug with MAX_VERSION blowing up array length.
be5a4d366786
·lintool on Mar 25 2014, 14:05.
Merge branch 'issue12' of github.com:milad621/warcbase into multi_version_cell
72b9558b3ca5
·lintool on Mar 25 2014, 13:37.
fixing branches
ef6650cb7bc4
·milad621 on Mar 25 2014, 06:47.
Fixed issue12 and some issues with dates in browser.
0d43c98ee177
·milad621 on Mar 25 2014, 06:39.

Mar 24 2014

Issue12 fixed
6c48ce81ee80
·milad621 on Mar 24 2014, 02:48.
test push
e0d6be1055f4
·milad621 on Mar 24 2014, 02:41.
issue12 fixed
252eb87988b1
·milad621 on Mar 24 2014, 02:30.

Mar 21 2014

update README file, add steps for getting URLs and extracting links
24370c4f051f
·Jeffyrao on Mar 21 2014, 00:32.

Mar 20 2014

add ExtractLinks and PrefixQuery(ExtractSiteLinks)
cb9e68d4f94d
·Jeffyrao on Mar 20 2014, 22:35.

Mar 19 2014

Adds Snappy compression.
374b6f2c2173
·lintool on Mar 19 2014, 02:09.

Mar 17 2014

Merge branch 'master' into disable_wal
d543a4957916
·lintool on Mar 17 2014, 14:04.
Merge branch 'jar_upgrade' into disable_wal
b430dc7c94f6
·lintool on Mar 17 2014, 13:57.
Fixed Issue #7
8c649e5336af
·lintool on Mar 17 2014, 13:53.
code reformat via eclipse imported project
53aacb335403
·Jeffyrao on Mar 17 2014, 03:04.

Mar 16 2014

Tried disabling WAL to see impact on performance.
dfb63a169b43
·lintool on Mar 16 2014, 21:22.
code reformat
1ad985045d9a
·Jeffyrao on Mar 16 2014, 20:58.
Simplified Maven dependencies; Moved from IA artifacts to openwayback artifacts.
073fd3206185
·lintool on Mar 16 2014, 13:50.
Light refactoring.
228ca87cd0c3
·lintool on Mar 16 2014, 01:04.
whitespace
19453fc7ac8f
·lintool on Mar 16 2014, 00:39.
Merge branch 'new_hbase_structure' of https://github.com/milad621/warcbase into…
1e0783ff8a6c
·lintool on Mar 16 2014, 00:32.

Dec 28 2013

servlet updated with the new hbase structure.
75a0820c382e
·milad621 on Dec 28 2013, 13:08.

Dec 11 2013

new hbase structure for ingest files
a80e87ca6f74
·milad621 on Dec 11 2013, 19:00.

Dec 8 2013

check text/html type and modify Jsoup.parse charset as ISO-8859-1
2b49ec425597
·Jeffyrao on Dec 8 2013, 19:55.

Dec 7 2013

modify UriMappingBuilder to read all files under given directory
2e7e0cb81775
·Jeffyrao on Dec 7 2013, 06:21.

Dec 6 2013

Extract links and Lucene FST for URLs.
8e1cadb7e70e
·Jeffyrao on Dec 6 2013, 20:26.

Dec 1 2013

started a new branch to extract text from warcbase data and organize the data…
944b138ee0d2
·milad621 on Dec 1 2013, 23:01.

Nov 23 2013

PrintAllUris add to appassembler which will output a urls.html file with all…
7e76e19f2547
·milad621 on Nov 23 2013, 20:25.

Nov 18 2013

Code refactoring after pair coding.
bf51f6b75bd8
·milad621 on Nov 18 2013, 13:22.

Nov 7 2013

Some code cleanup in servlet. DetectDuplicates fixed with new hbase table…
64ef175db2fd
·milad621 on Nov 7 2013, 05:43.
updated servlet.
bf1b3616e4c0
·milad621 on Nov 7 2013, 05:23.

Nov 6 2013

Added a seperate class to manage HBase connection and addRecord
76949eacabd5
·milad621 on Nov 6 2013, 23:27.

Oct 17 2013

Arc Processing tools added. Not working with hbase yet.
b1bee7dc3a8e
·milad621 on Oct 17 2013, 19:33.

Oct 9 2013

url style fixed. Can capture table names from url and home page changed to http…
b321544c8a1f
·milad621 on Oct 9 2013, 09:39.

Oct 2 2013

URL style changed
4b7ec6fab139
·milad621 on Oct 2 2013, 09:23.

Sep 24 2013

fixed a baseurl problem
127a533cf731
·milad621 on Sep 24 2013, 14:58.

Aug 17 2013

Some url bugs with senate data fixed. A few code refactoring done
24cf7be01bea
·milad621 on Aug 17 2013, 20:18.