Diffusion warcbase (master) History

History
mastera0305b4f7faf7f14af96021…

Apr 1 2014

Updated documentation.
a0305b4f7faf
·lintool on Apr 1 2014, 02:52.

Mar 31 2014

Exposed -getPrefix command-line option.
997a8e84e816
·lintool on Mar 31 2014, 18:46.
Light refactoring, fixed a few errors.
758948b463b9
·lintool on Mar 31 2014, 17:56.
Merge branch 'extract-links' of github.com:Jeffyrao/warcbase into fst
5227ba65232c
·lintool on Mar 31 2014, 17:40.
remove TestFSTPrefix class
e3ccb37b2e1d
·Jeffyrao on Mar 31 2014, 05:33.
add prefix search feature to UriMapping class; add test class for UriMapping…
71c68ecae0fa
·Jeffyrao on Mar 31 2014, 05:32.

Mar 29 2014

Fixes for Issues #18, #21, #22
a11b2d7221da
·lintool on Mar 29 2014, 01:29.
More work on Issue #22: prev/next buttons turns on and off appropriately.
2f1574575438
·lintool on Mar 29 2014, 01:28.

Mar 28 2014

add Lucene FST prefix search
a48c65e498ab
·Jeffyrao on Mar 28 2014, 21:23.
Fixed issue 18.
6d1747bd14af
·milad621 on Mar 28 2014, 08:08.
Fixed issues 21 and 22.
88e5d97286e4
·milad621 on Mar 28 2014, 07:52.
Merge branch 'master' of https://github.com/lintool/warcbase
dac5827fa1e5
·milad621 on Mar 28 2014, 06:48.
merged with master
f2f1e34837f3
·milad621 on Mar 28 2014, 06:46.

Mar 27 2014

Fixed Issues #12 and #14
7be8dc6f4e10
·lintool on Mar 27 2014, 13:53.
Minor code cleanup.
3da8131e077b
·lintool on Mar 27 2014, 13:47.

Mar 26 2014

Bug with getting close dates fixed.
d965c015a914
·milad621 on Mar 26 2014, 21:45.

Mar 25 2014

Fixed bug with MAX_VERSION blowing up array length.
be5a4d366786
·lintool on Mar 25 2014, 14:05.
Merge branch 'issue12' of github.com:milad621/warcbase into multi_version_cell
72b9558b3ca5
·lintool on Mar 25 2014, 13:37.
fixing branches
ef6650cb7bc4
·milad621 on Mar 25 2014, 06:47.
Fixed issue12 and some issues with dates in browser.
0d43c98ee177
·milad621 on Mar 25 2014, 06:39.

Mar 24 2014

Issue12 fixed
6c48ce81ee80
·milad621 on Mar 24 2014, 02:48.
test push
e0d6be1055f4
·milad621 on Mar 24 2014, 02:41.
issue12 fixed
252eb87988b1
·milad621 on Mar 24 2014, 02:30.

Mar 21 2014

trival update Readme file, modify format
b104adbd9a09
·Jeffyrao on Mar 21 2014, 00:35.
update README file, add steps for getting URLs and extracting links
24370c4f051f
·Jeffyrao on Mar 21 2014, 00:32.

Mar 20 2014

add ExtractLinks and PrefixQuery(ExtractSiteLinks)
cb9e68d4f94d
·Jeffyrao on Mar 20 2014, 22:35.

Mar 19 2014

Adds Snappy compression.
374b6f2c2173
·lintool on Mar 19 2014, 02:09.

Mar 17 2014

Fixed Issues #5 and #6
53fcae412c25
·lintool on Mar 17 2014, 14:12.
Reformatting source.
303860c9ae3b
·lintool on Mar 17 2014, 14:11.
Merge branch 'jar_upgrade' into disable_wal
b83b898c782c
·lintool on Mar 17 2014, 14:06.
Merge branch 'disable_wal' of github.com:lintool/warcbase into disable_wal
c52474128b38
·lintool on Mar 17 2014, 14:04.
Merge branch 'master' into disable_wal
d543a4957916
·lintool on Mar 17 2014, 14:04.
Merge branch 'jar_upgrade' into disable_wal
b430dc7c94f6
·lintool on Mar 17 2014, 13:57.
Fixed Issue #7
8c649e5336af
·lintool on Mar 17 2014, 13:53.
Added log4j properties.
48ac86cd8479
·lintool on Mar 17 2014, 13:51.
Added try/catch block in ingestion code.
a33f92f340e1
·lintool on Mar 17 2014, 13:28.
Merge branch 'working'
5ae6b8841cbe
·lintool on Mar 17 2014, 04:10.
code reformat via eclipse imported project
53aacb335403
·Jeffyrao on Mar 17 2014, 03:04.

Mar 16 2014

Ignores all files that aren't WARC/ARC files in ingest.
987f9518ebb8
·lintool on Mar 16 2014, 23:26.
Tried disabling WAL to see impact on performance.
dfb63a169b43
·lintool on Mar 16 2014, 21:22.
code reformat
1ad985045d9a
·Jeffyrao on Mar 16 2014, 20:58.
remove hadoop-job.xml
91d5e65858db
·Jinfeng Rao on Mar 16 2014, 20:27.
remove assembly plugin for hadoop job
5fb9d2a33a9e
·Jinfeng Rao on Mar 16 2014, 20:15.
openwayback-core pulls in hadoop-core; excludes
19e2ff05074a
·lintool on Mar 16 2014, 17:44.
Simplified Maven dependencies; Moved from IA artifacts to openwayback artifacts.
073fd3206185
·lintool on Mar 16 2014, 13:50.
Light refactoring.
228ca87cd0c3
·lintool on Mar 16 2014, 01:04.
Upgraded to CDH 4.4 jars for HBase and ZK.
c8577eac7bbb
·lintool on Mar 16 2014, 00:51.
whitespace
19453fc7ac8f
·lintool on Mar 16 2014, 00:39.
Merge branch 'new_hbase_structure' of https://github.com/milad621/warcbase into…
1e0783ff8a6c
·lintool on Mar 16 2014, 00:32.
Merge branch 'working'
267f165ba113
·lintool on Mar 16 2014, 00:23.
Update README.
854e5d13f12c
·lintool on Mar 16 2014, 00:22.
More POM tweaks to get MR jobs running correctly.
7a93263997f3
·lintool on Mar 16 2014, 00:03.

Mar 15 2014

Tweaked POM
2f51acbc5fbe
·lintool on Mar 15 2014, 21:05.
Whitespace.
fdb75decbdf4
·lintool on Mar 15 2014, 20:58.
Merge branch 'extract-links' of https://github.com/Jeffyrao/warcbase into…
d86dcb73c854
·lintool on Mar 15 2014, 20:51.

Jan 4 2014

Update README.md
5d382f5fb4f7
·Jeffyrao on Jan 4 2014, 06:58.
Update README.md
142082d20eec
·Jeffyrao on Jan 4 2014, 06:58.

Dec 28 2013

servlet updated with the new hbase structure.
75a0820c382e
·milad621 on Dec 28 2013, 13:08.

Dec 11 2013

new hbase structure for ingest files
a80e87ca6f74
·milad621 on Dec 11 2013, 19:00.

Dec 9 2013

modify UriMappingBuilder in pom.xml
3c1f4ccccc0c
·Jinfeng Rao on Dec 9 2013, 03:53.

Dec 8 2013

build hadoop job using Maven Assembly Plugin, modified pom.xml, added hadoop…
8a3cb8c12dca
·Jinfeng Rao on Dec 8 2013, 20:03.
check text/html type and modify Jsoup.parse charset as ISO-8859-1
2b49ec425597
·Jeffyrao on Dec 8 2013, 19:55.

Dec 7 2013

modify UriMappingBuilder to read all files under given directory
2e7e0cb81775
·Jeffyrao on Dec 7 2013, 06:21.
Simple MapReduce program to count number of unique URLs.
c4248d61b8b7
·lintool on Dec 7 2013, 04:21.

Dec 6 2013

Extract links and Lucene FST for URLs.
8e1cadb7e70e
·Jeffyrao on Dec 6 2013, 20:26.

Dec 5 2013

removed dead code
bbee2e38f787
·milad621 on Dec 5 2013, 17:10.

Dec 1 2013

Started a new branch to extract text from warcbase data and organize the data…
19735bbbbd86
·milad621 on Dec 1 2013, 23:08.
started a new branch to extract text from warcbase data and organize the data…
944b138ee0d2
·milad621 on Dec 1 2013, 23:01.

Nov 25 2013

Improved error checking for dates.
7ee8dbb88c3c
·lintool on Nov 25 2013, 00:44.

Nov 24 2013

Merge remote-tracking branch 'forkOrigin/master'
1bc2ea63e470
·milad621 on Nov 24 2013, 21:02.

Nov 23 2013

PrintAllUris add to appassembler which will output a urls.html file with all…
7e76e19f2547
·milad621 on Nov 23 2013, 20:25.
Tweaked browser; added MR programs for simple content analysis.
22dd0757c01e
·lintool on Nov 23 2013, 18:11.

Nov 22 2013

Hadoop InputFormats for ARC and WARC files + simple demos.
b5b4e211ea05
·lintool on Nov 22 2013, 17:17.

Nov 20 2013

Merge branch 'working'
0e4d1592c244
·lintool on Nov 20 2013, 15:47.
Tweaked README.
0aeaeb3a754b
·lintool on Nov 20 2013, 15:46.

Nov 18 2013

Code refactoring after pair coding.
bf51f6b75bd8
·milad621 on Nov 18 2013, 13:22.

Nov 8 2013

Fixed ava.lang.NegativeArraySizeException at org.apache.commons.io.output.
43d12364a70f
·milad621 on Nov 8 2013, 22:35.

Nov 7 2013

Created a new runnable to find a uri inside warc/arc files.
c3e37d50e856
·milad621 on Nov 7 2013, 19:32.
Uses HTablePool instead of creating a new connection each time.
33966c708354
·milad621 on Nov 7 2013, 06:28.
Some code cleanup in servlet. DetectDuplicates fixed with new hbase table…
64ef175db2fd
·milad621 on Nov 7 2013, 05:43.
updated servlet.
bf1b3616e4c0
·milad621 on Nov 7 2013, 05:23.

Nov 6 2013

Added a seperate class to manage HBase connection and addRecord
76949eacabd5
·milad621 on Nov 6 2013, 23:27.

Nov 5 2013

One runnable to process both arc and warc files in a folder.
01dff2a1f2a5
·milad621 on Nov 5 2013, 23:30.

Oct 30 2013

fixed some issues with the servlet. content and types might not follow eachother
1354c68684d9
·milad621 on Oct 30 2013, 13:40.
IngestWarcFiles fixed. Now it uses jwat-warc to ingest warcfiles to hbase.
e9e3d1c9d940
·milad621 on Oct 30 2013, 01:56.

Oct 24 2013

Updated WarcBrowser to work with the new structure of hbase table (supports…
70c00addd99e
·milad621 on Oct 24 2013, 05:02.

Oct 23 2013

IngestArc updated.
e8f7af4176f8
·milad621 on Oct 23 2013, 21:00.
IngestWarcFiles updated. Uses jwat and stores content type in htable
735b77efcad4
·milad621 on Oct 23 2013, 20:22.

Oct 17 2013

Arc Processing tools added. Not working with hbase yet.
b1bee7dc3a8e
·milad621 on Oct 17 2013, 19:33.

Oct 9 2013

No need for name arg
803daecd43ef
·milad621 on Oct 9 2013, 09:44.
url style fixed. Can capture table names from url and home page changed to http…
b321544c8a1f
·milad621 on Oct 9 2013, 09:39.

Oct 2 2013

URL style changed
c860aef8368c
·milad621 on Oct 2 2013, 09:24.
URL style changed
4b7ec6fab139
·milad621 on Oct 2 2013, 09:23.

Oct 1 2013

Merge branch 'milad-master'
79dee598b2df
·lintool on Oct 1 2013, 18:27.

Sep 27 2013

close button fixed
3770fab2cafe
·milad621 on Sep 27 2013, 22:18.

Sep 24 2013

fixed a baseurl problem
127a533cf731
·milad621 on Sep 24 2013, 14:58.

Sep 20 2013

A bug with swf files fixed
c27fb0fb72d9
·milad621 on Sep 20 2013, 21:31.

Sep 17 2013

header (navigation bar?) added. Pom update
8706a55f8e36
·milad621 on Sep 17 2013, 19:16.
header (navigation bar?) added
dc20e4fc805f
·milad621 on Sep 17 2013, 19:15.

Aug 17 2013

Some url bugs with senate data fixed. A few code refactoring done
24cf7be01bea
·milad621 on Aug 17 2013, 20:18.

Historymastera0305b4f7faf7f14af96021…

Apr 1 2014

Mar 31 2014

Mar 29 2014

Mar 28 2014

Mar 27 2014

Mar 26 2014

Mar 25 2014

Mar 24 2014

Mar 21 2014

Mar 20 2014

Mar 19 2014

Mar 17 2014

Mar 16 2014

Mar 15 2014

Jan 4 2014

Dec 28 2013

Dec 11 2013

Dec 9 2013

Dec 8 2013

Dec 7 2013

Dec 6 2013

Dec 5 2013

Dec 1 2013

Nov 25 2013

Nov 24 2013

Nov 23 2013

Nov 22 2013

Nov 20 2013

Nov 18 2013

Nov 8 2013

Nov 7 2013

Nov 6 2013

Nov 5 2013

Oct 30 2013

Oct 24 2013

Oct 23 2013

Oct 17 2013

Oct 9 2013

Oct 2 2013

Oct 1 2013

Sep 27 2013

Sep 24 2013

Sep 20 2013

Sep 17 2013

Aug 17 2013

History
mastera0305b4f7faf7f14af96021…