Diffusion warcbase (master) History

analysis
masterf972206db5168eb9c45ee1a…

Jun 16 2016

Created warcbase-core module.
f972206db516
·lintool on Jun 16 2016, 15:43.

Nov 25 2015

Slapped Apache License boilerplate -- now we're a *real* open-source project :)
2e88c1b19afb
·lintool on Nov 25 2015, 03:19.

Jun 7 2015

Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…
d9b553c76ced
·lintool on Jun 7 2015, 17:10.

Dec 23 2014

Renaming.
fb3edb4e40c7
·lintool on Dec 23 2014, 22:03.
Fixed OOM issues.
f6caf0f6f226
·lintool on Dec 23 2014, 22:00.
Fixed OOM errors.
59949b3d7af4
·lintool on Dec 23 2014, 04:59.

Dec 21 2014

Fixed minor code formatting issues.
12a00be93209
·lintool on Dec 21 2014, 03:58.

Dec 11 2014

Added WARC support to analysis/graph classes (except ExtractLinksJwat -- since…
8b1e82f16828
·Jeremy Wiebe on Dec 11 2014, 20:17.

Dec 10 2014

Added WARC support to analysis:
ec4285807f39
·Jeremy Wiebe on Dec 10 2014, 15:54.

Dec 8 2014

Fixed bug in ExtractLinksWac where fst.loadMapping() would choke on filename…
3d8672c631db
·Jeremy Wiebe on Dec 8 2014, 20:18.

Sep 14 2014

Added timing info.
05db518ccd83
·lintool on Sep 14 2014, 20:13.

Aug 23 2014

Switched ExtractSiteLinks and InvertAnchorText over to WacArcInputFormat; link…
98d7150f5eec
·lintool on Aug 23 2014, 17:53.
Minor refactoring, revised counters.
369ba2731f5b
·lintool on Aug 23 2014, 15:40.
Both the Jwat and Wac versions of ExtractLinks gives the same exact output.
74b023ed8b16
·lintool on Aug 23 2014, 06:46.

Aug 17 2014

Cleaned up programs for manipulating graphs; removed scanning HBase option for…
52cf7bb90940
·lintool on Aug 17 2014, 01:05.

Aug 16 2014

Removed outdated code; cleaned up WarcBrowserServlet.
f7eefd5da751
·lintool on Aug 16 2014, 21:27.
Refactoring packge org.warcbase.demo; added Jwat prefix to Hadoop InputFormats…
d58afca206fc
·lintool on Aug 16 2014, 18:23.

Aug 14 2014

Refactoring UrlUtil and related classes.
bf606c07d453
·lintool on Aug 14 2014, 23:31.
Refactoring on local UrlMappingBuilder.
6eeb0ea431f1
·lintool on Aug 14 2014, 16:24.
Uri -> Url classes renaming.
0bc5d4876199
·lintool on Aug 14 2014, 14:04.

Aug 13 2014

resolve conflicts
bc47be0fea19
·Jeffyrao on Aug 13 2014, 06:26.
reformat code
0623b52cd05c
·Jeffyrao on Aug 13 2014, 05:36.

Jul 25 2014

fix the bug of selecting webpage by date ineffective
59aa95aab857
·Jeffyrao on Jul 25 2014, 18:36.

Jul 22 2014

add Hadoop/HBase input choice for ExtractLinks and ExtractSiteLinks classes
7175239e0751
·Jeffyrao on Jul 22 2014, 23:43.

Jun 18 2014

issue #50
f3015cd7ba4b
·lintool on Jun 18 2014, 19:57.

Jun 17 2014

Fixed broken merge.
10d60bd28c75
·lintool on Jun 17 2014, 22:48.
Merge branch 'master' into admin
6c452cbb6b5d
·lintool on Jun 17 2014, 22:47.
Fixed issue #48
781fe4247b31
·lintool on Jun 17 2014, 22:41.
Added comment.
5f62c2f6fb9d
·lintool on Jun 17 2014, 22:15.
Minor refactoring.
d17dddca8deb
·lintool on Jun 17 2014, 21:44.
Initial working version of anchor text inversion program: issue #43
ee7f7749a30a
·lintool on Jun 17 2014, 20:06.
Started working on issue #46 cleanup of org.warcbase.data.Util
02c26d6d8ea4
·lintool on Jun 17 2014, 17:20.

Jun 13 2014

ExtractLinks using HBase appears to be working.
a6be4375e0c7
·lintool on Jun 13 2014, 03:08.
Refactored HDFS extractor; HBase extractor still broken.
21a07efb350e
·lintool on Jun 13 2014, 00:01.

Jun 12 2014

Merge branch 'master' into refactoring
bbc73ab64808
·lintool on Jun 12 2014, 19:54.
Light refactoring, pushed column family filter into scan.
273e5969e943
·lintool on Jun 12 2014, 19:51.
Alternative implementation based on iterating over maps... slightly slower.
3283bb8512ef
·lintool on Jun 12 2014, 16:18.
More light refactoring.
dbfbcb0b3c7e
·lintool on Jun 12 2014, 15:05.
add hbase option for ExtractLinks
34273fdb935a
·Jeffyrao on Jun 12 2014, 06:34.
reformat Jinfeng's code
cfb89d2d3379
·Jeffyrao on Jun 12 2014, 05:10.

Jun 11 2014

Refactoring, code cleanup.
b02f33c9b43a
·lintool on Jun 11 2014, 22:36.
Fixed issue #39
d4c29085ee12
·lintool on Jun 11 2014, 04:24.
Debugged HBase scan parameters so that they don't knock over region servers…
c3a4348e1250
·lintool on Jun 11 2014, 04:05.

Jun 5 2014

Moving ExtractLinks and ExtractSiteLinks into analysis.graph package, per Issue…
20ef503dbfe5
·lintool on Jun 5 2014, 22:41.
Extracts values at different timestamps.
dc58365a9579
·lintool on Jun 5 2014, 22:27.

May 26 2014

Initial MapReduce over HBase demo.
b4ab2ea0d499
·lintool on May 26 2014, 15:30.

May 22 2014

Simple program to find URL patterns in archives.
fcc4acf98a6f
·lintool on May 22 2014, 20:14.

Mar 16 2014

Light refactoring.
228ca87cd0c3
·lintool on Mar 16 2014, 01:04.
whitespace
19453fc7ac8f
·lintool on Mar 16 2014, 00:39.
Merge branch 'new_hbase_structure' of https://github.com/milad621/warcbase into…
1e0783ff8a6c
·lintool on Mar 16 2014, 00:32.

Mar 15 2014

Whitespace.
fdb75decbdf4
·lintool on Mar 15 2014, 20:58.

Dec 11 2013

new hbase structure for ingest files
a80e87ca6f74
·milad621 on Dec 11 2013, 19:00.

Dec 7 2013

Simple MapReduce program to count number of unique URLs.
c4248d61b8b7
·lintool on Dec 7 2013, 04:21.

Dec 5 2013

removed dead code
bbee2e38f787
·milad621 on Dec 5 2013, 17:10.

Dec 1 2013

Started a new branch to extract text from warcbase data and organize the data…
19735bbbbd86
·milad621 on Dec 1 2013, 23:08.
started a new branch to extract text from warcbase data and organize the data…
944b138ee0d2
·milad621 on Dec 1 2013, 23:01.

Nov 25 2013

Improved error checking for dates.
7ee8dbb88c3c
·lintool on Nov 25 2013, 00:44.

Nov 23 2013

PrintAllUris add to appassembler which will output a urls.html file with all…
7e76e19f2547
·milad621 on Nov 23 2013, 20:25.
Tweaked browser; added MR programs for simple content analysis.
22dd0757c01e
·lintool on Nov 23 2013, 18:11.

Nov 22 2013

Hadoop InputFormats for ARC and WARC files + simple demos.
b5b4e211ea05
·lintool on Nov 22 2013, 17:17.

Nov 18 2013

Code refactoring after pair coding.
bf51f6b75bd8
·milad621 on Nov 18 2013, 13:22.

Nov 7 2013

Some code cleanup in servlet. DetectDuplicates fixed with new hbase table…
64ef175db2fd
·milad621 on Nov 7 2013, 05:43.

Nov 5 2013

One runnable to process both arc and warc files in a folder.
01dff2a1f2a5
·milad621 on Nov 5 2013, 23:30.

Aug 17 2013

Some url bugs with senate data fixed. A few code refactoring done
24cf7be01bea
·milad621 on Aug 17 2013, 20:18.

Aug 15 2013

urls
e7a88b5211bb
·milad621 on Aug 15 2013, 08:04.

Aug 13 2013

Cleaned up analysis code. Removed dead code. Added README.
3a2aec9c5698
·lintool on Aug 13 2013, 10:54.

Aug 12 2013

Refactoring of browser; each archive is now stored in its own separate table…
4fffd3905e9e
·lintool on Aug 12 2013, 23:11.

R1473/src/main/java/org/warcbase/analysismasterf972206db5168eb9c45ee1a…

Jun 16 2016

Nov 25 2015

Jun 7 2015

Dec 23 2014

Dec 21 2014

Dec 11 2014

Dec 10 2014

Dec 8 2014

Sep 14 2014

Aug 23 2014

Aug 17 2014

Aug 16 2014

Aug 14 2014

Aug 13 2014

Jul 25 2014

Jul 22 2014

Jun 18 2014

Jun 17 2014

Jun 13 2014

Jun 12 2014

Jun 11 2014

Jun 5 2014

May 26 2014

May 22 2014

Mar 16 2014

Mar 15 2014

Dec 11 2013

Dec 7 2013

Dec 5 2013

Dec 1 2013

Nov 25 2013

Nov 23 2013

Nov 22 2013

Nov 18 2013

Nov 7 2013

Nov 5 2013

Aug 17 2013

Aug 15 2013

Aug 13 2013

Aug 12 2013

R1473/src/main/java/org/warcbase/analysis
masterf972206db5168eb9c45ee1a…