Diffusion warcbase (master) History

R1473/pom.xml
master35e440d9a761cc8a98958cc…

Jun 24 2016

Upgraded to CDH 5.7.1.
19be6edfa8cd
·lintool on Jun 24 2016, 23:08.

Jun 17 2016

Cleaned up and simplified dependencies, etc.
e8c43f1b9d24
·lintool on Jun 17 2016, 19:46.

Jun 16 2016

Integrating warcbase-hbase artifact.
6e8740a55414
·lintool on Jun 16 2016, 20:52.
Created warcbase-core module.
f972206db516
·lintool on Jun 16 2016, 15:43.

Mar 29 2016

downgraded guava to fix borked tests, cf https://issues.apache.
52eb2f96304d
·lintool on Mar 29 2016, 05:17.
Merge branch 'graphx'
2ffc60cc178e
·Jeremy Wiebe on Mar 29 2016, 04:09.

Mar 8 2016

Revert to old CDH (new version produces errors on cluster)
19eefb5d7f2c
·Jeremy Wiebe on Mar 8 2016, 02:44.

Mar 5 2016

Basic graph analysis, initial check-in
997d2405a84e
·Jeremy Wiebe on Mar 5 2016, 19:24.
Downgraded json4s, folds in parsing in the loader.
93686da4ff23
·lintool on Mar 5 2016, 16:46.
Initial pass at processing tweets in Warcbase.
f2abb2b5bbc2
·lintool on Mar 5 2016, 16:21.

Feb 13 2016

Use shapeless to flatten tuples of any arity
4b8f0fb7b482
·Jeremy Wiebe on Feb 13 2016, 06:48.

Nov 26 2015

Added matchbox function to NER-classify and generate JSON for visualizer, per…
c44118522844
·Jeremy Wiebe on Nov 26 2015, 16:05.

Nov 25 2015

Added jackson-databind to pom.xml
dd643f9fae5c
·Jeremy Wiebe on Nov 25 2015, 03:17.

Nov 24 2015

Killed all the Pig stuff.
f5d8edf50506
·lintool on Nov 24 2015, 22:05.

Nov 12 2015

Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.

Nov 9 2015

Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Nov 1 2015

Upgraded Spark to 1.3 in CDH 5.4.1
ade10b26bca4
·lintool on Nov 1 2015, 17:53.

Jul 22 2015

Fixed issue #137 Integrate with warc-hadoop-indexer for Shine
258a159c796e
·lintool on Jul 22 2015, 18:11.
Upgraded to a stable warc-hadoop-indexer artifact (2.2.0-BETA-5)
fac3312c1785
·lintool on Jul 22 2015, 18:03.

Jul 14 2015

Fixed: https://github.com/ukwa/webarchive-discovery/issues/64
7936dd172aae
·lintool on Jul 14 2015, 00:29.

Jul 13 2015

Copy dependencies into solr home.
34c47c6ba9bc
·lintool on Jul 13 2015, 06:46.
Upgrade CDH; fixed broken tests due classpath conflict issue and Tika upgrade.
151242d2d988
·lintool on Jul 13 2015, 05:59.
Refactoring to simplify code.
46d7cef43ee3
·lintool on Jul 13 2015, 02:39.
Starting to incorporate Hadoop WARC indexing code.
335d751b425c
·lintool on Jul 13 2015, 00:43.

Jun 30 2015

Added ExtractBoilerpipeText UDF
11dfe152d4cc
·Jeremy Wiebe on Jun 30 2015, 02:38.

Jun 9 2015

Fixed merge conflict that got committed to master accidentally.
988a2354d84d
·lintool on Jun 9 2015, 20:10.
testing fork
51abf7e86696
·ianmilligan1 on Jun 9 2015, 10:20.

Jun 7 2015

Removed JWAT jars.
2f462979da0d
·lintool on Jun 7 2015, 18:18.

May 30 2015

whitespace
a75d58d5b0ee
·lintool on May 30 2015, 04:25.

May 27 2015

Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.

May 26 2015

Added Stanford NER UDF
1c013a7b064f
·Jeremy Wiebe on May 26 2015, 18:41.

May 24 2015

Merge branch 'master' into extract-pdf-udf
9acc8ca74c0a
·lintool on May 24 2015, 19:37.

Dec 23 2014

Updated versions of some artifacts.
0fd8f3f72822
·lintool on Dec 23 2014, 02:44.

Dec 18 2014

Reformatting.
5c2540099370
·lintool on Dec 18 2014, 14:20.

Dec 9 2014

Updated PIG Tika Parser with new code that should work. Awaiting test on Hadoop
cea1f0cae974
·rwolniak on Dec 9 2014, 00:12.

Oct 18 2014

upgraded to webarchive-commons 1.1.4.
21397e4e4ff3
·lintool on Oct 18 2014, 14:30.

Sep 14 2014

Cleanup.
157df31c0f15
·lintool on Sep 14 2014, 19:48.

Sep 13 2014

Fixed build issues in upgrade to CDH 5.1.2.
159596e9b378
·lintool on Sep 13 2014, 03:14.

Aug 19 2014

Minor tweaks.
9cbd26b2e3bb
·lintool on Aug 19 2014, 22:59.

Aug 17 2014

Merge branch 'master' into selenium
c5b4973aba96
·lintool on Aug 17 2014, 15:10.
Cleaned up scripts.
4090318d2f99
·lintool on Aug 17 2014, 03:37.
Cute Selenium browser to conduct a random walk through the archive.
b02fd402870f
·lintool on Aug 17 2014, 03:32.

Aug 14 2014

Restored some sanity to versions and transitive dependencies.
ff34480981dd
·lintool on Aug 14 2014, 23:10.
Minor tweaks.
b6431bed6f8b
·lintool on Aug 14 2014, 18:09.
Merge branch 'master' into cneud-integration
72c1afbca77c
·lintool on Aug 14 2014, 17:35.
Bumped up memory, fixed class renaming.
ae5581b740fc
·lintool on Aug 14 2014, 14:30.
Uri -> Url classes renaming.
0bc5d4876199
·lintool on Aug 14 2014, 14:04.

Aug 12 2014

Commented out LibmagicJnaWrapper functionality because the jar isn't generally…
a14cda2217fa
·lintool on Aug 12 2014, 23:15.
Fixed compile and broken test issues.
93f6d42f4ff4
·lintool on Aug 12 2014, 21:47.

Aug 10 2014

Quick and dirty switch over to webarchive-commons API; stores raw ARC records.
8b5a6be9db61
·lintool on Aug 10 2014, 04:16.

Jun 26 2014

Merging with master.
60a9b96beebe
·Milad Gholami on Jun 26 2014, 16:16.

Jun 17 2014

Appears to have fixed issue #49, starting work on admin tool, issue #45.
c7a7247d5fa2
·lintool on Jun 17 2014, 04:18.

Jun 16 2014

openwayback upgraded to 2.0.0.BETA.2
37f9b1fce90f
·milad621 on Jun 16 2014, 22:49.

May 25 2014

remove javacsv dependency, add opencsv dependency
cad5eddceb19
·Jeffyrao on May 25 2014, 21:12.

May 6 2014

Minor refactoring.
52b39cfecc82
·lintool on May 6 2014, 03:40.
Merge branch 'master' of github.com:milad621/warcbase into display_issues
0856f3faddd1
·lintool on May 6 2014, 03:33.

Apr 19 2014

juniversalchardet removed.
627743751578
·milad621 on Apr 19 2014, 07:08.

Apr 16 2014

add ExtractSiteLinks in pom.xml
1e3b5543324b
·Jeffyrao on Apr 16 2014, 22:35.

Apr 6 2014

Fixed Issue 24.
fd511b6eaf7b
·milad621 on Apr 6 2014, 09:50.

Mar 31 2014

Light refactoring, fixed a few errors.
758948b463b9
·lintool on Mar 31 2014, 17:56.

Mar 17 2014

Merge branch 'jar_upgrade' into disable_wal
b83b898c782c
·lintool on Mar 17 2014, 14:06.
Merge branch 'master' into disable_wal
d543a4957916
·lintool on Mar 17 2014, 14:04.
Fixed Issue #7
8c649e5336af
·lintool on Mar 17 2014, 13:53.

Mar 16 2014

remove assembly plugin for hadoop job
5fb9d2a33a9e
·Jinfeng Rao on Mar 16 2014, 20:15.
openwayback-core pulls in hadoop-core; excludes
19e2ff05074a
·lintool on Mar 16 2014, 17:44.
Simplified Maven dependencies; Moved from IA artifacts to openwayback artifacts.
073fd3206185
·lintool on Mar 16 2014, 13:50.
Upgraded to CDH 4.4 jars for HBase and ZK.
c8577eac7bbb
·lintool on Mar 16 2014, 00:51.
Merge branch 'new_hbase_structure' of https://github.com/milad621/warcbase into…
1e0783ff8a6c
·lintool on Mar 16 2014, 00:32.
More POM tweaks to get MR jobs running correctly.
7a93263997f3
·lintool on Mar 16 2014, 00:03.

Mar 15 2014

Tweaked POM
2f51acbc5fbe
·lintool on Mar 15 2014, 21:05.

Dec 9 2013

modify UriMappingBuilder in pom.xml
3c1f4ccccc0c
·Jinfeng Rao on Dec 9 2013, 03:53.

Dec 8 2013

build hadoop job using Maven Assembly Plugin, modified pom.xml, added hadoop…
8a3cb8c12dca
·Jinfeng Rao on Dec 8 2013, 20:03.

Dec 6 2013

Extract links and Lucene FST for URLs.
8e1cadb7e70e
·Jeffyrao on Dec 6 2013, 20:26.

Dec 1 2013

started a new branch to extract text from warcbase data and organize the data…
944b138ee0d2
·milad621 on Dec 1 2013, 23:01.

Nov 23 2013

PrintAllUris add to appassembler which will output a urls.html file with all…
7e76e19f2547
·milad621 on Nov 23 2013, 20:25.

Nov 22 2013

Hadoop InputFormats for ARC and WARC files + simple demos.
b5b4e211ea05
·lintool on Nov 22 2013, 17:17.

Nov 7 2013

Created a new runnable to find a uri inside warc/arc files.
c3e37d50e856
·milad621 on Nov 7 2013, 19:32.

Nov 5 2013

One runnable to process both arc and warc files in a folder.
01dff2a1f2a5
·milad621 on Nov 5 2013, 23:30.

Oct 30 2013

IngestWarcFiles fixed. Now it uses jwat-warc to ingest warcfiles to hbase.
e9e3d1c9d940
·milad621 on Oct 30 2013, 01:56.

Oct 17 2013

Arc Processing tools added. Not working with hbase yet.
b1bee7dc3a8e
·milad621 on Oct 17 2013, 19:33.

Oct 2 2013

URL style changed
c860aef8368c
·milad621 on Oct 2 2013, 09:24.

Sep 17 2013

header (navigation bar?) added. Pom update
8706a55f8e36
·milad621 on Sep 17 2013, 19:16.

Aug 13 2013

Refactored dependencies.
19ab27133471
·lintool on Aug 13 2013, 15:17.
Cleaned up analysis code. Removed dead code. Added README.
3a2aec9c5698
·lintool on Aug 13 2013, 10:54.

Aug 12 2013

Refactoring of browser; each archive is now stored in its own separate table…
4fffd3905e9e
·lintool on Aug 12 2013, 23:11.
Interim check-in, refactoring of ingestion program.
0f580edb6186
·lintool on Aug 12 2013, 14:38.

Aug 6 2013

pom.xml updated
8c61d21d5167
·milad621 on Aug 6 2013, 21:00.

Jul 28 2013

pom updated
a65403484d61
·milad621 on Jul 28 2013, 22:25.
pom updated
e16c1d229b37
·milad621 on Jul 28 2013, 20:30.

Jul 18 2013

Refactored Jetty server into WarcBrowser.
41ff1fe630d6
·lintool on Jul 18 2013, 15:10.
pom.xml fixed
3d448add8f08
·milad621 on Jul 18 2013, 12:38.
pom
f3387f289d05
·milad621 on Jul 18 2013, 12:29.
testHbase program
54b8bb992ca6
·milad621 on Jul 18 2013, 12:19.
pom.xml updated
53ad25cf7c10
·milad621 on Jul 18 2013, 12:15.

Jul 17 2013

pom.xml updated
1dd61995bdf1
·milad621 on Jul 17 2013, 20:48.
added hadoop-core to pom.xml
1dd7ac9455cc
·milad621 on Jul 17 2013, 20:36.

Jul 16 2013

pom.xml updated
336a5b5d3009
·milad621 on Jul 16 2013, 08:32.

Jul 13 2013

Conversion into Maven artifact.
12c390826639
·lintool on Jul 13 2013, 22:26.