Diffusion warcbase (master) History

main
master70c55839d0c3a88004483ae…

Nov 23 2015

Moved initialization of NER3Classifier into map closure; switched from map() to…
70c55839d0c3
·Jeremy Wiebe on Nov 23 2015, 19:24.
Port named entities extractor Pig script over to Spark as per issue #158
4a3d31b8babf
·Alice-Z on Nov 23 2015, 02:05.
ExtractEntities takes a classifier file path
f9422c23efae
·Alice-Z on Nov 23 2015, 02:00.
Revert to object version of NER3Classifier
99583f793bc5
·Alice-Z on Nov 23 2015, 01:56.

Nov 21 2015

Turn test off; classifier is too large to be included
533b4152d534
·Alice-Z on Nov 21 2015, 19:50.
Pass classifier class to ExtractEntities UDF
582b21adefe6
·Alice-Z on Nov 21 2015, 18:59.
add test for ner3classifier
86733707e5a1
·Alice-Z on Nov 21 2015, 17:08.
Clean up, fix tests changed by new keepValidPages
14e521794754
·Alice-Z on Nov 21 2015, 16:33.

Nov 19 2015

Make keepValidPages smarter as in issue #163
2fd98dbfdb67
·Alice-Z on Nov 19 2015, 23:49.
Fix warcloader bug (issue #166)
758288fb2bd9
·Alice-Z on Nov 19 2015, 23:39.

Nov 12 2015

Use Jackson JSON serializers to write to String
cc274ed73b60
·Alice-Z on Nov 12 2015, 20:18.
Clean up string formatting
e9a3965e2389
·Alice-Z on Nov 12 2015, 15:01.
Merge branch 'ner3classifier' of github.com:lintool/warcbase into ner3classifier
d56d62cf1415
·Alice-Z on Nov 12 2015, 04:22.
Working extract entities with correct output string
9f7fa9f26b54
·Alice-Z on Nov 12 2015, 04:18.
WIP: port NER3 Classifier over to Java with example usage
c9ccf559fafc
·Alice-Z on Nov 12 2015, 02:56.
WIP: port NER3 Classifier over to Java with example usage
31269caa44b6
·Alice-Z on Nov 12 2015, 00:45.

Nov 11 2015

Refactor ExtractLinks to be called with the src url
c553ef0d853f
·Alice-Z on Nov 11 2015, 23:08.
Remove extract method as per referenced in issue #146
229d274d6c13
·Alice-Z on Nov 11 2015, 22:41.
add documentation
a599db1f45b9
·Alice-Z on Nov 11 2015, 22:27.
remove extract methods
1ee0455efdf6
·Alice-Z on Nov 11 2015, 22:24.
Add keepValidPages transformation and layer for counting in Spark
2c2607867ef1
·Alice-Z on Nov 11 2015, 22:08.

Nov 10 2015

Clean up extracting code, use pattern matching
e1be481cd782
·Alice-Z on Nov 10 2015, 02:58.

Nov 9 2015

Clean up enum to function mapping
3957b2cce7fb
·Alice-Z on Nov 9 2015, 18:07.
Fix imports and enable tests on JUnitRunner
9140519a9d50
·Alice-Z on Nov 9 2015, 18:07.

Nov 8 2015

Commit clean-up
3eb11b04e499
·Alice-Z on Nov 8 2015, 03:49.

Nov 3 2015

Add Spark support
8057c46945d0
·Alice-Z on Nov 3 2015, 02:58.

Oct 23 2015

add extractCrawldateDomainUrlBody
35e39da6e72d
·Alice-Z on Oct 23 2015, 18:21.

Oct 21 2015

update ArcRecords interface
e5821091ecca
·Alice-Z on Oct 21 2015, 19:56.

Oct 15 2015

extend RDD
cfe76f600c7d
·Alice-Z on Oct 15 2015, 21:13.
first commit
e77201892789
·Alice-Z on Oct 15 2015, 20:07.

Jul 22 2015

Fixed issue #137 Integrate with warc-hadoop-indexer for Shine
258a159c796e
·lintool on Jul 22 2015, 18:11.

Jul 21 2015

Complete revamped cmdline parameter handling.
17393c207934
·lintool on Jul 21 2015, 22:09.
Simplified specification of configs.
0be900c474ef
·lintool on Jul 21 2015, 21:45.
Config for WARCIndexer.
243004be2ded
·lintool on Jul 21 2015, 21:25.
Refactoring to simplify code.
b520faa88283
·lintool on Jul 21 2015, 21:25.

Jul 14 2015

Fixed: https://github.com/ukwa/webarchive-discovery/issues/64
7936dd172aae
·lintool on Jul 14 2015, 00:29.

Jul 13 2015

Rename.
38ac5fb5cba5
·lintool on Jul 13 2015, 06:46.
More code simplification and refactoring.
a3eb65567484
·lintool on Jul 13 2015, 03:09.
Refactoring to simplify code.
46d7cef43ee3
·lintool on Jul 13 2015, 02:39.
Reformat code.
befb3f9b39fb
·lintool on Jul 13 2015, 01:43.
Getting rid of external dependencies by checking in Solr configs.
04f416e56781
·lintool on Jul 13 2015, 00:59.
Starting to incorporate Hadoop WARC indexing code.
335d751b425c
·lintool on Jul 13 2015, 00:43.

Jun 30 2015

Added ExtractBoilerpipeText UDF
11dfe152d4cc
·Jeremy Wiebe on Jun 30 2015, 02:38.

Jun 14 2015

Merge branch 'master' into pyspark
c9f27489b093
·lintool on Jun 14 2015, 18:46.

Jun 9 2015

testing fork
51abf7e86696
·ianmilligan1 on Jun 9 2015, 10:20.
added Python scripts dir
7cb0c2e8d2cc
·Jeremy Wiebe on Jun 9 2015, 04:55.

Jun 7 2015

Removed JWAT; removed unneeded files in org.warcbase.analysis; switched…
d9b553c76ced
·lintool on Jun 7 2015, 17:10.
Added schema metadata in loader.
caf87fbecf45
·lintool on Jun 7 2015, 16:37.
better trapping of errors.
1f9d853f38a2
·lintool on Jun 7 2015, 16:31.

Jun 6 2015

Changed over to WacWarcInputFormatTest as the underlying InputFormat.
13619567667f
·lintool on Jun 6 2015, 22:58.

Jun 4 2015

Better integration of ARC readers in pyspark.
7c1ebde17fb4
·lintool on Jun 4 2015, 22:15.

May 27 2015

Converter to extract all metadata.
b220b0cc175a
·lintool on May 27 2015, 13:16.
Initial experiments with Python converters to use Hadoop InputFormat from…
649f1674352b
·lintool on May 27 2015, 01:31.

May 26 2015

Added Stanford NER UDF
1c013a7b064f
·Jeremy Wiebe on May 26 2015, 18:41.

May 24 2015

Cleanup.
eec01a41b18e
·lintool on May 24 2015, 20:31.
Wraps a try block around everything to catch all errors.
1c9b25027e73
·lintool on May 24 2015, 20:27.
Merge branch 'master' into extract-pdf-udf
9acc8ca74c0a
·lintool on May 24 2015, 19:37.

May 13 2015

tries to fix issues with relative links
a3a2cd7853c8
·lintool on May 13 2015, 00:29.

May 6 2015

Updated UDF to handle relative paths (with source page). Added test case.
5b10497c2d53
·lintool on May 6 2015, 18:19.
UDF for extracting top-level domain from URL.
fb95f3ae562e
·lintool on May 6 2015, 01:21.

Dec 25 2014

Added option to change MAX_CONTENT_SIZE in IngestFiles, Issues #112
f263769c86b7
·lintool on Dec 25 2014, 01:42.

Dec 24 2014

Fixed typo.
52b9696fba87
·lintool on Dec 24 2014, 21:41.
Add ingestion option to select either Snappy or GZ compression, Issue #110
5d471280e7db
·lintool on Dec 24 2014, 21:31.

Dec 23 2014

Renaming.
fb3edb4e40c7
·lintool on Dec 23 2014, 22:03.
Fixed OOM issues.
f6caf0f6f226
·lintool on Dec 23 2014, 22:00.
Fixed warnings.
b1c6995aa034
·lintool on Dec 23 2014, 21:21.
Fixed OOM errors.
59949b3d7af4
·lintool on Dec 23 2014, 04:59.
Fixed Issue #109: OOM when running UrlMappingBuilder
e6b21cc557d6
·lintool on Dec 23 2014, 04:01.

Dec 22 2014

Merge branch 'hbase-api-refactoring' into dev
6acf52f320cf
·lintool on Dec 22 2014, 21:34.

Dec 21 2014

Changed compression back from GZ to Snappy.
7f6d571424a9
·lintool on Dec 21 2014, 04:01.
Fixed minor code formatting issues.
12a00be93209
·lintool on Dec 21 2014, 03:58.

Dec 18 2014

Reformatting.
5c2540099370
·lintool on Dec 18 2014, 14:20.

Dec 11 2014

Updated regex for "Content-Type" (RFC2616 sec. 4.2 says HTTP header field names…
dbaa917eb8b2
·Jeremy Wiebe on Dec 11 2014, 22:26.
Use WarcRecordUtils.getWarcResponseMimeType() for ingest
329ddf9ce247
·Jeremy Wiebe on Dec 11 2014, 22:21.
Added WARC support to analysis/graph classes (except ExtractLinksJwat -- since…
8b1e82f16828
·Jeremy Wiebe on Dec 11 2014, 20:17.
Added getBodyContent(WARCRecord r) method to WarcRecordUtils
eb28dc02a8d2
·Jeremy Wiebe on Dec 11 2014, 20:15.

Dec 10 2014

Added WARC support to analysis:
ec4285807f39
·Jeremy Wiebe on Dec 10 2014, 15:54.

Dec 9 2014

Added WARC support to UrlMappingMapReduceBuilder. It can now accept a path…
65837e09aa3a
·Jeremy Wiebe on Dec 9 2014, 23:01.
ExtractTextFromPDFs updated
c49c58f76ed3
·rwolniak on Dec 9 2014, 00:38.
Updated PIG Tika Parser with new code that should work. Awaiting test on Hadoop
cea1f0cae974
·rwolniak on Dec 9 2014, 00:12.

Dec 8 2014

Added WARC support to WarcbaseResoureStore. OpenWayback replay of WARC files…
724eed27b771
·Jeremy Wiebe on Dec 8 2014, 20:26.
Added WARC file support to IngestFiles.
4239f1f9ae87
·Jeremy Wiebe on Dec 8 2014, 20:25.
Fixed bug in ExtractLinksWac where fst.loadMapping() would choke on filename…
3d8672c631db
·Jeremy Wiebe on Dec 8 2014, 20:18.
For building on OS X, changed HBase compression from Snappy to GZ.
5590850ab2f0
·Jeremy Wiebe on Dec 8 2014, 20:16.
Ooops. *Now* synced with lintool's master…
97550c3727c5
·Jeremy Wiebe on Dec 8 2014, 20:14.
Merge branch 'master' of https://github.com/lintool/warcbase
0379553cc55c
·Jeremy Wiebe on Dec 8 2014, 19:57.

Nov 12 2014

Still testing ExtractTextFromPDFs
8a3fed9342fc
·rwolniak on Nov 12 2014, 17:07.

Nov 10 2014

Extract text from PDF UDF. Currently still debugging
10b92d78f6ed
·rwolniak on Nov 10 2014, 04:07.

Oct 31 2014

Minor refactoring.
b8915343f5c9
·lintool on Oct 31 2014, 18:28.

Oct 22 2014

Refactoring to create method that extracts MIME from WARC response records.
0145d74ed35e
·lintool on Oct 22 2014, 18:42.

Oct 19 2014

Merge branch 'pig' into warc
d491020c8287
·lintool on Oct 19 2014, 19:01.
WARC-related Hadoop bindings.
410cfd81a069
·lintool on Oct 19 2014, 18:59.
Minor refactoring.
f4a249469cbc
·lintool on Oct 19 2014, 16:58.
Pig ArcLoader exports its own ResourceSchema.
0c52caed50cf
·lintool on Oct 19 2014, 02:02.
Merge branch 'master' into hbase-api-refactoring
f97523f440ff
·lintool on Oct 19 2014, 01:21.
Fixed test cases.
3af097ea655d
·lintool on Oct 19 2014, 00:50.
Pig loader materializes the actual content.
4ceefef78cdc
·lintool on Oct 19 2014, 00:47.

Oct 18 2014

Refactored Pig loaded to use WAC API.
58df7da00880
·lintool on Oct 18 2014, 17:05.

Oct 13 2014

Updated documentation.
de4267bf28f1
·lintool on Oct 13 2014, 14:21.

Sep 15 2014

Merge branch 'master' into warc
457a71345d2b
·lintool on Sep 15 2014, 01:57.

R1473/src/mainmaster70c55839d0c3a88004483ae…

Nov 23 2015

Nov 21 2015

Nov 19 2015

Nov 12 2015

Nov 11 2015

Nov 10 2015

Nov 9 2015

Nov 8 2015

Nov 3 2015

Oct 23 2015

Oct 21 2015

Oct 15 2015

Jul 22 2015

Jul 21 2015

Jul 14 2015

Jul 13 2015

Jun 30 2015

Jun 14 2015

Jun 9 2015

Jun 7 2015

Jun 6 2015

Jun 4 2015

May 27 2015

May 26 2015

May 24 2015

May 13 2015

May 6 2015

Dec 25 2014

Dec 24 2014

Dec 23 2014

Dec 22 2014

Dec 21 2014

Dec 18 2014

Dec 11 2014

Dec 10 2014

Dec 9 2014

Dec 8 2014

Nov 12 2014

Nov 10 2014

Oct 31 2014

Oct 22 2014

Oct 19 2014

Oct 18 2014

Oct 13 2014

Sep 15 2014

R1473/src/main
master70c55839d0c3a88004483ae…