Diffusion cop-mining-participants (master) History

cop-mining-participants/code
master

Nov 22 2020

Add one-name version with edit distance of experience script
103aa58f800e
·Jan Linder on Nov 22 2020, 21:34.

Nov 17 2020

Fix the bug with extracting the participants
c6b295092771
·Jan Linder on Nov 17 2020, 21:24.

Nov 16 2020

Reverse the change with pagenumbers because it inserted a lot more problems
f4dcd0f64366
·Jan Linder on Nov 16 2020, 23:56.
Fix two bugs with line_comes_first: i) clean category dict after each part of a…
5bee42e32c08
·Jan Linder on Nov 16 2020, 21:53.
Keep page numbers during extractions (bug fix) and other minor fixes
1db59f7ef7b2
·Jan Linder on Nov 16 2020, 18:44.
Adjust generate_complete_dataset to use the metadata file
6a0cb9b440d1
·Jan Linder on Nov 16 2020, 18:07.
Fix the Sri Lanka bug (was taken as a male participant (Sr)
b456f2b06428
·Jan Linder on Nov 16 2020, 13:24.
Adapt do_all.py s.t. it uses the metadata file
9e93900c46b3
·Jan Linder on Nov 16 2020, 13:12.
Rename the Analyzer files and create factory
86343b0a38b3
·Jan Linder on Nov 16 2020, 12:21.
Implement the MeetingAnalyzer class (abstract)
850a9306adc8
·Jan Linder on Nov 16 2020, 11:30.
Add metadata and sb48b
b14274a4a3cc
·Jan Linder on Nov 16 2020, 10:12.
Let OCR lists also start by default with parties as category
62f3c636c617
·Jan Linder on Nov 16 2020, 09:41.
Add script to find biggest delegations
d59172c8e7a5
·Jan Linder on Nov 16 2020, 09:36.

Nov 15 2020

Minor changes
d5a39367f24c
·Jan Linder on Nov 15 2020, 23:55.
Fix bug in DigitalPdfExtr that made pdfs with several list parts to fail
aa46a8340129
·Jan Linder on Nov 15 2020, 22:04.
Minor corrections for the parties over time
4ec3f290fe66
·Jan Linder on Nov 15 2020, 21:56.
Add the country library to unify parties
427418ee24c1
·Jan Linder on Nov 15 2020, 21:56.

Nov 13 2020

UNTESTED Refactoring and renaming of the text extractor part
165b9b13f17f
·Jan Linder on Nov 13 2020, 18:39.
Bugfix of the issue on pages with new categories (order was wrong)
bcde773d7e59
·Jan Linder on Nov 13 2020, 09:40.

Nov 10 2020

Add minor change for complete dataset
3ee4c9e68479
·Jan Linder on Nov 10 2020, 15:16.
Minor changes and todo
9dd37a46c53e
·Jan Linder on Nov 10 2020, 11:07.
Add experience graph s.t. it works
fa5ec0d659bc
·Jan Linder on Nov 10 2020, 08:49.

Nov 9 2020

Added experience graph but failed
4d6352e6b52f
·Jan Linder on Nov 9 2020, 22:10.
Add file to find average number of participants
90dfb209b07e
·Jan Linder on Nov 9 2020, 21:37.
Add delegation size plot
d3111e1d6608
·Jan Linder on Nov 9 2020, 17:25.
add gender analysis
9484071d42bd
·Jan Linder on Nov 9 2020, 16:13.

Nov 3 2020

Add hasTitle to complete dataset
5414f53caa32
·Jan Linder on Nov 3 2020, 11:34.
Add gender in the complete dataset
50cb623d6541
·Jan Linder on Nov 3 2020, 11:21.
Make other Analyzers unstatic.
ed5b12fba5f8
·Jan Linder on Nov 3 2020, 09:41.
Made Analyzer 1-5 unstatic and solved issue with encoding for cop5
85156b2d7027
·Jan Linder on Nov 3 2020, 08:49.

Nov 2 2020

Extraction of SB now possible. Now you need to give meeting label to…
71ee7f02893d
·Jan Linder on Nov 2 2020, 16:58.
Restructuring of repo
2eecd75d75b8
·Jan Linder on Nov 2 2020, 15:50.

Oct 26 2020

Add the description lines extraction
a46d37c100b6
·Jan Linder on Oct 26 2020, 21:44.
Add translation from french to english for cop2
73d06bab31b6
·Jan Linder on Oct 26 2020, 20:17.
Generate the complete dataset now possible
b5c1185e3a2e
·Jan Linder on Oct 26 2020, 20:17.

Oct 22 2020

Add the possibility to have more than one affiliation category per page
db6864b1a785
·Jan Linder on Oct 22 2020, 22:48.
Added new issues in source to readme
0ad97826fa8a
·Jan Linder on Oct 22 2020, 22:47.
Add presentation for Paula and Marlene, week6
2e7f25ecd1bc
·Jan Linder on Oct 22 2020, 16:11.

Oct 21 2020

add generate plots
5531d854d21a
·Jan Linder on Oct 21 2020, 20:43.

Oct 20 2020

Minor bugfixes and improvements
62f11a926efb
·Jan Linder on Oct 20 2020, 23:20.
Explain Analyzer in readme
0b5560f27517
·Jan Linder on Oct 20 2020, 22:55.
Rename cop1to5, included cop5 there
3fd706cf5bd0
·Jan Linder on Oct 20 2020, 22:50.
Improve Analyzers, especially cop7to8
97b00603c9f4
·Jan Linder on Oct 20 2020, 22:48.

Oct 19 2020

Implement cop7to8_analyzer and affilition list extractor.
67a5ee11403c
·Jan Linder on Oct 19 2020, 18:52.
Added party recognition for all cops
883bce5e3184
·Jan Linder on Oct 19 2020, 17:29.

Oct 17 2020

First edition of affiliation category extraction. Only works for cop5+ yet
8b9912f6d1fa
·Jan Linder on Oct 17 2020, 14:07.

Oct 14 2020

minor adaption op salutory addresses
fa3369344206
·Jan Linder on Oct 14 2020, 09:13.
Make pdfToText unstatic.
2186bf7609de
·Jan Linder on Oct 14 2020, 08:24.

Oct 13 2020

Add todo week5
7514566bb3e2
·Jan Linder on Oct 13 2020, 21:11.

Oct 12 2020

Improve pdftotext and its analysis.
dc91f390862e
·Jan Linder on Oct 12 2020, 22:39.
Implement pdfToTxt manually quite correctly
e22f3e528e6a
·Jan Linder on Oct 12 2020, 12:48.

Oct 10 2020

implement PDFPageDetailedAggregator to get the positions of LTContainers
a0f6ce99d856
·Jan Linder on Oct 10 2020, 12:14.

Oct 6 2020

bring copnewer_analyzer to work
c66cb28f7eac
·Jan Linder on Oct 6 2020, 17:32.
Remove old code files
0f3b244e9c77
·Jan Linder on Oct 6 2020, 13:50.
IMPLEMENT PDF TO TXT CORRECTLY Now use pdfminer with the laparam argument to…
29e001a8ff57
·Jan Linder on Oct 6 2020, 00:36.

Oct 5 2020

the ocr now works correctly
66d3e48b2e2a
·Jan Linder on Oct 5 2020, 21:38.
blacklist
154caa57dcc3
·Jan Linder on Oct 5 2020, 21:12.
whitelist ocr: not perfect
2f69d7e02c12
·Jan Linder on Oct 5 2020, 20:41.
Improve analysis for cop3 and cop4
9cc36860b206
·Jan Linder on Oct 5 2020, 14:16.
Small corrections for OCR
0d36b15c21a8
·Jan Linder on Oct 5 2020, 13:16.
finish modularization
67d367f6243d
·Jan Linder on Oct 5 2020, 00:14.

Oct 4 2020

File structure updated. (untested)
e2ddd912c5b8
·Jan Linder on Oct 4 2020, 15:15.
Begin with making a proper modularization
26f03fd369c3
·Jan Linder on Oct 4 2020, 14:22.

Sep 30 2020

inserted boxes for OCR and right parameters
2d93960b8990
·Jan Linder on Sep 30 2020, 21:52.

Sep 28 2020

minor changes of process, added raw of cop3
d37074c1d20f
·Jan Linder on Sep 28 2020, 16:55.
progress on cop2-4
5b6aa3b3ac8b
·Jan Linder on Sep 28 2020, 15:55.

Sep 27 2020

Use of process_copX.py precised in README
57cdc59e9210
·Jan Linder on Sep 27 2020, 12:16.
Implemented processing of cop2-4. Works good for countries, but has major…
355f0c163915
·Jan Linder on Sep 27 2020, 11:46.
try with pypdf2
a557a870c334
·Jan Linder on Sep 27 2020, 09:47.

Sep 26 2020

implemented process cop for 5 - 25 with textract but there are major errors in…
13691532eec5
·Jan Linder on Sep 26 2020, 18:44.

Sep 23 2020

progress on the class and process script
7f3f311f331c
·Jan Linder on Sep 23 2020, 08:40.

Sep 22 2020

Began the copx file
c747832a97fe
·Jan Linder on Sep 22 2020, 17:28.
added raw txt for cop25
aa92caa65e57
·Jan Linder on Sep 22 2020, 09:44.

Sep 17 2020

data complete
9514a37465f4
·Jan Linder on Sep 17 2020, 15:00.
data complete
77b17d5e7bd6
·Jan Linder on Sep 17 2020, 15:00.
first part of the lists
45310b6c42f1
·Jan Linder on Sep 17 2020, 11:34.

cop-mining-participants/codemaster

Nov 22 2020

Nov 17 2020

Nov 16 2020

Nov 15 2020

Nov 13 2020

Nov 10 2020

Nov 9 2020

Nov 3 2020

Nov 2 2020

Oct 26 2020

Oct 22 2020

Oct 21 2020

Oct 20 2020

Oct 19 2020

Oct 17 2020

Oct 14 2020

Oct 13 2020

Oct 12 2020

Oct 10 2020

Oct 6 2020

Oct 5 2020

Oct 4 2020

Sep 30 2020

Sep 28 2020

Sep 27 2020

Sep 26 2020

Sep 23 2020

Sep 22 2020

Sep 17 2020

cop-mining-participants/code
master