Diffusion cop-mining-participants (master) History

cop-mining-participants/code
masterbfaf644ce5dbea2ffe08592…

Dec 14 2020

Complete be able to create the dataset for interventions
bfaf644ce5db
·Jan Linder on Dec 14 2020, 18:41.
Start with ridge regression on interventions
2fbd43593dc9
·Jan Linder on Dec 14 2020, 15:02.
prepare tatianas data
55b1e910b671
·Jan Linder on Dec 14 2020, 12:49.

Dec 8 2020

Add todo week 13
e142ed229010
·Jan Linder on Dec 8 2020, 10:40.

Dec 7 2020

Some plot coloring changes
b2f8d4a7c209
·Jan Linder on Dec 7 2020, 18:06.
Introduce roles no description and no keyword found
bb055669c390
·Jan Linder on Dec 7 2020, 18:04.
Add absolute fossil fuel plot and modify experience score of affiliations
ea7760fc73b4
·Jan Linder on Dec 7 2020, 17:37.
Small update to count the number of detected valid countries
6c9935f135ae
·Jan Linder on Dec 7 2020, 14:49.

Dec 4 2020

After second meeting with Paula and Marlene
eb28513c4a9f
·Jan Linder on Dec 4 2020, 11:31.

Dec 3 2020

Little plot changements
79a294b0c664
·Jan Linder on Dec 3 2020, 09:23.
New plot plus some adjustments
88d774ac0654
·Jan Linder on Dec 3 2020, 08:27.
Bugfix that made all french meetings look the same
dcfbad132fff
·Jan Linder on Dec 3 2020, 08:27.

Dec 2 2020

Little color correction
8afee14b8b10
·Jan Linder on Dec 2 2020, 22:13.
Add new plot files
ccb733a2ee29
·Jan Linder on Dec 2 2020, 22:04.
Move all plots in seperate files
2a27b8d7dc3b
·Jan Linder on Dec 2 2020, 22:02.
Changes in the existing plots, order for role plot
21ce84dcd7c0
·Jan Linder on Dec 2 2020, 21:51.

Dec 1 2020

Add todo week 12
86382d8f2cff
·Jan Linder on Dec 1 2020, 11:02.
Add fossil fuel industry plot
4ec98cbf8c08
·Jan Linder on Dec 1 2020, 09:44.

Nov 30 2020

Improve government plot
5dbdc34e1e1c
·Jan Linder on Nov 30 2020, 16:31.

Nov 29 2020

Implement government plot
721b4f636bfc
·Jan Linder on Nov 29 2020, 23:09.
Add goverment plot file (empty)
bda2815268e4
·Jan Linder on Nov 29 2020, 22:45.
Minor changes
88ace802665c
·Jan Linder on Nov 29 2020, 22:45.

Nov 28 2020

Change the role function
0d61b283764e
·Jan Linder on Nov 28 2020, 15:44.
Change what the argument for the translate script is
97992315a056
·Jan Linder on Nov 28 2020, 13:22.
Translate the other sb meetings and insert that to the main script
9ca8fe345ef7
·Jan Linder on Nov 28 2020, 13:19.
Make all names lowercase in the total dataset
5890a105f1d9
·Jan Linder on Nov 28 2020, 11:54.
Implement plot for experience
765cae43bb6d
·Jan Linder on Nov 28 2020, 11:48.

Nov 27 2020

Begin with the experience plot
acc80fad4804
·Jan Linder on Nov 27 2020, 19:00.

Nov 23 2020

Add todo week 11.
f351007b4a67
·Jan Linder on Nov 23 2020, 14:47.
Improved one-person find experience script
970aa39095b1
·Jan Linder on Nov 23 2020, 08:49.

Nov 22 2020

Minor changes
80eadf9fdc6c
·Jan Linder on Nov 22 2020, 23:04.
Add one-name version with edit distance of experience script
103aa58f800e
·Jan Linder on Nov 22 2020, 21:34.

Nov 17 2020

Fix the bug with extracting the participants
c6b295092771
·Jan Linder on Nov 17 2020, 21:24.

Nov 16 2020

Reverse the change with pagenumbers because it inserted a lot more problems
f4dcd0f64366
·Jan Linder on Nov 16 2020, 23:56.
Fix two bugs with line_comes_first: i) clean category dict after each part of a…
5bee42e32c08
·Jan Linder on Nov 16 2020, 21:53.
Keep page numbers during extractions (bug fix) and other minor fixes
1db59f7ef7b2
·Jan Linder on Nov 16 2020, 18:44.
Adjust generate_complete_dataset to use the metadata file
6a0cb9b440d1
·Jan Linder on Nov 16 2020, 18:07.
Fix the Sri Lanka bug (was taken as a male participant (Sr)
b456f2b06428
·Jan Linder on Nov 16 2020, 13:24.
Adapt do_all.py s.t. it uses the metadata file
9e93900c46b3
·Jan Linder on Nov 16 2020, 13:12.
Rename the Analyzer files and create factory
86343b0a38b3
·Jan Linder on Nov 16 2020, 12:21.
Implement the MeetingAnalyzer class (abstract)
850a9306adc8
·Jan Linder on Nov 16 2020, 11:30.
Add metadata and sb48b
b14274a4a3cc
·Jan Linder on Nov 16 2020, 10:12.
Let OCR lists also start by default with parties as category
62f3c636c617
·Jan Linder on Nov 16 2020, 09:41.
Add script to find biggest delegations
d59172c8e7a5
·Jan Linder on Nov 16 2020, 09:36.

Nov 15 2020

Minor changes
d5a39367f24c
·Jan Linder on Nov 15 2020, 23:55.
Fix bug in DigitalPdfExtr that made pdfs with several list parts to fail
aa46a8340129
·Jan Linder on Nov 15 2020, 22:04.
Minor corrections for the parties over time
4ec3f290fe66
·Jan Linder on Nov 15 2020, 21:56.
Add the country library to unify parties
427418ee24c1
·Jan Linder on Nov 15 2020, 21:56.

Nov 13 2020

UNTESTED Refactoring and renaming of the text extractor part
165b9b13f17f
·Jan Linder on Nov 13 2020, 18:39.
Bugfix of the issue on pages with new categories (order was wrong)
bcde773d7e59
·Jan Linder on Nov 13 2020, 09:40.

Nov 10 2020

Add minor change for complete dataset
3ee4c9e68479
·Jan Linder on Nov 10 2020, 15:16.
Minor changes and todo
9dd37a46c53e
·Jan Linder on Nov 10 2020, 11:07.
Add experience graph s.t. it works
fa5ec0d659bc
·Jan Linder on Nov 10 2020, 08:49.

Nov 9 2020

Added experience graph but failed
4d6352e6b52f
·Jan Linder on Nov 9 2020, 22:10.
Add file to find average number of participants
90dfb209b07e
·Jan Linder on Nov 9 2020, 21:37.
Add delegation size plot
d3111e1d6608
·Jan Linder on Nov 9 2020, 17:25.
add gender analysis
9484071d42bd
·Jan Linder on Nov 9 2020, 16:13.

Nov 3 2020

Add hasTitle to complete dataset
5414f53caa32
·Jan Linder on Nov 3 2020, 11:34.
Add gender in the complete dataset
50cb623d6541
·Jan Linder on Nov 3 2020, 11:21.
Make other Analyzers unstatic.
ed5b12fba5f8
·Jan Linder on Nov 3 2020, 09:41.
Made Analyzer 1-5 unstatic and solved issue with encoding for cop5
85156b2d7027
·Jan Linder on Nov 3 2020, 08:49.

Nov 2 2020

Extraction of SB now possible. Now you need to give meeting label to…
71ee7f02893d
·Jan Linder on Nov 2 2020, 16:58.
Restructuring of repo
2eecd75d75b8
·Jan Linder on Nov 2 2020, 15:50.

Oct 26 2020

Add the description lines extraction
a46d37c100b6
·Jan Linder on Oct 26 2020, 21:44.
Add translation from french to english for cop2
73d06bab31b6
·Jan Linder on Oct 26 2020, 20:17.
Generate the complete dataset now possible
b5c1185e3a2e
·Jan Linder on Oct 26 2020, 20:17.

Oct 22 2020

Add the possibility to have more than one affiliation category per page
db6864b1a785
·Jan Linder on Oct 22 2020, 22:48.
Added new issues in source to readme
0ad97826fa8a
·Jan Linder on Oct 22 2020, 22:47.
Add presentation for Paula and Marlene, week6
2e7f25ecd1bc
·Jan Linder on Oct 22 2020, 16:11.

Oct 21 2020

add generate plots
5531d854d21a
·Jan Linder on Oct 21 2020, 20:43.

Oct 20 2020

Minor bugfixes and improvements
62f11a926efb
·Jan Linder on Oct 20 2020, 23:20.
Explain Analyzer in readme
0b5560f27517
·Jan Linder on Oct 20 2020, 22:55.
Rename cop1to5, included cop5 there
3fd706cf5bd0
·Jan Linder on Oct 20 2020, 22:50.
Improve Analyzers, especially cop7to8
97b00603c9f4
·Jan Linder on Oct 20 2020, 22:48.

Oct 19 2020

Implement cop7to8_analyzer and affilition list extractor.
67a5ee11403c
·Jan Linder on Oct 19 2020, 18:52.
Added party recognition for all cops
883bce5e3184
·Jan Linder on Oct 19 2020, 17:29.

Oct 17 2020

First edition of affiliation category extraction. Only works for cop5+ yet
8b9912f6d1fa
·Jan Linder on Oct 17 2020, 14:07.

Oct 14 2020

minor adaption op salutory addresses
fa3369344206
·Jan Linder on Oct 14 2020, 09:13.
Make pdfToText unstatic.
2186bf7609de
·Jan Linder on Oct 14 2020, 08:24.

Oct 13 2020

Add todo week5
7514566bb3e2
·Jan Linder on Oct 13 2020, 21:11.

Oct 12 2020

Improve pdftotext and its analysis.
dc91f390862e
·Jan Linder on Oct 12 2020, 22:39.
Implement pdfToTxt manually quite correctly
e22f3e528e6a
·Jan Linder on Oct 12 2020, 12:48.

Oct 10 2020

implement PDFPageDetailedAggregator to get the positions of LTContainers
a0f6ce99d856
·Jan Linder on Oct 10 2020, 12:14.

Oct 6 2020

bring copnewer_analyzer to work
c66cb28f7eac
·Jan Linder on Oct 6 2020, 17:32.
Remove old code files
0f3b244e9c77
·Jan Linder on Oct 6 2020, 13:50.
IMPLEMENT PDF TO TXT CORRECTLY Now use pdfminer with the laparam argument to…
29e001a8ff57
·Jan Linder on Oct 6 2020, 00:36.

Oct 5 2020

the ocr now works correctly
66d3e48b2e2a
·Jan Linder on Oct 5 2020, 21:38.
blacklist
154caa57dcc3
·Jan Linder on Oct 5 2020, 21:12.
whitelist ocr: not perfect
2f69d7e02c12
·Jan Linder on Oct 5 2020, 20:41.
Improve analysis for cop3 and cop4
9cc36860b206
·Jan Linder on Oct 5 2020, 14:16.
Small corrections for OCR
0d36b15c21a8
·Jan Linder on Oct 5 2020, 13:16.
finish modularization
67d367f6243d
·Jan Linder on Oct 5 2020, 00:14.

Oct 4 2020

File structure updated. (untested)
e2ddd912c5b8
·Jan Linder on Oct 4 2020, 15:15.
Begin with making a proper modularization
26f03fd369c3
·Jan Linder on Oct 4 2020, 14:22.

Sep 30 2020

inserted boxes for OCR and right parameters
2d93960b8990
·Jan Linder on Sep 30 2020, 21:52.

Sep 28 2020

minor changes of process, added raw of cop3
d37074c1d20f
·Jan Linder on Sep 28 2020, 16:55.
progress on cop2-4
5b6aa3b3ac8b
·Jan Linder on Sep 28 2020, 15:55.

Sep 27 2020

Use of process_copX.py precised in README
57cdc59e9210
·Jan Linder on Sep 27 2020, 12:16.
Implemented processing of cop2-4. Works good for countries, but has major…
355f0c163915
·Jan Linder on Sep 27 2020, 11:46.
try with pypdf2
a557a870c334
·Jan Linder on Sep 27 2020, 09:47.

cop-mining-participants/codemasterbfaf644ce5dbea2ffe08592…

Dec 14 2020

Dec 8 2020

Dec 7 2020

Dec 4 2020

Dec 3 2020

Dec 2 2020

Dec 1 2020

Nov 30 2020

Nov 29 2020

Nov 28 2020

Nov 27 2020

Nov 23 2020

Nov 22 2020

Nov 17 2020

Nov 16 2020

Nov 15 2020

Nov 13 2020

Nov 10 2020

Nov 9 2020

Nov 3 2020

Nov 2 2020

Oct 26 2020

Oct 22 2020

Oct 21 2020

Oct 20 2020

Oct 19 2020

Oct 17 2020

Oct 14 2020

Oct 13 2020

Oct 12 2020

Oct 10 2020

Oct 6 2020

Oct 5 2020

Oct 4 2020

Sep 30 2020

Sep 28 2020

Sep 27 2020

cop-mining-participants/code
masterbfaf644ce5dbea2ffe08592…