Homec4science
Diffusion cop-mining-participants (master) Browse

cop-mining-participants/report/predictive_modelling.tex
eb1f439034a7master

Last Change
Raw File

predictive_modelling.tex
View Options

\section{Predictive Modelling} \label{predictive_modelling}
Having extracted and processed the data contained in the participant lists, we use them to build predictive models for other data.
First, we build linear models for the data on interventions at UNFCCC meetings collected by Tatiana Cogne and Victor Kristof (see \ref{tatiana}).
Note that we can't go further on this topic due to time constraints, but there is more potential for creating models with our data,
especially for the interaction dataset also collected by Tatiana Cogne and Victor Kristof.
\subsection{Predict Interventions}
The data on interventions lists for different UNFCCC meetings how many times a party intervenes in this meeting.
We build a model that predicts for a party and a given meeting the number of interventions of the party at this meeting.
Figure \ref{fig:interv_distr} plots the distribution of the interventions, i.e., the distribution of the labels of the complete dataset.
Most parties don't have any or only one interventions, while some parties intervene a lot more.
\begin{figure}[ht]
\centering
\includegraphics[width=0.7\textwidth]{distr_interventions.png}
\caption{Distribution of the intervention labels}
\label{fig:interv_distr}
\end{figure}
\subsubsection{Data samples}
We define a data sample $x_i$ as the participation of a party at a meeting. Note that we only consider parties and no other
affiliations as only parties are able to make interventions in the official negotiations. We thus define the features of
a data sample.
\begin{center}
\begin{tabularx}{\textwidth}{|c|c|X|}
\hline
Name & Value Range & Description \\
\hline\hline
year & $\{0, 1, \dots, 24\}$ & The year the meeting took place minus 1995 \\
\hline
number\_of\_delegates & $\{1, 2, \dots, 1589\}$ & Number of participants of this delegation \\
\hline
meeting\_type & $\{0, 1\}$ & 0 if the meeting is a COP, 1 if it's an SB \\
\hline
government\_rate & $[0,1]$ & Proportion of delegates with role "Government" \\
\hline
diplomacy\_rate & $[0,1]$ & Proportion of delegates with role "Diplomacy" \\
\hline
security\_rate & $[0,1]$ & Proportion of delegates with role "Security" \\
\hline
press\_rate & $[0,1]$ & Proportion of delegates with role "Press" \\
\hline
university\_rate & $[0,1]$ & Proportion of delegates with role "Universities" \\
\hline
no\_description\_rate & $[0,1]$ & Proportion of delegates with no description \\
\hline
no\_keyword\_rate & $[0,1]$ & Proportion of delegates with no detected keyword \\
\hline
nb\_fossil\_fuel\_industry\_associations & $\{0, 1, \dots, 26\}$ & Absolute number of delegates with association to the fossil fuel industry \\
\hline
woman\_proportion & $[0,1]$ & The proportion of female participants in the delegation \\
\hline
experience\_score\_cop & $[0,18]$ & The experience score on previous COPs of the delegation \\
\hline
experience\_score\_sb & $[0,17]$ & The experience score on previous COPs of the delegation \\
\hline
experience\_score\_parties\_rate & $[0,1]$ & The proportion of the total experience score that has been acquired in the category "Parties" \\
\hline
is\_Afghanistan & $\{0, 1\}$ & 1 if the delegation is Afghanistan \\
\hline
is\_Albania & $\{0, 1\}$ & 1 if the delegation is Albania \\
\hline
$\vdots$ & $\vdots$ & $\vdots$ \\
\hline
is\_Zimbabwe & $\{0, 1\}$ & 1 if the delegation is Zimbabwe \\
\hline
is\_unrecognized\_country & $\{0, 1\}$ & 1 if no party has been detected \\
\hline
\end{tabularx}
\end{center}
There is a total of $213$ features. The feature \textit{year} is the year the meeting took place and is substracted 1995 which is the year of the first meeting (SB1)
to get values closer to zero.
The features \textit{government\_rate} to \textit{no\_keyword\_rate} correspond to the proportion of each role that we assign
(see \ref{roles}). For the experience score, we provide COP and SB experience in total numbers, they sum up to the total
experience score of an affiliation. The \textit{experience\_score\_parties\_rate} denotes the rate of the total experience score
that has been acquired in parties (see \ref{experience}).
The information about the parties are converted into 198 binary features, one for each of the 197 Parties to the Convention
and one for an invalid or unrecognized country. \\
In total, we have 9218 data samples. We randomly pick about 80\% of these samples, i.e. 7400 samples, as our training set.
The resting samples form our test set.
\subsubsection{Models}
% baseline models
We first build two \textbf{baseline models}, such that we are later able to compare our models to those simple models.
The first baseline model is a majority-vote predictor which always predicts zero interventions, as this is the most common label. Hence,
\begin{equation}
\hat{y_i} = 0 \text{,}
\end{equation}
where $\hat{y_i}$ is the label that we predict for sample $\boldsymbol{x}_i$.
The second, slightly more sophisticated baseline model consists in computing the average number of interventions a
party did over all meetings in the training data
and always predicting this average. Formally,
\begin{equation}
\hat{y_i} = \frac{1}{N_{p(i)}} \sum_{k=1}^{N_{p(i)}} y_k^{p(i)} ,
\end{equation}
where $p(i)$ is the party of sample $\boldsymbol{x}_i$ and $\{y_1^{p(i)}, y_2^{p(i)}, \dots, y_{N_{p(i)}}^{p(i)}\}$ are the number of interventions
of party $p(i)$ in the training samples. \\
Let $N = 7400$ be the number of training samples and let $D = 213$ be the number of features. Let
\begin{equation}
\tilde{\boldsymbol{x}}_i =
\begin{bmatrix}
\boldsymbol{x}_i \\
1
\end{bmatrix}
\in \mathbb{R}^{D + 1}
\end{equation}
be a feature vector augmented with 1 to include the global bias.
Let
\begin{equation}
\boldsymbol{X} =
\begin{bmatrix}
\tilde{\boldsymbol{x}}_1^T \\
\tilde{\boldsymbol{x}}_2^T \\
\vdots \\
\tilde{\boldsymbol{x}}_N^T
\end{bmatrix}
\end{equation}
be a matrix that contains all the augmented feature vectors.
And let
\begin{equation}
\boldsymbol{y} =
\begin{bmatrix}
y_1 \\
y_2 \\
\vdots \\
y_N
\end{bmatrix}
\in \mathbb{R}^N
\end{equation}
be the vector of training labels. \\
% linear model
Now, we introduce a \textbf{ridge regression} model.
For this reason, we normalize the features before training the model. For a feature $x_{i,j}$ we compute
\begin{equation}
x_{i,j} ' = \frac{x_{i,j} - \mu}{l},
\end{equation}
where $\mu \in \mathbb{R}$ is the mean of all the features $x_{k,j}$ and $l \in \mathbb{R}_{\ge 0}$ is the l2-norm of all the features $x_{k,j} \enspace \forall k \in \{1, 2, \hdots, N\}$.
To fit this model, we compute the optimal parameters $\boldsymbol{w}^* \in \mathbb{R}^{D + 1}$ as
\begin{equation}
\boldsymbol{w}^* = (\boldsymbol{X}^T \boldsymbol{X} + \alpha \boldsymbol{I}_N)^{-1} \boldsymbol{X}^T \boldsymbol{y} \text{,}
\end{equation}
where $\alpha \in \mathbb{R}$ is a regularizer.
The predictions are then computed as
\begin{equation}
\hat{y_i} = \tilde{\boldsymbol{x}}_i^T \boldsymbol{w}^* \text{.}
\end{equation}\\
% linear model with logarithmic transformation
As our labels seem to be distributed exponentially, we try another approach where
we perform a \textbf{logarithmic transformation} to our labels before
applying the same linear model as before. We transform each label $y$ s.t. $y' = log(c + y)$ with $c > 0$.
We arbitrarily choose $c = 1$ so that $y' = 0$ when $y = 0$.
The fitting of the model and the predictions are then made exactly as for the ridge regression model. \\
% mixed model
A next approach is to try to handle the large amount of zero interventions better.
Similar to others before us \cite{ridout1998models, fletcher2005modelling, martin2005modelling},
we introduce a \textbf{two-step model} to better handle the massive count of zero labels.
Our two-step model works as follows:
\begin{enumerate}
\item Classify each sample as zero interventions or non-zero.
\item For the non-zero samples, predict the number of interventions (at least 1).
\end{enumerate}
In the first step, we use a logistic regressor with regularization.
In the second step, we use a Poisson regressor with regularization.
Mathematically, the prediction is
\begin{equation}
\hat{y_i} =
\begin{cases}
0 & \text{if } \sigma (\boldsymbol{x}^T \boldsymbol{w}^*) < 0.5 \\
\exp(\boldsymbol{x}^T \boldsymbol{z}^*) & \text{otherwise}
\end{cases} \text{,}
\end{equation}
where $\boldsymbol{w}^*$ is learned as the parameters of a logistic regression and $\boldsymbol{z}^*$ is learned as the parameters of a Poisson regression on those
training samples $\boldsymbol{x}_i$ for which $y_i > 0$. Formally, the fitting of the Poisson regressor is done as follows.
We model the count data $ y_i $ as
\begin{equation}
P(y_i | \boldsymbol{x}_i, \boldsymbol{z}^*) = \frac{\lambda^{y_i} e^{\lambda}}{y_i !} \text{,}
\end{equation}
where $ \lambda = \exp(\boldsymbol{x}_i^T \boldsymbol{z}^*) $ is the rate of interventions.
Let $ \boldsymbol{X}' = [\boldsymbol{x}_i | y_i > 0] \in \mathbb{R}^{N' \times (D + 1)} $ be the training samples
for which the number of interventions is not zero.
Then the log-likelihood of this model is
\begin{equation}
\Likeh (\boldsymbol{z}^* | \boldsymbol{X}', \boldsymbol{y}) = \sum_{i=1}^{N'} [y_i \log(\lambda) - \lambda - \log(y_i!)],
\end{equation}
and the optimal parameters are obtained as
\begin{equation}
\boldsymbol{z}^* = \underset{\boldsymbol{z}}{\mathrm{argmin}} [\Likeh (\boldsymbol{z} | \boldsymbol{X}', \boldsymbol{y}) + \alpha \| \boldsymbol{z} \|] \text{,}
\end{equation}
where $\alpha \in \mathbb{R}$ is the regularizer.
\subsubsection{Results}
We will compare our models by the root-mean-square error (RMSE) between the predicted number of interventions $\hat{y_i}$ and the true values $y_i$.
The root-mean-square error for $n$ samples is computed as
\begin{equation}
RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (\hat{y_i} - y_i)^2}
\end{equation}
Figure \ref{fig:RMSE_results} shows the RMSE of the different models.
\begin{figure}[ht]
\centering
\includegraphics[width=0.9\textwidth]{resultsRMSE.png}
\caption{Resulting RMSE of the different models}
\label{fig:RMSE_results}
\end{figure}
First, we consider the \textbf{baseline models}. When always predicting zero interventions, the test data yields $ RMSE = 9.54 $.
When we always predict the average number of interventions of the party in question during all samples in the training data,
the test data yields $ RMSE = 5.02 $. This shows that the information of the party already gives a lot of information about the behavior
during meetings. \\
The \textbf{ridge regression} model with all features yields an $ RMSE = 5.01 $.
The optimal solution was found with cross-validation at regularizer $\alpha = 0.0101 $.
We can analyze features with the strongest influence on the prediction.
The bias of the whole dataset is at $ w^*_0 = 3.281 $. The features with the strongest influence on the predictions are parties, as we expect seeing that the second
baseline model works pretty good. The highest tendency to many interventions per meetings is showed by the European Union ($ + 74.7 $), United States ($ + 53.3 $) and China ($ + 48.0 $).
Cote d'Ivoire ($ - 2.58 $), San Marino ($ - 2.56 $) and Greece ($ - 2.48 $) are the parties that bias the most towards little interventions.
When considering only non-party features, the top of the list towards more interventions are \textit{press\_rate} ($ + 2.51 $), \textit{university\_rate} ($ + 1.11 $)
and \textit{experience\_score\_parties\_rate} ($ + 0.70 $).
The non-party features that are lowering the predicted number of interventions the most are \textit{no\_description\_rate} ($ - 1.67 $), \textit{diplomacy\_rate} ($ - 0.82 $)
and \textit{no\_keyword\_rate} ($ - 0.43 $).
Interestingly, the year and the number of delegates are the features with the weakest influence on the prediction. Apparently, time and delegation size do
have a rather small influence on the activity of a party. \\
The \textbf{logarthmic transformation} unfortunately doesn't help the model to make better predictions.
It yields an $ RMSE = 5.76 $. The problem is that even with a logarithmic transformation, the labels still don't follow a Gaussian distribution
at all, as Figure \ref{fig:interv_distr_logtransf} shows. \\
\begin{figure}[ht]
\centering
\includegraphics[width=0.7\textwidth]{distr_interventions_logtransf.png}
\caption{Distribution of the intervention labels after the logarithmic transformation}
\label{fig:interv_distr_logtransf}
\end{figure}
The \textbf{two-step model} only slightly improves the prediction, it yields an $ RMSE = 4.94 $.
The first step correctly classifies 79.2\% of the test samples into zero or non-zero, with an optimal regularizer of $\alpha = 1.035 $.
The second step predicts the number of interventions on the samples that have been classified as non-zero by the logistic regressor.
For only those samples, i.e., the ones that have been predicted to be non-zero, it yields an $ RMSE = 9.89 $.
When looking at the final prediction of all the test samples, the two-step model yields an $ RMSE = 4.94 $.
This is a slight improvement compared to the previous models.
c4science · Help