"/home/walch/miniconda3/envs/py3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
" from ._conv import register_converters as _register_converters\n"
+ "/home/walch/miniconda3/envs/py3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
+ " from ._conv import register_converters as _register_converters\n"
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
"\u001b[0;31mKeyError\u001b[0m: 'mmh_mse_0'"
]
}
],
"source": [
"a = pd.read_csv(os.path.join(my_ds.ds_path, 'log.txt'))\n",
"/home/walch/miniconda3/envs/py3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
" from ._conv import register_converters as _register_converters\n"
"The _features_-file is more high-level than the meteo_data file, and has a well-defined file structure. In the given directory (_data_path_), there should be one subfolder with _raw_data_, and one subfolder with _datasets_. All models are stored here, together with the feature and target tables for training, validation, testing and query (which is what I call a set of __unlabelled__ features, for which we cannot compute any error but which gives us the output that we are finally interested in)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# data_path = os.path.abspath(\"/Users/alinawalch/Documents/EPFL/data/meteo\") # folder in which raw data is stored\n",
"print(new_set.train_path) # this will contain both training and validation feature tables\n",
"print(new_set.test_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So far they are empty; in order to make a new feature table, we must run *make_dataset*. We can either create training and testing at the same time (i.e. only run through the data once), or we can run *make_testset* seperately."
"You can now find the following files in the model folder:<br>\n",
"- **norm_features.csv** and **norm_targets.csv** are excel files containing the normalisation data (mean, std, min, max)\n",
"- Inside *train*: **features.hdf5** and **targets.hdf5** as well as csv files with training locations and training dates\n",
"- Inside *test*: similar information labelled with **_test...**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Normalisation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The __normalise_all__ function checks if training and testing features exist, and creates a copy of the files with normalised data. The training data is randomly split into __training__ and __validation__ data. The inputs are the following:\n",
"- **feature_norm** and **target_norm**: Set to *'mean'* to standardize the data to zero mean and unit variance, and to *'range'* to scale to [0,1]. Anything else will just copy the data and not perform any normalisation\n",
"- **val_ratio**: Ratio, according to which the tables are split into training and validation sets (default 0.8, i.e. 80% training and 20% testing)\n",
"- **force_normalization**: Overwrites files if they already exists"
"Now, some files were added with the appendix **_norm_mean** for the features and **_norm_range** for the targets and the prefix __train___ and __val___ in the *train*-folder. Alternatively, you can also normalise training and testing data seperately:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"set normalization status to mean\n",
"set normalization status to range\n",
"Normalised feature files already exist - set force_normalization to true to re-compute the file\n"
"Normalised file already exists - set force_normalization to true to re-compute the file\n",
"Normalised file already exists - set force_normalization to true to re-compute the file\n"
]
}
],
"source": [
"new_set.normalize_test()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Query data (the actual modelling)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I have created the fishnets that gives the locations for the final grid in ArcGIS, so all I am doing here is to convert the locations from csv into a list of features and adding the additional information that I want to add (e.g. hour and month). The query locations are expected to be in the subfolder _locations_ in the data path."
"In order to get back to an original data format so we can plot it, we need to rescale the targets. These must be computed by the model, but just for demo we can __copy the targets__ from the test set (*test/test_targets*) and then run this:"
"To reconvert the tables to xarray, we first need to concatenate again the features and targets. This example takes for this the original feature and target tables:"
"As some information, such as the year or the day in this case, is not present in the data, it means that if we trying to convert anything else than the **query**-data back to an xarray (which actually we should not do or want to do anyways), then there may be duplicates - multiple entries with the exact same index. This causes trouble and cannot be converted to an xarray. What we can do to make sure that there are no duplicates is to take the mean over all index variables:"