Homec4science
Diffusion PDM-Nicola-Oulu (master) Browse

R11149/main_project/report/Chapters_old/2_n1_data_structures.tex
2e2e9f4b5145master

Last Change
Raw File

2_n1_data_structures.tex
View Options

Data is stored in a variety of formats. The most common format to store data is still related to table formats, as it has well structured entries. This enables to generate in example automatic queries or simple rule based transformations.
The main difference is probably how well defined the structure is. Some have no official rules, while others have a strict syntax. Some use predefined value types, others relay on the user to use consistent formatation and accept any input. Some databases are also encoded to save memory, which slightly reduces readability as it can not just be opened in a text editor.
For the presented data-structures the properties are summerized in the table \ref{tab:data-format-table}.
\captionsetup[table]{justification=raggedright,singlelinecheck=off}
\begin{table}[!ht]
% Add some padding to the table cells:
\def\arraystretch{1.1}%
\begin{center}
\caption{Different data formats and their features. "x" means it is supported. "$\cdot$" symbolises it can be done, but is not the common usecase. }
\vspace{100pt}
\label{tab:data-format-table}
\begin{tabular}{l|llllll}
data format &
\tabrotate{easy readable} &
\tabrotate{standardised syntax} &
\tabrotate{nested structures} &
\tabrotate{defined value types} &
\tabrotate{schema} &
\tabrotate{memory efficient} \\ \hline
"EU-ADR" \cite{data_harmoization_health}
%e s n d s m
CSV
&x& &$\cdot$& & & \\ \hline
%e s n d s m
JSON
&x&x&x& & & \\ \hline
%e s n d s m
XML
&x&x&x& & & \\ \hline
%e s n d s m
ARVO
& &x&x&x&x&x \\ \hline
%e s n d s m
Parquet
& &x&$\cdot$&x&x&x \\ \hline
%e s n d s m
Protobuf
& &x&x&x&x&x \\ \hline
%e s n d s m
CDM
& & & & & & \\ \hline
"example" \cite{data_harmonization_health_dnn}
%e s n d s m
csv
& & & & & & \\ \hline
\end{tabular}
\end{center}
\end{table}
\subsection{CSV}
CSV stands for "comma seperated values". It is a table saved in a text file. The file extension is ".csv".
Different entries are usually separated by a line break. The exact symbol for the line break depends on the users operating system. Rows are separated by the comma "," sign. This convention is not strict and it is also possible to use other symbols such as spaces, minus signs or tabulators. When using tabulators the file extension might be changed to ".tsv".
The values have no strict formatation. Users of a document need to agree on a specific format. This is especially important as some information such as dates can be ambiguous without knowing the used format.
It is not very common to have nested array structures because there clear dedicated structure.
An example of a csv-file could look like this:
\begin{python}
first name, birth date, nationality, married, nobel prices
Carol, 07/11/1867, poland/france, true, 2
\end{python}
Despite the lack of formal rules this format is very wide spread. \cite{csv_format}
\subsection{JSON}
JSON(JavaScript Object Notation) is used to transmit structured data. It is mainly applied in web applications that are using JavaScript, Ajax or WebSockets. It is defined by the standards published by ECMA-404 and RFC 8259 who have equivalent definitions.
It supports the formats "zero value", "bool", "number", "string","Array" and "Objects". Arrays do start and end with square brackets \[\] and objects are marked with curly braces \{\}.
It does not support certain values used in Java such as "NaN" or "Infinity" which will be convertet to "null". Also there is no dedicated format for dates, they will be converted to a string representation.
The structure of JSON also allows nasted arrays. \cite{json_format}
An example could look like this:
\begin{python}
{
"first name": "Carol",
"birth date": "07/11/1867",
"nationality": {
"Place1": "poland",
"Place2": "france"
}
"married": true
"# nobel prices": 2
}
\end{python}
\subsection{XML}
XML (Extensible Markup Language) is a language to represent tree structured data. It is readable by humans as well as by machines. The original version was published in 1998, but the newest version is released in 2008.
Examples of XML-languages are RSS, MathML, GraphML, XHTML, XAML, SVG or GPX.
A characteristic of the document type is that all information is structured by opening and closing tags, that are marked by angle brackets <>. The information in between the tags seems to have no specific format requirements (??). \cite{xml_format}
An example could look like this:
\begin{python}
<person>
<first name>Carol</first name>
<birth date>07/11/1867</birth date>
<nationality>
<Place1>poland</Place1>
<Place2>france</Place2>
</nationality>
<married>true</married>
<nobel prices>2</nobel prices>
</person>
\end{python}
\subsection{ARVO}
Arvo is simply said a JSON structure with an added schema. This means it specifies the type of each entry. Additionally it stores the data serialized, which requires less memory space. I.e there is no need to store the data-label for each entry, as it is defined in the schema. It also supports changes in the schema. \cite{arvo_format}
An example of the schema and an created entry could look like this:
\begin{python}
{ "namespace": "example.avro",
"type": "record",
"name": "Person",
"fields": [
{"name": "first name", "type": "string"},
{"name": "birth date", "type": "string"},
{ "name": "nationality",
"type": "array",
"items": {
"namespace": "example.avro",
"type": "record",
"fields": [
{"name": "Place1", "type": "string"}
{"name": "Place2", "type": "string"}]
}
}
{"name": "# nobel prices", "type": "int"},
{"name": "married", "type": "bool"}]
}
# read the schema
schema = avro.schema.parse(open("user.avsc", "rb").read())
writer = DataFileWriter(open("users.avro", "wb"),
DatumWriter(), schema)
writer.append({
"first name": "Carol",
"birth date": "07/11/1867",
"Place1": "poland",
"Place2": "france",
"married": true,
"# nobel prices": 2,
})
writer.close()
\end{python}
\subsection{Parquet}
Similar to ARVO Paraquet provides a schema for the datastructure. This means that all types are well defined. It is a very memory efficient structure, as it can use different encoding algorithm for each column. This means a column containing only integers and one containing strings can be encoded both optimal to represent their values.
In contrast to ARVO the dataformat seems to be more related to SQL databases and simple csv tables, and is not optimized to contain nested data, although they can be stored if necessary. \cite{parquet_format}
\subsection{Protobuf}
Protobuf, brief for protocol buffer, supports a schema and is thus fully typed. The schema has additionally to data formats like ARVO like the keywords "required", "optional" or "repeated". It also gives the possibility to access a field with a tag instead of the keyword, which is indicated after the "=" sign.
The format also supports nested formats. This can be done either by defining a message inside a message, or by using a type defined in another message. \cite{protobuf_format}
A schema could look like the following:
\begin{python}
message Person {
required string "first name" = 1;
optional string "birth date" = 2;
message nationality {
repeated string "Place" = 3;
}
optional bool "married" = 4;
optional int32 "# nobel prices" = 5;
}
\end{python}
\subsection{CDM}
\label{sec:CDM}
CDM, short for common data model, aims to uniform data structures across multiple applications. Across multiple platforms there are very often common data base entries, such as "username". However they used to be saved separately in each database, which makes it prone to incompatibility. This can be because of different labels, or simply because the user miss-spelled the entry in one data-base.
In CDM databases are connected with each other and elements are not defined in multible locations. Further on there are standards developed how a database should be structured. There are multiple examples of CDM approaches, microsoft \cite{cdm_format_microsoft} is using it in their products and provides services to implement it on your own tools, ISO/TC211 is a specific example for the domain geographic information \cite{cdm_format_geo} or INSPIRE is a EU-project \cite{cdm_format_inspire} that aims to harmonize data across countries.
c4science · Help