julian.kunkel
/
mistral-io-datasets
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Weiter

This commit is contained in:
Julian M. Kunkel 2020-12-04 18:45:20 +00:00
parent 94e13ed4f3
commit 7db55f1279
1 changed files with 7 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
paper/main.tex
View File

@ -608,15 +608,14 @@ However, we believe that they are still well suited for a visual inspection and
This job represents post-processing (CMORization) which is a typical step. This job represents post-processing (CMORization) which is a typical step.
It is executed for different simulations and variables across timesteps. It is executed for different simulations and variables across timesteps.
The job name of Job-S suggests that is applied to the control variable. The job name suggests that is applied to the control variable.
In the metadata, we found 22,580 jobs with “cmor” in the name of which 367 jobs mention “control”. In the metadata, we found 22,580 jobs with “cmor” in the name of which 367 jobs mention “control”.
The bin and KS algorithms identify one job which name doesn't include “cmor”, The B and KS algorithms identify one job which name doesn't include “cmor”,
All other algorithms identify only “cmor” jobs and 26-38 of these jobs are applied to “control” (see \Cref{tbl:control-jobs}) -- only the KS algorithm doesn't identify any job with control. All other algorithms identify only “cmor” jobs and 26-38 of these jobs are applied to “control” (see \Cref{tbl:control-jobs}) -- only the KS algorithm doesn't identify any job with control.
A selection of job timelines is given in \Cref{fig:job-S-hex-lev}; all of these jobs are jobs on control variables. A selection of job timelines on control variables is given in \Cref{fig:job-S-hex-lev}.
The single non-cmor job and a high-ranked non-control cmor job is shown in \Cref{fig:job-S-bin-agg}. The single non-cmor job and a high-ranked non-control cmor job is shown in \Cref{fig:job-S-bin-agg}.
While we cannot visually see much differences between these two jobs compared to the cmor job processing the control variables, the algorithms indicate that jobs processing the control variables must be more similar as they appear much more frequently in the Top\,100 jobs than in all jobs labeled with “cmor”. While we cannot visually see much differences between these two jobs compared to the the control job, the algorithms indicate that jobs processing the control variables are more similar as they are more frequent in the Top\,100 jobs.
For Job-S, we found that all algorithms work well and, therefore, omit further timelines. For Job-S, we found that all algorithms work well and, therefore, omit further timelines.
\begin{table}[bt] \begin{table}[bt]
@ -734,11 +733,11 @@ The number of unique names is 19, 38, 49, and 51 for B-aggzero, Q-phases, Q-nati
The jobs that are similar according to the B algorithms (see \Cref{fig:job-M-bin-aggzero}) differ from our expectations. The jobs that are similar according to the B algorithms (see \Cref{fig:job-M-bin-aggzero}) differ from our expectations.
The other algorithms like Q-lev (\Cref{fig:job-M-hex-lev}) and Q-native (\Cref{fig:job-M-hex-native}) seem to work as intended: The other algorithms like Q-lev (\Cref{fig:job-M-hex-lev}) and Q-native (\Cref{fig:job-M-hex-native}) seem to work as intended:
While jobs exhibit short bursts of other active metrics even for low similarity we can eyeball a relevant similarity. While jobs exhibit short bursts of other active metrics even for low similarity, we can eyeball a relevant similarity.
The KS algorithm working on the histograms ranks the jobs correctly on the similarity of their histograms. The KS algorithm working on the histograms ranks the jobs correctly on the similarity of their histograms.
However, as it does not deal with the length of the jobs, it may identify jobs of very different length. However, as it does not deal with the length of the jobs, it may identify jobs of very different length.
In \Cref{fig:job-M-ks}, we see the 3rd ranked job, which profile is indeed quite similar but the time series differs but it is just running for 10min (1 segment) on 10\,nodes. In \Cref{fig:job-M-ks}, we see the 3rd ranked job, which profile is indeed quite similar but the time series differs but it is just running for 10min (1 segment) on 10\,nodes.
Remember, for the KS algorithm, we concatenate the metrics of all nodes together instead of averaging it in order to explore if node-specific information helps to draw further information about similarity. Remember, for the KS algorithm, we concatenate the metrics of all nodes together instead of averaging it in order to explore if node-specific information helps the similarity.
\begin{figure}[bt] \begin{figure}[bt]
\begin{subfigure}{0.5\textwidth} \begin{subfigure}{0.5\textwidth}
@ -988,7 +987,7 @@ As expected, the histograms mimics the profile of the reference job, and thus, t
\section{Conclusion} \section{Conclusion}
\label{sec:summary} \label{sec:summary}
In this article, we conducted a study to identify similar jobs based on timelines of nine I/O statistics. We conducted a study to identify similar jobs based on timelines of nine I/O statistics.
Therefore, we applied six different algorithmic strategies developed before and included this time as well a distance metric based on the Kolmogorov-Smirnov-Test. Therefore, we applied six different algorithmic strategies developed before and included this time as well a distance metric based on the Kolmogorov-Smirnov-Test.
The quantitative analysis shows that a diverse set of results can be found and that only a tiny subset of the 500k jobs is very similar to each of the three reference jobs. The quantitative analysis shows that a diverse set of results can be found and that only a tiny subset of the 500k jobs is very similar to each of the three reference jobs.
For the small post-processing job, which is executed many times, all algorithms produce suitable results. For the small post-processing job, which is executed many times, all algorithms produce suitable results.