Page Menu
Home
c4science
Search
Configure Global Search
Log In
Files
F64632246
diffusion.slurm
No One
Temporary
Actions
Download File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Subscribers
None
File Metadata
Details
File Info
Storage
Attached
Created
Tue, May 28, 07:36
Size
985 B
Mime Type
text/x-shellscript
Expires
Thu, May 30, 07:36 (1 d, 23 h)
Engine
blob
Format
Raw Data
Handle
17935868
Attached To
R1979 Scitas Spark Cluster Mode Service
diffusion.slurm
View Options
#!/bin/bash
# author: gilles foureste (EPFL)
#
#SBATCH --nodes=2
# ntasks per node MUST be one, because multiple slaves per work doesn't
# work well with slurm + spark in this script (they would need increasing
# ports among other things)
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=24
#SBATCH --mem=8192
# Beware! $HOME will not be expanded and invalid paths will result Slurm jobs
# hanging indefinitely with status CG (completing) when calling scancel!
##SBATCH --time=96:00:00
#SBATCH --partition=scitas
#SBATCH --qos=scitas
#set -x
#
echo "---- starting $0 on $HOSTNAME"
echo
#
MASTER_NODE=""
start-spark.sh
echo "configuration done..."
set -x
#
#echo $MASTER_NODE
#echo $MASTER
#
#MASTER_IP=$(cat ./sparklogs_${SLURM_JOBID}/spark_master)
MASTER_IP=$(cat ${SLURM_JOBID}_spark_master)
#
echo $MASTER_IP
time time spark-submit \
--executor-memory 5G \
--master $MASTER_IP \
./diffusion.py
#
stop-spark.sh
#scancel -u foureste
#spark-submit --master $MASTER ./pi.py
Event Timeline
Log In to Comment