diff --git a/README.md b/README.md index c866e84..b5cf661 100644 --- a/README.md +++ b/README.md @@ -1,364 +1,363 @@ Genocrunch ========== A web-based platform for mining metagenomic data **Official web server:** ## Rights - **Copyright:** All rights reserved. ECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE, Switzerland, Laboratory of Intestinal Immunology, 2016-2018 - **License:** GNU AGPL 3 (See LICENSE.txt for details) - **Authors:** AR Rapin, FPA David, C Pattaroni, J Rougemont, BJ Marsland and NL Harris ## Resources - **Git clone URL:** - **Documentation:** - **License:** - **Dockerfile:** ## Framework Genocrunch uses the ruby on Rails framework with a PostgreSQL database. ## Supported platforms - **Linux** (tested on Ubuntu 16.04 LTS and CentOS 7) - **macOS** (tested on 10.12 Sierra) ## Supported web browsers - **Mozilla Firefox** (Mobile versions are not supported) ## Requirements - **Ruby version 2.3.1** - **Rails version 5.0.0** - **Python version >=2.7.0 <3.0.0** - **R version >3.2.2** ## Installation (Debian Linux and macOS) ### Ruby, Rails and PostgreSQL **Debian Linux** Uninstall possible pre-installed versions of ruby: ``` $ sudo apt-get purge ruby ``` (Re-)install ruby and Rails using rbenv (see [here](https://www.digitalocean.com/community/tutorials/how-to-install-ruby-on-rails-with-rbenv-on-ubuntu-16-04#prerequisites)). Install PostgreSQL and start PostgreSQL server: ``` $ sudo apt-get install postgresql postgresql-contrib libpq-dev $ sudo service postgresql start ``` **macOS** Install ruby and Rails using homebrew and rbenv (see [here](https://www.gorails.com/setup/osx/10.12-sierra)). Install PostgreSQL and start PostgreSQL server with homebrew: ``` $ brew install postgresql $ brew services start postgresql ``` ### Python **Debian Linux** ``` $ sudo apt-get install build-essential python-dev python-pip $ pip install numpy ``` Check that python version is between 2.7.0 and 3.0.0: ``` $ python -V ``` **macOS** If not done yet, install Homebrew: ``` $ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" ``` Install python 2.7 with Homebrew: ``` $ brew install python ``` Check that python version is between 2.7.0 and 3.0.0: ``` $ python -V ``` ### R **Debian Linux** ``` $ sudo apt-get install r-base-core libnlopt-dev libcurl4-openssl-dev libxml2 libxml2-dev ``` Open the R environment and check that R version is above 3.2.2: ``` $ R > R.version.string ``` **macOS** Update XQuartz and Xcode if needed. Download the R binary from CRAN at . Click on the downloaded .pkg file and follow the instructions. ### R dependencies Install required R packages from CRAN and bioconductor: ``` $ sudo R > install.packages(c("ineq", "rjson", "fpc", "multcomp", "FactoMineR", "colorspace", "vegan", "optparse", "gplots", "fossil", "coin", "SNFtool", "devtools")) > source("https://bioconductor.org/biocLite.R") > biocLite("sva") > library(devtools) > install_github("igraph/rigraph") > q() ``` ### Genocrunch web application Create a new rails project and add the Genocrunch files: ``` $ rails new genocrunch -d postgresql -B $ git clone https://git@c4science.ch/source/genocrunch-2.1.git /tmp/genocrunch $ rsync -r /tmp/genocrunch/ /genocrunch $ sudo rm -r /tmp/genocrunch $ cd genocrunch \ && cp gitignore.keep .gitignore \ && cp config/config.yml.keep config/config.yml \ && cp config/database.yml.keep config/database.yml \ && cp config/initializers/devise.rb.keep config/initializers/devise.rb \ && cp config/environments/development.rb.keep config/environments/development.rb \ && cp db/seeds.rb.keep db/seeds.rb ``` Run the `install.sh` script (this is not essential for the application): ``` $ chmod 755 install.sh $ ./install.sh $ source .bashrc # or .bash_profile for macOS ``` The Genocrunch web app will store data files in `users/`. To store data in another location, use a simlink: ``` $ rmdir users && ln -s /path/to/your/custom/storage/location users ``` ### Ruby libraries (gems) Use the Gemefile to install required gems: ``` $ bundle install ``` ### Set application configuration variables Set the application configuration variables in the `config/config.yml` file to fit the current installation: ``` #config/config.yml development: # Genocrunch main directory data_dir: /path/to/genocrunch # Additional link(s) that should be included in the Infos menu of the topbar info_links: [{name: 'link_name', href: 'link_url', target: '_blank'}] # Webmaster email webmaster_email: 'webmaster_email' # Send a validation link to user email to confirm registration? user_confirmable: false production: data_dir: /path/to/genocrunch info_links: [{name: 'link_name', href: 'link_url', target: '_blank'}] webmaster_email: 'webmaster_email' user_confirmable: false ``` ### Set genocrunch emails Set the email details that will be used by Genocrunch to send information such as registration confirmation link or password recovery link to users. The following example would set Genocrunch to use an hypothetical gmail address (`app_email@gmail.com`) in development. ``` #config/initializers/devise.rb Devise.setup do |config| ... config.mailer_sender = "app_email@gmail.com" ... ``` ``` #config/environments/development.rb Rails.application.configure do ... config.action_mailer.default_url_options = { :host => 'localhost:3000' } config.action_mailer.smtp_settings = { :address => "smtp.gmail.com", :port => 587, :domain => "mail.google.com", :user_name => "app_email@gmail.com", :password => "app_email_password", :authentication => :plain, :enable_starttls_auto => true } ... ``` ### Setup the PostgreSQL server Create a new role and a new database (you can create different users and databases for development, test and/or production): ``` $ sudo su postgres # for macOS, replace postgres by _postgres $ psql postgres=# CREATE ROLE myusername WITH LOGIN PASSWORD 'mypassword'; postgres=# CREATE DATABASE my_application_db_name OWNER myusername; postgres=# \q $ exit ``` Set the `config/database.yml` file: In development, test and/or production sections, set the `database`, `username` and `password` to fit the corresponding PostgreSQL database. Also make sure to uncomment `host: localhost`: ``` #config/database.yml ... database: my_application_db_name ... username: myusername ... password: mypassword ... host: localhost ... ``` ### Initialize the database Two default users will be created: guest and admin. The guest user is required to try the application without registering/signing-in. The admin user is optional. Seting the guest and admin passwords and emails can be done prior to seeding the database, by editing the `db/seeds.rb` file: ``` #db/seeds.rb User.create!([{username: 'guest', role: 'guest', email: 'guest@guestmailbox.com', # <- HERE confirmed_at: '2017-01-01 00:00:00.000000', password: 'guest_account_password'}, # <- HERE {username: 'admin', role: 'admin', email: 'admin@adminmailbox.com', # <- HERE confirmed_at: '2017-01-01 00:00:00.000000', password: 'admin_account_password'}]) # <- AND THERE ... ``` Run the following commands to create and seed the database. Caution: This will erase previous database tables. Use it for installation, not update. For updates, use migrations or SQL queries. ``` $ rake db:schema:load $ rake db:seed ``` ### Run the Rails server - * Development mode ``` $ rails server ``` You can now access the application in your browser at on your machine and `your.ip.address:3000` on your network. **By default, the server runs in development mode.** ### Start workers * Prefered way: ``` $ RAILS_ENV=development bin/delayed_job -n 2 start ``` OR ``` $ RAILS_ENV=development bin/delayed_job -n 2 restart ``` * Alternative way (not recommanded): ``` $ rake jobs:work ``` You can now create new jobs (run analysis). Read the documentation () for details. ### Create a new version Versions of installed R packages can be referenced in the version page (). For this, run the `get_version.py` script: ``` $ get_version.py ``` This will create a .json file in the genocrunch main directory with a name looking like `version_2017-12-18_18:03:08.898906.json`. Sign in as admin and navigate to Infos>Versions Click on the **New Version** button and fill the form. In the JSON field, copy the json string contained in the .json file previously created using the `get_version.py` script. Finally, click on the **Create Version** button. ### Terms of service Terms of service can be edited in `public/app/TERMS_OF_SERVICE.txt`. ## Usage See **Infos>Doc** in the application web page (). ## Running on Docker See [here](https://c4science.ch/source/genocrunch_docker). diff --git a/app/assets/javascripts/form.js b/app/assets/javascripts/form.js index c8220d0..16efeaf 100644 --- a/app/assets/javascripts/form.js +++ b/app/assets/javascripts/form.js @@ -1,159 +1,200 @@ function update_bin_levels(col_name, val, url){ val = (val) ? val : [1] var ori_filename = $("#p2_primary_dataset").val(); getColumn("bin_levels", $("#p_primary_dataset")[0].files[0], col_name, val, url); } function populate_multiselect(id, new_data, value, url){ $('#p_'+id).multiselect('enable') .multiselect('dataprovider', new_data) .multiselect('select', value) .multiselect("refresh"); $('#p_'+id+'-container').removeClass("hidden"); $('#p_'+id+'-placeholder').addClass("hidden"); if (id == 'category_column'){ var val = $("#default_bin_levels").val() var new_url = (url) ? ($("#url_read_file_column").val() + "?file_key=primary_dataset") : null; update_bin_levels($("#p_category_column").val(), JSON.parse(val), new_url); } } function getColumn(id, file, col_name, value, url) { var new_data = []; if (!url){ var reader = new FileReader; reader.onload = function (event) { var lines = event.target.result.split(/\r\n|\r|\n/g); var i=0; for (var i = 0; i < (lines.length-1); i++) { if (lines[i].charAt(0) != '#') break; } i = (i > 1) ? i-1 : 0; var header_els = lines[i].split("\t") var pos_col = 0 for (var k = 0; k max_val){ max_val = n } } if (max_val > 0){ var new_data=[]; for (var j=1 ; j < max_val+1; j++) { new_data.push({label: j, value: j}); } populate_multiselect(id, new_data, value, url) } }; // console.log(file); reader.readAsText(file); }else{ $.ajax({ url: url + "&col_name=" + col_name, type: "get", dataType: "html", beforeSend: function(){ }, success: function(new_data){ populate_multiselect(id, JSON.parse(new_data), value, url) }, error: function(e){ } }); } }; +function setCategoryColumn(id, file, value = 0, add_blank = null, url = null) { + + var new_data = []; + + if (!url){ + + var reader = new FileReader; + reader.onload = function (event) { + var lines = event.target.result.split(/\r\n|\r|\n/g); + + var i = 0; + for (i = 0; i < (lines.length-1); i++) { + if (lines[i].charAt(0) != '#') + break; + } + i = (i > 0) ? i-1 : 0; + + var content = lines[i].split("\t") + + new_data = [{label:([' ', ''].indexOf(content[0]) == -1) ? 'first column ('+content[0]+')': 'first column', value:content[0]}, + {label:([' ', ''].indexOf(content[content.length-1]) == -1) ? 'last column ('+content[content.length-1]+')': 'last column', value:content[content.length-1]}]; + populate_multiselect(id, new_data, value, url) + }; + reader.readAsText(file); + }else{ + $.ajax({ + url: url + ((add_blank == null) ? '' : '&add_blank=1'), + type: "get", + dataType: "html", + beforeSend: function(){ + }, + success: function(new_data){ + populate_multiselect(id, JSON.parse(new_data), value, url) + }, + error: function(e){ + } + }); + + } +}; + function setSelectFromFileRow(id, file, value = 0, add_blank = null, url = null) { var new_data = []; if (!url){ var reader = new FileReader; reader.onload = function (event) { var lines = event.target.result.split(/\r\n|\r|\n/g); var i = 0; for (i = 0; i < (lines.length-1); i++) { if (lines[i].charAt(0) != '#') break; } i = (i > 0) ? i-1 : 0; var content = lines[i].split("\t") if (add_blank) new_data.push({label:"", value:""}); if (content.length > 0){ for (var i in content) { new_data.push({label:content[i], value:content[i]}); }; } populate_multiselect(id, new_data, value, url) }; reader.readAsText(file); }else{ $.ajax({ url: url + ((add_blank == null) ? '' : '&add_blank=1'), type: "get", dataType: "html", beforeSend: function(){ }, success: function(new_data){ populate_multiselect(id, JSON.parse(new_data), value, url) }, error: function(e){ } }); } }; // Transform a select input into a nice multiselect field function setMultiselect(id, nonSelectedText = 'Select', where){ $(id).multiselect({ includeSelectAllOption: false, // Would append a value currently not supported by genocrunch_console nonSelectedText: nonSelectedText, numberDisplayed: 1, maxHeight: 150, buttonClass: 'btn btn-secondary', templates: { li: '
    • ', } }); $(id + ".form-control").addClass('hidden'); } function fold_section(fold_bar_class) { $(fold_bar_class).click(function(){ var e = $(this).parent().parent().parent().parent().parent().children().filter(".card-block").first(); //(".field_group_content").first(); var angle = (e.hasClass('hidden')) ? 180 : 0; $(this).filter("i.fa-chevron-down").css({transform:'rotate(' + angle + 'deg)'}); e.toggleClass('hidden'); }); }; diff --git a/app/assets/stylesheets/backgrounds.css.scss b/app/assets/stylesheets/backgrounds.css.scss index 044f1cc..74fc0b9 100644 --- a/app/assets/stylesheets/backgrounds.css.scss +++ b/app/assets/stylesheets/backgrounds.css.scss @@ -1,15 +1,8 @@ -.bg-welcome, -.bg-error { +.bg-welcome { min-height:100vh; height:auto; background-size: cover; background-repeat: no-repeat; -} - -.bg-welcome { background-image: asset-url("bg-welcome.jpg") } -.bg-error { - background-image: asset-url("bg-error.jpg") -} diff --git a/app/assets/stylesheets/buttons.css.scss b/app/assets/stylesheets/buttons.css.scss index 474b9bb..d47f7b1 100644 --- a/app/assets/stylesheets/buttons.css.scss +++ b/app/assets/stylesheets/buttons.css.scss @@ -1,51 +1,10 @@ .custom-btn-container { margin:0; padding:0; } .custom-btn-container>.btn { color:inherit; margin:inherit; } -.btn-topbar { - margin:3px; - float:left; -} - -.form-buttons-container { - position: fixed; - width:150px; - top:auto; - left:auto; - margin-left:-180px; -} - -.form-button { - width:100%; - font-size: 18px; - line-height: 24px; - -webkit-box-shadow: -1px 1px 2px 1px #888888; - -moz-box-shadow: -1px 1px 2px 1px #888888; - box-shadow: -1px 1px 2px 1px #888888; -} - -.figure-button { - margin: 3px; - display: inline-block; -} - - -.select-button { - color: #fff; - text-shadow: 1px 1px 0 #fff; - text-shadow: -1px -1px 0 rgba(0,0,0,0.3); - border-color: #B33030; - border-bottom-color: #A02A2A; - background-color: #CF3C3C; - background-image: -webkit-gradient(linear, 0 0, 0 100%, from(#E45D5D), to(#B54C4C)); - background-image: -moz-linear-gradient(#E45D5D, #B54C4C); - background-image: -ms-linear-gradient(#E45D5D, #B54C4C); - background-image: -o-linear-gradient(#E45D5D, #B54C4C); - background-image: linear-gradient(#E45D5D, #B54C4C); -} diff --git a/app/assets/stylesheets/errors.css.scss b/app/assets/stylesheets/errors.css.scss index 3d20d96..cf5d6be 100644 --- a/app/assets/stylesheets/errors.css.scss +++ b/app/assets/stylesheets/errors.css.scss @@ -1,72 +1,30 @@ -.error-page { - position: static; - margin: 0; - margin-right: auto; - margin-left: auto; - display: table; - color: #fff; - text-align:center; - max-width:75%; - width:auto; - height:100%; - transform:translate(0, 50%); - - .error-text { - width:auto; - height:auto; - transform:translate(0, -50%); - text-shadow: 3px 3px 3px #333; - text-align:center; - } - - h1 { - font-size: 40px; - line-height: 90px; - } - - p { - font-size: 20px; - line-height: 20px; - } - -} - #notice { color: #fff; text-align:center; } - -.field_with_errors { - height:100%; - padding: 2px; - box-shadow: #ff3385 0px 0px 2px, inset #ff3385 0px 0px 2px; -} - #error_explanation { display: table; width: 100%; border: 2px solid #ff3385; padding: 7px; padding-bottom: 0; background-color: #eee; box-sizing: border-box; -moz-box-sizing: border-box; -webkit-box-sizing: border-box; - - h2 { - text-align: left; - font-weight: bold; - padding: 5px 5px 5px 15px; - font-size: 16px; - margin: -7px; - margin-bottom: 0px; - background-color: #ff3385; - color: #fff; - } - - ul li { - font-size: 16px; - list-style: circle; - } - } +#error_explanation h2 { + text-align: left; + font-weight: bold; + padding: 5px 5px 5px 15px; + font-size: 16px; + margin: -7px; + margin-bottom: 0px; + background-color: #ff3385; + color: #fff; +} +#error_explanation ul>li { + font-size: 16px; + list-style: circle; +} + diff --git a/app/assets/stylesheets/figures.css.scss b/app/assets/stylesheets/figures.css.scss index 2728bd3..a308ef3 100644 --- a/app/assets/stylesheets/figures.css.scss +++ b/app/assets/stylesheets/figures.css.scss @@ -1,404 +1,101 @@ +.fig-container { + height:auto; + min-height:600px; + max-height:90vh; + overflow:auto; +} + +.fig-container .svg-figure { + display:table; + margin:auto; +} + +#fig-legend { + height:auto; + max-height:90vh; + overflow:auto; +} + #fig-description-btn { color:#333; cursor: pointer; position:relative; display:inline-block; &:hover { color:#014c8c; } - &:hover:before { - border-bottom:.5rem solid #014c8c; - } - &:hover:after { - border-top:.5rem solid #014c8c; - } -} - -#fig-description-btn:before, -#fig-description-btn:after { - position:absolute; - content:""; - width:0; - height:0; - bottom:.75rem; - left:15px; -} - -.range-displayed-value:before { - position:relative; - display:table; - margin-left:auto; - margin-right:auto; - content:''; - width:0; - height:0; - margin-bottom:-0.25rem; - top:0; - border-top:.25rem solid #ccc; - border-left:.25rem solid transparent; - border-right:.25rem solid transparent; } - -.range-displayed-value:after { - position:relative; - display:table; - margin-left:auto; - margin-right:auto; - content:''; +#fig-description-btn:before { + position:absolute; + display:block; + content:""; width:0; height:0; - margin-top:-0.25rem; - top:0; - border-bottom:.25rem solid #ccc; - border-left:.25rem solid transparent; - border-right:.25rem solid transparent; -} + bottom:.75rem; + left:15px; + border-top:.5rem solid #333; + border-left:.5rem solid transparent; + border-right:.5rem solid transparent; -#fig-description-btn:before { - display:none; - border-bottom:.5rem solid #333; - border-top:0; - border-left:.5rem solid transparent; - border-right:.5rem solid transparent; -} - -#fig-description-btn:after { - border-bottom:0; - border-top:.5rem solid #333; - border-left:.5rem solid transparent; - border-right:.5rem solid transparent; + -ms-transform: rotate(-180deg); + transition: transform 0.3s linear 0.3s; + -moz-transition: transform 0.3s linear 0.3s; + -webkit-transition: transform 0.3s linear 0.3s; } #fig-description-btn.active:before { - display:block; -} -#fig-description-btn.active:after { - display:none; + transform: rotate(-180deg); + -moz-transform: rotate(-180deg); + -webkit-transform: rotate(-180deg); } + #fig-description { position:relative; display:inline-block; height:auto; max-height:100vh; overflow:auto; transition: max-height 1s ease; -webkit-transition: max-height15s ease; -moz-transition: max-height 1s ease; -ms-transition: max-height 1s ease; } #fig-description.inactive { max-height:1rem; overflow:hidden; margin-bottom:10px; } #fig-description.inactive>ul:after { - position:absolute; - bottom:0; - left:0; - content:""; - width:100%; - height:100%; - background:linear-gradient(to bottom, rgba(255,255,255,0), rgba(255,255,255,1)); -} - -.network-well { - border:1px solid rgba(0,0,0,.125); -} - - -.figure-layout { - display: -webkit-box; - display: -webkit-flex; - display: -ms-flexbox; - display: flex; - flex-wrap: wrap; - min-height:100vh; -} - -.fig-container { - min-height:600px; - max-height:90vh; - height:auto; - - #fig { - width:100%; - height:100%; - overflow:auto; - } - - .svg-figure { - display:table; - margin:auto; - } -} - -.btn-container { - margin-top:10px; - margin-bottom:10px; -} - -.btn-sep { - border-top:1px solid #666; - display:inline-block; + position:absolute; + bottom:0; + left:0; + content:""; width:100%; - margin-bottom:5px -} - -.fig-legend-container { - max-height:90vh; - height:auto; - padding:10px; - - #fig-legend { - width:100%; - height:100%; - overflow:auto; - } - - .columns-1 { - columns: 1; - -webkit-columns: 1; - -moz-columns: 1; - } - - .columns-2 { - columns: 2; - -webkit-columns: 2; - -moz-columns: 2; - } - - .columns-4 { - columns: 4; - -webkit-columns: 4; - -moz-columns: 4; - } -} - -// BAR FOR FIGURE MODIFICATION/SEARCH TOOLS -.figure-btn-container { - padding-top:7px; - padding-bottom:7px; - background-color:#333; - color:#fff; - display: flex; - flex-direction: column; - overflow:visible; - - .export .dropdown>ul>li { - cursor:pointer; - } - - .figure-btn { - border-top:1px solid #666; - display:inline-block; - - .search-input{ - margin-top:5px; - - i { - color:#666; - } - } - - .icon-sim-link { - float:left; - margin-right:5px; - } - - .icon-dissim-link { - float:left; - margin-right:5px; - } - } - - -} - -// TOP AND BOTTOM SCROLL BARS -.scrollbar-top-container, .double-scrolled-container { - overflow-x: auto; - overflow-y:hidden; -} - -.scrollbar-top-container { - height: 20px; -} -.double-scrolled-container { - height: 100%; -} - -.scrollbar-top { - height: 100%; -} - -.double-scrolled { height:100%; - overflow: visible; -} - -// THE FIGURE ITSELF (PLOTS, LEGEND, ETC...) -.figure-container { - background-color:#fff; - display: flex; - flex-direction: column; - overflow:hidden; - height:auto; - min-height:100vh; - - .figure-container-margin { - padding-bottom:25px; - margin-right:5px; - margin-left:-15px; - } - - .label-sm { - font-size:12px; - } - - .svg-figure { - display:table; - margin:auto; - - } - - .network-well { - padding:0; - } - - - // FIGURE LEGEND - .legend-container { - padding:5px; - padding-bottom:15px; - height:100%; - overflow:hidden; - overflow-y:auto; - overflow-x:auto; - - .legend-container-margin { - padding:10px; - } - - p { - margin-top:10px; - font-weight:bold; - } - - .sidebar-sub-title { - font-style: italic; - } - - .svg-legend { - padding:0; - padding-left:20px; - margin:5px; - height:100%; - } - - ul { - padding:5px; - &>li { - cursor:pointer; - padding:5px; - } - - .legend-no-interaction { - cursor:auto; - } - } - - .svg-legend-with-title { - margin-top:40px; - } - - ul>li>span { - width:100%; - display:inline; - white-space:nowrap; - - i { - margin-right:5px; - } - } - } + background:linear-gradient(to bottom, rgba(255,255,255,0), rgba(255,255,255,1)); } -// MOVING SIDEBAR FOR THE DESCRIPTION OF THE FIGURE -.sidebar { - position:absolute; - right:0px; - width:20px; - box-sizing: border-box; - -moz-box-sizing: border-box; - -webkit-box-sizing: border-box; - background-color:rgba(255, 0, 102, 0.9); - padding:2px; - -webkit-transition:width .3s ease-out, padding .3s ease; - -moz-transition:width .3s ease-out, padding .3s ease; - -ms-transition:width .3s ease-out, padding .3s ease; - transition:width .3s ease-out, padding .3s ease; - overflow:hidden; - border-left:4px solid #e6005c; - -webkit-box-shadow: -2px 0px 3px -2px #888888; - -moz-box-shadow: -2px 0px 3px -2px #888888; - box-shadow: -2px 0px 3px -2px #888888; - - .sidebar-icon { - transform:rotate(0deg); - color:#fff; - -webkit-transition:transform .3s linear .4s; - -moz-transition:transform .3s linear .4s; - -ms-transition:transform .3s linear .4s; - transition:transform .3s linear .4s; - } - - .description{ - visibility:hidden; - opacity:0; - -webkit-transition-delay:visibility .3s; - -moz-transition-delay:visibility .3s; - -ms-transition-delay:visibility .3s; - transition-delay:visibility .3s; - -webkit-transition:opacity .1s ease .3s; - -moz-transition:opacity .1s ease .3s; - -ms-transition:opacity .1s ease .3s; - transition:opacity .1s ease .3s; - word-break: normal; - text-align:justify; - h3 { - margin-top:4px; - color:#fff; - small { - color:#fff; - } - } - } -} - -// LOGS -.log-title { - text-align:center; - color:#fff; - margin-right: auto; - margin-left: auto; - display: table; - text-shadow: 3px 3px 3px #333; - - small { - color:#fff; - } -} - -.log-pre { - background-color: rgba(0, 0, 0, 0.4); - color:#fff; - width:75%; +.range-displayed-value:before, +.range-displayed-value:after { + position:relative; + display:table; margin-left:auto; margin-right:auto; + content:''; + width:0; + height:0; + margin-bottom:-0.25rem; + top:0; + border-left:.25rem solid transparent; + border-right:.25rem solid transparent; +} +.range-displayed-value:before { + border-top:.25rem solid #ccc; +} +.range-displayed-value:after { + border-bottom:.25rem solid #ccc; } - - diff --git a/app/assets/stylesheets/general.css.scss b/app/assets/stylesheets/general.css.scss index 3727f8d..3479762 100644 --- a/app/assets/stylesheets/general.css.scss +++ b/app/assets/stylesheets/general.css.scss @@ -1,343 +1,334 @@ -#main{ - margin:5px; - height:100vh; -} - .main-container { - min-height:100vh; height:auto; - padding-top:3px; + min-height:100vh; + padding-top:1rem; } .doc-table tr td, .doc-table tr th, .version-table tr td { text-align:left } - .border-left { - border-left:2px solid grey; + border-left:1px solid #ccc; } .border-right { - border-right:2px solid grey; + border-right:1px solid #ccc; +} +.border-top { + border-top:1px solid #ccc; +} +.border-bottom { + border-top:1px solid #ccc; } -.navbar-bottom { - display:table; - width:100%; - margin: 0; +.navbar.navbar-top { + -moz-box-shadow: 0 2px 2px #808080; + -webkit-box-shadow: 0 2px 2px #808080; + box-shadow: 0 2px 2px #808080; } -.navbar-bottom > li { - float:none; - display:table-cell; - text-align:center; + +.navbar.navbar-bottom { + border-top:1px solid #ccc; } .side-index-container { padding:0; z-index:0; - - .side-index { - height:100vh; - overflow:auto; - width:100%; - - .side-index-list { - margin-bottom:50vh; - padding-left:5px; - padding-right:5px; - } - } +} +.side-index-container .side-index { + height:100vh; + overflow:auto; + width:100%; +} +.side-index-container .side-index-list { + margin-bottom:50vh; + padding-left:5px; + padding-right:5px; + padding-top:1rem; } .topbar-padding { - padding-top:56px + padding-top:56px; } .topbar-margin { - margin-top:56px + margin-top:56px; } .dropdown-item { cursor:pointer; } -.step-card{ - margin-bottom:5px -} .float-right { float:right } .align-center { text-align:center; } .align-justify { text-align:justify; } .align-left { text-align:left; } - .align-right { text-align:right; } + ul.no-bullets li { list-style-type:none; } -.border-top { - border-top:1px solid #ccc; -} - .full-width { width:100%; } .full-height { height:100%; } .align-items-center { align-items:center; } .hidden { display:none; } .video-wrapper { position: relative; padding-bottom: 56.25%; /* 16:9 */ padding-top: 25px; height: 0; iframe { position: absolute; top: 0; left: 0; width: 100%; height: 100%; } } #popup_window { position: absolute; z-index:100000; padding:15px; display: none; background: #ccc; border: 1px solid; } #popup_window_close { float:right; font-size:20px; margin-top:-19px; margin-right:-10px; cursor:pointer; } .tip-window-container { position:relative; display:block; width:0; height:0; } .tip-window { position: absolute; display:block; z-index:1; margin: 15px 0 0 0; padding:10px; background: #e6e6e6; border: 1px solid #333; color:#333; width:300px; text-align:left; white-space:normal; } .tip-window:before, .tip-window:after { position:absolute; content:""; width:0; height:0; } .tip-window.right:before, .tip-window.right:after { right:10px; } .tip-window.left:before, .tip-window.left:after { left:10px; } .tip-window:before { top:-15px; border-bottom:15px solid #333; border-left:15px solid transparent; border-right:15px solid transparent; } .tip-window:after { top:-14px; z-index:1; border-bottom:15px solid #e6e6e6; border-left:15px solid transparent; border-right:15px solid transparent; } .tip-window-close { cursor:pointer; } .title_popup {white-space:nowrap;font-weight:bold} .infos { position:static; } .background-col-1 { background-color:#e6e6e6; } .background-col-2 { color:#fff; background-color:#999; } .row-padding-10 { padding-top:10vh; padding-bottom:10vh; } .row-padding-5 { padding-top:5vh; padding-bottom:5vh; } .row-padding-2 { padding-top:2vh; padding-bottom:2vh; } .row-padding-bottom-20 { padding-bottom:20vh; } .v-align-middle { height:100%; transform:translate(0, 35%); } .back-to-top { color:#999; text-decoration: none; &:hover, &:focus, &:active { color:#ccc; text-decoration: none; } } .content { position:static; display: table; margin-right: auto; margin-left: auto; margin-top: 60px; margin-bottom: 60px; .left { float:left; margin-right:20px; } .right { float:right; margin-left:20px; } .align-left { text-align:left; } .align-right { text-align:right; } .centered-div { width: 260px; line-height: 300px; } .centered-span { display: inline-block; vertical-align: middle; line-height: normal; } } a { color: #333; &.white { color: #fff; } &:link, &:visited { text-decoration: none; } &:hover { text-decoration: underline; &.button, &.tab, &.subtab, &.help-button, &.slider, &.slide-open, &.slide-close, &.test-button { text-decoration: none; } &.slide-open, &.slide-close { text-shadow: 1px 0px #333, -1px -0px #333, 0px -1px #333, 0px 1px #333; } } } h1 { font-weight: 500; font-size: 28px; line-height: 34px; } h2 { font-weight: 500; font-size: 24px; line-height: 30px; } h3 { font-weight: 500; font-size: 20px; line-height: 25px; } pre { font-family: verdana, arial, helvetica, sans-serif; background-color: #eee; padding: 10px; font-size: 12px; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word; } diff --git a/app/assets/stylesheets/show.css.scss b/app/assets/stylesheets/show.css.scss index 8d82bea..0b11e5d 100644 --- a/app/assets/stylesheets/show.css.scss +++ b/app/assets/stylesheets/show.css.scss @@ -1,57 +1,35 @@ -.menu { - width:100%; - margin-top:5px; - margin-bottom:5px; - border-color:#ccc; +.show-navbar { + background-color:#f7f7f9; } -.menu.selected>.btn:before, -.menu.selected>.btn:after { - position:absolute; - content:""; - width:0; - height:0; - top:0.5rem; -} -.menu.selected>.btn:before { - right:-5px; - border-bottom:5px solid transparent; - border-left:5px solid #ccc; - border-top:5px solid transparent; -} -.menu.selected>.btn:after { - right:-4px; - z-index:1; - border-bottom:5px solid transparent; - border-left:5px solid #e6e6e6; - border-top:5px solid transparent; +.menu.btn { + border-color:#ccc; + border-top:1px solid #ccc; } - -.menu.selected a { - position:relative; +.menu.btn.selected { background-color:#e6e6e6; color:inherit; - border-color:inherit; } - -.menu a { - width:100%; - border-color:inherit; +.menu.btn:focus, +.menu.btn-secondary:focus { + -moz-box-shadow:none; + -webkit-box-shadow:none; + box-shadow:none; } #job_header_right { position:absolute; top:0px; right:10px; } .job-description { max-height:100px; height:auto; overflow:auto; } .show-element { margin:10px; padding:10px; } diff --git a/app/controllers/jobs_controller.rb b/app/controllers/jobs_controller.rb index 8a79ea9..b079cf5 100644 --- a/app/controllers/jobs_controller.rb +++ b/app/controllers/jobs_controller.rb @@ -1,773 +1,773 @@ class JobsController < ApplicationController before_action :set_job, only: [:show, :edit, :update, :destroy, :serve, :view, :refresh, :clone] # before_action :authenticate_user!, only: [:show, :edit, :update, :destroy, :serve] def kill_job j tmp_dir = Pathname.new(APP_CONFIG[:data_dir]) + "users" + j.user_id.to_s + j.key main_pid = File.read(tmp_dir + ".pid") if File.exist?(tmp_dir + ".pid") # kill job if one already running existing_main_job = `ps -ef | grep #{j.key} | grep #{main_pid} | grep -v 'grep'` puts "Existing main job: " + existing_main_job.to_json # pids=[] if main_pid and !existing_main_job.empty? lines = `ps -ef | grep #{j.key} | grep -v 'grep'`.split("\n").select{|l| !l.empty?} pids = lines.map{|l| t= l.split(/\s+/); t[1]} pids.unshift(main_pid) if pids.size > 0 pids.each do |pid| cmd = "kill #{pid}" `#{cmd}` end end end # delete log.json log_json = tmp_dir + "output" + "log.json" logger.debug("DEBUG: " + log_json.to_s) File.delete log_json if File.exist? log_json end def clone #if current_user tmp_dir = Pathname.new(APP_CONFIG[:data_dir]) + "users" + @job.user_id.to_s + @job.key @new_job = nil if current_user @new_job = @job.dup session[:current_key] = create_key() @new_job.key = session[:current_key] else current_job = Job.where(:key => session[:current_key]).first if current_job ### kill current job kill_job(current_job) @new_job = current_job @new_job.update_attributes(:form_json => @job.form_json, :output_json => @job.output_json, :name => @job.name, :description => @job.description, :status => @job.status) else @new_job = @job.dup @new_job.key = session[:current_key] # @new_job.read_access = true # @new_job.write_access = true end end # kill_job(@job) ### clone @new_job.sandbox = (current_user) ? false : true @new_job.user_id = (current_user) ? current_user.id : 1 new_tmp_dir = Pathname.new(APP_CONFIG[:data_dir]) + "users" + @new_job.user_id.to_s Dir.mkdir(new_tmp_dir) if !File.exist? new_tmp_dir new_tmp_dir += @new_job.key if File.exist? new_tmp_dir FileUtils.rm_r new_tmp_dir Dir.mkdir new_tmp_dir end FileUtils.cp_r tmp_dir.to_s + "/.", new_tmp_dir ### rename cloned tar.gz FileUtils.mv (new_tmp_dir + (@job.key + ".tar.gz")), (new_tmp_dir + (@new_job.key + ".tar.gz")) if File.exist?(new_tmp_dir + (@job.key + ".tar.gz")) ### change filepaths in the result file cmd = "perl -i -pe 's/\\/#{@job.user_id}\\/#{@job.key}\\//\\/#{@new_job.user_id}\\/#{@new_job.key}\\//g' #{new_tmp_dir}/output/log.json" logger.debug("CMD: #{cmd}") `#{cmd}` ### delete tmp_dir FileUtils.rm_r new_tmp_dir + "tmp" if File.exist?(new_tmp_dir + "tmp") @new_job.name += " cloned" if @new_job.save redirect_to job_path(@new_job.key) end #else # render :nothing => true # end end def serve if readable? @job tmp_dir = Pathname.new(APP_CONFIG[:data_dir]) + 'users' + @job.user.id.to_s + @job.key #params[:key] # + params[:step] # tmp_dir += params[:item_id].to_s if params[:item_id] filename = params[:filename] #|| 'dl_output.tab' filepath = tmp_dir + filename send_file filepath.to_s, type: params[:content_type] || 'text', disposition: (!params[:display]) ? ("attachment; filename=" + filename.gsub("/", "_")) : '' end end def read_file_header if params[:file_key] new_data = [] user_id = (current_user) ? current_user.id.to_s : "1" dir = Pathname.new(APP_CONFIG[:data_dir]) + "users" + user_id + session[:current_key] + 'input' filename = params[:file_key] + ".txt" filepath = dir + filename test = "" if File.exist? filepath lines = [] File.open(filepath, "r") do |f| while(l = f.gets) do lines.push(l.chomp) end end j=0 (0 .. lines.size-1).to_a.each do |i| if !lines[i].match(/^#/) j=i break end end i = (j > 1) ? j-1 : 0 headers = lines[i].split("\t") if params[:add_blank] new_data.push({label:"", value:""}); end if headers.size > 0 headers.each do |header| new_data.push({:label => header, :value => header}); end end end render :text => new_data.to_json else render :nothing => true end end def read_file_column if params[:file_key] new_data = [] user_id = (current_user) ? current_user.id.to_s : "1" dir = Pathname.new(APP_CONFIG[:data_dir]) + "users" + user_id + session[:current_key] + 'input' filename = params[:file_key] + ".txt" filepath = dir + filename if File.exist? filepath lines = File.readlines(filepath) j=0 (0 .. lines.size-1).each do |i| if !lines[i].match(/^#/) j = i break end end i = (j > 1) ? j-1 : 0 header_els = lines[i].chomp.split("\t") pos_col = 0 (0 .. header_els.size).each do |k| if header_els[k] == params[:col_name] pos_col = k break end end max_val=0 (i .. lines.size-1).each do |j| t = lines[j].split("\t") n = t[pos_col].split(";").size if n > max_val max_val = n end end if max_val > 0 (1 .. max_val+1).each do |j| new_data.push({:label => j, :value => j}); end end end render :text => new_data.to_json else render :nothing => true end end # BUILD THE INDEX OF JOBS FROM CURRENT USER OR ALL (ADMIN ONLY) def index if current_user if current_user.role == "admin" #and session[:which] == "all" @jobs = Job.all @users = User.all else#if current_user @jobs = current_user.jobs.all end end respond_to do |format| format.html{ if !current_user redirect_to "/welcome" end } end end def get_views @h_views = {} View.all.each do |v| @h_views[v.name]= v end end def get_statuses log_file = Pathname.new(APP_CONFIG[:data_dir]) + "users" + @user.id.to_s + @job.key + 'output' + 'log.json' #stdout_file = Pathname.new(APP_CONFIG[:data_dir]) + "users" + @user.id.to_s + @job.key + 'output' + 'stdout.log' #stderr_file = Pathname.new(APP_CONFIG[:data_dir]) + "users" + @user.id.to_s + @job.key + 'output' + 'stderr.log' #if File.exist? stdout_file and !File.size(stdout_file) != 0 # @stdout_log = JSON.parse(File.read(stdout_file)).select {|e| e[0] != nil} #end #if File.exist? stderr_file and !File.size(stderr_file) != 0 # @stderr_log = JSON.parse(File.read(stderr_file)).select {|e| e[0] != nil} #end @h_statuses = {}#'completed' => 1, 'pending' => 2, 'running' => 3, 'failed' => 4} Status.all.each do |s| @h_statuses[s.name]= s end if File.exist? log_file and !File.size(log_file) != 0 @log_json = JSON.parse(File.read(log_file)) @final_json = { :global_status => nil, :status_by_step => {}, :status_by_substep => {}, :global_status_by_step => {}, :messages_by_step => {} } @h_icons={ 'description' => 'fa fa-info-circle', 'description-item' => 'ml-3', 'comment' => 'fa fa-info-circle', 'output' => 'fa fa-file-text-o', 'warning' => 'fa fa-exclamation-triangle icon-warning', 'error' => 'fa fa-exclamation-triangle icon-danger' } @test = "" ### datasets @log_json.select{|e| ['dataset', 'map'].include?(e['type'])}.each do |cat| @final_json[:status_by_step][cat['name']] ||= [] @final_json[:global_status_by_step][cat['name']] ||= nil # @final_json[:messages_by_step][cat['name']] ||= [] cat['log'].select{|e| e['type'] != 'file'}.each do |el| @final_json[:status_by_substep][el['name']]=[] step_key = cat['name'] + "_" + el['name'] @final_json[:messages_by_step][step_key] ||= [] tmp_global_status = nil # @test += el['operations'].to_json if el['operations'] el['operations'].each do |el2| # @test += el2.to_json @final_json[:status_by_substep][el['name']].push({:name => el2['name'], :status => el2['status'], :execution_time => el2['execution_time'], :cat => cat['name']}) @final_json[:messages_by_step][step_key].push({:name => el2['name'], :messages => el2['messages'], :cat => cat['name']}) if !tmp_global_status or (@h_statuses[el2['status']] and @h_statuses[el2['status']].precedence > @h_statuses[tmp_global_status].precedence) tmp_global_status = el2['status'] # @test += el2['status'].to_json end end end @final_json[:status_by_step][cat['name']].push({:name => el['name'], :status => (el['status'] || tmp_global_status), :execution_time => el['execution_time']}) if !@final_json[:global_status_by_step][cat['name']] or (@h_statuses[el['status']] and @h_statuses[el['status']].precedence > @h_statuses[@final_json[:global_status_by_step][cat['name']]].precedence) @final_json[:global_status_by_step][cat['name']] = el['status'] end if !@final_json[:global_status] or (@h_statuses[el['status']] and @h_statuses[el['status']].precedence > @h_statuses[@final_json[:global_status]].precedence) @final_json[:global_status] = el['status'] end @final_json[:messages_by_step][step_key].push({:name => el['name'], :messages => el['messages']}) end # @final_json[:messages_by_step][cat['name']].push({:name => cat['name'], :messages => cat['messages']}) end ### analyses analyses = @log_json.select{|e| e['type'] == 'analysis'} if analyses.size > 0 analyses.first['log'].each do |el| @final_json[:global_status_by_step][el['name']] ||= nil @final_json[:status_by_step][el['name']] ||= [] # @final_json[:status_by_substep][el['name']]=[] @final_json[:messages_by_step][el['name']] ||= [] tmp_global_status = nil if el['levels'] el['levels'].each do |el2| @final_json[:status_by_step][el['name']].push({:name => el2['name'], :status => el2['status'], :execution_time => el2['execution_time']}) @final_json[:messages_by_step][el['name']].push({:name => el2['name'], :messages => el2['messages']}) if !tmp_global_status or (@h_statuses[el2['status']] and @h_statuses[el2['status']].precedence > @h_statuses[tmp_global_status].precedence) tmp_global_status = el2['status'] end end end if !@final_json[:global_status_by_step][el['name']] or (@h_statuses[el['status']] and @h_statuses[el['status']].precedence > @h_statuses[@final_json[:global_status_by_step][cat['name']]].precedence) @final_json[:global_status_by_step][el['name']] = el['status'] end if !@final_json[:global_status] or (@h_statuses[el['status']] and @h_statuses[el['status']].precedence > @h_statuses[@final_json[:global_status]].precedence) @final_json[:global_status] = el['status'] end @final_json[:status_by_step][el['name']].push({:name => el['name'], :status => (el['status'] || tmp_global_status), :execution_time => el['execution_time']}) @final_json[:messages_by_step][el['name']].push({:name => el['name'], :messages => el['messages']}) end end @update = 0 @status_vector = [] if @final_json if ['primary_dataset', 'map', 'secondary_dataset'].include? session[:selected_view] if @final_json[:status_by_step][session[:selected_view]] @status_vector += @final_json[:status_by_step][session[:selected_view]].map{|e| @h_statuses[e[:status]].id} @final_json[:status_by_substep].keys.sort.each do |k| @status_vector += @final_json[:status_by_substep][k].select{|e| e[:cat] == session[:selected_view]}.map{|e| @h_statuses[e[:status]].id} end end @final_json[:messages_by_step].keys.sort.each do |k| @status_vector += @final_json[:messages_by_step][k].select{|e| e[:cat] == session[:selected_view] and e[:messages]}.map{|e| e[:messages].size} end else if @final_json[:status_by_step][session[:selected_view]] @status_vector += @final_json[:status_by_step][session[:selected_view]].map{|e| @h_statuses[e[:status]].id} end if @final_json[:messages_by_step][session[:selected_view]] @status_vector += @final_json[:messages_by_step][session[:selected_view]].select{|e| e[:messages]}.map{|e| e[:messages].size} end end end # @final_json[:status_by_step].keys.sort.each do |name| # @status_vector.push(@final_json[:status_by_step][name].map{|e| @h_statuses[e[:status]].id}) # end # @final_json[:messages_by_step].keys.sort.each do |name| # @status_vector.push(@final_json[:messages_by_step][name].select{|e| e[:messages]}.map{|e| e[:messages].size}) # end if session[:status_vector] != @status_vector @update = 1 end end ## end file exist end def refresh # @user = (current_user) ? current_user : User.where(:username => 'guest').first # @user = @project. get_basic_info() get_statuses() render :partial => "refresh" end def view # @user = (current_user) ? current_user : User.where(:username => 'guest').first get_basic_info() get_statuses() get_views() @form_json = JSON.parse @job.form_json if @job.form_json and !@job.form_json.empty? if !session[:current_level] session[:current_level] = (@form_json['bin_levels'] and @form_json['bin_levels'].size > 0) ? @form_json['bin_levels'][0] : nil end session[:current_level] = params[:current_level].to_i if params[:current_level] @data_json = nil @filename = nil @imagename = nil @description = '' @i = 0 @log = '' - if !['primary_dataset', 'secondary_dataset', 'map'].include?(params[:partial]) + if !['details', 'primary_dataset', 'secondary_dataset', 'map'].include?(params[:partial]) e = @log_json.select{|el| el['type'] == 'analysis'}.first['log'].select{|el| el['name'] == params[:partial]}.first if e['levels'] #and @form_json['bin_levels'] @i = @form_json['bin_levels'].index(session[:current_level].to_s) @form_json['bin_levels'].each_index do |li| l = @form_json['bin_levels'][li] @log += ">" + l.to_s + ";" if l == session[:current_level].to_s @i = li @log += '!!' end end e2 = e['levels'][@i] @filename = e2['messages'].select{|el| el['output'] and el['output'].match(/#{@h_views[params[:partial]].data_format}$/)}.map{|el| el['output']}.first @imagename = e2['messages'].select{|el| el['output'] and el['output'].match(/pdf|png|jpg$/)}.map{|el| el['output']}.first @description = e2['messages'].select{|el| el['description']}.map{|el| el['description']} else @i = -1 @filename = e['messages'].select{|el| el['output'] and el['output'].match(/#{@h_views[params[:partial]].data_format}$/)}.map{|el| el['output']}.first @imagename = e['messages'].select{|el| el['output'] and el['output'].match(/pdf|png|jpg$/)}.map{|el| el['output']}.first @description = e['messages'].select{|el| el['description']}.map{|el| el['description']} end end #end if @filename and File.exist? @filename @data_json = File.read(@filename) end session[:selected_view] = params[:partial] json_file = Rails.root.join('lib', 'genocrunch_console', 'etc', 'genocrunchlib.json') file_content = File.read(json_file) h = JSON.parse(file_content) @h_form_choices = h['choices'] render :partial => "view_" + params[:partial] end # ALLOW SESSION VARIABLE UPDATE ON SHOW # THIS ALLOW CONDITIONAL RENDERING OF EITHER FIGURES OR EDIT FORM PARTIALS def show # @user = (current_user) ? current_user : User.where(:username => 'guest').first # if current_user.role == "admin" and session[:which] == "all" # @jobs = Job.all # @users = User.all # else#if current_user # @jobs = current_user.jobs.all # end # session[:context] = params[:context] # logger.debug("JOB: " + @job.to_json) #check_box belongs_to get_basic_info() get_statuses() get_views() @analyses = [] @inputs = [] @h_job_form = JSON.parse(@job.form_json) session[:current_level] = (@h_job_form['bin_levels'] && @h_job_form['bin_levels'][0]) || nil @h_form['fields']['Inputs'].select{|f| f['type'] == 'file'}.each do |f| if @h_job_form[f['id']] @inputs.push({:id => f['id'], :label => f['label'] || f['id'].gsub(/_/, ' ').capitalize}) end end @h_form['fields']['Analysis'].select{|f| f['type'] == 'check_box' and !f['belongs_to']}.each do |f| if @h_job_form[f['id']] #and @h_job_form[f['id']]==true @analyses.push({:id => f['id'], :label => f['label'] || f['id'].gsub(/_/, ' ').capitalize}) end end if !readable? @job redirect_to root_path, notice: 'Cannot access this resource.' end end # ALLOW UPDATE OF INDEX UPON CHANGE AND ADMIN TO ACCESS ALL JOBS AND USERS def manage # session[:which] = params[:which] # session[:what] = params[:what] @previous_jobs = nil if current_user.role == "admin" #and params[:which] == "all" @jobs = Job.all @users = User.all @partial = "/admins/index" if defined?(params[:previous_jobs]) @previous_jobs = params[:previous_jobs] end else @jobs = current_user.jobs @users = current_user @partial = "/jobs/index" if defined?(params[:previous_jobs]) @previous_jobs = params[:previous_jobs] end end @id = "indexPartial" respond_to do |format| format.js end end def get_basic_info json_file = Rails.root.join('lib', 'genocrunch_console', 'etc', 'genocrunchlib.json') file_content = File.read(json_file) @h_form = JSON.parse(file_content) @h_tips = JSON.parse(File.read(Rails.root.join('public', 'app', 'tips.json'))) @h_help = {} @h_field_groups = {} @h_form['fields'].each_key{|card_title| @h_form['fields'][card_title] = @h_form['fields'][card_title].select{|field| field['scope'] != 'cli_only'} @h_form['fields'][card_title].map{|field| @h_help[field['id']] = field['help'] if field['belongs_to'] @h_field_groups[field['belongs_to']]||=[] @h_field_groups[field['belongs_to']].push(field) end } } end # SET A NEW JOB def new @user = (current_user) ? current_user : User.where(:username => 'guest').first @job = @user.jobs.new session[:current_key] = create_key() @job.key = session[:current_key] get_basic_info @default = {} end def set_fields p @missing_fields = [] @present_fields = [] get_basic_info @h_fields ={} @h_form['fields'].each_key{|card_title| @h_form['fields'][card_title].map{|field| @h_fields[field['id']]=field } } @log = '' flag=0 if p flag=1 ### check if some parameters are not allowed p.each_key do |k| if !@h_fields[k] flag = 0 break end end ### check if all parameters required are submitted @h_fields.keys.select{|k| @h_fields[k]['optional'] == false}.each do |k| logger.debug("EXPLORE field " + k) if (@h_fields[k]['type']== 'file' and ((p[k] and !p[k].blank? ) or !params[:p2][k].empty?) ) or (p[k] and !p[k].empty?) @present_fields.push(@h_fields[k]) else @missing_fields.push(@h_fields[k]) flag = 0 end end end return flag end def create_dirs user_id = (current_user) ? current_user.id.to_s : "1" dir = Pathname.new(APP_CONFIG[:data_dir]) + "users" + user_id Dir.mkdir dir if !File.exist? dir dir = Pathname.new(APP_CONFIG[:data_dir]) + "users" + user_id + @job.key Dir.mkdir dir if !File.exist? dir end def write_files p user_id = (current_user) ? current_user.id.to_s : "1" dir = Pathname.new(APP_CONFIG[:data_dir]) + "users" + user_id + @job.key + 'input' Dir.mkdir dir if !File.exist? dir form_json = JSON.parse @job.form_json if @job.form_json and !@job.form_json.empty? # fields = @h_fields.keys.select{|k| @h_fields[k]['type'] == 'file' and p[k]} fields = ['primary_dataset', 'map', 'secondary_dataset'] fields.each do |k| logger.debug("Key:" + k) filepath = dir + ( k + '.txt') content = (p[k]) ? p[k].read : nil if content and !content.empty? params[:p2][k] = p[k].original_filename File.open(filepath, 'w') do |f| logger.debug("CONTENT FILE: " + content) f.write content end `dos2unix #{filepath}` `mac2unix #{filepath}` elsif form_json and form_json[k.to_s] params[:p2][k] = form_json[k.to_s]['original_filename'] params[:p][k] = form_json[k.to_s] end end end # CREATE A NEW JOB def create # @job = Job.new(job_params) @job = Job.new(:key => params[:tmp_key]) create_dirs() write_files(params[:p]) @valid_job = set_fields params[:p] @job.name = params[:p][:name] @job.description = params[:p][:description] @job.form_json = params[:p].to_json @default = params[:p] if current_user @job.user_id = current_user.id @job.sandbox = false end ## not possible to create a new job with an existing key @valid_job = 0 if !@job.key.empty? and Job.where(:key => @job.key).first respond_to do |format| if @valid_job==1 and @job.save @job.delay.perform # run analysis as delayed job session[:agree_with_terms] = true session[:current_key]=create_key() if current_user format.html { redirect_to job_path(@job.key) } format.json { render action: 'new', status: :created, location: @job } else format.html { render action: 'new' } format.json { render json: @job.errors, status: :unprocessable_entity } end end end def edit get_basic_info @default = JSON.parse(@job.form_json) params[:p2]={} ['primary_dataset', 'map', 'secondary_dataset'].select{|k| @default[k]}.each do |k| params[:p2][k] = @default[k]['original_filename'] end end # PATCH/PUT /jobs/1 # PATCH/PUT /jobs/1.json def update write_files(params[:p]) valid_job = set_fields params[:p] h_job = { :name => params[:p][:name], :description => params[:p][:description], :form_json => params[:p].to_json } @default = JSON.parse(@job.form_json) @default = params[:p] respond_to do |format| if valid_job==1 and @job.update_attributes(h_job) #kill job kill_job(@job) @job.delay.perform # run analysis as delayed job session[:agree_with_terms] = true # if current_user.role == "admin" and @job.user.role != "admin" # format.html { redirect_to jobs_url } # else # format.html { redirect_to jobs_url } format.html {redirect_to job_path(@job.key)} # end format.json { head :no_content } else format.html { render action: 'edit'} format.json { render json: @job.errors, status: :unprocessable_entity } end end end # DELETE /jobs/1 # DELETE /jobs/1.json def destroy @job.destroy respond_to do |format| format.html { redirect_to jobs_url } format.json { head :no_content } end end private # Use callbacks to share common setup or constraints between actions. def set_job # if current_user.role == "admin" # @job = Job.all.find(params[:key]) # else # @job = current_user.jobs.find(params[:key]) # end @job = Job.where(:key => params[:key] || params[:tmp_key]).first session[:current_key] = @job.key if action_name != 'clone' #if current_user @user = @job.user logger.debug("JOB: " + @job.to_json) # @job = nil if !readable? @job end # Never trust parameters from the scary internet, only allow the white list through. def job_params params.require(:job).permit() end end diff --git a/app/views/home/_doc.html.erb b/app/views/home/_doc.html.erb index ac1ed48..9015ed1 100644 --- a/app/views/home/_doc.html.erb +++ b/app/views/home/_doc.html.erb @@ -1,505 +1,508 @@

    Documentation

    Version 1.3 / 04.05.2018

    Overview

    Genocrunch is a web-based data analysis platform dedicated to metagenomics and metataxonomics. It is tailored to process counts datasets derived from high-throughput nucleic acids sequencing of microbial communities <%= image_tag('genocrunch_pipeline-1.png', height:'14px') %>, such as gene counts or counts of operational taxonomic units (OTUs) <%= image_tag('genocrunch_pipeline-2.png', height:'14px') %>. In addition to such primary dataset, it also allows the integration of a secondary dataset (e.g. metabolites levels) <%= image_tag('genocrunch_pipeline-3.png', height:'14px') %>.
    Genocrunch provides tools covering data pre-processing and transformation, diversity analysis, multivariate statistics, dimensionality reduction, differential analysis, clustering as well as similarity network analysis and associative analysis. The results of clustering are automatically inferred as additional models into other analyses <%= image_tag('genocrunch_pipeline-4.png', height:'14px') %>.
    Interactive visualization is offered for all figures.

    <%= image_tag('genocrunch_pipeline.png', width:'100%') %>

    New users and registration

    Running an analysis session does not require any registration, however, managing multiple analysis (recovering/editing previous work) is done through a personnal index that requires users to sign in. Before signing in, new users need to register.

    Registering

    Registering allows users to recover and edit their previous work via a personal index page. Registering is done via the Register button of the top-bar. The registration process requires new users to chose a password and provide an email address. The email address is only used to recover forgotten passwords. <% if APP_CONFIG[:user_confirmable] %> In order to complete the registration process, new users need to follow a confirmation link which is automatically sent to the provided email address. In case of issue, this confirmation link can be resent to the provided email address via the Resend confirmation instruction link on the welcome page. <% end %>

    Signing in

    Signing in is done from the welcome page, which is accessible via the Sign In button of the top-bar. Before signing in, new users need to complete the registration process.

    Trying Genocrunch

    The best way to try Genocrunch is to load an example and edit it.

    -

    Examples

    +

    Examples

    Examples of analyses are provided via the Examples link of the top-bar. Loading Example data into a new analysis session is possible via the Load button. This allows to visualize and edit Examples without restriction. For signed-in users, it will also copy the example into their personal analysis index.

    Analyzing data

    Running a new analysis

    New analyses can be submitted via the New Analysis button of the topbar. If the user is signed out, the analysis session will be associated to the browser session and remain available as long as the browser is not closed. If the user had been signed-in, the analysis will be stored in his personal index. Analyses parameters are set by filling a single form comprising four parts for Inputs, Pre-processing, Transformation and Analysis. Analyses are then submitted via the Start Analysis button located at the bottom the form. See the Inputs and Tools section below for details.

    Editing an analysis

    Analyses can be edited via the associated Edit button present on the analysis page and via the edit icons () of the analysis index (for signed-in users only). Editing is performed by updating the analysis form and submiting the modifications via the Restart Analysis button located at the bottom of the form.

    Cloning an analysis

    Analyses can be copied via the copy icons () of the analysis index (for signed-in users only).

    Deleting an analysis

    Analyses can be deleted via the delete icons () of the analysis index (for signed-in users only). This process is irreversible.

    Inputs and Tools

    The analyses are set from a single form composed of four sections: Inputs, Pre-processing, Transformation, Analysis. These sections and their associated parameters are described below.

    Inputs

    The Inputs section of the analysis creation form allows to upload data files.

    • General information
      • Name (mandatory)

        The name will appear in the analysis index. It should be informative enough to differentiate from other analyses.

      • Description

        The description can be used to provide additional information about the analysis.

    • Data files
      • Primary dataset (mandatory) -

        The primary dataset is the principal data file. It must contain a data table in the tab-delimited text format with columns representing samples and rows representing observations. The first row must contain the names of samples. The first column must contain the names of the observations. Both columns and rows names must be unique. A column containing a description of each observation, in the form of semi-column-delimited categories, can be added. This format can be obtained from the BIOM format using the biom_convert command.

        +

        The primary dataset is the principal data file. It must contain a data table in the tab-delimited text format with columns representing samples and rows representing observations. The first row must contain the names of samples. The first column must contain the names of the observations. Both columns and rows names must be unique. A column containing a description of each observation, in the form of semi-column-delimited categories, can be added at the end. This format can be obtained from the BIOM format using the biom_convert command.

        Format example:

         #OTU ID	Spl1	Spl2	Spl3	Spl4	taxonomy
 1602	1	2	4	4	k__Bacteria; p__Actinobacteria; c__Actinobacteria; o__Actinomycetales; f__Propionibacteriaceae; g__Propionibacterium; s__acnes
 1603	0	0	24	0	k__Bacteria; p__Firmicutes; c__Clostridia; o__Clostridiales; f__Clostridiaceae; g__Clostridium; s__difficile
 1604	0	12	23	0	k__Bacteria; p__Firmicutes; c__Bacilli; o__Lactobacillales; f__Enterococcaceae; g__Enterococcus; s__casseliflavus
 1605	23	4	2	14	k__Bacteria; p__Firmicutes; c__Clostridia; o__Clostridiales; f__Veillonellaceae; g__Veillonella; s__parvula
 1606	45	10	42	12	k__Bacteria; p__Bacteroidetes; c__Bacteroidia; o__Bacteroidales; f__Bacteroidaceae; g__Bacteroides; s__fragilis
 1607	8	15	20	13	k__Bacteria; p__Bacteroidetes; c__Bacteroidia; o__Bacteroidales; f__Prevotellaceae; g__Prevotella; s__copri
      • Category column (mandatory) -

        To increase compatibility, the category column specifies which column in the pimary dataset contains a categorical description of the observations.

        +

        To increase compatibility, the category column specifies which column in the pimary dataset contains a categorical description of the observations. This must be either the first or the last column.

      • Map (mandatory)

        The map contains information about the experimental design. It must contain a table in the tab-delimited text format with columns representing experimental variables and rows representing samples. The first column must contain the names of the samples as they appear in the primary dataset. The first row must contain the names of experimental variables. Both columns and rows names must be unique. This format is compatible with the mapping file used by the Qiime pipeline.

        Format example:

         ID	Sex	Treatment
 Spl1	M	Treated
 Spl2	F	Treated
 Spl3	M	Control
 Spl4	F	Control
      • Secondary dataset

        The secondary dataset is an optional data file containing additional observations. It must contain a data table in the tab-delimited text format with columns representing samples and rows representing observations. The first row must contain the names of samples as they appear in the primary dataset and the map. The first column must contain the names of the observations. Both columns and rows names must be unique.

        Format example:

         Metabolites	Spl1	Spl2	Spl3	Spl4
 metabolite1	0.24	0.41	1.02	0.92
 metabolite2	0.98	0.82	1.12	0.99
 metabolite3	0.05	0.11	0.03	0.02

    Pre-processing

    The Pre-processing section of the analysis creation form specifies modifications that will be applied to the primary dataset prior to analysis.

    • Filtering

      Data in the primary dataset can be filtered based on Relative or Absolute abundance and presence.

      • Abundance threshold

        Minimal abundance per sample to be retained.

      • Presence thresholds

        Minimal presence among samples to be retained.

      Example: Filtering data with an absolute abundance threshold of 5 and and an absolute presence threshold of 2.

       #Before filtering
 #OTU ID	Spl1	Spl2	Spl3	Spl4	taxonomy
 1602	1	2	4	4	k__Bacteria; p__Actinobacteria; c__Actinobacteria; o__Actinomycetales; f__Propionibacteriaceae; g__Propionibacterium; s__acnes
 1603	0	0	24	0	k__Bacteria; p__Firmicutes; c__Clostridia; o__Clostridiales; f__Clostridiaceae; g__Clostridium; s__difficile
 1604	0	12	23	0	k__Bacteria; p__Firmicutes; c__Bacilli; o__Lactobacillales; f__Enterococcaceae; g__Enterococcus; s__casseliflavus
 1605	23	4	2	14	k__Bacteria; p__Firmicutes; c__Clostridia; o__Clostridiales; f__Veillonellaceae; g__Veillonella; s__parvula
 1606	45	10	42	12	k__Bacteria; p__Bacteroidetes; c__Bacteroidia; o__Bacteroidales; f__Bacteroidaceae; g__Bacteroides; s__fragilis
 1607	8	15	20	13	k__Bacteria; p__Bacteroidetes; c__Bacteroidia; o__Bacteroidales; f__Prevotellaceae; g__Prevotella; s__copri
 
       #After filtering
 #OTU ID	Spl1	Spl2	Spl3	Spl4	taxonomy
 1604	0	12	23	0	k__Bacteria; p__Firmicutes; c__Bacilli; o__Lactobacillales; f__Enterococcaceae; g__Enterococcus; s__casseliflavus
 1605	23	4	2	14	k__Bacteria; p__Firmicutes; c__Clostridia; o__Clostridiales; f__Veillonellaceae; g__Veillonella; s__parvula
 1606	45	10	42	12	k__Bacteria; p__Bacteroidetes; c__Bacteroidia; o__Bacteroidales; f__Bacteroidaceae; g__Bacteroides; s__fragilis
 1607	8	15	20	13	k__Bacteria; p__Bacteroidetes; c__Bacteroidia; o__Bacteroidales; f__Prevotellaceae; g__Prevotella; s__copri
    • Category binning

      Data in primary dataset can be binned by category thanks to the category column.

      • Binning levels

        Category levels at which data should be binned.

      • Category binning function

        Chose whether binning should be performed by summing or averaging values.

      Example: Binning data at level 2 with sum.

       #Before binning
 #OTU ID	Spl1	Spl2	Spl3	Spl4	taxonomy
 1604	0	12	23	0	k__Bacteria; p__Firmicutes; c__Bacilli; o__Lactobacillales; f__Enterococcaceae; g__Enterococcus; s__casseliflavus
 1605	23	4	2	14	k__Bacteria; p__Firmicutes; c__Clostridia; o__Clostridiales; f__Veillonellaceae; g__Veillonella; s__parvula
 1606	45	10	42	12	k__Bacteria; p__Bacteroidetes; c__Bacteroidia; o__Bacteroidales; f__Bacteroidaceae; g__Bacteroides; s__fragilis
 1607	8	15	20	13	k__Bacteria; p__Bacteroidetes; c__Bacteroidia; o__Bacteroidales; f__Prevotellaceae; g__Prevotella; s__copri
 
       #After binning
 	ID	Spl1	Spl2	Spl3	Spl4	taxonomy
 1	23	16	25	14	k__Bacteria; p__Firmicutes
 2	53	25	62	25	k__Bacteria; p__Bacteroidetes

    Transformation

    The Transformation section of the analysis creation form propose additional modifications for both primary and secondary datasets.

    • Rarefaction

      Although controversial (McMurdie and Holmes, 2014), rarefaction is a popular method amongst microbial ecologists used to correct for variations in sequencing depth, inherent to high-throughput sequencing methods. It consists in random sampling (without replacement) of a fixed number of count from each sample to be compared. The same number of count being drawn out of each sample, this corrects for differences in sequencing depth while conserving the original count distribution among observations. Note that the analysis of diversity is particularly sensitive to variations in sequencing depth.

      • Sampling depth

        This specifies the number of count to be drawn from each sample.

        Tip: The maximal sampling depth corresponds to the minimal sequencing depth. First check the sequencing depth of your data by suming the counts in each sample. Then chose a sampling depth accordingly.

      • N samplings

        This specifies how many times the random drawing should be repeated. If more than 1 random sampling is done, the result is the average of all random samplings.

    • Transformation

      Whether it is to correct for sequencing depth, to stabilize the variance or simply to improve the visualization of skewed data, a transformation step may be needed. Common methods include transforming counts into proportions (percent or counts per million) and applying a log. We are working to also propose more advanced transformations, including those developed for RNA sequencing (RNA-seq) as part of the DESeq2 (Love et al., 2014) and the Voom (Law et al., 2014) pipelines as well as methods to limit the effect of known experimental bias (or batch-effect), such as Combat.

      • Transformation

        Specifies which transformation method should be applied.

    Analysis

    The Analysis section of the analysis creation form sets which statistics to apply and which figures to generate.

    • Experimental design

      This sets the default statistical method to apply for the analysis. Two options are available: Basic and Advanced.

      • Statistics (Advanced only)

        Chose a statistical method to compare groups of samples. If Basic is chosen, this defaults to ANOVA.

      • Model

        Chose a model defining groups of samples. If Basic is chosen, a model can be picked among headers of the map. If Advanced is chosen, the model must be typed by the user in the form of an R formula. The formula must be compatible with the specified statistics. All terms of the formula must refer to column headers of the map.

      Examples:
      1. Simple design: 
         ID	Subject	Site
 Spl1	subject1	Hand
 Spl2	subject2	Hand
 Spl3	subject3	Hand
 Spl4	subject4	Foot
 Spl5	subject5	Foot
 Spl6	subject6	Foot
        Statistics Model
        Comparing sites (parametric) T-test Site
        Comparing sites (non-parametric) Wilcoxon rank sum test Site
      2. Simple paired design: The order of paired samples in the map is important! 
         ID	Subject	Site
 Spl1	subject1	Hand
 Spl2	subject2	Hand
 Spl3	subject3	Hand
 Spl4	subject1	Foot
 Spl5	subject2	Foot
 Spl6	subject3	Foot
        Statistics Model
        Comparing sites (parametric) Paired t-test Site
        Comparing sites (non-parametric) Wilcoxon signed rank test Site
      3. Multiple comparisons: 
         ID	Subject	Site
 Spl1	subject1	Hand
 Spl2	subject2	Hand
 Spl3	subject3	Hand
 Spl4	subject4	Foot
 Spl5	subject5	Foot
 Spl6	subject6	Foot
 Spl7	subject7	Mouth
 Spl8	subject8	Mouth
 Spl9	subject9	Mouth
        Statistics Model
        Comparing sites (parametric) ANOVA Site
        Comparing sites (non-parametric) Kruskal-Wallis rank sum test Site
      4. Multiple comparisons with nesting: 
         ID	Subject	Site
 Spl1	subject1	Hand
 Spl2	subject2	Hand
 Spl3	subject3	Hand
 Spl4	subject1	Foot
 Spl5	subject2	Foot
 Spl6	subject3	Foot
 Spl7	subject1	Mouth
 Spl8	subject2	Mouth
 Spl9	subject3	Mouth
        Statistics Model
        Comparing sites Friedman test Site | Subject
      5. Additive model: 
         ID	Treatment	Gender
 Spl1	Treated	M
 Spl2	Treated	M
 Spl3	Treated	M
 Spl4	Treated	F
 Spl5	Treated	F
 Spl6	Treated	F
 Spl7	Control	M
 Spl8	Control	M
 Spl9	Control	M
 Spl10	Control	F
 Spl11	Control	F
 Spl12	Control	F
        Statistics Model
        Assessing effects of treatment and gender ANOVA Treatment+Gender
    • Proportions

      For each sample, display the proportions of each observation (as percent) in the form of a stacked barchart. This gives an overview of the primary dataset. This analysis is performed before applying any selected transformation.

    • Diversity

      Perform a diversity analysis of the primary dataset. This will display rarefaction curves for the selected diversity metric(s). Rarefaction curves are used to estimate the diversity in function of the sampling depth. A rarefaction curve showing an asymptotic behavior is considered as an indicator of sufficient sampling depth to observe the sample diversity. This analysis is performed before applying any selected transformation.

      • metric

        Chose a diversity metric. Choices include the richness as well as metrics included in the R vegan, ineq and fossil packages. The richness simply represents the number of different observations that are seen within a particular sample.

      • Compare groups

        If selected, diversity between groups will be assessed using statistics and model specified in the experimental design.

    • perMANOVA

      Perform a permutational multivariate analysis of the variance using the Adonis method from R package vegan. This method uses a distance matrix based on the primary dataset.

      • Distance metric

        Chose a distance metric. Choices include metrics available in the R vegan package as well as distances based on correlation coefficients.

      • Model

        For compatibility purpose, this model may differ from the model specified in the experimental design section. This model must be in the form of an R formula compatible with the Adonis function of the R vegan package. All terms of the formula must refer to column headers of the map.

      • Strata

        This is used to specify any nesting in the model. the strata must be compatible with the Adonis function of the R vegan package.

    • PCA

      Perform a principal component analysis (PCA) on the primary dataset. PCA is a commonly used dimensionality reduction method that projects a set of observations onto a smaller set of components that capture most of the variance between samples.

    • PCoA

      Perform a principal coordinate analysis (PCoA) on the primary dataset. The PCoA is a form of dimensionality reduction based on a distance matrix.

      • Distance metric

        Chose a distance metric. Choices include metrics available in the R vegan package as well as distances based on correlation coefficients.

    • Heatmap

      This displays the primary dataset on a heatmap of proportions. The columns (samples) and rows (observations) of the heatmap are re-ordered using a hierarchical clustering. Each observation will be individually compared between groups of samples based on the specified experimental design and associated p-values will be displayed on the side of the heatmap. Individual correlations between observations from the primary dataset and observations from the secondary dataset will also be displayed on the side of the heatmap when available. Samples will be color-coded according to the experimental design.

    • Changes

      This performs a differential analysis. Each observation will be individually compared between groups of samples based on the specified experimental design. Fold-change between groups as well as the associated p-values and mean abundance will be displayed on MA plots and volcano plots.

    • Correlation network

      This builds a network with nodes representing observations and edges representing the correlations. The network will be based on the primary dataset and will include the secondary dataset if available. Observations belonging to the primary and the secondary dataset will be represented using different node shapes. Nodes will be colored to include a number of additional information.

      • Correlation method

        Chose a correlation method. Choices include the Pearson correlation and the Spearman correlation.

    • Clustering

      This applies a clustering algorithm to separate samples into categories. The categories will be added as a new column to the map. Categories will automatically be compared with ANOVA in relevant analysis.

      • Algorithm

        Chose a clustering algorithm. Choices include the k-means algorithm, the k-medoids algorithm (also known as the partitioning around medoids of pam algorithm) and versions of the k-medoids based on distance matrices.

    • Similarity network

      This builds a network with nodes representing samples and edges representing a similarity score. If a secondary dataset is available, three networks will be build: one based on the primary dataset, one based on the secondary dataset and a third based on the fusion of these two networks. The similarity network fusion (SNF) is based on the R SNFtool package. A categorical clustering based on each network is applied.

      • Metric for primary dataset

        Chose the similarity metric to apply on the primary dataset.

      • Metric for secondary dataset

        Chose the similarity metric to apply on the secondary dataset.

      • Clustering algorithm

        Chose the clustering algorithm to assign samples to categories based on each similarity network.

    Versions

    Genocrunch data analysis tools are based on R libraries. Used libraries are mentionned in the analysis report. The version of the libraries are available on the Versions page via the Infos menu of the topbar).

    -

    Analysis report

    +

    Analysis report

    A detailed analysis report is generated for each analysis. It is available directly after creating/updating a new analysis via the My Analysis button of the topbar or via the corresponding icon () in the analysis index (for signed-in users only). The analysis report includes detailed descriptions of the tools and methods used, possible warning and error logs as well as intermediate files, final files and interactive figures as they become available. Information for the pre-processing and data transformation are available for the primary dataset and the optional secondary dataset in their respective sections. Figures and descriptions for each analysis are available in separate sections. If multiple binning levels were specified, figures for each levels can be accessed via the associated Level button.

    Exporting figures

    Figures can be exported via the associated Export menu. All figures can be exported as a vector image in the SVG format. Some figures are also available in the PDF format, but PDFs are generated in the same time as the figure raw data and cannot be edited with the interactive visualization tools. Raw figure data are also available either in the JSON format, the TXT format or both. Figure legends are currently only available in the HTML format.

    -

    Archive

    +

    Archive

    Files containing initial, intermediate and final data as well as the analysis log and logs for standard output and standard error are automatically included into a .tar.gz archive. This archive can be downloaded via the button of the analysis report page or via the corresponding icon () in the analysis index (for signed-in users only). Although some figures may be automatically included in the archive in the form of a minimal PDF file, the archive does generally not contain figure images. These should be exported separately via the Export menu.

    + +

    Bugs report

    +

    During computation (while the analysis is running), the standard error (stderr) stream is redirected into a bugs report that can be downloaded using the link located at the bottom-right of the details page in the analysis report.

    diff --git a/app/views/home/_doc_index.html.erb b/app/views/home/_doc_index.html.erb index 1d87b33..15e9d09 100644 --- a/app/views/home/_doc_index.html.erb +++ b/app/views/home/_doc_index.html.erb @@ -1,273 +1,265 @@ diff --git a/app/views/jobs/_field_file.html.erb b/app/views/jobs/_field_file.html.erb index a77fc6a..455240e 100644 --- a/app/views/jobs/_field_file.html.erb +++ b/app/views/jobs/_field_file.html.erb @@ -1,61 +1,62 @@ <%= javascript_tag do %> $(document).ready(function() { function change_<%= field['id'] %>(where){ <% if field['id'] == 'primary_dataset' %> var url = (where == 'server') ? '<%= read_file_header_jobs_path() %>?file_key=<%= field['id'] %>' : null; - setSelectFromFileRow("category_column", $("#p_primary_dataset")[0].files[0], '<%= @default['category_column'] || '' -%>', null, url); + // setSelectFromFileRow("category_column", $("#p_primary_dataset")[0].files[0], '<%= @default['category_column'] || '' -%>', null, url); + setCategoryColumn("category_column", $("#p_primary_dataset")[0].files[0], '<%= @default['category_column'] || '' -%>', null, url); <% elsif field['id'] == 'map' %> var url = (where == 'server') ? '<%= read_file_header_jobs_path() %>?file_key=<%= field['id'] %>' : null; setSelectFromFileRow("prim_batch_effect_suppression", $("#p_map")[0].files[0], '<%= @default['prim_batch_effect_suppression'] || '' -%>', null, url); setSelectFromFileRow("sec_batch_effect_suppression", $("#p_map")[0].files[0], '<%= @default['sec_batch_effect_suppression'] || '' -%>', null, url); setSelectFromFileRow("basic_model", $("#p_map")[0].files[0], '<%= @default['basic_model'] || '' -%>', null, url); <% end %> } $("#p_<%= field['id'] %>").on('change',function(){ var filename = $(this).val().replace(/.*?fakepath\\/, ''); $(this).next('.form-control-file').addClass("selected").html(filename); // the file on the server is obsolete: $("#p2_<%= field['id'] %>").val("") // reset the default value of bin_levels $("#default_bin_levels").val("[]"); // refresh the fields depending on the file field change_<%= field['id'] %>("client"); }) var ori_filename = $("#p2_<%= field['id'] %>").val(); if (ori_filename != ''){ $("#p_<%= field['id'] %>").next('.form-control-file').addClass("selected").html(ori_filename); change_<%= field['id'] %>("server"); <% if field['id'] == 'map' %> $('#p_prim_batch_effect_suppression_fun-container').removeClass("hidden"); $('#p_prim_batch_effect_suppression_fun-placeholder').addClass("hidden"); $('#p_sec_batch_effect_suppression_fun-container').removeClass("hidden"); $('#p_sec_batch_effect_suppression_fun-placeholder').addClass("hidden"); $('#p_basic_model-container').removeClass("hidden"); $('#p_basic_model-placeholder').addClass("hidden"); <% end %> <% if field['id'] == 'primary_dataset' %> $('#p_category_column-container').removeClass("hidden"); $('#p_category_column-placeholder').addClass("hidden"); $('#p_bin_levels-container').removeClass("hidden"); $('#p_bin_levels-placeholder').addClass("hidden"); <% end %> } }); <% end %> diff --git a/app/views/jobs/_jobs.html.erb b/app/views/jobs/_jobs.html.erb index bead1dd..df3c0c2 100644 --- a/app/views/jobs/_jobs.html.erb +++ b/app/views/jobs/_jobs.html.erb @@ -1,135 +1,135 @@

    Analyses

    - + <% if current_user.role == "admin" %> <% end %> - + <% @jobs.compact.each do |job| %> <% date = job.updated_at.year.to_s[-2..-1].to_s + "-" + job.updated_at.month.to_s.rjust(2, '0') + "-" + job.updated_at.day.to_s.rjust(2, '0') + " " + job.updated_at.hour.to_s.rjust(2, '0') + ":" + job.updated_at.min.to_s.rjust(2, '0') + " " + job.updated_at.zone.to_s %> <% if current_user.role == "admin" %> <% end %> <% destroy_text = (['pending', 'running'].include? job.status) ? 'abort' : 'delete' %> - <% end %>
    Key Name DateUser TypeStatus Data Map Results | |
    <%= link_to job.key, job_path(job.key) %>
    <%= link_to job.name, job_path(job.key) %>
    		<%= link_to date, job_path(job.key) %> <%= (job.user) ? job.user.username : 'NA' %> <%= (Example.where(:job_key => job.key).all.size > 0) ? 'Example' : 'Std' %> <% if job.status %> <% end %> <%= link_to raw(''), serve_job_path(job.key, :filename => 'input/primary_dataset.txt'), title: "primary dataset" %> <% if File.exist? Pathname.new(APP_CONFIG[:data_dir]) + 'users' + job.user_id.to_s + job.key + 'input' + 'secondary_dataset.txt' %> | <%= link_to raw(''), serve_job_path(job.key, :filename => 'input/secondary_dataset.txt'), title: "secondary dataset" %> <% end %> <%= link_to raw(''), serve_job_path(job.key, :filename => 'input/map.txt') %> <%= link_to raw(''), job_path(job.key), :title => 'show' %> <% if (File.exist? Pathname.new(APP_CONFIG[:data_dir]) + 'users' + job.user_id.to_s + job.key + (job.key + '.tar.gz').to_s) and (!['pending', 'running'].include? job.status) %> | <%= link_to raw(''), serve_job_path(job.key, :filename => job.key + '.tar.gz'), :title => 'download archive' %> <% end %> - <%= link_to raw(''), clone_job_path(job.key) %> + + <%= link_to raw(''), clone_job_path(job.key) %> | -<%= link_to raw(''), edit_job_path(job.key) %> +<%= link_to raw(''), edit_job_path(job.key) %> | <% destroy_text = (['pending', 'running'].include? job.status) ? 'abort' : 'delete' %> <%= link_to(job_path(job.key), method: :delete, data: { confirm: 'Are you sure you want to ' + destroy_text + ' "' + (job.name || 'NA') + '" ?'}) do %> - + <% end %>

    Status summary

    <%= @jobs.select{ |i| i.status == "completed" }.length %> completed ()
    <%= @jobs.select{ |i| i.status == "pending" }.length %> pending ()
    <%= @jobs.select{ |i| i.status == "running" }.length %> running ()
    <%= @jobs.select{ |i| i.status == "failed" }.length %> failed ()
    Total: <%= @jobs.length %>
    <%= javascript_tag do %> $(document).ready(function(){ var table = $('#myTable').DataTable({ sDom: 'ltp', "aLengthMenu": [[10, 25, 50, 100, -1], [10, 25, 50, 100, "All"]], "iDisplayLength" : 10, "order": [[ 2, 'desc' ], [ 1, 'asc' ]] }); $('#filter-input').insertAfter('#myTable_length') }); function filterTable() { var input = document.getElementById("filter-input"), filter = input.value.toUpperCase(), table = document.getElementById("myTable"), tr = table.getElementsByTagName("tr"), td, kept; for (var i = 1; i < tr.length; i++) { td = tr[i].getElementsByTagName("td"); kept = false; for (var j = 0; j < td.length; j++) { if (td[j].innerHTML.toUpperCase().indexOf(filter) > -1) { tr[i].style.display = ""; kept = true; break; } } if (kept == false) { tr[i].style.display = "none"; } } } <% end %> diff --git a/app/views/jobs/_standard_fig_layout.html.erb b/app/views/jobs/_standard_fig_layout.html.erb index f98e4b0..c461410 100644 --- a/app/views/jobs/_standard_fig_layout.html.erb +++ b/app/views/jobs/_standard_fig_layout.html.erb @@ -1,102 +1,102 @@
    <% if @form_json['bin_levels'] %>
    <% l = @form_json['bin_levels'].map{|e| [e, e]} %> <%= select_tag 'current_level', options_for_select(l, session[:current_level].to_s), {:class => 'form-control multiselect figtool-multiselect'} %>
    <% end %>
    <%= render :partial => 'export_btn' %>
    <% if @description and !@description.empty? %>
    <% if @description.length() > 1 %> - +
      <% @description.each do |d| %>
    • <%= d %>
      • <% end %>
    <% else %> <%= @description.first %> <% end %>
    <% end %>
    -
    +
    -
    +
    <%= javascript_tag do %> $('#fig-description-btn').click(function(e) { $('#fig-description-btn').toggleClass('active'); $('#fig-description').toggleClass('inactive'); }); $("#current_level").change(function(){ var div = $('#show_content'); var url = '<%= view_job_path(:key => @job.key) %>?partial=<%= params[:partial] %>¤t_level=' + $(this).val(); $.ajax({ url: url, type: "get", beforeSend: function(){ }, success: function(returnData){ div.empty(); div.html(returnData); }, error: function(e){ } }); }); setMultiselect('#current_level'); <% end %> diff --git a/app/views/jobs/_view_adonis.html.erb b/app/views/jobs/_view_adonis.html.erb index 9c2e3e4..4df42c0 100644 --- a/app/views/jobs/_view_adonis.html.erb +++ b/app/views/jobs/_view_adonis.html.erb @@ -1,9 +1,11 @@ <%= render :partial => 'standard_fig_layout' %> <%= javascript_tag do %> + $("#view-title").html("perMANOVA") + var fig_data = <%= raw @data_json %>; adonisPieChart('fig', 'fig-legend', fig_data, W = 600, H = 600, font_family = "verdana, arial, helvetica, sans-serif"); <% end %> diff --git a/app/views/jobs/_view_change.html.erb b/app/views/jobs/_view_change.html.erb index d4203fb..85bc566 100644 --- a/app/views/jobs/_view_change.html.erb +++ b/app/views/jobs/_view_change.html.erb @@ -1,9 +1,11 @@ <%= render :partial => 'standard_fig_layout' %> <%= javascript_tag do %> + $("#view-title").html("Changes") + var fig_data = <%= raw @data_json %>; foldChange('fig', 'fig-legend', fig_data, W = 600, H = 600, font_family = "verdana, arial, helvetica, sans-serif", color_palette = d3.schemeCategory10); <% end %> diff --git a/app/views/jobs/_view_clustering.html.erb b/app/views/jobs/_view_clustering.html.erb index 65c1f80..27adc45 100644 --- a/app/views/jobs/_view_clustering.html.erb +++ b/app/views/jobs/_view_clustering.html.erb @@ -1,66 +1,64 @@ -
    -
    -
    <% if @form_json['bin_levels'] %>
    <% l = @form_json['bin_levels'].map{|e| [e, e]} %> <%= select_tag 'current_level', options_for_select(l, session[:current_level].to_s), {:class => 'form-control multiselect figtool-multiselect'} %>
    <% end %> +
    +
    + <%= @form_json['bin_levels'] ? 'Clustering details (level '+session[:current_level].to_s+')' : 'Clustering details' %> +
    +
    <% if @description and !@description.empty? %> -
    - - - - <% if @description.length() > 1 %>
      <% @description.each do |d| %>
    • <%= d %>
      • <% end %>
    <% else %> <%= @description.first %> <% end %> -     -
    <% end %> +
    +
    + -
    -
    <%= javascript_tag do %> +$("#view-title").html("Clustering") + $("#current_level").change(function(){ var div = $('#show_content'); var url = '<%= view_job_path(:key => @job.key) %>?partial=<%= params[:partial] %>¤t_level=' + $(this).val(); $.ajax({ url: url, type: "get", beforeSend: function(){ }, success: function(returnData){ div.empty(); div.html(returnData); }, error: function(e){ } }); }); setMultiselect('#current_level'); <% end %> diff --git a/app/views/jobs/_view_correlation_network.html.erb b/app/views/jobs/_view_correlation_network.html.erb index 5d854f5..80d4b50 100644 --- a/app/views/jobs/_view_correlation_network.html.erb +++ b/app/views/jobs/_view_correlation_network.html.erb @@ -1,9 +1,11 @@ <%= render :partial => 'standard_fig_layout' %> <%= javascript_tag do %> + $("#view-title").html("Correlation network") + var fig_data = <%= raw @data_json %>; correlationNetwork('fig', 'fig-legend', fig_data, W = 600, H = 600, font_family = "verdana, arial, helvetica, sans-serif", color_palette = d3.schemeCategory10); <% end %> diff --git a/app/views/jobs/_view_details.html.erb b/app/views/jobs/_view_details.html.erb new file mode 100644 index 0000000..9b76a03 --- /dev/null +++ b/app/views/jobs/_view_details.html.erb @@ -0,0 +1,25 @@ + +
      +
    • Creation: <%= @job.created_at.year.to_s[-2..-1].to_s + "-" + @job.created_at.month.to_s.rjust(2, '0') + "-" + @job.created_at.day.to_s.rjust(2, '0') + " " + @job.created_at.hour.to_s.rjust(2, '0') + ":" + @job.created_at.min.to_s.rjust(2, '0') + " " + @job.created_at.zone.to_s %> +
      • +
    • Last update: <%= @job.updated_at.year.to_s[-2..-1].to_s + "-" + @job.updated_at.month.to_s.rjust(2, '0') + "-" + @job.updated_at.day.to_s.rjust(2, '0') + " " + @job.updated_at.hour.to_s.rjust(2, '0') + ":" + @job.updated_at.min.to_s.rjust(2, '0') + " " + @job.updated_at.zone.to_s %> +
      • +
    • Owner: <%= @job.user.username %>
      • +
    • <%= @job.key %>
      • +
    + +
    +
    Description
    +
    + <%= (@job.description != '') ? @job.description : 'You did not enter any description for this analysis' %> +
    +
    + +<%= link_to serve_job_path(@job.key, :filename => 'output/stderr.log'), :title => 'Bugs report', :class => 'p-1 float-right' do %> + +<% end %> + + +<%= javascript_tag do %> + $("#view-title").html("Details") +<% end %> diff --git a/app/views/jobs/_view_diversity.html.erb b/app/views/jobs/_view_diversity.html.erb index 188b857..64da23b 100644 --- a/app/views/jobs/_view_diversity.html.erb +++ b/app/views/jobs/_view_diversity.html.erb @@ -1,11 +1,13 @@ <%= render :partial => 'standard_fig_layout' %> <%= javascript_tag do %> + $("#view-title").html("Diversity") + var fig_data = <%= raw @data_json %>, RfunctionsDiversity = <%= @h_form_choices['diversity'].to_json.html_safe %>; diversity('fig', 'fig-legend', fig_data, RfunctionsDiversity, W = 600, H = 600, font_family = "verdana, arial, helvetica, sans-serif", color_palette = d3.schemeCategory10); <% end %> diff --git a/app/views/jobs/_view_heatmap.html.erb b/app/views/jobs/_view_heatmap.html.erb index e1f1b0a..70b905e 100644 --- a/app/views/jobs/_view_heatmap.html.erb +++ b/app/views/jobs/_view_heatmap.html.erb @@ -1,16 +1,18 @@ <%= render :partial => 'standard_fig_layout' %> <%= javascript_tag do %> + $("#view-title").html("Heatmap") + var fig_data = <%= raw @data_json %>, cmax = 200, rmax = 200; if (fig_data.colnames.length > cmax || fig_data.rownames.length > rmax) { alert("Sorry, the heatmap is too big to be displayed:\n\ Its size is "+fig_data.colnames.length+"x"+fig_data.rownames.length+" but the max is "+cmax+"x"+rmax+"\n\n\ You can still download the PDF file from the Export button.") } else { heatMap('fig', 'fig-legend', fig_data, W = 750, H = 750, font_family = "verdana, arial, helvetica, sans-serif"); } <% end %> diff --git a/app/views/jobs/_view_map.html.erb b/app/views/jobs/_view_map.html.erb index f48e95f..8ae9bd6 100644 --- a/app/views/jobs/_view_map.html.erb +++ b/app/views/jobs/_view_map.html.erb @@ -1,5 +1,11 @@ <% if @final_json and @final_json[:status_by_step]['map'] %> <%= render :partial => 'list_messages', :locals => {:data => 'map'} %> <% end %> + +<%= javascript_tag do %> + + $("#view-title").html("Map") + +<% end %> diff --git a/app/views/jobs/_view_pca.html.erb b/app/views/jobs/_view_pca.html.erb index 8c8fc2b..e5ad65f 100644 --- a/app/views/jobs/_view_pca.html.erb +++ b/app/views/jobs/_view_pca.html.erb @@ -1,9 +1,11 @@ <%= render :partial => 'standard_fig_layout' %> <%= javascript_tag do %> + $("#view-title").html("PCA") + var fig_data = <%= raw @data_json %>; pca('fig', 'fig-legend', fig_data, W = 600, H = 600, font_family = "verdana, arial, helvetica, sans-serif", color_palette = d3.schemeCategory10); <% end %> diff --git a/app/views/jobs/_view_pcoa.html.erb b/app/views/jobs/_view_pcoa.html.erb index f8e626a..50cec52 100644 --- a/app/views/jobs/_view_pcoa.html.erb +++ b/app/views/jobs/_view_pcoa.html.erb @@ -1,9 +1,11 @@ <%= render :partial => 'standard_fig_layout' %> <%= javascript_tag do %> + $("#view-title").html("PCoA") + var fig_data = <%= raw @data_json %>; pcoa('fig', 'fig-legend', fig_data, W = 600, H = 600, font_family = "verdana, arial, helvetica, sans-serif", color_palette = d3.schemeCategory10); <% end %> diff --git a/app/views/jobs/_view_primary_dataset.html.erb b/app/views/jobs/_view_primary_dataset.html.erb index 44c9a78..48f81bb 100644 --- a/app/views/jobs/_view_primary_dataset.html.erb +++ b/app/views/jobs/_view_primary_dataset.html.erb @@ -1,5 +1,9 @@ <% if @final_json and @final_json[:status_by_step]['primary_dataset'] %> <%= render :partial => 'list_messages', :locals => {:data => 'primary_dataset'} %> <% end %> + +<%= javascript_tag do %> + $("#view-title").html("Primary dataset") +<% end %> diff --git a/app/views/jobs/_view_proportions.html.erb b/app/views/jobs/_view_proportions.html.erb index 5073bfe..74c619a 100644 --- a/app/views/jobs/_view_proportions.html.erb +++ b/app/views/jobs/_view_proportions.html.erb @@ -1,10 +1,12 @@ <%= render :partial => 'standard_fig_layout' %> <%= javascript_tag do %> + $("#view-title").html("Proportions") + var fig_data = <%= raw @data_json %>; barchart('fig', 'fig-legend', fig_data, W = 600, H = 600, font_family = "verdana, arial, helvetica, sans-serif", color_palette = d3.schemeCategory10); <% end %> diff --git a/app/views/jobs/_view_secondary_dataset.html.erb b/app/views/jobs/_view_secondary_dataset.html.erb index d9050e2..eab1bf3 100644 --- a/app/views/jobs/_view_secondary_dataset.html.erb +++ b/app/views/jobs/_view_secondary_dataset.html.erb @@ -1,6 +1,9 @@ <% if @final_json and @final_json[:status_by_step]['secondary_dataset'] %> <%= render :partial => 'list_messages', :locals => {:data => 'secondary_dataset'} %> <% end %> +<%= javascript_tag do %> + $("#view-title").html("Secondary dataset") +<% end %> diff --git a/app/views/jobs/_view_similarity_network.html.erb b/app/views/jobs/_view_similarity_network.html.erb index 4035c46..c8f7026 100644 --- a/app/views/jobs/_view_similarity_network.html.erb +++ b/app/views/jobs/_view_similarity_network.html.erb @@ -1,9 +1,11 @@ <%= render :partial => 'standard_fig_layout' %> <%= javascript_tag do %> + $("#view-title").html("Similarity network") + var fig_data = <%= raw @data_json %>; similarityNetwork('fig', 'fig-legend', fig_data, W = 600, H = 600, font_family = "verdana, arial, helvetica, sans-serif", color_palette = d3.schemeCategory10); <% end %> diff --git a/app/views/jobs/show.html.erb b/app/views/jobs/show.html.erb index 1478768..8dfcba6 100644 --- a/app/views/jobs/show.html.erb +++ b/app/views/jobs/show.html.erb @@ -1,200 +1,194 @@
    -
    <%= javascript_tag do %> $(document).ready(function() { - $('#job-description-btn').click(function(e) { - $('#job-description').toggleClass('hidden'); - }); - var window_width = $(window).width(); <% @h_tips['show'].each_with_index do |tip, index| %> <% if !current_user and !session["tip#{tip['id']}".to_sym] and tip['target'] %> var target = $("#<%= tip['target'] %>"), offset = target.offset(), tip = $('
    ', { id: "tip_window<%= index %>", class: "tip-window" }), tip_container = $("
    ", { class: "tip-window-container" }).append(tip), tip_text = $("

    ").html("<%= raw tip['html'] %>"), tip_icon = $('', { class: "fa fa-info-circle" }), tip_close = $('
    ', { class: "tip-window-close" }); tip_close.html(' Got it!') tip_close.click(function() { $("#tip_window<%= index %> p").html(''); $("#tip_window<%= index %>").css({display:'none'}); }) tip.css({ left: (window_width-offset.left < window_width/2) ? -260 : 0, }) tip.addClass((window_width-offset.left < window_width/2) ? "right" : "left"); tip.append(tip_icon); tip.append(tip_text); tip.append(tip_close); target.append(tip_container); <% session["tip#{tip['id']}".to_sym] = true %> <% end %> <% end %> var selected_view = $("#selected_view").val(); $("#menu-"+selected_view).addClass('selected'); }); $(".menu").click(function(){ var name = this.id.split("-")[1]; $("#selected_view").val(name); $(".menu").removeClass('selected'); $(this).addClass('selected'); }); var timer = setInterval(function(){ refresh_data() }, 5000); function refresh_data(){ var url = '<%= refresh_job_path(:key => @job.key) %>'; $.ajax({ url: url, type: "get", dataType: "script", beforeSend: function(){ }, success: function(returnData){ // div.empty(); // div.html(returnData); }, error: function(e){ } }); } $(".menu").click(function(){ var t = this.id.split("-"); var div = $("#show_content"); var url = '<%= view_job_path(:key => @job.key) %>?partial=' + t[1]; $.ajax({ url: url, type: "get", beforeSend: function(){ }, success: function(returnData){ div.empty(); div.html(returnData); }, error: function(e){ } }); }); refresh_data(); <% end %> diff --git a/app/views/layouts/application.html.erb b/app/views/layouts/application.html.erb index 688d4a7..d8136df 100644 --- a/app/views/layouts/application.html.erb +++ b/app/views/layouts/application.html.erb @@ -1,155 +1,155 @@ <%= Rails.application.class.parent_name %> <%= stylesheet_link_tag "application", media: "all", "data-turbolinks-track" => true %> <%= javascript_include_tag "application", "data-turbolinks-track" => true %> <%= csrf_meta_tags %> -