SCIENTIFIC FILESYSTEM

Examples: Snakemake and the Scientific Filesystem

Tue, 30 Jan 2018 11:13:01 +0000

Here is a story of snakemake, the workflow manager that could, and his partner in crime, the Scientific Filesystem!

What do you get when you combine a Scientific Filesystem, an organized, programmatically accessible, and discoverable specification for scientific applications, and Snakemake, a workflow management system is a tool to create scalable data analyses? In a container? Reproducibility, of course! Here we will go through a small tutorial to generate:

a scientific filesystem
in a container (Docker and Singularity!)
with Snakemake as a workflow manager.

Overview

The Scientific Filesystem (SCIF) is a host (or container) agnostic specification for a filesystem organization, set of environment variables, and functions to control them to make scientific applications discoverable, and easy to use. We are installing a SCIF via a shared SCIF recipe file, snakemake_tutorial.scif that will install your scientific applications bwa, samtools, and the workflow management engine Snakemake workflow in both Docker and Singularity containers. If we need to make changes, we can just edit the one file. For a quick look at how the same commands are run between Docker and Singularity - below we have a Singularity container called snakemake.simg and a docker container vanessa/snakemake.scif. We will first start with a side by side quickstart, and then for interested readers, go through the details of each one with building separately for Singularity and then Docker.

Get recipe, workflow and test data

The repository contains all you need to reproduce the following tutorial. It contains the SCIF recipe file, the Snakemake tutorial workflow, and test data.

git clone https://github.com/sci-f/snakemake.scif.git

Containers Side by Side

In summary:

Two containers! One Recipe!

Building containers

sudo singularity build snakemake.simg Singularity

docker build -t vanessa/snakemake.scif .

Run the workflow

The Snakefile specifies rules that should be executed. State a target file and Snakemake will build a directed acyclic graph (DAG) from the rules until the target file is reached. The rules are then executed to create the target file. The Snakefile hides the details of the environment provided by SCIF in the container as you can see from the commands below. All commands can be executed from outside the container or from within the container.

In all cases we bind the example dataset to the data directory provided by SCIF. SCIF provides the environment in which the workflow steps are executed. The workflow steps themself and therefore the interaction with SCIF is implemented in the Snakefile.

Inside container, Singularity

singularity shell --bind data/:/scif/data snakemake.simg
snakemake all

Inside container, Docker

To run the whole workflow we need the Snakefile which is not part of the container. To access the Snakefile from within the Docker container, we need an additional mount.

docker run -v $PWD/data:/scif/data:z -v $PWD:/working_dir -it -w /working_dir --entrypoint /bin/bash vanessa/snakemake.scif
snakemake all

Outside Container, Singularity

singularity exec --bind data/:/scif/data snakemake.simg snakemake all

Outside Container, Docker

Because Docker does not mount the current working directory, we mount it to a the SCIF data directory of the SCIF snakemake app. The snakemake app will execute snakemake in /scif/data/snakemake, thus snakemake can find the Snakefile.

docker run -v $PWD/data:/scif/data:z -v $PWD:/scif/data/snakemake:z -it vanessa/snakemake.scif run snakemake all

Generate graphical representation of the workflow

The whole workflow should finish within 5 seconds … too fast to realize what is happening. A visualization of the whole workflow we just ran would be nice …

The SCIF app graphviz_create_dag generates a directed acyclic graph of the workflow execution plan. This is a feature of Snakemake, but needs Graphviz as additional dependency. Therefore Graphviz is installed in the appinstall section of this app.

rm -r data/calls/ /data/mapped_reads/ data/sorted_reads/ data/report.html

Now we can generate the graph of the execution plan. The plan is computed from the Snakefile in our current working directory and the target file we specify. In this case we want to create the final report file report.html. The directed acyclic graph of the workflow should be saved atdata/dag.svg. The commands for Singularity and Docker are as follows:

Singularity

singularity run --bind data/:/scif/data snakemake.simg run graphviz_create_dag $PWD report.html dag.svg
> [graphviz_create_dag] executing /bin/bash /scif/apps/graphviz_create_dag/scif/runscript /home/fbartusch/github/snakemake_tutorial report.html dag.svg
> Building DAG of jobs...

Docker

In contrast to Singularity, Docker cannot access the current working directory. Therefore we mount the current working directory to the SCIF data folder of the graphviz_create_dag app.

docker run -v $PWD/data:/scif/data:z -v $PWD:/scif/data/graphviz_create_dag:z -it vanessa/snakemake.scif run graphviz_create_dag /scif/data/graphviz_create_dag report.html dag.svg
> [graphviz_create_dag] executing /bin/bash /scif/apps/graphviz_create_dag/scif/runscript /scif/data/graphviz_create_dag report.html dag.svg
> Building DAG of jobs...

Map with bwa mem

Instead of running the whole workflow at once with Snakemake, we can also use the SCIF environment to execute computations. The following examples also illustrate special SCIF syntax for working with environment variables or piping output to a file. First, we map the reads with bwa mem to the reference genome.

Inside container, Singularity

Note the use of [e] so we can easily pass the environment variable to the SCIF. Otherwise, it would be evaluated on the host.

singularity shell --bind data/:/scif/data snakemake.simg
mkdir /scif/data/mapped_reads
scif run bwa mem -o [e]SCIF_DATA/mapped_reads/A.sam [e]SCIF_DATA/genome.fa [e]SCIF_DATA/samples/A.fastq

Inside container, Docker

docker run -v $PWD/data:/scif/data:z -it --entrypoint /bin/bash vanessa/snakemake.scif
mkdir -p /scif/data/mapped_reads
scif run bwa mem -o [e]SCIF_DATA/mapped_reads/A.sam [e]SCIF_DATA/genome.fa [e]SCIF_DATA/samples/A.fastq

Outside container, Singularity

mkdir -p data/mapped_reads
singularity run --bind data:/scif/data snakemake.simg run bwa mem -o [e]SCIF_DATA/mapped_reads/A.sam [e]SCIF_DATA/genome.fa [e]SCIF_DATA/samples/A.fastq

Outside the container, Docker

mkdir -p data/mapped_reads
docker run -v $PWD/data:/scif/data:z vanessa/snakemake.scif run bwa mem -o [e]SCIF_DATA/mapped_reads/A.sam [e]SCIF_DATA/genome.fa [e]SCIF_DATA/samples/A.fastq

Sam to Bam Conversion

Inside the container, Singularity

Note the use of [out] as a substitute for >. If you wanted to use > you could put the entire thing in quotes.

singularity shell --bind data/:/scif/data snakemake.simg
scif run samtools view -Sb [e]SCIF_DATA/mapped_reads/A.sam [out] [e]SCIF_DATA/mapped_reads/A.bam    # or
scif run samtools 'view -Sb $SCIF_DATA/mapped_reads/A.sam > $SCIF_DATA/mapped_reads/A.bam'
> [samtools] executing /bin/bash /scif/apps/samtools/scif/runscript view -Sb $SCIF_DATA/mapped_reads/A.sam > $SCIF_DATA/mapped_reads/A.bam

Inside the container, Docker

docker run -v $PWD/data:/scif/data:z -it --entrypoint /bin/bash vanessa/snakemake.scif
scif run samtools view -Sb [e]SCIF_DATA/mapped_reads/A.sam [out] [e]SCIF_DATA/mapped_reads/A.bam    # or
scif run samtools 'view -Sb $SCIF_DATA/mapped_reads/A.sam > $SCIF_DATA/mapped_reads/A.bam'
> [samtools] executing /bin/bash /scif/apps/samtools/scif/runscript view -Sb $SCIF_DATA/mapped_reads/A.sam > $SCIF_DATA/mapped_reads/A.bam

Outside the container, Singularity

singularity run --bind data:/scif/data snakemake.simg run samtools view -Sb [e]SCIF_DATA/mapped_reads/A.sam [out] [e]SCIF_DATA/mapped_reads/A.bam   # or
singularity run --bind data:/scif/data snakemake.simg run samtools 'view -Sb $SCIF_DATA/mapped_reads/A.sam > $SCIF_DATA/mapped_reads/A.bam'

Outside the container, Docker

docker run -v $PWD/data:/scif/data vanessa/snakemake.scif run samtools view -Sb [e]SCIF_DATA/mapped_reads/A.sam [out] [e]SCIF_DATA/mapped_reads/A.bam   # or
docker run -v $PWD/data:/scif/data vanessa/snakemake.scif run samtools 'view -Sb $SCIF_DATA/mapped_reads/A.sam > $SCIF_DATA/mapped_reads/A.bam'

Interactive development

This can be done for Docker or Singularity, just with different commands to shell into the container!

docker run -it -v $PWD/data:/scif/data:z vanessa/snakemake.scif pyshell
singularity run --bind data/:/scif/data snakemake.simg pyshell

Found configurations for 4 scif apps
bwa
graphviz_create_dag
samtools
snakemake
[scif] /scif bwa | graphviz_create_dag | samtools | snakemake
Python 3.6.2 |Anaconda, Inc.| (default, Sep 22 2017, 02:03:08) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> 

# Activate bwa 
client.activate('samtools')

# Environment variables active
client.environment

# Run bwa interactively
args = ['mem', '-o', '[e]SCIF_DATA/mapped_reads/a.sam', '[e]SCIF_DATA/genome.fa', '[e]SCIF_DATA/samples/A.fastq']
client.run('bwa', args=args)

# Run sam--bam interactively
args = ["view", "-Sb", "/scif/data/mapped_reads/r1_subset.sam", ">", "/scif/data/mapped_reads/r1_subset.bam"]
client.run('samtools', args=args)

See the Github repository snakemake.scif if you have any questions or issues.

Examples: CarrierSeq: sequence analysis workflow with SCIF

Mon, 29 Jan 2018 17:12:01 +0000

Some time ago we featured the CarrierSeq workflow run with Singularity. At the time, SCIF was available only for Singularity, and with great feedback from reviewers and the community, SCIF (the Scientific Filesystem) was re-fashioned to be install-able in any container technology (Docker or Singularity) or on your host!

Just to hammer it into your brain, I’ll remind you what the Scientific Filesystem, referred to as SCIF, is. SCIF is a specification for a filesystem organizational, a set of environment variables, and functions that control the two in order to optimize the usability and discoverability of scientific applications. Thus, in this second tutorial we are again going to install a Scientific Filesystem into a Docker container and the same one into a Singularity container using the same scif recipe file.

Overview

The Scientific Filesystem (SCIF) is a host (or container) agnostic specification for a filesystem organization, set of environment variables, and functions to control them to make scientific applications discoverable, and easy to use. We are installing a SCIF via a shared SCIF recipe file, carrierseq.scif that defines applications for mapping, poisson, and other helpers into both a Docker and Singularity container. If we need to make changes, we can just edit the one file. For each, we will build a development container (with tools like samtools and bwa exposed as entrypoints) and a pipeline production container (to put those commands together to run a step like “mapping.” For a quick look at how the same commands are run between Docker and Singularity - below we have a Singularity container called cseq and a docker container vanessa/cseq. For the detailed walkthroughs, jump down to Details.

CarrierSeq Pipeline Container

List apps in container

$ ./cseq apps
$ docker run vanessa/cseq apps

  download
      help
   mapping
   poisson
    readme
 reference
   sorting

Ask for help for the application called “readme”

$ ./cseq help readme
$ docker run vanessa/cseq help readme

Print the repository’s README.md to the console

$ ./cseq run readme
$ docker run vanessa/cseq run readme

#### CarrierSeq
#### About

bioRxiv doi: https://doi.org/10.1101/175281

CarrierSeq is a sequence analysis workflow for low-input nanopore
            sequencing which employs a genomic carrier.

           Github Contributors: Angel Mojarro (@amojarro), 
                                Srinivasa Aditya Bhattaru (@sbhattaru), 
                                Christopher E. Carr (@CarrCE), 
                                and Vanessa Sochat (@vsoch).
 
fastq-filter from: https://github.com/nanoporetech/fastq-filter

[MORE]

Inspect the container

$ ./cseq inspect
$ docker run vanessa/cseq inspect

or a single scientific application

$ cseq inspect mapping
$ docker run vanessa/cseq inspect mapping

Run three entrypoints in a row, and bind a data directory

$ singularity run --bind data:/scif/data cseq run mapping
$ singularity run --bind data:/scif/data cseq run poisson
$ singularity run --bind data:/scif/data cseq run sorting

$ docker run -v $PWD/data:/scif/data vanessa/cseq run mapping
$ docker run -v $PWD/data:/scif/data vanessa/cseq run poisson
$ docker run -v $PWD/data:/scif/data vanessa/cseq run sorting

Create an interactive session with an application context

$ ./cseq shell mapping
$ docker run -it vanessa/cseq shell mapping

CarrierSeq Development Container

List applications

$ docker run vanessa/cseq:dev apps
$ ./cseq-dev apps
       bwa
    fqtrim
      help
    python
     seqtk
sra-toolkit

Open interactive python

$ ./cseq-dev run python
$ docker run -it vanessa/cseq:dev run python
[python] executing /bin/bash /scif/apps/python/scif/runscript
Python 2.7.9 (default, Jun 29 2016, 13:08:31) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Load container with bwa on path

$ ./cseq-dev shell bwa
$ docker run -it vanessa/cseq:dev shell bwa
[bwa] executing /bin/bash 
$ which bwa
$ /scif/apps/bwa/bin/bwa

Details

This tutorial assumes that you are familiar with and have Docker installed on your machine. You also need to download some data files to use with your container. If you aren’t familar with genomic analysis (as I’m not) you will be utterly confused about how to download data. The reference is provided in the “reference” folder of the repository, but the input data isn’t. The container was originally designed to download it’s own data, but due to a change in the sri-toolkit the original data is provided as a manual download link from Dropbox.

The SCIF Recipe

Let’s take a look at a (truncated) application definition in the scif carrierseq recipe, carrierseq.scif.

%appinstall mapping
    # Install bwa
    git clone https://github.com/lh3/bwa.git build
    cd build && git checkout v0.7.15
    make
    mv -t ../bin bwa bwakit   
    # Install fqtrim
    cd .. 
    wget https://ccb.jhu.edu/software/fqtrim/dl/fqtrim-0.9.5.tar.gz
    tar xvfz fqtrim-0.9.5.tar.gz && cd fqtrim-0.9.5 && make release
    mv fqtrim ../bin

%appenv mapping
    all_reads=$SCIF_DATA/all_reads.fastq
    bwa_threads=${CSEQ_BWATHREADS:-1}
    q_score=${CSEQ_QSCORE:-9}
    output_folder=$SCIF_DATA
    export all_reads reference_genome bwa_threads q_score output_folder

%appfiles mapping
    /python/quality_score_filter.py bin/quality_score_filter.py
%apprun mapping
    source $SCIF_APPENV_reference

    # Create all input directories
    mkdir -p $SCIF_DATA/00_bwa # map all reads to carrier reference genome
...

We have sections like these for each internal module (a SCIF application) that we want to create. The above is for the application “mapping.” These definitions are in the file carrierseq.scif and when we create the SCIF in the container it starts empty. We then install the recipe to the container. Here is the recipe ( carrierseq.scif) and installing it to (any) container comes down to installing the scif client, and pointing it at the recipe.

pip install scif
scif install carrierseq.scif

After installation, all the steps in appinstalls are performed, the various metadata, environments, and entry point run scripts are created, and we are ready to go! Let’s see what this looks like when done in a Singularity and Docker container.

CarrierSeq Scientific Filesystem (Singularity)

Singularity is a container, similar to Docker, that is secure to run in HPC environments. By way of using a Scientific Filesystem (SCIF) with Singularity, we have a lot of freedom in deciding on what level of functions we want to expose to the user. A developer will want easy access to the core tools (e.g., bwa, seqtk) while a user will likely want one level up, on the level of a collection of steps associated with some task (e.g., mapping). We will walk through the steps of building and using each one.

Setup

You first need to install Singularity. For this tutorial, we are using the development branch with the current version 2.4.x. You can install it doing the following.

git clone -b development https://www.github.com/singularityware/singularity.git
cd singularity
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install

Now you are ready to go! If the above steps tell you to install any missing dependencies, you should do that.

Build the image

Building looks like this:

sudo singularity build cseq Singularity

This will build a read only (essentially frozen) image, so your pipeline is forever preserved.

Exploring the Container

If you didn’t know anything about the image, you would want to explore it. SCIF provides easy commands to do that. When we just execute the container, the SCIF shows us it’s default help:

./cseq 
[help] executing /bin/bash /scif/apps/help/scif/runscript
    CarrierSeq is a sequence analysis workflow for low-input nanopore
    sequencing which employs a genomic carrier.
    Github Contributors: Angel Mojarro (@amojarro),
                         Srinivasa Aditya Bhattaru (@sbhattaru),
                         Christopher E. Carr (@CarrCE),
                         Vanessa Sochat (@vsoch).
    fastq-filter from: https://github.com/nanoporetech/fastq-filter
    To see applications installed in the Scientific Filesystem:
    scif apps
    To run a typical pipeline, you might do:
    scif run mapping
    scif run poisson
    scif run sorting
    If you install in a container, the entrypoint should be scif, and then
    issue the above commands to it.

SCI-F Apps

You can see apps in the image, as instructed:

./cseq apps
  download
      help
   mapping
   poisson
    readme
 reference
   sorting

or see help for a specific app. For example, readme exists only to capture and then print the entire readme of the repository (or any repository container README.md for which the SCIF is installed for) to the console:

./cseq run readme | tail
The matrix illustrates the reads/channel distribution of B. subtilis, contamination, and HQNRs across all 512 nanopore channels. Here we are able to visually identify overly productive channels (e.g., 191 reads/channel, etc) producing likely HQNRs.
![alt text](https://github.com/amojarro/carrierseq/blob/master/example/carrierseq_roi_q9_p005.png)

### HQNR Pore Occupancy
“Bad” channels identified by CarrierSeq as HQNR-associated (reads/channel > 7).
![alt text](https://github.com/amojarro/carrierseq/blob/master/example/carrierseq_hqnrs_q9_p005.png)

### Target Reads Pore Occupancy
“Good” channels identified by CarrierSeq as non-HQNR-associated (reads/channel ≤ 7). Channels producing 6 or more reads yield HQNRs that have satisfied our CarrierSeq parameters. By imposing a stricter p-value, CarrierSeq may be able to reject more HQNRs (e.g., xcrit = 5).
![alt text](https://github.com/amojarro/carrierseq/blob/master/example/carrierseq_target_reads_q9_p005.png)

And then we can ask for help for any of the pipeline steps:

./cseq help download

We can also look at metadata for the entire image, or for an app. The inspect command can expose environment variables, labels, the definition file, tests, and runscripts.

# Returns a large data structure with all apps
./cseq inspect

# Returns for a particular app
./cseq inspect download
{
    "download": {
        "apphelp": [
            "   The original sra-toolkit does not serve the correct data, so for now you ",
            "   should download data from ",
            "   https://www.dropbox.com/sh/vyor82ulzh7n9ke/AAC4W8rMe4z5hdb7j4QhF_IYa?dl=0) and then move into some data folder you intend to mount:",
            "      mv $HOME/Downloads/all_reads.fastq data/"
        ]
    }
}

Overall Strategy

Since the data is rather large, we are going to map a folder to our $PWD where the analysis is to run. This directory, just like the modular applications, has a known and predictable location. So our steps are going to look like this:

# 0. Make an empty random folder to bind to for data.
mkdir data

# 1. Download data, map the data base to an empty folder on our local machine
#     (we actually will do this from the browser as the sri toolkit is changed.
# 2. Perform mapping step of pipeline, mapping the same folder.
# 3. perform poisson regression on filtered reads
# 4. Finally, sort results

# Download data from https://www.dropbox.com/sh/vyor82ulzh7n9ke/AAC4W8rMe4z5hdb7j4QhF_IYa?dl=0
# See issue --> https://github.com/amojarro/carrierseq/issues/1
cseq="singularity run --bind data:/scif/data cseq"

# $cseq run <app>

CarrierSeq Pipeline

The common user might want access to the larger scoped pipeline that the software provides. In the case of CarrierSeq, this means (optionally, download) mapping, poisson, and then sorting. If the image isn’t provided (e.g., a Singularity Registry or Singularity Hub) the user can build from the build recipe file, Singularity.devel. If you haven’t done this already:

sudo singularity build cseq Singularity.devel

The author is still working on updating the automated download, for now download from here and then move into some data folder you intend to mount:

mv $HOME/Downloads/all_reads.fastq data/

Let’s be lazy and put the bind and singularity command into an environment variable so we don’t need to type it many times.

cseq="singularity run --bind data:/scif/data cseq"

And then the various steps of the pipeline are run as was specified above:

$cseq run mapping
$cseq run poisson
$cseq run sorting

There are even helper functions that the main ones use to get various environments. For example, the reference app serves only to return the path to the reference genome!

$ ./cseq exec reference echo [e]reference_genome
[reference] executing /bin/echo $reference_genome
$SCIF_APPROOT_reference/lambda_ecoli.fa

or run quietly

$ ./cseq --quiet exec reference echo [e]reference_genome
$SCIF_APPROOT_reference/lambda_ecoli.fa

and what is that evaluated to?

./cseq --quiet exec reference echo [e]SCIF_APPROOT_reference/lambda_ecoli.fa
/scif/apps/reference/lambda_ecoli.fa

Cool!

1. Mapping

To run mapping, bind the data folder to the image, and specify the app to be mapping:

singularity run --bind data:/scif/data cseq run mapping

2. Poisson

singularity run --bind data:/scif/data cseq run poisson

3. Sorting

singularity run --bind data:/scif/data cseq run sorting

How Can I Change It?

Given two containers that share the same input and output schema, I could swap out of the steps:

...
singularity run --bind data:/scif/data container1 run sorting
singularity run --bind data:/scif/data container2 run sorting

or even provide a single container with multiple options for the same step

...
singularity run --bind data:/scif/data cseq run sorting1
singularity run --bind data:/scif/data cseq run sorting2

As a user, you want a container that exposes enough metadata to run different steps of the pipeline, but you don’t want to need to know the specifics of each command call or path within the container. In the above, I can direct the container to my mapped input directory and specify a step in the pipeline, and I dont need to understand how to use bwa or grep or seqtk, or any of the other software that makes up each.

Interactive Environment

If you are curious about all the Scientific Filesystem variables available to you:

$cseq exec mapping env | grep SCIF_

and read the specification for a complete list.

Or if you wanted an interactive shell to explore running commands inside the container:

singularity shell --bind data:/scif/data cseq

or enter a shell in context of a scif application (mapping)

$ singularity run cseq shell mapping
$ echo $SCIF_APPNAME
mapping

CarrierSeq Development

The developer has a different use case - to have easy command line access to the lowest level of executables installed in the container. Given a global install of all software, without SCI-F I would need to look at $PATH to see what has been added to the path, and then list executables in path locations to find new software installed to, for example, /usr/bin. There is no way to easily and programatically “sniff” a container to understand the changes, and the container is a black development box, perhaps only understood by the creator or with careful inspection of the build recipe.

In this use case, we want to build the development container.

sudo singularity build carrierseq.dev.img Singularity.devel

Now when we look at apps, we see all the core software that can be combined in specific ways to produce a pipeline step like “mapping”.

singularity apps carrierseq.dev.img
bwa
fqtrim
python
seqtk
sra-toolkit

each of which might be run, exec to activate the app environment, or shell to shell into the container under the context of a specific app:

# Open interactive python
singularity run --app python carrierseq.dev.img

# Load container with bwa on path
$ singularity shell --app bwa carrierseq.dev.img
$ which bwa
$ /scif/apps/bwa/bin/bwa

These two images that serve equivalent software is a powerful example of the flexibility of SCI-F. The container creator can choose the level of detail to expose to a user that doesn’t know how it was created, or perhaps has varying levels of expertise. A lab that is using core tools for working with sequence data might have preference for the development container, while a finalized pipeline distributed with a publication would have preference for the first. In both cases, the creator doesn’t need to write custom scripts for a container to run a particular app, or to expose environment variables, tests, or labels. By way of using SCI-F, this happens automatically.

CarrierSeq Scientific Filesystem (Docker)

Build the image

Building looks like this. Notice that we aren’t using the primary Dockerfile](https://github.com/vsoch/carrierseq/blob/master/Dockerfile) in the repository, we are using the Dockerfile.scif that is the version for the Scientific Filesystem.

docker build -f Dockerfile.scif -t vanessa/cseq .

Exploring the Container

If you didn’t know anything about the image, you would want to explore it. SCIF provides easy commands to do that. Let’s also define a quicker entry point to using the container:

cseq="docker run vanessa/cseq"

If we didn’t know anything and executed the container, we would see the scif client, and it would give us the following commands.

$cseq

SCI-F Apps

You can see apps in the image:

$cseq apps
  download
      help
   mapping
   poisson
    readme
 reference
   sorting

$cseq run readme | tail
The matrix illustrates the reads/channel distribution of B. subtilis, contamination, and HQNRs across all 512 nanopore channels. Here we are able to visually identify overly productive channels (e.g., 191 reads/channel, etc) producing likely HQNRs.
![alt text](https://github.com/amojarro/carrierseq/blob/master/example/carrierseq_roi_q9_p005.png)

### HQNR Pore Occupancy
"Bad" channels identified by CarrierSeq as HQNR-associated (reads/channel > 7).
![alt text](https://github.com/amojarro/carrierseq/blob/master/example/carrierseq_hqnrs_q9_p005.png)

### Target Reads Pore Occupancy
"Good" channels identified by CarrierSeq as non-HQNR-associated (reads/channel ≤ 7). Channels producing 6 or more reads yield HQNRs that have satisfied our CarrierSeq parameters. By imposing a stricter p-value, CarrierSeq may be able to reject more HQNRs (e.g., xcrit = 5).
![alt text](https://github.com/amojarro/carrierseq/blob/master/example/carrierseq_target_reads_q9_p005.png)

And then we can ask for help for any of the pipeline steps:

$cseq help download

We can also look at metadata for the entire image, or for an app. The inspect command can expose environment variables, labels, the definition file, tests, and runscripts.

# Returns a large data structure with all apps
$cseq inspect

# Returns for a particular app
$cseq inspect download
{
    "download": {
        "apphelp": [
            "   The original sra-toolkit does not serve the correct data, so for now you ",
            "   should download data from ",
            "   https://www.dropbox.com/sh/vyor82ulzh7n9ke/AAC4W8rMe4z5hdb7j4QhF_IYa?dl=0) and then move into some data folder you intend to mount:",
            "      mv $HOME/Downloads/all_reads.fastq data/"
        ]
    }
}

Overall Strategy

# 0. Make an empty random folder to bind to for data.
mkdir data

Download data, map the data base to an empty folder on our local machine (we actually will do this from the browser as the sri toolkit is changed.
Perform mapping step of pipeline, mapping the same folder.
perform poisson regression on filtered reads
Finally, sort results

Let’s add a volume to our command:

cseq="docker run -v $PWD/data:/scif/data vanessa/cseq"

CarrierSeq Pipeline

docker build -t vanessa/cseq .

The author is still working on updating the automated download, for now download from here and then move into some data folder you intend to mount:

mv $HOME/Downloads/all_reads.fastq data/

Let’s be lazy and put the bind and singularity command into an environment variable so we don’t need to type it many times.

cseq="docker run -v $PWD/data:/scif/data vanessa/cseq"

And then the various steps of the pipeline are run as was specified above:

$cseq run mapping
$cseq run poisson
$cseq run sorting

1. Mapping

To run mapping, bind the data folder to the image, and specify the app to be mapping:

cseq="docker run -v $PWD/data:/scif/data vanessa/cseq"
$cseq run mapping

2. Poisson

cseq="docker run -v $PWD/data:/scif/data vanessa/cseq"
$cseq run poisson

3. Sorting

cseq="docker run -v $PWD/data:/scif/data vanessa/cseq"
$cseq run sorting

How Can I Change It?

Given two containers that share the same input and output schema, I could swap out of the steps:

...
docker run -v $PWD/data:/scif/data vanessa/cseq run sorting
docker run -v $PWD/data:/scif/data vanessa/another run sorting

or even provide a single container with multiple options for the same step

...
docker run -v $PWD/data:/scif/data vanessa/cseq run sorting1
docker run -v $PWD/data:/scif/data vanessa/cseq run sorting2

Interactive Environment

If you are curious about all the Scientific Filesystem variables available to you:

$CSEQ exec mapping env | grep SCIF_

Or if you wanted an interactive shell to explore running commands inside the container:

singularity shell --bind data:/scif/data cseq

or enter a shell in context of a scif application (mapping)

$ singularity run cseq shell mapping
$ echo $SCIF_APPNAME
mapping

CarrierSeq Development

In this use case, we want to build the development container.

$ docker build -f Dockerfile.devel -t vanessa/cseq:dev .

We can run the container and get some default help printed

docker run vanessa/cseq:dev
[help] executing /bin/bash /scif/apps/help/scif/runscript
    This is a development image for CarrierSeq, meaning that it exposes the
    underlying tools (bwa, samtools) as applications in the Scientific
    Filesystem.
    If you want to run the CarrierSeq workflow, use the carrierseq.scif recipe.
    Github Contributors: Angel Mojarro (@amojarro),
                         Srinivasa Aditya Bhattaru (@sbhattaru),
                         Christopher E. Carr (@CarrCE),
                         Vanessa Sochat (@vsoch).
    To see applications installed in the Scientific Filesystem:
    scif apps
    To run a tool, you might do:
    scif run bwa
    scif run samtools
    Or get help or metadata
    scif help bwa
    scif inspect bwa
    If you install in a container, the entrypoint should be scif, and then
    issue the above commands to it.

Now when we look at apps, we see all the core software that can be combined in specific ways to produce a pipeline step like “mapping”.

$ docker run vanessa/cseq:dev apps
bwa
fqtrim
python
seqtk
sra-toolkit

each of which might be run, exec to activate the app environment, or shell to shell into the container under the context of a specific app:

# Open interactive python
$ docker run -it vanessa/cseq:dev run python

# Load container with bwa on path
$ docker run -it vanessa/cseq:dev shell bwa
$ which bwa
$ /scif/apps/bwa/bin/bwa

These two images that serve equivalent software is a powerful example of the flexibility of SCIF. The container creator can choose the level of detail to expose to a user that doesn’t know how it was created, or perhaps has varying levels of expertise. A lab that is using core tools for working with sequence data might have preference for the development container, while a finalized pipeline distributed with a publication would have preference for the first. In both cases, the creator doesn’t need to write custom scripts for a container to run a particular app, or to expose environment variables, tests, or labels. By way of using SCIF, this happens automatically.

See the original repository CarrierSeq here or the Singularity CarrierSeq here.

Examples: An Introduction to SCI-F Apps

Mon, 29 Jan 2018 15:15:01 +0000

In this tutorial, we are going to get started with the Scientific Filesystem (SCIF), which is a specification for a filesystem organizational, a set of environment variables, and functions that control the two in order to optimize the usability and discoverability of scientific applications. Specifically, we will build a “foobar” container, and a “cowsay” container that prints colored fortunes.</a>. We don’t even have to make containers (we can install on the host) but we will for this tutorial to keep the applications isolated from the host.

Why do we need SCIF?

A Scientific Filesystem (SCIF) provides entrypoints, metadata, and environments that are separate from the traditional host filesystem. When used in the context of containers, SCIF provides internal modularity. For example, installing a set of libraries, defining environment variables, or adding lables that belong to application foo in the SCIF makes a strong assertion that those dependencies belong to foo. When I run foo, I can be confident that the container is running in this context, meaning with foo’s custom environment, and with foo’s libraries and executables on the path. This is drastically different from serving many executables in a single container, because there is no way to know which are associated with which of the container’s intended functions. For example, let’s take a look at this series of steps to install dependencies for software foo and bar.

%post

# install dependencies 1
# install software A (foo)
# install software B (bar)
# install software C (foo)
# install software D (bar)

The creator may know that A and C were installed for foo and B and D for bar, but down the road, when someone discovers the container, if they can find the software at all, the intention of the container creator would be lost. As many are now, containers without any form of internal organization and predictibility are black boxes. We don’t know if some software installed to /opt, or to /usr/local/bin, or to their custom favorite folder /code. We could assume that the creator added important software to the path and look in these locations, but that approach is still akin to fishing in a swamp. We might only hope that the container’s main entry point is enough to make the container perform as intended.

Mixed up Modules

If your container truly runs one script, the traditional model of having a single entry point works well. Even in the case of having two functions like foo and bar you probably have something like this.

%runscript

if some logic to choose foo:
   check arguments for foo
   run foo
else if some logic to choose bar:
   run bar

and maybe your environment looks like this:

%environment
    BEST_GUY=foo
    export BEST_GUY

but what if you run into this issue, with foo and bar?

%environment
    BEST_GUY=foo
    BEST_GUY=bar
    export BEST_GUY

You obviously can’t have them at separate times. You’d have to source some custom environment file (that you make on your own) and it gets hard easily with issues of using shell and sourcing. We don’t know who the best guy is! You probably get the general idea. Without internal organization and modularity:

You have to do a lot of manual work to expose the different software to the user via a custom runscript (and be a generally decent programmer).
All software must share the same metadata, environment, and labels.

Under these conditions, containers are at best block boxes with unclear delineation between software provided, and only one context of running anything. The container creator shouldn’t need to spend inordinate amounts of time writing custom runscripts to support multiple functions and inputs. Each of foo and bar should be easy to define, and have it’s own runscript, environment, labels, tests, and help section.

Container Transparency

Using a SCIF makes foo and bar transparent, and solve this problem of mixed up modules. Our simple issue of mixed up modules could be solved if we could separate them like this:

%appenv foo
    BEST_GUY=foo
    export BEST_GUY

%appenv bar
    BEST_GUY=bar
    export BEST_GUY

%apprun foo
    echo The best guy is $BEST_GUY

%apprun bar
    echo The best guy is $BEST_GUY

In the above, each chunk that starts with %app delineates a particular section. The section %appenv is for environment variables, and %apprun is the entrypoint for the application of interest. Which application do they belong to? The name comes after the section name, (%apprun foo). Importantly, each application has its own modular location. When you do an %appinstall foo, the commands are all done in context of that base. The organization looks like this:

/scif/apps/

     foo/
        bin/
        lib/
        scif/
            runscript.help
            runscript
            environment.sh

     bar/

     ....

The bin and lib are also automatically generated. The really cool thing is that we can define an application with many different strategies. An application could be…

A single file to run! If I just add a file to a bin, make it executable, and then make it the entrypoint, I’m done.

%appfiles foo
    runfoo.sh   bin/runfoo.sh

%appinstall foo
    chmod u+x bin/runfoo.sh

%apprun foo
    exec foo

You don’t even need files! You could just do this.

%appinstall foo
    echo 'echo "Hello Foo."' >> bin/runfoo.sh
    chmod u+x bin/runfoo.sh

An application might just be a particular environment, to be predominantly used with shell

%appenv foo
    BEST_APP=foo
    export BEST_APP

For a complete list of the sections defined, see the full specification. You can define any number of the sections for an application.

Containers are already reproducible in that they package dependencies. This basic format adds to that by making the software inside of them modular, predictable, and programmatically accessible. We can say confidently that some set of steps, labels, or variables in one of the entry points is associated with a particular action of the container. We can better reveal how dependencies relate to each step in a scientific workflow. And the scientific workflow doesn’t necessary have to be in a container - with SCIF this recipe is host agnostic. I can install foo and bar on my host, or in a Docker or Singularity container. Let’s do this for two containers, so we don’t muck with our personal machines.

Docker

You can start with a Docker base that has scif installed, and then has the scif executable as the entrypoint and added to the path. With this base, all you need to do to build your recipe is add the file to the container, and install it with scif:

FROM vanessa/scif
ADD foobar.scif /
RUN scif install /foobar.scif

and then build normally:

docker build -t vanessa/foobar .

You can do the exact same thing with Singularity, and even use the same base!

Singularity

Bootstrap:docker
From: vanessa/scif

%files
    foo-bar.scif

%post
    /opt/conda/bin/scif install /cowsay.scif install /foo-bar.scif

and build

sudo singularity build foobar.simg Singularity.foobar

Commands

Now we can run the containers side by side, and see the SCIF respond consistently regardless of the technology wrapping it. Using SCIF “apps”, a user can easily discover both foo and bar without knowing anything about the container:

# The following two are equivalent for Singularity
$ singularity run foobar.img apps
$ ./foobar.simg apps

# Docker
$ docker run -it vanessa/foobar apps
bar
foo

and inspect each one:

$ ./foobar.simg inspect foo
$ docker run -it vanessa/foobar inspect foo
{
    "foo": {
        "appenv": [
            "BEST_GUY=foo",
            "export BEST_GUY"
        ],
        "apprun": [
            "BEST_GUY=foo",
            "export BEST_GUY",
            "BEST_GUY=foo",
            "export BEST_GUY",
            "    echo The best guy is $BEST_GUY"
        ]
    }
}

and of course, run them!

$ docker run -it vanessa/foobar run foo
[foo] executing /bin/bash /scif/apps/foo/scif/runscript
The best guy is foo

$ ./foobar.simg run foo
[foo] executing /bin/bash /scif/apps/foo/scif/runscript
The best guy is foo

$ ./foobar.simg --quiet run foo
The best guy is foo

Container Modularity

The next goal is container modularity! For this example, we are going to switch to a more fun build recipe - cowsay! If you want to grab the recipe to follow along, you can get it here. Let’s again build a Docker and Singularity container.

FROM vanessa/scif
ADD cowsay.scif /
RUN scif install /cowsay.scif

docker build -f Dockerfile.cowsay -t vanessa/cowsay .

and Singularity

Bootstrap:docker
From: vanessa/scif

%files
    cowsay.scif

%post
    /opt/conda/bin/scif install /cowsay.scif install /cowsay.scif

and build

sudo singularity build cowsay Singularity.cowsay

So how is it that the different SCIF apps find one another? They discover each other by way of environment variables. For the filesystem, the name and base scif is chosen intentionally to be something short, and likely to be unique. The same is true for the environment namespace. You can always find the “active” application with the variables prefixed with SCIF_ (without ending in an application name) and any (sleeping / not active) application via the same variables, but ending in the application name. If you needed to discover all applications with labels, for example, you would search the list of environment names to find those that start with SCIF_APPLABELS_

 ./cowsay exec lolcat env | grep SCIF_APPLABELS_
SCIF_APPLABELS_fortune=/scif/apps/fortune/scif/labels.json
SCIF_APPLABELS_cowsay=/scif/apps/cowsay/scif/labels.json
SCIF_APPLABELS_lolcat=/scif/apps/lolcat/scif/labels.json

And here we are looking at all the environment active for lolcat:

 ./cowsay exec lolcat env | grep lolcat
[lolcat] executing /usr/bin/env 
SCIF_APPDATA=/scif/data/lolcat
SCIF_APPNAME_lolcat=lolcat
SCIF_APPRUN=/scif/apps/lolcat/scif/runscript
SCIF_APPRECIPE=/scif/apps/lolcat/scif/lolcat.scif
SCIF_APPROOT=/scif/apps/lolcat
SCIF_APPLIB_lolcat=/scif/apps/lolcat/lib
SCIF_APPMETA_lolcat=/scif/apps/lolcat/scif
SCIF_APPBIN_lolcat=/scif/apps/lolcat/bin
SCIF_APPNAME=lolcat
SCIF_APPHELP_lolcat=/scif/apps/lolcat/scif/runscript.help
SCIF_APPLIB=/scif/apps/lolcat/lib
SCIF_APPMETA=/scif/apps/lolcat/scif
SCIF_APPBIN=/scif/apps/lolcat/bin
SCIF_APPHELP=/scif/apps/lolcat/scif/runscript.help
SCIF_APPENV_lolcat=/scif/apps/lolcat/scif/environment.sh
SCIF_APPENV=/scif/apps/lolcat/scif/environment.sh
SCIF_APPLABELS_lolcat=/scif/apps/lolcat/scif/labels.json
SCIF_APPTEST_lolcat=/scif/apps/lolcat/scif/test.sh
SCIF_APPDATA_lolcat=/scif/data/lolcat
SCIF_APPRUN_lolcat=/scif/apps/lolcat/scif/runscript
SCIF_APPRECIPE_lolcat=/scif/apps/lolcat/scif/lolcat.scif
SCIF_APPROOT_lolcat=/scif/apps/lolcat
SCIF_APPLABELS=/scif/apps/lolcat/scif/labels.json
SCIF_APPTEST=/scif/apps/lolcat/scif/test.sh
...
PATH=/scif/apps/lolcat/bin:/opt/conda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/scif/apps/lolcat
LD_LIBRARY_PATH=/scif/apps/lolcat/lib:/.singularity.d/libs
BEST_APP=lolcat

Notice a few things!

the specific environment (%appenv lolcat) is active because BEST_APP is lolcat
the lib folder in lolcat’s base is added to the LD_LIBRARY_PATH
the bin folder is added to the path
locations for lolcat data are exposed. It’s up to you how you use these, but you can predictably know that a well made app will look for it’s data in it’s specific folder.
environment variables are provided for the applications’s root, it’s data, and it’s name

As mentioned previously, if you need to interact with other application’s during the execution of an application, the base and data folders are provided too (note that below we are filtering to “fortune” when lolcat is active):

./cowsay exec lolcat env | grep fortune
SCIF_APPLABELS_fortune=/scif/apps/fortune/scif/labels.json
SCIF_APPTEST_fortune=/scif/apps/fortune/scif/test.sh
SCIF_APPDATA_fortune=/scif/data/fortune
SCIF_APPRUN_fortune=/scif/apps/fortune/scif/runscript
SCIF_APPRECIPE_fortune=/scif/apps/fortune/scif/fortune.scif
SCIF_APPROOT_fortune=/scif/apps/fortune
SCIF_APPNAME_fortune=fortune
SCIF_APPLIB_fortune=/scif/apps/fortune/lib
SCIF_APPMETA_fortune=/scif/apps/fortune/scif
SCIF_APPBIN_fortune=/scif/apps/fortune/bin
SCIF_APPHELP_fortune=/scif/apps/fortune/scif/runscript.help
SCIF_APPENV_fortune=/scif/apps/fortune/scif/environment.sh

Also provided are more global paths for data and apps:

SCIF_APPS=/scif/apps
SCIF_DATA=/scif/data

Cowsay Container

Now let’s go through the tutorial to use our cowsay container! We will show the equivalent commands for Docker and Singularity, using a Scientific Filesystem entrypoint.

$ ./cowsay apps
$ docker run vanessa/cowsay apps
  cowsay
  fortune
  lolcat

Ask for help for a specific app!

$ ./cowsay help fortune
$ docker run vanessa/cowsay help fortune
fortune is the best app

Ask for help for all apps, without knowing in advance what they are:

for app in $(./cowsay apps)
   do
     ./cowsay help $app
done
    cowsay is the best app
    fortune is the best app
    lolcat is the best app

Run a particular app

$ ./cowsay run fortune
$ docker run vanessa/cowsay run fortune
[fortune] executing /bin/bash /scif/apps/fortune/scif/runscript
	My dear People.
	My dear Bagginses and Boffins, and my dear Tooks and Brandybucks,
and Grubbs, and Chubbs, and Burrowses, and Hornblowers, and Bolgers,
Bracegirdles, Goodbodies, Brockhouses and Proudfoots.  Also my good
Sackville Bagginses that I welcome back at last to Bag End.  Today is my
one hundred and eleventh birthday: I am eleventy-one today!"
		-- J. R. R. Tolkien

You can add --quiet to suppress verbosity from scif.

$ ./cowsay --quiet run fortune
$ docker run vanessa/cowsay --quiet run fortune

Advanced running - pipes!

echo "Moo Moo Milk, it's the best" | ./cowsay run cowsay

[cowsay] executing /bin/bash /scif/apps/cowsay/scif/runscript
 _____________________________
< Moo Moo Milk, it's the best >
 -----------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Examples: CarrierSeq: sequence analysis workflow with Singularity + SCIF

Tue, 26 Sep 2017 17:12:01 +0000

Singularity is a container, similar to Docker, that is secure to run in HPC environments. For this pipeline, we take a previously implemented pipeline (in Docker and for host) and provide two example Singularity containers, each of which provides modular access to the different software inside. By way of using the Scientific Filesystem (SCI-F) with Singularity, we have a lot of freedom in deciding on what level of functions we want to expose to the user. A developer will want easy access to the core tools (e.g., bwa, seqtk) while a user will likely want one level up, on the level of a collection of steps associated with some task (e.g., mapping). These two use cases enhance our traditional use of a container as a single executable. We will walk through the steps of building and using each one.

Setup

You first need to install Singularity. For this tutorial, we are using the development branch with the upcoming version 2.4. You can install it doing the following.

git clone -b development https://www.github.com/singularityware/singularity.git
cd singularity
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install

Now you are ready to go!

Get Data

If you aren’t familar with genomic analysis (as I’m not) you will be utterly confused about how to download data. The reference is provided, but the input data isn’t. The container was originally designed to download it’s own data, but due to a change in the sri-toolkit the original data is provided as a manual download link from Dropbox. The good news is, if/when the data download works, we can provide another app in the continer to get it, without needing to understand how the sri-toolkit works.

Build the image

Building looks like this:

sudo singularity build --writable carrierseq.img Singularity

If you prefer to have a “sandbox” directory, you can do this:

sudo singularity build --sandbox carrierseq.img Singularity

I don’t understand why we have both. I’m ok with this. If you don’t need a writable image (for testing) just build a final one:

sudo singularity build carrierseq.img Singularity

This will build an essentially frozen image, so your pipeline is forever preserved.

Listing Pipeline Apps

If you didn’t know anything about the image, you would want to explore it. SCI-F provides easy commands to do that.

Global Help

singularity help carrierseq.img

    CarrierSeq is a sequence analysis workflow for low-input nanopore 
    sequencing which employs a genomic carrier.

    Github Contributors: Angel Mojarro (@amojarro), 
                         Srinivasa Aditya Bhattaru (@sbhattaru),   
                         Christopher E. Carr (@CarrCE),
                         Vanessa Sochat (@vsoch).

    fastq-filter from: https://github.com/nanoporetech/fastq-filter
    see:
           singularity run --app readme carrierseq.img | less for more detail

    To run a typical pipeline, you might do:

    # Download data
    singularity run --app mapping --bind data:/scif/data carrierseq.img
    singularity run --app poisson --bind data:/scif/data carrierseq.img
    singularity run --app sorting --bind data:/scif/data carrierseq.img
   

SCI-F Apps

You can see apps in the image, as instructed:

singularity apps carrierseq.img
download
mapping
poisson
readme
reference
sorting

The command mentioned at the body is an internal provided only to add, preserve, and make the README accessible. Cool!

singularity run --app readme carrierseq.img | less

# CarrierSeq

## About

bioRxiv doi: https://doi.org/10.1101/175281

CarrierSeq is a sequence analysis workflow for low-input nanopore sequencing which employs a genomic carrier.

Github Contributors: Angel Mojarro (@amojarro), Srinivasa Aditya Bhattaru (@sbhattaru), Christopher E. Carr (@CarrCE), and Vanessa Sochat (@vsoch).</br> 
fastq-filter from: https://github.com/nanoporetech/fastq-filter

.......etc

We can see all apps in the image:

singularity apps carrierseq.img
mapping
poisson
readme
sorting
download

And then we can ask for help for any of the pipeline steps: https://github.com/amojarro/carrierseq/issues/1

singularity help --app mapping carrierseq.img
singularity help --app poisson carrierseq.img
singularity help --app sorting carrierseq.img

We can also look at metadata for the entire image, or for an app. The inspect command can expose environment variables, labels, the definition file, tests, and runscripts.

See singularity inspect --help for more examples:

singularity inspect carrierseq.img

{
    "org.label-schema.usage.singularity.deffile.bootstrap": "docker",
    "org.label-schema.usage.singularity.deffile": "Singularity",
    "org.label-schema.usage": "/.singularity.d/runscript.help",
    "org.label-schema.schema-version": "1.0",
    "org.label-schema.usage.singularity.deffile.from": "ubuntu:14.04",
    "org.label-schema.build-date": "2017-09-20T18:16:50-07:00",
    "BIORXIV_DOI": "https://doi.org/10.1101/175281",
    "org.label-schema.usage.singularity.runscript.help": "/.singularity.d/runscript.help",
    "org.label-schema.usage.singularity.version": "2.3.9-development.gaaab272",
    "org.label-schema.build-size": "1419MB"
}


singularity inspect --app mapping carrierseq.img
{
    "FQTRIM_VERSION": "v0.9.5",
    "SEQTK_VERSION": "v1.2",
    "BWA_VERSION": "v0.7.15",
    "SINGULARITY_APP_NAME": "mapping",
    "SINGULARITY_APP_SIZE": "9MB"
}

Overall Strategy

Since the data is rather large, we are going to map a folder to our present working directory ($PWD) where the analysis is to run. This directory, just like the modular applications, has a known and predictable location. So our steps are going to look like this:

# 0. Make an empty random folder to bind to for data.
mkdir data

# 1. Download data, map the data base to an empty folder on our local machine
#     (we actually will do this from the browser as the sri toolkit is changed.
# 2. Perform mapping step of pipeline, mapping the same folder.
# 3. perform poisson regression on filtered reads
# 4. Finally, sort results

# Download data from https://www.dropbox.com/sh/vyor82ulzh7n9ke/AAC4W8rMe4z5hdb7j4QhF_IYa?dl=0
# See issue --> https://github.com/amojarro/carrierseq/issues/1
singularity run --app mapping --bind data:/scif/data carrierseq.img
singularity run --app poisson --bind data:/scif/data carrierseq.img
singularity run --app sorting --bind data:/scif/data carrierseq.img

For any of the above steps, see the singularity help --app <appname> carrierseq.img for how to customize settings and environment variables. This demo is intended to use the defaults.

CarrierSeq Pipeline

sudo singularity build carrierseq.img Singularity

The author is still working on updating the automated download, for now download from here and then move into some data folder you intend to mount:

mv $HOME/Downloads/all_reads.fastq data/

And then the various steps of the pipeline are run as was specified above:

singularity run --app mapping --bind data:/scif/data carrierseq.img
singularity run --app poisson --bind data:/scif/data carrierseq.img
singularity run --app sorting --bind data:/scif/data carrierseq.img

1. Mapping

To run mapping, bind the data folder to the image, and specify the app to be mapping:

singularity run --app mapping --bind data:/scif/data carrierseq.img

2. Poisson

singularity run --app poisson --bind data:/scif/data carrierseq.img

3. Sorting

singularity run --app sorting --bind data:/scif/data carrierseq.img

How Can I Change It?

Given two containers that share the same input and output schema, I could swap out of the steps:

...
singularity run --app sorting --bind data:/scif/data carrierseq.img
singularity run --app sorting --bind data:/scif/data another.img

or even provide a single container with multiple options for the same step

...
singularity run --app sorting1 --bind data:/scif/data sorting.img
singularity run --app sorting2 --bind data:/scif/data sorting.img

CarrierSeq Development

In this use case, we want to build the development container.

sudo singularity build carrierseq.dev.img Singularity.devel

Now when we look at apps, we see all the core software that can be combined in specific ways to produce a pipeline step like “mapping”.

singularity apps carrierseq.dev.img
bwa
fqtrim
python
seqtk
sra-toolkit

each of which might be run, exec to activate the app environment, or shell to shell into the container under the context of a specific app:

# Open interactive python
singularity run --app python carrierseq.dev.img

# Load container with bwa on path
$ singularity shell --app bwa carrierseq.dev.img
$ which bwa
$ /scif/apps/bwa/bin/bwa

See the original repository CarrierSeq here or the Singularity CarrierSeq here.

Examples: Singularity Scientific Workflow with SCI-F Example

Mon, 25 Sep 2017 03:12:01 +0000

Here we are going to do a comparison between a scientific analysis provided in standard Singularity container, versus the same analysis implemented with the Scientific Filesystem (SCI-F). This analysis was originally used to compare Singularity vs. Docker on cloud providers via this container with interesting results pertaining to resources usage between the different cases. If you are interested in a Docker vs. Singularity implementation, see that project. For this small example, we want to give rationale for taking a SCI-F apps approach over a traditional Singularity image. We compare the following equivalent (but different!) implementations:

The original image is served on Singularity Hub:

singularity pull shub://vsoch/singularity-scientific-example

And the image with SCI-F app supports must be built with (not yet released) Singularity 2.4:

sudo singularity build scif.img Singularity

If you want to build locally, you can install the development branch:

git clone -b development https://www.github.com/singularityware/singularity.git
cd singularity
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install

Evaluation

The containers use the same software to perform the same functions, but notably, the software and executables are organized differently, and called differently. Singularity standard (the first without SCI-F) relies on external scripts, and the container is a bit of a black box. Singularity with SCI-F has no external dependencies beyond data, and is neatly organized. So how do we evaluate this?

The aim here is to qualitatively evaluate SCI-F on its ability to expose container metadata, and information about the pipeline and executables inside. As each evaluation is scoped to the goal’s of the container, for this example we focus on the purpose of deploying a set of steps that encompass a pipeline to perform variant calling.

First, the goals of SCI-F:

Containers are consistent to allow for comparison. I am able to easily discover relevant software and data for one or more applications defined by the container creator.
Containers are transparent. If i discover a container and do not have any prior knowledge or metadata, the important executables and metadata are revealed to me.
Container contents are easily available for introspection because they are programmatically parseable. I can run a function over a container, and know exactly the functions available to me, ask for help, and know where to interact with inputs and outputs.
Container internal infrastructure is modular. Given a set of SCI-F apps from different sources, I can import different install routines and have assurance that environment variables defined for each are sourced correctly for each, and that associated content does not overwrite previous content. Each software and data module must carry, minimally, a unique name and install location in the system.

For each of the above, let’s define some basic tests.

1. Development Evaluation

For this use case, we are a container developer, and we are writing a singularity build recipe.

Can I easily define multiple entrypoints?

Standard: Singularity standard defaults to a single runscript per container. If I need to expose multiple functions, I either need to write a more complicated single entrypoint to direct the call, or I need to write detailed docs on different exec commands (and executables inside) for the user to run. For this real world use case, at the time when this runscript was written, SCI-F was not yet conceptualized. Given the sheer number of tools in the container, the runscript served to list a long list of executables, written to a text file, added to the path:

%runscript

    if [ $# -eq 0 ]; then
        echo "\nThe following software is installed in this image:"
        column -t /Software/.info | sort -u --ignore-case        
        echo "\Note that some Anaconda in the list are modules and note executables."
        echo "Example usage: analysis.img [command] [args] [options]"  
    else
        exec "$@"
    fi

Given that this container was intended to run a scientific workflow, this list doesn’t help to make its usage transparent. It would be useful for an individual familiar with the core software, perhaps developing a specific workflow with it. Arguably, this complete listing should be provided, perhaps not as a main entrypoint, but a separate app to print metadata, or the software names and versions added as labels to the app(s) where they are relevant. It’s important to note that this first container did not include any logic for performing the analysis, this was provided by the included scripts. If the scripts are separated from the container, reproducing the analysis is no longer possible.

SCI-F: SCI-F has the advantage here because I can give names to different apps, and write a different executable runscript for each. I can still provide a main runscript, and it could either give the user a listing of possible options (below) or run the entire workflow the container provides. Here is a main runscript that instructs the user on how to use the apps, dynamically generating the list:

%runscript

    if [ $# -eq 0 ]; then
        echo "\nThe following software is installed in this image:"
        ls /scif/apps | sort -u --ignore-case        
        echo "Example usage: singularity --app <name> <container> [command] [args] [options]"  
    else
        exec "$@"
    fi

And here is an example app, specifically to downlad a component of the data:

%apprun download-reference
    mkdir -p $REF_DIR
    wget -P $REF_DIR ftp://ftp.sanger.ac.uk/pub/gencode/Gencode_human/release_25/gencode.v25.transcripts.fa.gz
    gzip -d $REF_DIR/gencode.v25.transcripts.fa.gz
    wget -P $REF_DIR ftp://ftp.ensembl.org/pub/release-85/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz
    gzip -d $REF_DIR/Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz

Importantly, the scripts that were previously in scripts are now cleanly organized into sections in the build recipe. This modular organization and easy accessibility would have been very challening given the first container organization. The runscript would have needed to be long and complicated to infer what the user wanted to do, or the same functionality achieved by executing different scripts (inside the container), which is a non-trivial (or minimally more annoying to write) than simply writing lines into sections.

Can I easily install content known to modules?

Given that I have two functions for my container to perform, foo and bar, can I generate an install routine that will allow for both shared dependencies (i.e. global) and distinct dependencies?

Standard: Singularity standard has one mode to install global dependencies, everything goes into the %post script and any organization of required data, files, and libraries is scattered around the image. Other than coming up with a manual organization and documenting it, there is no way to cleanly define boundaries that will be easily discovered by the user. If you take a look at the Standard Singularity recipe, you will see this reflected in one huge, single install procedure. As an example, for this container the tools bwa and samtools were generally used for an alignment step, and there is no way of knowing this. They are installed globally:

cd /Software
su -c 'git clone https://github.com/Linuxbrew/brew.git' singularity
su -c '/Software/brew/bin/brew install bsdmainutils parallel util-linux' singularity
su -c '/Software/brew/bin/brew tap homebrew/science' singularity
su -c '/Software/brew/bin/brew install art bwa samtools' singularity
su -c 'rm -r $(/Software/brew/bin/brew --cache)' singularity

In fact, the art tools are installed with the same manager (brew), but they belong to an entirely different step. If a research scientist (or user) were parsing this build recipe, or using an NLP algorithm that took distance into account, there would be no strong signal about how these software were used or related to one another.

SCI-F: With SCI-F, by simply defining an environment, labels, install, or runscript to be in the context of an app, the modularity is automatically generated. When I add a list of files to an app foo, I know they are added to the container’s predictable location for foo. If I add a file to bin I know it goes into foo’s bin, and is added to the path when foo is run. If I add a library to lib, I know it is added to LD_LIBRARY_PATH when foo is run. I don’t need to worry about equivalently named files under different apps getting mixed up, or being called incorrectly because both are on the path. For example, in writing these sections, a developer can make it clear that bwa and samtools are used together for alignment:

# =======================
# bwa index and align
# =======================

%appinstall bwa-index-align
    git clone https://github.com/lh3/bwa.git build
    cd build && git checkout v0.7.15 && make
    mv -t ../bin bwa bwakit

    apt-get install -y liblzma-dev
    cd .. && wget https://github.com/samtools/samtools/releases/download/1.5/samtools-1.5.tar.bz2
    tar -xvjf samtools-1.5.tar.bz2
    cd samtools-1.5 && ./configure --prefix=${SINGULARITY_APPROOT}
    make && make install

%apprun bwa-index-align
    mkdir -p $DATADIR/Bam
    bwa index -a bwtsw $DATADIR/Reference/Homo_sapiens.GRCh38.dna.primary_assembly.fa
    bwa mem -t $NUMCORES $DATADIR/Reference/Homo_sapiens.GRCh38.dna.primary_assembly.fa $DATADIR/Fastq/dna_1.fq.gz $DATADIR/Fastq/dna_2.fq.gz | samtools view -bhS - > $DATADIR/Bam/container.bam  

%applabels bwa-index-align
    bwa-version v0.7.15
    samtools-version v1.5

and that art is used to simulate reads:

# =======================
# simulate reads
# =======================

%apphelp simulate-reads
    Optionally set any of the following environment variables (defaults shown)
    READS (100000000)
    READ_LEN (150)
    GENOME_SIZE (3400000000)

%appenv simulate-reads
    READS=${READS:-100000000}
    READ_LEN=${READ_LEN:-150}
    GENOME_SIZE=${GENOME_SIZE:-3400000000}
    export GENOME_SIZE READ_LEN READS

%appinstall simulate-reads   
    wget https://www.niehs.nih.gov/research/resources/assets/docs/artbinmountrainier20160605linux64tgz.tgz
    tar -xzvf artbinmountrainier20160605linux64tgz.tgz 
    mv art_bin_MountRainier/* bin/
    chmod u+x bin/art_*

%apprun simulate-reads
    GENOME="$REF_DIR/Homo_sapiens.GRCh38.dna.primary_assembly.fa"
    FOLD_COVERAGE=$(python -c "print($READS*$READ_LEN/$GENOME_SIZE)")
    art_illumina --rndSeed 1 --in $GENOME --paired --len 75 --fcov $FOLD_COVERAGE --seqSys HS25 --mflen 500 --sdev 20 --noALN --out $FASTQ_DIR/dna_ && gzip $FASTQ_DIR/dna_1.fq && gzip $FASTQ_DIR/dna_2.fq

Whether I glanced at the recipe, or did some kind of text processing, I could easily see the relationships and purpose of the software in the container.

Can I associate environment and metadata with modules?

Given two different functions for my container to perform, foo and bar, can I define environment variables and labels (metadata) that I know will be sourced (environment) or exposed (inspect labels) in the context of the app?

Standard: Singularity standard also has one global shared %environment, and %labels section. If two functions in the container share the same environment variable and the value is different, this must be resolved manually. For this example, the first container didn’t have any labels or environment, however in practice these global sections are usually used for high level, global variables like author and version (of the container). When I run the container, regardless of if different contexts or variables are needed for executables inside, I get the same environment.

SCI-F: With SCI-F, I simply write the different variables to their sections, and have confidence that they will be sourced (environment) or inspected (labels) with clear association to the app.

%appenv run-rtg
    MEM=${MEM:-4g}
    THREADS=${THREADS:2}
    export MEM THREADS

%applabel run-rtg
    rtg-version 3.6.2

Do I need to know standard locations in advance?

Given that a container has conformance to SCI-F, do I need to know how it works to use it?

Standard: With a standard location, we would be relying on Linux File System conventions (e.g., installation of content to /usr/local or intuitively infer that a folder called /Software (as with this scientific example) or /code is likely where the creator put important content.

SCI-F: Instead of requiring the user to know that an app’s base might be at /scif/apps/foo, we instead expose environment variables (e.g., SINGULARITY_APPBASE) that can be referenced at runtime to refer to different apps. This is especially important if, for example, I need to reference the output of one app as input for another, or simply know it’s install location. Regardless of which app is running, the container will also expose the top level folder for all apps installations, and data, SINGULARITY_DATA and SINGULARITY_APPS at /scif/data and /scif/apps, respectively.

2. Production Evaluation

For this use case, we operate under the scenario that we are familiar with Singularity and the commands to use SCi-F, but we know nothing about the containers. We are a user.

Do I know what the container does?

The most natural thing to do with a Singularity container, knowing that it is possible to execute, is to do exactly that. For this evaluation, we want to assess how well executing the container reveals the intentions of the creator.

Standard: From the runscript we evaluated earlier, we are presented with a list of software and versions installed in the image, without detail to where or for what purpose. While this listing is comprehensive, it’s most appropriate for a developer than a scientific workflow. In this listing, it isn’t clear how the software is used or combine in the analysis. We are reliant on some external script or controller that drives the container. We don’t have any hints about possible analysis steps the container can serve.

The following software is installed in this image:
alabaster           0.7.9     Anaconda
anaconda            4.3.0     Anaconda
anaconda-client     1.6.0     Anaconda
anaconda-navigator  1.4.3     Anaconda
argcomplete         1.0.0     Anaconda
art                 20160605  Homebrew
astroid             1.4.9     Anaconda
astropy             1.3       Anaconda
babel               2.3.4     Anaconda
backports           1.0       Anaconda
beautifulsoup4      4.5.3     Anaconda
bitarray            0.8.1     Anaconda
blaze               0.10.1    Anaconda
bokeh               0.12.4    Anaconda
boto                2.45.0    Anaconda
bottleneck          1.2.0     Anaconda
bsdmainutils        9.0.10    Homebrew
bwa                 0.7.15    Homebrew
cairo               1.14.8    Anaconda
cffi                1.9.1     Anaconda
chardet             2.3.0     Anaconda
chest               0.2.3     Anaconda
click               6.7       Anaconda
cloudpickle         0.2.2     Anaconda
clyent              1.2.2     Anaconda
colorama            0.3.7     Anaconda

...

sqlalchemy          1.1.5     Anaconda
sqlite              3.13.0    Anaconda
statsmodels         0.6.1     Anaconda
sympy               1.0       Anaconda
terminado           0.6       Anaconda
tk                  8.5.18    Anaconda
toolz               0.8.2     Anaconda
tornado             4.4.2     Anaconda
traitlets           4.3.1     Anaconda
unicodecsv          0.14.1    Anaconda
util-linux          2.29      Homebrew
wcwidth             0.1.7     Anaconda
werkzeug            0.11.15   Anaconda
wheel               0.29.0    Anaconda
widgetsnbextension  1.2.6     Anaconda
wrapt               1.10.8    Anaconda
xlrd                1.0.0     Anaconda
xlsxwriter          0.9.6     Anaconda
xlwt                1.2.0     Anaconda
xz                  5.2.2     Anaconda
xz                  5.2.3     Homebrew
yaml                0.1.6     Anaconda
zeromq              4.1.5     Anaconda
zlib                1.2.8     Anaconda
\Note that some Anaconda in the list are modules and note executables.
Example usage: analysis.img [command] [args] [options]

SCI-F: When we run the SCi-F container, a different kind of information is presented. By listing the container contents at /scif/apps, the user knows what pipeline steps are included in the analysis.

singularity run scif.img

and in fact, this basic listing is generally useful for apps, so it’s provided as it’s own command, if the user doesn’t write a runscript at all:

singularity apps scif.img

The runscript also hints that I can direct the %help command to better understand an app. While both SCiF and standard Singularity allows for specification of a global %help section, providing documentation on the level of the application is more focused and offers specificity to the user. Both also allow for global %labels that might point to documentation, code repositories, or other informative information.

Does moduarity come naturally?

For this metric, we want to know if using different functions of the container (modules) is intuitive. As we have seen above, the definition of a “module” could be anything from a series of script calls to perform a pipeline step (alignment using bwa and samtools), a single script call (such as a module just for bwa), or even the same function applied to a specific set of content (e.g., download “A” v.s. download “B”).

Standard: Singularity proper represents a module on the level of the container. The entire container is intended for one entrypoint, and any deviation from that requires customization of that entrypoint, or special knowledge about other executables in the container to call with exec. The container is modular only in context of being a single step in a pipeline.

SCI-F: SCI-F, in that the software inside is defined and installed in a modular fashion, makes it easy to find three different modules:

download-fastq
download-rtg
download-reference

and without looking further, infer that likely these three downloads can be run in parallel. While this doesn’t represent any kind of statement or assurance of this, it allows for the container have a natural modularity. Consider steps that are named according to an order:

1-download
2-preprocess
3-analysis

The creator of the container, in choosing a careful naming, can deliver a huge amount of information about different modules.

Do I know what executables are important in the container?

Without much effort, I should have a high level understanding of the different functions that the container performs, as defined by the creator. For example, a container intended for development of variant calling will expose low level tools (e.g, bwa, seqtk) while a container that is intended will expose a pipeline (e.g., mapping).

Standard: For Singularity standard, if the container performs one function (and one only), then a single runscript / entrypoint is sufficient. Having multiple functions is completely reliant on the programming ability of the creator. If the creator is able to write a more complex runscript that explains different use cases, the container can satisfy this goal. If not, the container is a black box.

SCI-F For SCI-F, without any significant programming ability, the different apps are easily exposed to the user.

Can I easily get help for an executable?

I should be able to run a command to get help or a summary of what the container does (introspection).

Standard: For Singularity standard, you can ask a container for help, but it’s a single global help.

SCI-F: For SCI-F apps, you can define help sections for all of the functions that your container serves, along with a global help.

3. Research Evaluation

An important attribute of having modular software apps is that it allows for separation of files and executables for research, and those that belong to the base system. From a machine learning standpoint, it provides labels for some subset of content in the container that might be used to better understand how different software relates to a pipeline. Minimally, it separates important content from the base, allowing, for example, a recursive tree generated at /scif to capture a large majority of additions to the container. Or simple parsing of the build recipe to see what software (possibly outside of this location) was intended for each app. Equally important, having container software installed at a global at %post also says important things about it - that it perhaps is important for more than one software module, or is more of a system library.

Conclusion

SCI-F clearly has the advantage when it comes to a container creator embedding his or her work with implied metadata about software and container contents. SCI-F also makes it easier to package different run scripts with the container, and expose them easily to the user. However, this does not mean that the standard approach of using a container as a general toolbox and distributing it with a series of external callers is bad or wrong. The choice to use (or not use) SCI-F apps is largely dependent on the goals of the creator, and the intended users.

News: Introducing SCI-F Apps

Fri, 15 Sep 2017 16:54:46 +0000

SCI-F refers to the Scientific Filesystem specifically designed to allow for internal modularity and reproducibility of scientific containers (or anywhere else a filesystem can be installed).

SCI-F is an organizational format that is intended to make it easy to generate internally modular containers. This means any of the following:

Record metrics and resource usage for the container's primary function.
Modular (different) software within the same container.
General linters, tests, or logging.
Helpers for running under different contexts.

For each of the above, this functionality comes by way of different "apps" that are installed in the container. An app can carry it's own set of files, environment variables, metadata, help, and runscript. This website is a base for sharing, and finding apps that could be useful to you!

To get started, you should read about the standard, and clone the development branch of Singularity to try creating a container with apps. If you want your app to appear here, simply add a folder to the repo in the _apps folder, which should be linked to _api. That's it!

SCI-F Apps are a sibling to Singularity Hub and a proud part of the open source Singularity containers project. Thanks for visiting!

Ask a Question

Examples: Container Metrics: Evaluating a container from Different Angles via SCIF Apps

Wed, 13 Sep 2017 22:16:01 +0000

For this next use case, a scientist is interested in running a series of metrics over an analysis of interest (the primary entry point for a rperoducible container. He has been given a container with a runscript, and several installed supporting metrics (SCIF apps also in the container), and knows nothing beyond that. You can find the dependency files at https://github.com/sci-f/metrics.scif.

First, clone the repository!

git clone https://github.com/sci-f/metrics.scif
cd metrics.scif

Building the image

Let’s first build the container. You can use the Makefile to build the image:

make

# Does make clean followed by make build

or manually:

sudo singularity build metrics Singularity

Running the Image

And now run it. This should perform the container’s main function, calling it’s runscript:

./metrics
Hello-World!

Works great! But then what if we wanted to know what tools (SCIF apps) come with the container? That’s easy to do:

./metrics apps

custom
linter
parallel
strace
time

Each of these is suited for a particular use case, discussed next.

Use Case 1: Evaluate software across different metrics

A system admin or researcher concerned about evaluation of different software could add relevant metrics apps to the software containers, and then easily evaluate each one with the equivalent command to the container. As an example, here is a simple app to return a table of system traces for some main SCIF app, or a user specific name runscript:

%apprun strace
    if [ $# -eq 0 ]
        then
        exec strace -c -t scif run main
    else
        exec strace -c -t scif run "$@"
    fi

The table returned shows the traces:

$ ./metrics run strace
[strace] executing /bin/bash /scif/apps/strace/scif/runscript
[main] executing /bin/bash /scif/apps/main/scif/runscript
Hello World!
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.000008           0        40           munmap
  0.00    0.000000           0       707           read
  0.00    0.000000           0         1           write
  0.00    0.000000           0       426        42 open
  0.00    0.000000           0       447           close
...

or a set of metrics from “time”

./metrics run time
[time] executing /bin/bash /scif/apps/time/scif/runscript
COMMAND    ELAPSED_TIME_HMS    AVERAGE_MEM    FS_INPUTS    MAX_RES_SIZE_KB    FS_OUTPUTS    PERC_CPU_ALLOCATED    CPU_SECONDS_USED    W_TIMES_SWAPPED    SHARED_TEXT_KB    ELAPSED_TIME_SECONDS    NUMBER_SIGNALS_DELIVERED    AVG_UNSHARED_STACK_SIZE    SOCKET_MSG_RECEIVED    SOCKET_MSG_SENT    AVG_RESIDENT_SET_SIZE    CONTEXT_SWITCHES
scif run main    0:00.22    0    74    28120    0    100%    0.21    0    0    0.22    0    0    0    0    0    29

The user can also specify a name of another app in the container to run a system trace for it instead (truncated):

./metrics run strace custom
[strace] executing /bin/bash /scif/apps/strace/scif/runscript custom
[custom] executing /bin/bash /scif/apps/custom/scif/runscript
Beware of a dark-haired man with a loud tie.
                 (__) 
                 (oo) 
           /------\/ 
          / |    ||   
         *  /\---/\ 
            ~~   ~~   
..."Have you mooed today?"...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 58.33    0.000014           4         4           wait4
 41.67    0.000010           0       426        42 open
  0.00    0.000000           0       710           read
  0.00    0.000000           0         1           write
  0.00    0.000000           0       447           close
...

Regardless of what your runscript does, this app will provide a consistent way to produce this metric. Who knew there were so many open and read calls to just echo-ing a line to the console!

Use Case 2: Custom Functions and Metrics

When a container is intended to only perform one function, this use case maps nicely to having a single runscript. As the number of possible functions increase, however, the user is forced to either:

have a runscript that can take command line options to call different executables
use the exec command with some known path (to the user)

SCI-F apps allow for an easy way to define custom helper metrics or functions for the container. For example, let’s say I created some custom, special metric. Or in this case, it’s more of a container easter egg.

%apprun custom
    apt-get moo

and then the resulting output

./metrics run custom
"I wonder", he said to himself, "what's in a book while it's closed.  Oh, I
know it's full of letters printed on paper, but all the same, something must
be happening, because as soon as I open it, there's a whole story with people
I don't know yet and all kinds of adventures and battles."
		-- Bastian B. Bux
                 (__) 
                 (oo) 
           /------\/ 
          / |    ||   
         *  /\---/\ 
            ~~   ~~   
..."Have you mooed today?"...

This simple ability to create general, modular applications for containers means that we can move toward the possibility that some researchers can specialize in the development of the metrics, and others the analyses.

Use Case 3: Code Quality and Linting

A SCIF app can be used for general tests that are generalizable to other containers. The example is provided here with the “linter”

./metrics run linter <file>

The app can perform a linting of some default script provided by the container, or a user specified file.

$ ./metrics run linter
[linter] executing /bin/bash /scif/apps/linter/scif/runscript
No config file found, using default configuration
************* Module runscript
E:  1, 0: invalid syntax (<string>, line 1) (syntax-error)
\end{lstlisting}

$ ./metrics run linter script.py

Use Case 4: Runtime Evaluation

In that a metric can call a runscript, it could be easy to evaluate running the main analysis under various levels or conditions. As a simple proof of concept, here we are creating an app to execute the same exact script in parallel.

%apprun parallel
    parallel /bin/bash ::: $SCIF_APPRUN_main $SCIF_APPRUN_main $SCIF_APPRUN_main

./metrics run parallel
[parallel] executing /bin/bash /scif/apps/parallel/scif/runscript
Hello World!
Hello World!
Hello World!

And you might imagine a similar loop to run an analysis, and modify a runtime or system variable for each loop, and save the output (or print to console).

Run them all!

And we don’t need to know anything in advance (paths to hidden executables, how paths or environment should be handled) to run all the container applications, if we wanted to do that. We can use a loop

for app in $(./metrics apps)
   do
      ./metrics run $app
done

This metrics implementation is available for use and documentation provided in entirety on Github. To see (an older) asciicast in its home and original form, you can see it here.

Examples: Hello World, Using SCI-F for Modular Software Evaluation

Wed, 13 Sep 2017 22:16:01 +0000

A common question pertains to evaluation of different solutions toward a common goal. An individual might ask “How does implementation “A” compare to implementation “B” as evaluated by one or more metrics?” For a systems admin, the metric might pertain to running times, resource usage, or efficiency. For a researcher, he or she might be interested in looking at variability (or consistency) of outputs. Importantly, it should be possible to give a container serving such a purpose to a third party that does not know locations of executables, or environment variables to load, and the container runs equivalently. SCI-F allows for this by way of providing modular software applications, each corresponding to a custom environment, libraries, and potentially files.

Method

To demonstrate this use case, we developed a container that implements the most basic function for a program, a print to the console, for each of 19 different languages (R, awk, bash, c, cat, chapel, clisp, cpp, csh, go, julia, octave, perl, python, ruby, rust, tcsh, zsh). The container is designed as a means to collect a series of metrics relative to timing and system resources for each language. The metrics pertain to system resources provided by the time (time(1) - Linux manual page ) and strace (strace(1): trace system calls/signals…) utilities. A user that did not create the container could ask it for global help:

$ singularity help container.img


This container will say hello-world (in dinosaur)!

Examples:

     # See all installed languages
     singularity apps <container>

     # See help for a specific language
     singularity --app <language> <container> help to see help  |or|

     # Run a specific languages
     singularity --app <language> <container>

or see all applications installed:

$ singularity apps container.img
   awk
   bash
   …
   zsch

Or run any particular language:

$ singularity run --app bash container.img
RaawwWWWWWRRRR!!

Importantly, in the example above, using “run” for the application “bash” handles loading environment variables, adding the application folders to the path, and executing the associated runscript. The application also optionally can serve it’s own labels, environment metadata, and specifics about its size with the “inspect” command:

singularity inspect --app bash container.img

{
    "SINGULARITY_APP_NAME": "bash",
    "SINGULARITY_APP_SIZE": "1MB"
}

Therefore, the metric evaluation could be run, across modules, without knowing the applications installed with a simple for loop.

for app in $(singularity apps container.img)
    do
    singularity run --app $app container.img
done

In the case of metric evaluation, it would be up to the implementor to decide to evaluate the software internally (above) or externally. For example, an external evaluation might look like the following:

for app in $(singularity apps container.img)
    do
    /usr/bin/time -a singularity run --app $app container.img
done

In practice, for general metrics like timing and host resources, it is advantageous to perform tests externally, as the containers themselves can be agnostic to the tests. For tests that look at system calls (e.g., strace as in the example above) calling externally would mean needing to properly account for the call to the singularity software itself in the results.

Results

To demonstrate the value of using SCI-F containers, we ran a simple function to print to the command line in 19 languages, and were able to run the analysis in entirety without knowing the specific commands for each language. The resulting table of features pertaining to times and resources (Supplementary Table 1) demonstrates a wide span of differences between the seemingly identical calls. A principle component analysis (PCA) of these features shows distinct groups (see full results here)