Scientific Filesystem Quick Start

This is the quick start for using a Scientific Filesystem. We won’t be designing or creating one, but using a pre-generated Docker or Singularity container. The awesome part is that despite two different container technologies, the guts inside, and interacting with them, will be the same! Thus, we will first show how to pull the different containers, and then show usage side by side. If you do want to learn these first steps, see the preview and install tutorial. If you want a REALLY quick start, skip the prose and go to it! If you want code for a quick start, see our hello-world.scif Github repository.

What you should understand for this quick start is what we’ve done thus far. We (as the creator of a container with a scientific filesystem) wrote a recipe, a text file with instructions for interacting with different scientific filesystem software modules (“apps”) and then exposed a single entrypoint that is the controller for the different apps. Then we build the container.

Then the user that doesn’t have knowledge to the creation is able to interact with the software modules defined by the filesystem in a consistent and user friendly way. Right now I am playing the role of the creator, and you the user for doing this quick start. Let’s jump in!

Singularity

For our first container we are using the Singularity scientific filesystem container that is built in the preview and install tutorial. First, you should pull the image, either with the sregistry client or singularity natively:

singularity pull --name scif-cli shub://vsoch/scif:scif
Progress |===================================| 100.0% 
Done. Container is at: /home/vanessa/Desktop/scif-cli

# or using sregistry client
# sregistry pull --name scif-cli vsoch/scif:scif

Importantly note that we are pulling the tag scif. If you pull latest (e.g., vsoch/scif then you will get a container with the scif client installed, but no filesystem.

Docker

For our second container, we are starting with the Dockerized scientific filesystem container that is built in the preview and install tutorial. We will use the latest for this example, and you can also see the tags available for other versions. First, pull the image:

docker pull vanessa/scif:hw

Note that the container vanessa/scif is a base without any SCIF installed, but you would use it to install your own SCIF.

FROM vanessa/scif
ADD recipe.scif /
RUN scif install /recipe.scif

What you should understand from the install tutorial is that we (as the creator of the container) wrote a recipe, a text file with instructions for interacting with different scientific filesystem software modules (“apps”) and then exposed a single entrypoint that is the controller for the different apps. Then we build the container.

Then the user that doesn’t have knowledge to the creation is able to interact with the software modules defined by the filesystem in a consistent and user friendly way. Right now I am playing the role of the creator, and you the user. Let’s jump in! For each example below, you can test with Docker, Singularity, or both. For some, the $PS1 prompt might not match, but the output is equivalent.

Test the entrypoint

We can first test the entrypoint. We have just pulled the container, and we know nothing. So we run it.

docker run vanessa/scif:hw
./scif-cli 

Scientific Filesystem [v0.0.3]
usage: scif [-h] [--debug] [--quiet] [--writable]
            {version,pyshell,shell,preview,help,install,inspect,run,apps,dump,exec}
            ...

scientific filesystem tools

optional arguments:
  -h, --help            show this help message and exit
  --debug               use verbose logging to debug.
  --quiet               suppress print output
  --writable, -w        for relevant commands, if writable SCIF is needed

actions:
  actions for Scientific Filesystem

  {version,pyshell,shell,preview,help,install,inspect,run,apps,dump,exec}
                        scif actions
    version             show software version
    pyshell             Interactive python shell to scientific filesystem
    shell               shell to interact with scientific filesystem
    preview             preview changes to a filesytem
    help                look at help for an app, if it exists.
    install             install a recipe on the filesystem
    inspect             inspect an attribute for a scif installation
    run                 entrypoint to run a scientific filesystem
    apps                list apps installed
    dump                dump recipe
    exec                execute a command to a scientific filesystem

Apps

We are familiar with the scientific filesystem, so we can use the apps command to see what is installed. Since this conainer had the scientific filesystem installed on build, it already has three apps for us to play with:

docker run vanessa/scif:hw apps
./scif-cli apps

SCIF [app]              [root]
1  hello-world-echo	/scif/apps/hello-world-echo
2  hello-world-env	/scif/apps/hello-world-env
3  hello-world-script	/scif/apps/hello-world-script

Help

We can then ask for help for a particular app. This section is important for the creator to put some time into describing the basic important things that should be known.

docker run vanessa/scif:hw help hello-world-env
./scif-cli help hello-world-env
This is the help section for hello-world-env! This app
does not have anything other than an environment installed.
It just defines the environment variable `OMG=TACOS`. Try issuing
a command to the scif entrypoint to echo this variable:
scif exec hello-world-env echo [e]OMG
docker run vanessa/scif:hw exec hello-world-env echo [e]OMG
[hello-world-env] executing /bin/echo $OMG
TACOS

And you see a hasty help message that I wrote for the module hello-world-env telling you how to run it. We will do that soon.

Inspect

We can also inspect an app of interest, which will spit out a metadata structure for it. You can think of the help command as returning a human friendly thing, and inspect something that can be programmatically parsed.

docker run vanessa/scif:hw inspect hello-world-env
./scif-cli inspect hello-world-env
{
    "hello-world-env": {
        "appenv": [
            "OMG=TACOS"
        ],
        "apphelp": [
            "This is the help section for hello-world-env! This app",
            "does not have anything other than an environment installed.",
            "It just defines the environment variable `OMG=TACOS`. Try issuing",
            "a command to the scif entrypoint to echo this variable:",
            "scif exec hello-world-env echo [e]OMG",
            "docker run vanessa/scif:hw exec hello-world-env echo [e]OMG",
            "[hello-world-env] executing /bin/echo $OMG",
            "TACOS"
        ]
    }
}

Yes, it really just is an environment, and a help message for it! Now that we’ve seen this instruction twice, let’s give run a try with specification of an environment variable, $OMG in the container.

Run

We can run the hello-world-echo app like this:

docker run vanessa/scif:hw run hello-world-echo
./scif-cli run hello-world-echo
[hello-world-echo] executing /bin/bash /scif/apps/hello-world-echo/scif/runscript
The best app is hello-world-echo

What about our example above with hello-world-env? It can be weird trying to pass an environment variable into a container from the host, because it gets evaluated (and then winds up something unexpected or empty!) To help this, with scif we use a modified syntax to pass the variable into the container. We replace $ with [e] so that $VARIABLE is [e]VARIABLE. Here is an example:

docker run vanessa/scif:hw exec hello-world-env echo [e]OMG
./scif-cli exec hello-world-env echo [e]OMG
[hello-world-env] executing /bin/echo $OMG
TACOS

If we had done that with $ it would have evaluated the variable on our host shell, and passed nothing into the container (unless in fact $OMG was defined on the host)/

Test

When the recipe has an %apptest section for an app, the content is written to a file test.sh in the metadat folder. In this example application, we have a bash script that will print a message and exit with 0 (success) given no argument, or exit with a return code set to a provided argument:

%apptest hello-world-script
    echo "Running tests!"
    if [ $# -eq 0 ]
        then
            echo "No arguments supplied, tests pass!"
            exit 0
        fi
    echo "Argument supplied, exiting with ${1}"
    exit ${1}

Thus, we can test the hello-world-script app like this to get a return code of 0:

docker run vanessa/scif:hw test hello-world-script
[hello-world-script] executing /bin/bash /scif/apps/hello-world-script/scif/test.sh
Running tests!
No arguments supplied, tests pass!

$ echo $?
0

And like this to get a non-zero return code (e.g., 255)

docker run vanessa/scif:hw test hello-world-script 255
[hello-world-script] executing /bin/bash /scif/apps/hello-world-script/scif/test.sh 255
Running tests!
Argument supplied, exiting with 255

ERROR Return code 255
$ echo $?
255

If an app doesn’t have tests, it prints this:

$ docker run vanessa/scif:hw test hello-world-echo
No tests defined for this app.

And for now, I’ve decided to provide a return code of 1, because it should be encouraged to write tests.

$ echo $?
1

Exec

You can also execute a command:

docker run vanessa/scif:hw exec hello-world-echo echo "Another hello!"
./scif-cli exec hello-world-echo echo "Another hello!"
[hello-world-echo] executing /bin/echo Another hello!
Another hello!

Bash Shell

If you want to interact with your container in the context of an app, there is a command for that! We can either shell into the container with the global scif environment (and no activated apps):

./scif-cli shell
WARNING No app selected, will run default ['/bin/bash']
executing /bin/bash 
vanessa@thinkpad:/scif$ 

Notice how the path ($PS1) in the terminal window changed to /scif? The same happens for Docker of course:

docker run -it vanessa/scif:hw shell
WARNING No app selected, will run default ['/bin/bash']
executing /bin/bash 
root@1ab15ba4cc3b:/scif

And the main difference here is that inside of the Docker image, we are (usually always) root :) Let’s see where we are:

$ echo $PWD
/scif

$ ls
apps
data

Since the scientific filesystem is not an entire environment like a container (it is rooted at /scif this is where we shell to. If we shelled in context of an app (next) we would be in the apps root. Before exit, try looking at the environment to see that we have activated the global SCIF environment variables.

env | grep SCIF
...
SCIF_APPBIN_hello_world_echo=/scif/apps/hello-world-echo/bin
SCIF_MESSAGELEVEL=INFO
exit
...

After exiting, we can do the same in the context of a specific app:

 ./scif-cli shell hello-world-env
[hello-world-env] executing /bin/bash 
vanessa@thinkpad:/scif/apps/hello-world-env$ 

Notice how we are in the root of hello-world-env. It works the same for Docker.

docker run -it vanessa/scif:hw shell hello-world-env
[hello-world-env] executing /bin/bash 
root@1ab15ba4cc3b:/scif/apps/hello-world-env# echo $OMG
TACOS
root@1ab15ba4cc3b:/scif/apps/hello-world-env# 

This is a great example of how a single container can be used to serve different interactive environments.

Python Shell

We can enter an interactive shell for exploring the container filesystem, if we want to do more than execute commands. For docker, we have to append an -it to mean we want an “interactive terminal” to the run command, using “pyshell” as the entrypoint:

./scif-cli pyshell
Found configurations for 2 scif apps
hello-world-echo
hello-world-script
[scif] /scif hello-world-echo | hello-world-script
Python 3.6.3 |Anaconda, Inc.| (default, Oct 13 2017, 12:02:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.
docker run -it vanessa/scif:hw pyshell
Found configurations for 3 scif apps
hello-world-env
hello-world-script
hello-world-echo
[scif] /scif hello-world-env | hello-world-script | hello-world-echo
Python 3.6.3 |Anaconda, Inc.| (default, Oct 13 2017, 12:02:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

We can now look at details for the client:

In [1]: client
Out[1]: [scif]

In [2]: client.apps()
Out[2]: ['hello-world-env', 'hello-world-script', 'hello-world-echo']

The “preview” command is most useful when you have a recipe (and haven’t installed yet) and want to look at what will be created on the filesystem. But you can also run it for an already installed scif:

In [3]: client.preview()
[base] /scif 
[apps] /scif/apps 
[data] /scif/data
 

[root] /scif/apps/hello-world-env 
[lib] /scif/apps/hello-world-env/lib 
[bin] /scif/apps/hello-world-env/bin 
[data] /scif/data/hello-world-env 
+ appenv hello-world-env
/scif/apps/hello-world-env/scif/environment.sh
OMG=TACOS
+ apprecipe hello-world-env
/scif/apps/hello-world-env/scif/hello-world-env.scif

...

You can also get the full set of environment variables:

client.get_env()
Out[5]: 
{'SCIF_APPBIN_hello_world_echo': '/scif/apps/hello-world-echo/bin',
 'SCIF_APPBIN_hello_world_env': '/scif/apps/hello-world-env/bin',
 'SCIF_APPBIN_hello_world_script': '/scif/apps/hello-world-script/bin',
 'SCIF_APPDATA_hello_world_echo': '/scif/data/hello-world-echo',
 'SCIF_APPDATA_hello_world_env': '/scif/data/hello-world-env',
 'SCIF_APPDATA_hello_world_script': '/scif/data/hello-world-script',
 'SCIF_APPENV_hello_world_echo': '/scif/apps/hello-world-echo/scif/environment.sh',
 'SCIF_APPENV_hello_world_env': '/scif/apps/hello-world-env/scif/environment.sh',
 'SCIF_APPENV_hello_world_script': '/scif/apps/hello-world-script/scif/environment.sh',
 'SCIF_APPHELP_hello_world_echo': '/scif/apps/hello-world-echo/scif/runscript.help',
 'SCIF_APPHELP_hello_world_env': '/scif/apps/hello-world-env/scif/runscript.help',
 'SCIF_APPHELP_hello_world_script': '/scif/apps/hello-world-script/scif/runscript.help',
 'SCIF_APPLABELS_hello_world_echo': '/scif/apps/hello-world-echo/scif/labels.json',
 'SCIF_APPLABELS_hello_world_env': '/scif/apps/hello-world-env/scif/labels.json',
 'SCIF_APPLABELS_hello_world_script': '/scif/apps/hello-world-script/scif/labels.json',
 'SCIF_APPLIB_hello_world_echo': '/scif/apps/hello-world-echo/lib',
 'SCIF_APPLIB_hello_world_env': '/scif/apps/hello-world-env/lib',
 'SCIF_APPLIB_hello_world_script': '/scif/apps/hello-world-script/lib',
 'SCIF_APPMETA_hello_world_echo': '/scif/apps/hello-world-echo/scif',
 'SCIF_APPMETA_hello_world_env': '/scif/apps/hello-world-env/scif',
 'SCIF_APPMETA_hello_world_script': '/scif/apps/hello-world-script/scif',
 'SCIF_APPNAME_hello_world_echo': 'hello-world-echo',
 'SCIF_APPNAME_hello_world_env': 'hello-world-env',
 'SCIF_APPNAME_hello_world_script': 'hello-world-script',
 'SCIF_APPRECIPE_hello_world_echo': '/scif/apps/hello-world-echo/scif/hello-world-echo.scif',
 'SCIF_APPRECIPE_hello_world_env': '/scif/apps/hello-world-env/scif/hello-world-env.scif',
 'SCIF_APPRECIPE_hello_world_script': '/scif/apps/hello-world-script/scif/hello-world-script.scif',
 'SCIF_APPROOT_hello_world_echo': '/scif/apps/hello-world-echo',
 'SCIF_APPROOT_hello_world_env': '/scif/apps/hello-world-env',
 'SCIF_APPROOT_hello_world_script': '/scif/apps/hello-world-script',
 'SCIF_APPRUN_hello_world_echo': '/scif/apps/hello-world-echo/scif/runscript',
 'SCIF_APPRUN_hello_world_env': '/scif/apps/hello-world-env/scif/runscript',
 'SCIF_APPRUN_hello_world_script': '/scif/apps/hello-world-script/scif/runscript',
 'SCIF_APPS': '/scif/apps',
 'SCIF_DATA': '/scif/data'}

or activate an app (and then see how the enviroment variables change, they will have added a few!

client.activate('hello-world-env')

In [7]: client.get_env()
Out[7]: 
{'OMG': 'TACOS',
 'SCIF_APPBIN': '/scif/apps/hello-world-env/bin',
 'SCIF_APPDATA': '/scif/data/hello-world-env',
 'SCIF_APPENV': '/scif/apps/hello-world-env/scif/environment.sh',
 'SCIF_APPHELP': '/scif/apps/hello-world-env/scif/runscript.help',
 'SCIF_APPLABELS': '/scif/apps/hello-world-env/scif/labels.json',
 'SCIF_APPLIB': '/scif/apps/hello-world-env/lib',
 'SCIF_APPMETA': '/scif/apps/hello-world-env/scif',
 'SCIF_APPNAME': 'hello-world-env',
  ...
 'SCIF_APPRECIPE': '/scif/apps/hello-world-env/scif/hello-world-env.scif',
 'SCIF_APPROOT': '/scif/apps/hello-world-env',
 'SCIF_APPRUN': '/scif/apps/hello-world-env/scif/runscript',
 'SCIF_APPS': '/scif/apps',
 'SCIF_DATA': '/scif/data'}

And then deactivate to undo that.

client.deactivate()

I’m skiing over sunshine with happiness to be working on this! The response to reviewers (with this update) will be submit in early March. Please contribute feedback (no matter how small!) to the docs and spec or the client (all served from that repo) and add your name to the specification. If you have already contributed, then please submit a pull request there and add your name! Onwards to scientific filesystem galaxies, friends!