docker-environment-setup by nextflow-io

---
name: Docker Environment Setup
description: Set up a Docker container for running Nextflow training examples. Handles basic setup, Docker-outside-of-Docker (DooD) for containerized processes, ARM Mac platform emulation, and troubleshooting. Use when you need to run Nextflow examples in a consistent environment.
---

# Docker Environment Setup

Set up a Docker container environment for running Nextflow training examples. This skill handles all Docker configuration needed to match the Codespaces/Gitpod environment that learners use.

## Initial Questions

Use `AskUserQuestion` to determine the setup type:

**Which Docker setup do you need?**

- **Basic setup (Recommended)** - For tutorials without containerized processes (e.g., hello_nextflow basics, plugin_development)
- **DooD setup** - For tutorials with containerized processes (e.g., genomics, essential_scripting_patterns)
- **Check/restart existing** - Verify or restart an existing nf-training container

---

## Determine NXF_VER and training image

Before any Docker commands, read the Nextflow version and the training image reference from devcontainer.json:

```bash
NXF_VER=$(grep -o '"NXF_VER":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4)
TRAINING_IMAGE=$(grep -o '"image":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4)
echo "Using NXF_VER=${NXF_VER}, TRAINING_IMAGE=${TRAINING_IMAGE}"
```

Always use `${TRAINING_IMAGE}` as read from the **currently checked-out branch's** `devcontainer.json` — do not hardcode `ghcr.io/nextflow-io/training:latest`. The tag a branch pins is the version that corresponds to what that branch's docs were written/tested against; pulling whatever the registry's `latest` tag happens to be right now can silently diverge from it. Pull that exact reference (`docker pull "${TRAINING_IMAGE}"`) before starting the container, so a stale local cache doesn't ship an outdated image under the same tag name — but never substitute a different tag than what the branch declares.

---

## Basic Setup

For tutorials that don't use containerized processes:

```bash
# Clean up any existing container
docker stop nf-training 2>/dev/null; docker rm nf-training 2>/dev/null

# Read the image reference pinned by this branch's devcontainer.json and pull it
NXF_VER=$(grep -o '"NXF_VER":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4)
TRAINING_IMAGE=$(grep -o '"image":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4)
docker pull "${TRAINING_IMAGE}"

# Start fresh container with UTF-8 locale support
docker run -d --name nf-training \
  -e NXF_VER=${NXF_VER} \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -v "${PWD}:/workspaces/training" \
  -w /workspaces/training \
  "${TRAINING_IMAGE}" \
  sleep infinity
```

**Important**: The `LANG=C.UTF-8` and `LC_ALL=C.UTF-8` environment variables are critical for handling non-ASCII characters (like "Holà", "Grüß Gott") in file names and content.

### Running Commands

```bash
docker exec -e LANG=C.UTF-8 -e LC_ALL=C.UTF-8 \
  -w /workspaces/training/<working-dir> \
  nf-training \
  <command>
```

---

## Docker-outside-of-Docker (DooD) Setup

For tutorials with containerized processes (FASTP, BWA, SAMTOOLS, etc.):

```bash
# Clean up any existing container
docker stop nf-training 2>/dev/null; docker rm nf-training 2>/dev/null

# Read the image reference pinned by this branch's devcontainer.json and pull it
NXF_VER=$(grep -o '"NXF_VER":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4)
TRAINING_IMAGE=$(grep -o '"image":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4)
HOST_PATH="${PWD}"
docker pull "${TRAINING_IMAGE}"

# Start container with DooD support
docker run -d --name nf-training \
  -e NXF_VER=${NXF_VER} \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "${HOST_PATH}:${HOST_PATH}" \
  -w "${HOST_PATH}" \
  "${TRAINING_IMAGE}" \
  sleep infinity

# Create symlink for Codespaces paths
docker exec nf-training bash -c "rm -rf /workspaces/training && mkdir -p /workspaces && ln -sf ${HOST_PATH} /workspaces/training"
```

**Critical differences from basic setup:**

1. **Docker socket mount** (`-v /var/run/docker.sock:/var/run/docker.sock`) - Allows Nextflow to spawn sibling containers
2. **Matching host paths** (`-v "${HOST_PATH}:${HOST_PATH}"`) - Work directories resolve correctly between containers
3. **Symlink** - Makes `/workspaces/training/...` paths work locally

### Running Commands with DooD

```bash
docker exec -e LANG=C.UTF-8 -e LC_ALL=C.UTF-8 -e USER=testuser \
  -w "${HOST_PATH}/<working-dir>" \
  nf-training \
  nextflow run <script.nf> [options]
```

### When DooD is Needed

Any tutorial where processes specify containers:

- `hello_nextflow` (later lessons with containers)
- `nf4_science/genomics` and other domain modules
- Side quests: `essential_scripting_patterns`, `metadata`, etc.

---

## Apple Silicon (ARM) Macs

Most bioinformatics containers are built for x86_64/amd64. On ARM Macs, create a platform config:

```bash
docker exec nf-training bash -c 'cat > /tmp/platform.config << EOF
docker.runOptions = "--platform linux/amd64"
EOF'
```

Include when running:

```bash
docker exec -e LANG=C.UTF-8 -e LC_ALL=C.UTF-8 -e USER=testuser \
  -w "${HOST_PATH}/<working-dir>" \
  nf-training \
  nextflow run <script.nf> -c /tmp/platform.config
```

**Note**: Platform emulation uses more memory. For OOM errors (exit code 137), increase Docker Desktop memory in Preferences → Resources.

---

## Troubleshooting

| Error                                    | Cause               | Solution                                             |
| ---------------------------------------- | ------------------- | ---------------------------------------------------- |
| `Cannot connect to Docker daemon`        | Socket not mounted  | Add `-v /var/run/docker.sock:/var/run/docker.sock`   |
| `.command.sh: No such file or directory` | Path mismatch       | Use matching paths: `-v "${HOST_PATH}:${HOST_PATH}"` |
| `exec format error`                      | ARM/x86 mismatch    | Add `--platform linux/amd64` to docker.runOptions    |
| Exit code 137 (OOM)                      | Insufficient memory | Increase Docker Desktop memory allocation            |
| `Malformed input or unmappable chars`    | Missing UTF-8       | Add `-e LANG=C.UTF-8 -e LC_ALL=C.UTF-8`              |
| `Error: No such container: nf-training`  | Container stopped   | Restart container (see below)                        |

---

## Container Restart Procedure

The container may stop during long sessions. To restart:

```bash
# 1. Check if container is running
docker ps | grep nf-training

# 2. If not running, restart with DooD setup
docker stop nf-training 2>/dev/null; docker rm nf-training 2>/dev/null

HOST_PATH="${PWD}"
NXF_VER=$(grep -o '"NXF_VER":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4)
TRAINING_IMAGE=$(grep -o '"image":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4)
docker pull "${TRAINING_IMAGE}"

docker run -d --name nf-training \
  -e NXF_VER=${NXF_VER} \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "${HOST_PATH}:${HOST_PATH}" \
  -w "${HOST_PATH}" \
  "${TRAINING_IMAGE}" \
  sleep infinity

# 3. Recreate symlink (critical!)
docker exec nf-training bash -c "rm -rf /workspaces/training && mkdir -p /workspaces && ln -sf ${HOST_PATH} /workspaces/training"

# 4. Recreate platform config if needed (ARM Macs)
docker exec nf-training bash -c 'cat > /tmp/platform.config << EOF
docker.runOptions = "--platform linux/amd64"
EOF'
```

---

## Cleanup

When done with testing:

```bash
docker stop nf-training && docker rm nf-training
```

---

## Notes

- Always verify you're in the repository root before starting (check for `docs/en/mkdocs.yml`)
- The container uses `sleep infinity` so it persists across multiple command executions
- Symlink must be recreated each time the container restarts
- For long sessions, periodically check container is still running: `docker ps | grep nf-training`