Building your own JuliaHub Job Images

What are JuliaHub Job Images?

JuliaHub Job Images are Docker containers that JuliaHub uses to execute batch jobs and applications. These images provide the runtime environment for your code, including the operating system, Julia installation, system libraries, and any pre-installed packages or dependencies.

Job images serve as the foundation for running computational workloads on JuliaHub, whether you're running batch processing jobs, interactive applications, or distributed computing tasks. They ensure consistent, reproducible execution environments across the JuliaHub platform.

Versioning and Compatibility

Job image versioning follows semantic versioning (MAJOR.MINOR.PATCH):

  • MAJOR version changes indicate incompatible API changes to the job image interfaces
  • MINOR version changes add new features or update embedded components (Julia, packages, OS components)
  • PATCH version changes contain bug fixes

JobAPI Version Mapping

JuliaHub uses JobAPI versions to define the interface between job images and the JuliaHub platform. Here's how they map to image versions:

  • JobAPI v0.1.0: Image versions < 10.6 (Version 1 images)
  • JobAPI v0.2.0: Image versions >= 10.6 (Late Version 1 images)
  • JobAPI v0.3.0: Image versions >= 10.X (Version 2 images)

Understanding Version Types

  • Job Image Versions: These are the Docker image release versions that follow semantic versioning and track incremental improvements, bug fixes, and component updates
  • JobAPI Versions: These define the interface/protocol versions between images and the JuliaHub platform, establishing compatibility boundaries
  • Relationship: Multiple job image versions can share the same JobAPI version, but breaking changes in the interface require a new JobAPI version

Each JuliaHub platform release specifies which JobAPI versions it supports. The platform maintains reasonable backward compatibility - for example, an image built with JobAPI version v0.2.0 will work with JuliaHub installations supporting JobAPI in the range [0.2.0 - 1.0.0).

Job Images Version 2: Customer-Managed Independence

Version 2 of JuliaHub Job Images give batch jobs and applications complete control over their execution environments while still enabling their integration with JuliaHub services.

JuliaHub Managed Services for Job Images

Version 2 introduces a sidecar, which is a separate process that runs alongside, but isolated from the main container running user code. It runs in a separate container. It provides JuliaHub services through REST APIs running alongside your container. This eliminates the need to integrate these services directly into your image.

Core Infrastructure Services

  • Registry Management
    • Automatic provisioning of Julia registries in your depot during startup
    • Supports both public and private registries based on user permissions
    • No user code changes required - registries are available when Julia starts
  • Credentials Management
    • Automatic provisioning and periodic refresh of JuliaHub tokens and AWS credentials
    • Credentials are made available via shared volumes at runtime
    • Transparent to user code - existing credential access patterns continue to work
  • Job Status Reporting
    • APIs for reporting job progress, completion, and failure states
    • Automatic status detection by an injected monitoring process

Data Management Services

  • Inputs and Outputs
    • Inputs automatically downloaded before job start
    • Outputs uploaded from a pre-decided the directory
  • Datasets Integration
    • Requires explicit use of JuliaHubData.jl package
    • Available through JuliaHub's private registry
    • Provides programmatic access to JuliaHub datasets and data management features
  • Runtime Secrets Access
    • REST API access to user-configured and project level secrets

Distributed Computing

  • Managed through JuliaHubDistributed.jl package available in JuliaHub registry
  • Key Methods:
    • JuliaHubDistributed.start() - Initiates worker provisioning
    • JuliaHubDistributed.wait_for_workers() - Waits for cluster readiness
    • JuliaHubDistributed.Elastic module for elastic worker pools
  • Automatic reporting of distributed status (workers started, joined, etc.)
  • Works on local development machines for testing

Logging

  • Automatic collection of stdout/stderr streams
  • Optional structured logging via JuliaHubLogging.jl package for enhanced UI rendering

Migration Considerations

The managed services architecture introduces several breaking changes that require user code updates:

  • Secrets: Must use JuliaHubSecrets.jl instead of direct access
  • Datasets: Must explicitly import JuliaHubData.jl
  • Distributed Computing: Must use JuliaHubDistributed.jl instead of built-in functions
  • Output Handling: Must use pre-set environment variables instead of setting them
  • Structured Logging: Must use JuliaHubLogging.jl for enhanced log rendering

These changes enable much greater flexibility and performance while maintaining the full feature set of JuliaHub's platform services. Refer to the migration guide for detailed instructions on updating your code.

Key Improvements

Complete Image Independence: Unlike version 1, you no longer need to build your images on top of JuliaHub base images. You can create completely custom Docker images from any base you choose, as long as they comply with the interface requirements.

Sidecar Architecture: Version 2 introduces a sidecar-based architecture where JuliaHub functionality is provided through REST APIs running alongside your main container. This eliminates the complexity of wrapping your job process and allows for cleaner separation of concerns.

Simplified Dependency Management: Version 2 eliminates the problematic stacked Julia depot/environment issues that occurred in version 1. You no longer need to layer or include base Manifests in your sysimage builds, giving you complete control over your Julia environment.

Flexible Integration: Jobs can choose which JuliaHub integrations they need and don't pay the penalty for unused features. For example, if your job doesn't need Julia registries, it won't download or refresh all available registries.

This approach enables truly customer-managed images where you maintain full control over the software stack while benefiting from JuliaHub's platform services through well-defined APIs.

Version 2 Interface Details

Container Image Requirements

Version 2 job images must meet specific requirements to integrate properly with JuliaHub's execution environment:

Platform Requirements

  • Architecture: Must be based on amd64 Linux
  • Operating System: Any Linux distribution (JuliaHub base images use Ubuntu 22.04, but that is not a restriction)

User Account Requirements

Your image must include a user account with these exact specifications:

  • Username: jrun
  • User ID: 8000
  • Group ID: 8000
  • Groups: 8000(jrun), 50(staff)
  • Home Directory: /home/jrun

Julia Environment Requirements

  • Default Depot: Julia depot must be at the default location /home/jrun/.julia
  • No Internal Depots: Version 2 eliminates internal Julia depots, giving you complete control

Startup Command Requirements

Your image must specify the startup command via the JULIAHUB_RUN_CMD environment variable:

  • Required: ENV JULIAHUB_RUN_CMD=/path/to/your/startup/script
  • Common Location: /opt/juliahub/v2/jhrun/run.sh
  • Permissions: The startup script must be executable by the jrun user

File Structure Requirements

Version 2 images must follow a specific directory structure to integrate properly with JuliaHub:

├── home
│   └── jrun
│       ├── JuliaHubResultsSummary.json # reserved: job summary file
│       ├── data                        # reserved: user EFS symlink path
│       ├── outputs                     # reserved: job output directory
│       └── workspace                   # reserved: PWD for batch jobs, job files get copied here
├── mnt                                 # reserved: for mounting various volumes
│   ├── (other mounted secrets)
│   └── data                            # reserved: user EFS mount path
├── opt
│   ├── julia                           # reserved: julia provisioning
│   ├── juliaup                         # reserved: julia provisioning
│   └── juliahub
│       └── v2
│           ├── registries-input        # reserved for registry input from the sidecar
│           ├── files-input             # reserved for file input from the sidecar
│           ├── batchcommon.sh          # appears in many images
│           ├── tracing-utilities.sh    # appears in many images
│           ├── env_utils.sh            # appears in many images
│           ├── worker.sh               # appears in batch images
│           └── jhrun                   # reserved for image entrypoints, copy your run.sh here
├── juliahubbundles                     # reserved: sysimage bundle mount path
├── var
│   └── run
│       ├── config                      # reserved: configmap mount path
│       └── secrets
│           └── jobsecrets              # reserved: secrets mount path

Environment Variables

Version 2 implements a three-stage environment variable processing system:

Stage 1: Initial Container Environment

The main container starts with only three essential environment variables:

  • JRUN_NAME: Job instance identifier
  • JRUN_CLUSTER_COOKIE: Cluster authentication token
  • JULIAHUB_CONFIG_MAP_MOUNT: Configuration file location

Stage 2: Startup Executable Environment

The startup executable reads configuration and prepares a comprehensive environment for your application, including:

  • HOME: User home directory path
  • HOSTNAME: Container hostname
  • JRUN_CLUSTER_COOKIE: Cluster authentication token
  • JRUN_NAME: Job instance identifier
  • JULIAHUB_API_PROXY_URL: JuliaHub API endpoint URL
  • JULIAHUB_APP_PORT: Application hosting port number
  • JULIAHUB_APP_URL: Application public access URL
  • JULIAHUB_CODEJL_FILE_NAME: Code file name
  • JULIAHUB_CODESERVER_CONFIG_SUFFIX: Code server configuration suffix
  • JULIAHUB_CODESERVER_EXTENSIONS_SUFFIX: Code server extensions suffix
  • JULIAHUB_DISTRIBUTED_TOPOLOGY: Distributed computing configuration
  • JULIAHUB_ELASTIC_WORKERS: Elastic worker pool enabled
  • JULIAHUB_JOB_NAME: Job name without worker suffix
  • JULIAHUB_JOB: Running in JuliaHub environment flag
  • JULIAHUB_JULIA_HOME_SUFFIX: Julia home directory suffix
  • JULIAHUB_LOCAL_DEPOT_PATH: Local Julia depot path
  • JULIAHUB_LOCAL_WORKSPACE: Local workspace directory
  • JULIAHUB_MASTER_POD_HOST: Master pod hostname
  • JULIAHUB_MIN_WORKERS_REQUIRED: Minimum required workers
  • JULIAHUB_NTHREADS: Number of Julia threads
  • JULIAHUB_NWORKERS: Number of worker processes
  • JULIAHUB_PKG_SERVER_URL: Julia package server URL
  • JULIAHUB_PRODUCT_NAME: Product name identifier
  • JULIAHUB_PROJECT_PATH: Project directory path
  • JULIAHUB_PROJECT_UUID: Unique project identifier
  • JULIAHUB_RESULTS_SUMMARY_FILE: Job results summary file path
  • JULIAHUB_RESULTS_UPLOAD_DIR: Output files directory path
  • JULIAHUB_RUN_MODE: Job execution mode
  • JULIAHUB_SHARED_VOLUME_DEPOT_PATH: Shared depot path
  • JULIAHUB_SHARED_VOLUME_PATH: Shared volume mount path
  • JULIAHUB_USE_SYSIMAGE: Use custom sysimage flag
  • JULIAHUB_USER_EMAIL: User email address
  • JULIAHUB_USER_ENVS: User environment variables JSON
  • JULIAHUB_USER_FULL_NAME: User display name
  • JULIAHUB_USER_PUBLIC_USERNAME: User public identifier
  • JULIAHUB_WORKER_WAIT_TIMEOUT: Worker connection timeout seconds
  • PATH: System executable search paths
  • PWD: Current working directory
  • SHLVL: Shell nesting level
  • Additional variables for specialized features

Here's a visual representation of the three-stage environment processing:

┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐
│   Stage 1: Initial  │    │  Stage 2: Startup   │    │ Stage 3: User App   │
│   Container Env     │───▶│  Executable Env     │───▶│   Environment       │
│                     │    │                     │    │                     │
│ • JRUN_NAME         │    │ • All Stage 1 vars  │    │ • Julia-specific    │
│ • JRUN_CLUSTER_     │    │ • JULIAHUB_API_*    │    │ • JULIA_DEPOT_PATH  │
│   COOKIE            │    │ • JULIAHUB_USER_*   │    │ • JULIA_NUM_THREADS │
│ • JULIAHUB_CONFIG_  │    │ • JULIAHUB_JOB_*    │    │ • Custom app vars   │
│   MAP_MOUNT         │    │ • + 25+ more vars   │    │ • (Implementation   │
│                     │    │                     │    │    dependent)       │
└─────────────────────┘    └─────────────────────┘    └─────────────────────┘

Stage 3: User Application Environment

When the startup executable invokes your application via JULIAHUB_RUN_CMD, the environment state is up to your image implementation. Default JuliaHub images usually provide the following environment variables in batch environment. Note that this is only indicative and you can customize it as per your needs.

  • GKSwstype: Graphics backend type
  • HOME: User home directory path
  • HOSTNAME: Container hostname
  • JH_APP_URL: Application URL (deprecated, use JULIAHUBAPPURL)
  • JRUN_NAME: Job instance identifier
  • JULIA_DEPOT_PATH: Julia package depot locations
  • JULIA_GR_PROVIDER: Julia GR plotting backend
  • JULIA_NUM_THREADS: Number of Julia threads
  • JULIA_PKG_SERVER: Julia package registry server
  • JULIA_PKG_USE_CLI_GIT: Use CLI Git for packages
  • JULIA_WORKER_TIMEOUT: Worker connection timeout seconds
  • JULIAHUB_APP_PORT: Application hosting port number
  • JULIAHUB_APP_URL: Application public access URL
  • JULIAHUB_DISTRIBUTED_TOPOLOGY: Distributed computing configuration
  • JULIAHUB_MIN_WORKERS_REQUIRED: Minimum required workers
  • JULIAHUB_ELASTIC_WORKERS: Elastic worker pool enabled
  • JULIAHUB_NWORKERS: Number of worker processes
  • JULIAHUB_API_PROXY_URL: JuliaHub API endpoint URL
  • JULIAHUB_JOB_NAME: Job name without worker suffix
  • JULIAHUB_JOB: Running in JuliaHub environment flag
  • JULIAHUB_PRODUCT_NAME: Product name identifier
  • JULIAHUB_PROJECT_PATH: Project directory path
  • JULIAHUB_PROJECT_UUID: Unique project identifier
  • JULIAHUB_RESULTS_SUMMARY_FILE: Job results summary file path
  • JULIAHUB_RESULTS_UPLOAD_DIR: Output files directory path
  • JULIAHUB_RUN_MODE: Job execution mode
  • JULIAHUB_USER_EMAIL: User email address
  • JULIAHUB_USER_FULL_NAME: User display name
  • JULIAHUB_WORKER_WAIT_TIMEOUT: Worker connection timeout seconds
  • NVIDIA_DRIVER_CAPABILITIES: NVIDIA GPU capabilities
  • NVIDIA_REQUIRE_CUDA: Required CUDA version
  • NVIDIA_VISIBLE_DEVICES: Visible GPU devices
  • PATH: System executable search paths
  • PWD: Current working directory
  • SHLVL: Shell nesting level

Execution Flow

The version 2 execution flow follows these steps:

  1. InitContainer Injection: An initContainer injects the startup executable into your container before it starts
  2. Configuration Reading: The startup executable reads all configuration from JULIAHUB_CONFIG_MAP_MOUNT
  3. Environment Preparation: The executable prepares the comprehensive environment state described above
  4. Application Invocation: The executable calls the command specified in JULIAHUB_RUN_CMD (typically pointing to your run.sh script)
  5. User Application Start: Your application starts with the prepared environment and full access to JuliaHub services through sidecar APIs

Here's a visual representation of the v2 architecture and execution flow:

┌─────────────────┐    ┌────────────────────┐    ┌─────────────────┐
│ Init Container  │    │ Main Container     │    │  Sidecar        │
│                 │    │                    │    │                 │
│ • Injects       │───>│ 1. Startup         │<──>│ • REST APIs     │
│   startup       │    │    executable      │    │ • Secrets       │
│   executable    │    │                    │    │ • Files         │
│                 │    │ 2. Reads config    │    │ • Registries    │
└─────────────────┘    │                    │    │ • Status        │
                       │ 3. Sets up env     │    │                 │
                       │                    │    └─────────────────┘
                       │ 4. Calls           │
                       │   JULIAHUB_RUN_CMD │
                       │                    │
                       │ 5. Your app starts │
                       │    starts          │
                       │ ┌────────────────┐ │
                       │ │ Your Batch     │ │
                       │ │ or Interactive │ │
                       | | application    │ │
                       │ └────────────────┘ │
                       └────────────────────┘

The key environment variable JULIAHUB_RUN_CMD should point to your main application startup script (e.g., /opt/juliahub/v2/jhrun/run.sh).

Migrating from Version 1 to Version 2

For Batch Images (Julia-based)

If you've been maintaining a batch image built on top of a JuliaHub batch image:

  1. Update Base Image: Move to a v2 base image and verify everything works correctly
  2. Environment Changes: Most changes are user-facing additions, so the transition should be straightforward if your image only extended the base with sysimages or additional executables
  3. Depot Management: Since there are no internal depots, you no longer need to layer or include the base Manifest in your sysimage builds

For Applications

Version 2 provides much more flexibility for application developers:

Key Changes:

  • You no longer need to build on any JuliaHub base image - completely custom images are supported
  • The entrypoint mechanism has changed - create a startup script and point to it with JULIAHUB_RUN_CMD
  • New environment variables are available at runtime

TLDR Migration Checklist

  1. Drop JuliaHub Base Dependency (optional): You can build completely custom images or continue using JuliaHub base images
  2. Update Base Image: If staying with JuliaHub bases, adjust your Dockerfile for v2 base images
  3. Remove Base Manifests: Sysimages no longer need to include any base Manifest or depot dependencies
  4. Create Startup Script: Prepare your run.sh type script and set JULIAHUB_RUN_CMD to point to it
  5. Handle Environment Changes: Adjust for the new environment variable structure

Environment Variable Changes

New Environment Variables (non-breaking additions):

  • HOSTNAME added
  • JRUN_CLUSTER_COOKIE used in JuliaHubDistributed
  • JULIAHUB_API_PROXY_URL for use in user code/juliahub supplied libraries to invoke APIs
  • JULIAHUB_APP_PORT used in apps hosting servers
  • JULIAHUB_DISTRIBUTED_TOPOLOGY used in JuliaHubDistributed
  • JULIAHUB_ELASTIC_WORKERS used in JuliaHubDistributed
  • JULIAHUB_JOB_NAME jobname without -worker suffix
  • JULIAHUB_JOB whether this is running in juliahub environment
  • JULIAHUB_MIN_WORKERS_REQUIRED
  • JULIAHUB_NWORKERS used in JuliaHubDistributed
  • JULIAHUB_RESULTS_SUMMARY_FILE for use in user code, points at the summary file
  • JULIAHUB_RESULTS_UPLOAD_DIR for use in user code, points at the outputs directory
  • JULIAHUB_RUN_MODE used in JuliaHubDistributed
  • JULIAHUB_USER_EMAIL renamed from JULIAHUB_USEREMAIL
  • JULIAHUB_WORKER_WAIT_TIMEOUT used in JuliaHubDistributed elastic mode
  • PWD added
  • SHLVL added

Removed Environment Variables (breaking changes):

  • LD_LIBRARY_PATH - removed
  • INTERNAL_DEPOT_PATH - removed
  • JULIA_DATASETS_PATH - removed
  • JULIAHUB_JULIA_HOME_SUBDIR - removed
  • JULIAHUB_USEREMAIL - renamed to JULIAHUB_USER_EMAIL
  • REGISTRIES - removed

Example Migration

Here's a simple example of setting up a custom v2 image:

FROM ubuntu:22.04

# Create required user
RUN groupadd -g 8000 jrun && useradd -u 8000 -g 8000 -m -s /bin/bash jrun

# Install your dependencies
RUN apt-get update && apt-get install -y \
    julia \
    python3 \
    # your other dependencies
    && rm -rf /var/lib/apt/lists/*

# Create required directories
RUN mkdir -p /opt/juliahub/v2/jhrun \
    /home/jrun/workspace \
    /home/jrun/outputs \
    /mnt/data \
    /var/run/config \
    /var/run/secrets/jobsecrets

# Copy your startup script
COPY run.sh /opt/juliahub/v2/jhrun/run.sh
RUN chmod +x /opt/juliahub/v2/jhrun/run.sh

# Set the required environment variable
ENV JULIAHUB_RUN_CMD=/opt/juliahub/v2/jhrun/run.sh

USER jrun

Your run.sh script then has access to all JuliaHub environment variables and can integrate with JuliaHub services as needed.