Job submission

struct NodeSpec

Stores information about a compute node that can be allocated for JuliaHub jobs. The list of all available node specifications can be accessed with nodespecs, or specific ones searched with nodespec.

julia> JuliaHub.nodespec()
Node: 3.5 GHz Intel Xeon Platinum 8375C
 - GPU: no
 - vCores: 2
 - Memory: 8 Gb
 - Price: 0.17 $/hr

They can be used to contruct explicit compute configuration objects when submitting JuliaHub jobs.

See also: submit_job, ComputeConfig.

JuliaHub.nodespecs(; auth::Authentication) -> Vector{NodeSpec}

Query node specifications available on the current server, returning a list of NodeSpec objects.

JuliaHub.nodespec(
    [nodes::Vector{NodeSpec}];
    ncpu::Integer=1, ngpu::Integer=false, memory::Integer=1,
    exactmatch::Bool=false, throw::Bool=true,
    [auth::Authentication]
) -> Union{NodeSpec, Nothing}

Finds the node matching the specified node parameters. Throws an InvalidRequestError if it is unable to find a node with the specific parameters. However, if throw is set to false, it will return nothing instead in that situation.

By default, it searches for the smallest node that has the at least the specified parameters (prioritizing GPU count, CPU count, and memory in this order when deciding). If exactmatch is set to true, it only returns a node specification if it can find one that matches the parameters exactly.

A list of nodes (e.g. from nodespecs) can also be passed, so that the function does not have to query the server for the list. When this method is used, it is not necessary to pass auth.

struct BatchImage

Represents an available JuliaHub batch job image. These can be passed to BatchJob to specify which underlying job image will be used for the job.

A list of available batch images can be accessed with batchimages and specific images can be constructed with batchimage.

See also: batchimages, batchimage, BatchJob, script, appbundle.

JuliaHub.batchimages([product::AbstractString]; [auth::Authentication]) -> Vector{BatchImage}

Return the list of all batch job images available to the currently authenticated user, as a list of BatchImage objects. These can be passed to BatchJob.

Optionally, by passing a product identifier, the list can be narrowed down to images available for that specific product.

See also: BatchImage, batchimage, BatchJob, script, appbundle.

JuliaHub.batchimage(
    [product::AbstractString, [image::AbstractString]];
    throw::Bool=true, [auth::Authentication]
) -> BatchImage

Pick a product job batch image from the list of all batch image, returning a BatchImage object. If image is omitted, it will return the default image corresponding to product. If product is omitted as well, it will return the default image of the instance (generally the standard Julia batch image).

Will throw an InvalidRequestError if the specified image can not be found. If throw=false, it will return nothing instead in this situation.

See also: BatchImage, batchimages, BatchJob, script, appbundle.

abstract type AbstractJobConfig

Abstract supertype of all application configuration types that can be passed to submit_job for submission as a JuliaHub job. The package has built-in support for the following application configurations:

• JuliaHub.BatchJob
• JuliaHub.ApplicationJob
• JuliaHub.PackageApp

struct BatchJob <: AbstractJobConfig

Represents the application configuration of a JuliaHub batch job. A batch job is defined by the following information:

• The Julia code that is to be executed in the job.
• Julia package environment (i.e. Project.toml, Manifest.toml) and other files, such as the appbundle.
• The underlying batch job container image (see also batchimages), which defaults to the standard Julia image by default.

Instances of this types should normally not be constructed directly, and the following functions should be used instead:

• script or @script_str: for submitting simple Julia scripts or code snippets
• appbundle: for submitting more complex "appbundles" that include additional file, private or modified package dependencies etc.

Optional arguments

• image :: Union{BatchImage, Nothing}: can be used to specify which product's batch job image will be used when running the job, by passing the appropriate BatchImage object (see also: batchimage and batchimages). If set to nothing (the default), the job runs with the default Julia image.

• sysimage :: Bool: if set to true, requests that a system image is built from the job's Manifest.toml file before starting the job. Defaults to false.

Constructors

BatchJob(::BatchJob; [image::BatchImage], [sysimage::Bool]) -> BatchJob

Construct a BatchJob, but override some of the optional arguments documented above. When the argument is omitted, the value from the underlying BatchJob object is used. This is the only constructor that is part of the public API.

This method is particularly useful when used in in combination with the @script_str string macro, to be able to specify the job image or trigger a sysimage build. For example, the following snippet will set a different batch image for the script-type job:

JuliaHub.BatchJob(
    JuliaHub.script"""
    @info "Hello World"
    """,
    image = JuliaHub.batchimage("...")
)

JuliaHub.script(...) -> BatchJob

Constructs the configuration for a script-type batch job, returning the respective BatchJob object that can then be passed to submit_job. A script-type batch job is defined by the following:

• A user-provided Julia script that gets executed on the server. Note that no validation of the input code is done.

• An optional Julia package environment (e.g. Project.toml, Manifest.toml and Artifacts.toml). If any of the TOML files are provided, they must parse as valid TOML files, but no further validation is done client-side.

  If the manifest is not provided, the project environment must be instantiated from scratch, generally pulling in the latest versions of all the dependencies (although [compat] sections are honored).

  It is also fine to omit the project file, and just provide the manifest, and the environment defined by the manifest still gets instantiated. If both are omitted, the job runs in an empty environment.

• A JuliaHub job image, which determines the container environment that will be used to execute the code in (see batchimage, batchimages, BatchImage). If omitted, the default Julia image is used.

See also the @script_str string macro to more easily submit simple scripts that are defined in code.

Methods

script(
    scriptfile::AbstractString;
    [project_directory::AbstractString], [image::BatchImage], [sysimage::Bool]
) -> BatchJob

Constructs a script-type batch job configuration the will execute the code in scriptfile. Optionally, a path to a project environment directory can be passed via project_directory, which will be searched for the environment TOML files, and a job image can be specified via image.

script(;
    code::AbstractString,
    [project::AbstractString], [manifest::AbstractString], [artifacts::AbstractString],
    [image::BatchImage], [sysimage::Bool]
) -> BatchJob

A lower-level method that can be used to construct the script-type BatchJob configuration directly in memory (i.e. without having to write out intermediate files).

The code keyword argument is mandatory and will specify contents of the Julia script that gets executed on the server. The Julia project environment can be specified by passing the contents of the TOML files via the corresponding arguments (project, manifest, artifacts). The job image can be specified via image.

JuliaHub.@script_str -> JuliaHub.BatchJob

A string macro to conveniently construct a script-type batch job configuration (BatchJob) that can be submitted as a JuliaHub job.

script = JuliaHub.script"""
@info "Hello World!"
"""

This allows for an easy submission of simple single-script jobs to JuliaHub:

JuliaHub.submit_job(
    JuliaHub.script"""
    @info "Hello World!"
    """
)

By default, the macro picks up the currently active Julia project environment (via Base.active_project()), and attaches the environment .toml files to the script. To disable this, you can call the macro with the noenv suffix, e.g.

script = JuliaHub.script"""
@info "Hello World!"
"""noenv

However, if your local environment has development dependencies, you likely need to use an appbundle instead (see appbundle).

JuliaHub.submit_job(
    JuliaHub.BatchJob(
        JuliaHub.script"""
        @info "Hello World!"
        """,
        image = JuliaHub.batchimage(...)
    )
)

JuliaHub.appbundle(
    directory::AbstractString, codefile::AbstractString;
    [image::BatchImage], [sysimage::Bool]
) -> BatchJob
JuliaHub.appbundle(
    directory::AbstractString;
    code::AbstractString, [image::BatchImage], [sysimage::Bool]
) -> BatchJob

Construct an appbundle-type JuliaHub batch job configuration. An appbundle is a directory containing a Julia environment that is bundled up, uploaded to JuliaHub, and then unpacked and instantiated as the job starts.

The primary, two-argument method will submit a job that runs a file from within the appbundle (specified by codefile, which must be a path relative to the root of the appbundle). The code that gets executed is read from codefile, which should be a path to Julia source file relative to directory.

JuliaHub.appbundle(@__DIR__, "my-script.jl")

Alternatively, if codefile is omitted, the code must be provided as a string via the code keyword argument.

JuliaHub.appbundle(
    @__DIR__,
    code = """
    @show ENV
    """
)

See BatchJob for a description of the optional arguments.

Extended help

The following should be kept in mind about how appbundles are handled:

• The bundler looks for a Julia environment (i.e. Project.toml and/or Manifest.toml files) at the root of the directory. If the environment does not exist (i.e. the files are missing), the missing files are created. If the manifest is missing, then the environment is re-instantiated from scratch based on the contents of Project.toml. The generated files will also be left in the user-provided directory directory.

• Development dependencies of the environment (i.e. packages added with pkg> develop or Pkg.develop()) are also bundled up into the archive that gets submitted to JuliaHub (including any current, uncommitted changes). Registered packages are installed via the package manager via the standard environment instantiation, and their source code is not included in the bundle directly.

• When the JuliaHub job starts, the bundle is unpacked and the job's starting working directory is set to the root of the unpacked appbundle directory, and you can e.g. load the data from those files with just read("my-data.txt", String).

  JuliaHub 6.2 and older

  On some JuliaHub versions (6.2 and older), the working directory was set to the parent directory of unpacked appbundle (with the appbundle directory called appbundle), and so it was necessary to do joinpath("appbundle", "mydata.dat") to load files.

• When submitting appbundles with the two-argument codefile method, you can expect @__DIR__ and include to work as expected.

  However, when submitting the Julia code as a string (via the code keyword argument), the behavior of @__DIR__ and include should be considered undefined and subject to change in the future.

• The one-argument + code keyword argument method is a lower-level method, that more closely mirrors the underlying platform API. The custom code that is passed via code is sometimes referred to as the "driver script", and the two-argument method is implemented by submitting an automatically constructed driver script that actually loads the specified file.

struct ComputeConfig

This type encapsulates the configuration of a jobs's compute cluster, including the hardware configuration and the cluster topology.

See also: submit_job.

Constructors

JuliaHub.ComputeConfig(
    node::NodeSpec;
    nnodes::Integer = 1,
    process_per_node::Bool = true,
    elastic::Bool = false,
)

• node: a NodeSpec object that specifies the hardware of a single node.

• nnodes::Union{Integer, Tuple{Integer, Integer}} = 1: specifies the number of nodes of type node that will be allocated. Alternatively, a two-integer tuple can also be passed, where the first value specifies the minimum number of nodes required to start a job. By default, a single-node job is started.

• process_per_node::Bool = true: if true, there will only be a single Julia process per node, and the total number of Julia processes will be nnodes. If set to false, however, each core on each node will be allocated a separate Julia process (running in an isolated container on the same node), and so the total number of Julia processes will be nnodes × ncpu, and it will essentially always be a multi-process job.

• elastic::Bool = false: if set, the job will be started in an elastic cluster mode. In this case, a minimum number of nnodes must not be passed.

JuliaHub.submit_job(
    app::Union{AbstractJuliaHubApp, AbstractJobConfig},
    [compute::ComputeConfig];
    # Compute keyword arguments
    ncpu::Integer = 1, ngpu::Integer = 0, memory::Integer = 1,
    nnodes::Integer = 1, minimum_nnodes::Union{Integer,Nothing} = nothing,
    elastic::Bool = false,
    process_per_node::Bool = true,
    # Runtime configuration keyword arguments
    [alias::AbstractString], [env], [expose::Integer],
    [project::Union{UUID, AbstractString}], timelimit::Limit = Hour(1),
    # General keyword arguments
    dryrun::Bool = false,
    [auth :: Authentication]
) -> Job

Submits the specified application config app as a job to JuliaHub. Returns a Job object corresponding to the submitted job.

Compute arguments. If compute is passed, the compute keyword arguments can not be passed. If compute is not passed, the following arguments can be used to specify the compute configration via keyword arguments:

• ncpu, ngpu and memory are used to pick a node type that will be used to run the job. The node type will be a minimum one that satisfies the constraints, but may have more compute resources than specified by the arguments (it corresponds to the exactmatch = false case of nodespec).

• nnodes, minimum_nnodes, process_per_node, and elastic specify the corresponding arguments in ComputeConfig.

Runtime configuration. These are used to set the Runtime configuration of the job.

• alias :: Union{AbstractString, Nothing}: can be used to override the name of the job that gets displayed in the UI. Passing nothing is equivalent to omitting the argument.

• timelimit :: Limit: sets the job's time limit (see Limit for valid values)

• env: an iterable of key-value pairs that can be used to set environment variable that get set before, the job code gets executed.

• project :: Union{UUID, AbstractString, Nothing}: the UUID of the project that the job will be associated with. If a string is passed, it must parse as a valid UUID. Passing nothing is equivalent to omitting the argument.

• expose :: Union{Integer, Nothing}: if set to an integer in the valid port ranges, that port will be exposed over HTTPS, allowing for (authenticated) HTTP request to be performed against the job, as long as the job binds an HTTP server to that port. The allowed port ranges are 1025-9008,9010-23399,23500-32767(in other words,<= 1024,9009,23400-23499, and>= 32768` can not be used). See the relevant manual section for more information.

General arguments.

• auth :: Authentication: optional authentication object (see the authentication section for more information)

• dryrun :: Bool: if set to true, submit_job does not actually submit the job, but instead returns a WorkloadConfig object, which can be used to inspect the configuration that would be submitted.

  The WorkloadConfig object can then be submitted to JuliaHub with the additional submit_job method:

  JuliaHub.submit_job(::WorkloadConfig; [auth::Authentication])

JuliaHub.Limit

Type-constraint on JuliaHub job timelimit arguments in submit_job.

The job time limit can either be a time period (an instance of Dates.Period), an Integer, (interpreted as the number of hours), or JuliaHub.Unlimited().

Only an integer number of hours are accepted by JuliaHub, and fractional hours from get rounded up to the next full integer number of hours (e.g. Dates.Minute(90) will be interpreted as 2 hours).

struct Unlimited

An instance of this type can be passed as the [timelimit] option to submit_job to start jobs that run indefinitely, until killed manually.

JuliaHub.submit_job(..., timelimit = JuliaHub.Unlimited())

struct WorkloadConfig

Represents a full job configuration, including the application, compute and runtime configuration.

Instances of this type can be constructed by passing dryrun = true to submit_job, and can also be directly submitted to JuliaHub with the same function.

abstract type AbstractJuliaHubApp

Abstract supertype for JuliaHub applications object types.

JuliaHub.applications([category::Symbol]; [auth::Authentication]) -> Vector{AbstractJuliaHubApp}

Returns the list of applications enabled for the authenticated user, optionally in the specified category only. Returns a vector of AbstractJuliaHubApp instances.

julia> JuliaHub.applications()
7-element Vector{JuliaHub.AbstractJuliaHubApp}:
 JuliaHub.application(:default, "Linux Desktop")
 JuliaHub.application(:default, "Julia IDE")
 JuliaHub.application(:default, "Pluto")
 JuliaHub.application(:default, "Windows Workstation")
 JuliaHub.application(:package, "RegisteredPackageApp")
 JuliaHub.application(:package, "CustomDashboardApp")
 JuliaHub.application(:user, "ExampleApp.jl")

julia> JuliaHub.applications(:default)
4-element Vector{JuliaHub.DefaultApp}:
 JuliaHub.application(:default, "Linux Desktop")
 JuliaHub.application(:default, "Julia IDE")
 JuliaHub.application(:default, "Pluto")
 JuliaHub.application(:default, "Windows Workstation")

julia> JuliaHub.applications(:package)
2-element Vector{JuliaHub.PackageApp}:
 JuliaHub.application(:package, "RegisteredPackageApp")
 JuliaHub.application(:package, "CustomDashboardApp")

julia> JuliaHub.applications(:user)
1-element Vector{JuliaHub.UserApp}:
 JuliaHub.application(:user, "ExampleApp.jl")

JuliaHub.application(
    category::Symbol, name::AbstractString;
    throw::Bool=true, [auth::Authentication]
) -> AbstractJuliaHubApp

Returns the application corresponding to name from the specified category of applications. Will throw an InvalidRequestError if the application can't be found, or returns nothing in this situation if throw=false is passed.

category specifies the application category and must be one of: :default, :package, or :user. This is necessary to disambiguate apps with the same name in the different categories.

See also: applications.

Examples

julia> JuliaHub.applications()
7-element Vector{JuliaHub.AbstractJuliaHubApp}:
 JuliaHub.application(:default, "Linux Desktop")
 JuliaHub.application(:default, "Julia IDE")
 JuliaHub.application(:default, "Pluto")
 JuliaHub.application(:default, "Windows Workstation")
 JuliaHub.application(:package, "RegisteredPackageApp")
 JuliaHub.application(:package, "CustomDashboardApp")
 JuliaHub.application(:user, "ExampleApp.jl")

struct DefaultApp <: AbstractJuliaHubApp

Represents a default JuliaHub instance application, and they can be started as jobs with submit_job.

The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.

julia> apps = JuliaHub.applications(:default)
4-element Vector{JuliaHub.DefaultApp}:
 JuliaHub.application(:default, "Linux Desktop")
 JuliaHub.application(:default, "Julia IDE")
 JuliaHub.application(:default, "Pluto")
 JuliaHub.application(:default, "Windows Workstation")

julia> JuliaHub.application(:default, "Pluto")
DefaultApp
 name: Pluto
 key: pluto

struct PackageApp <: AbstractJuliaHubApp

Represents a JuliaHub package application that is available in one of the instance's package registries. These packages can be started as JuliaHub jobs with submit_job.

The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.

julia> apps = JuliaHub.applications(:package)
2-element Vector{JuliaHub.PackageApp}:
 JuliaHub.application(:package, "RegisteredPackageApp")
 JuliaHub.application(:package, "CustomDashboardApp")

julia> JuliaHub.application(:package, "RegisteredPackageApp")
PackageApp
 name: RegisteredPackageApp
 uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5
 registry: General (23338594-aafe-5451-b93e-139f81909106)

See also: help.juliahub.com on applications

struct UserApp <: AbstractJuliaHubApp

Represents a private application that has been added to the user account via a Git repository. These applications can be started as JuliaHub jobs with submit_job.

The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.

julia> apps = JuliaHub.applications(:user)
1-element Vector{JuliaHub.UserApp}:
 JuliaHub.application(:user, "ExampleApp.jl")

julia> JuliaHub.application(:user, "ExampleApp.jl")
UserApp
 name: ExampleApp.jl
 repository: https://github.com/JuliaHubExampleOrg/ExampleApp.jl

See also: help.juliahub.com on applications

struct ApplicationJob <: AbstractJobConfig

AbstractJobConfig that wraps a DefaultApp. This is primarily used internally and should rarely be constructed explicitly.

struct PackageJob <: AbstractJobConfig

AbstractJobConfig that wraps a PackageApp or UserApp. This is primarily used internally and should rarely be constructed explicitly.

Constructors

JuliaHub.PackageJob(app::Union{JuliaHub.PackageApp,JuliaHub.UserApp}; [sysimage::Bool = false])

Can be used to construct a PackageApp or UserApp based job, but allows for some job parameters to be overridden. Currently, only support the enabling of a system image based job by setting sysimage = true.

julia> app = JuliaHub.application(:package, "RegisteredPackageApp")
PackageApp
 name: RegisteredPackageApp
 uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5
 registry: General (23338594-aafe-5451-b93e-139f81909106)

julia> JuliaHub.submit_job(JuliaHub.PackageJob(app; sysimage = true))
JuliaHub.Job: jr-xf4tslavut (Submitted)
 submitted: 2023-03-15T07:56:50.974+00:00
 started:   2023-03-15T07:56:51.251+00:00
 finished:  2023-03-15T07:56:59.000+00:00