Jobs

JuliaHub.jl provides various methods to inspect and interact jobs that have been submitted to JuliaHub. It is also possible to get access to job outputs programmatically.

Job status states

A job will be in various "states" during its execution, and its current state can be inspected via the [.status] field.

The following diagram illustrates the possible states a job can be in, and how it moves between them during a job lifecycle.

stateDiagram-v2 direction LR [*] --> SUBMIT:::user SUBMIT --> Submitted Submitted --> Running Running --> FINISHED state FINISHED { direction LR [*] --> Failed [*] --> Stopped [*] --> Completed } FINISHED:::user classDef user fill:lightgray
Refreshing a Job object

An instance of a Job object reflects the state of the job when the job function was called. If you want to inspect the current state of a job, you must first "refresh" the job object, which can simply be done with the help of the job function.

job = JuliaHub.job(job)

Jobs with exposed ports

Some JuliaHub jobs may expose ports and can be communicated with from the outside over the network (e.g. batch jobs that expose ports).

If the job exposes a port, it can be accessed at a dedicated hostname (see the .hostname property of the Job object). The server running on the job is always exposed on port 443 on the public hostname, and the communication is TLS-wrapped (i.e. you need to connect to it over the HTTPS protocol). In most cases, your requests to the job also need to be authenticated (see also the JuliaHub.request function).

See also: the guide on submitting batch jobs with open ports, expose argument for JuliaHub.submit_job, JuliaHub.request

Reference

JuliaHub.JobReferenceType
const JobReference :: Type

A type constraint on the arguments of many jobs-related functions that is used to specify the job. A job reference must be either a Job object, or an AbstractString containing the unique job ID.

source
JuliaHub.jobsFunction
JuliaHub.jobs(; [limit::Integer], [auth::Authentication]) -> Vector{Job}

Retrieve the list of jobs, latest first, visible to the currently authenticated user.

By default, JuliaHub only returns up to 20 jobs. However, this default limit can be overridden by passing the limit keyword (which must be a positive integer).

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

source
JuliaHub.jobFunction
JuliaHub.job(job::JobReference; throw::Bool=true, [auth::Authentication]) -> Job

Fetch the details of a job based on the job reference ref. Will throw an InvalidRequestError if the job does not exist, or returns nothing if throw=false is passed.

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

source
JuliaHub.isdoneFunction
JuliaHub.isdone(::Job)

A helper function to check if a Job is "done", i.e. its status is one of Completed, Stopped, or Failed.

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. As such, the result from this function may not represent the current live state of the job. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

source
JuliaHub.wait_jobFunction
wait_job(
    job::AbstractString;
    interval::Integer = 30, [auth::Authentication]
) -> Job

Blocks until remote job referred to by the job reference job has completed, by polling it with every interval seconds. Returns an updated Job object.

source
JuliaHub.extend_jobFunction
JuliaHub.extend_job(job::JobReference, extension::Limit; [auth::Authentication]) -> Job

Extends the time limit of the job referred to by the job reference ref by extension (Dates.Period, or Integer number of hours). Returns an updated Job object.

See Limit for more information on how the extension argument is interpreted. Note that Unlimited is not allowed as extension.

See also: Job.

source
JuliaHub.JobLogMessageType
struct JobLogMessage

Contains a single JuliaHub job log message, and has the following fields:

  • timestamp :: Union{ZonedDateTime, Nothing}: log message timestamp (in UTC)
  • message :: Union{String, Nothing}: log message string. This generally corresponds to one line of printed output

Fields that can also be nothing may be missing for some log messages.

See also: job_logs, job_logs_buffered.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.job_logsFunction
JuliaHub.job_logs(job; offset::Integer = 0, [limit::Integer], [auth::Authentication]) -> Vector{JobLogMessage}

Fetches the log messages for the specified JuliaHub job. The job is specifed by passing the job name as a string, or by passing a Job object (i.e. job::Union{AbstractString,Job}). Returns the log messages as an array of JobLogMessage objects.

Optionally, the function takes the following keyword arguments:

  • offset::Integer: the offset of the first log message fetched (0 corresponds to the first message); for the first method, this defaults to 0; however, in the second (callback) case, if offset is not specified, any existing logs will be ignored.

  • limit::Integer: the maximum number of messages fetched (all by default)

No default limit

The limit keyword does not have a default limit, and so by default job_logs fetches all the log messages. This may take a while and require many requests to JuliaHub if the job has a huge number of log messages.

source
JuliaHub.job_logs_bufferedFunction
JuliaHub.job_logs_buffered(
    [f::Base.Callable], job::Union{Job,AbstractString};
    streaming::Bool=true, [offset::Integer],
    [auth::Authentication]
) -> AbstractJobLogsBuffer

A lower-level function to work with log streams, and is particularly useful when working with jobs that have not finished yet and are actively producing new log messages.

The function accepts the following arguments:

  • f :: Base.Callable: an optional callback function that gets called every time the buffer is updated. The callback must take two arguments: f(::AbstractJobLogsBuffer, ::AbstractVector). The first argument is the buffer object itself, and the second argument will be passed a read-only view of all the logs that have been loaded into the buffer, including the new ones.
  • job :: Union{Job,AbstractString}: either the job name or a Job object.
  • streaming :: Bool: if set to true, the buffer object The streaming can be stopped with interrupt!.
  • offset :: Integer: optional non-negative value to specify the starting point of the buffer

Interface of the returned object

Returns an instance of the abstract AbstractJobLogsBuffer type. These objects contain log messages (of type JobLogMessage), but not all the log messages are immediately available. Instead, at any given time the buffer represents a continous section of logs that can be extended in either direction.

The following functions can be used to interact with log buffers: job_logs_newer!, job_logs_older!, JuliaHub.hasfirst, JuliaHub.haslast. Additionally, the objects will have a .logs :: Vector{JobLogMessage} property that can be used to access the log messages that have been loaded into the buffer.

See also: job_logs, Job.

source
JuliaHub.job_logs_older!Function
JuliaHub.job_logs_older!(
    buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]
) -> AbstractJobLogsBuffer

Updates the AbstractJobLogsBuffer object by adding up to count log messages to the beginning of the buffer. If count is omitted, it will seek all the way to the beginning of the logs.

If all the logs have already been loaded into the buffer (i.e. JuliaHub.hasfirst(buffer) is true), the function is a no-op.

See also: job_logs_buffered, job_logs_newer!.

source
JuliaHub.job_logs_newer!Function
JuliaHub.job_logs_newer!(
    buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]
) -> AbstractJobLogsBuffer

Updates the AbstractJobLogsBuffer object by adding up to count log messages to the end of the buffer. If count is omitted, it will seek all the way to the end of the current logs.

For a finished job, if all the logs have already been loaded into the buffer (i.e. JuliaHub.haslast(buffer) is true), the function is a no-op. If the buffer is actively streaming new logs for a running job, then the function is also a no-op.

See also: job_logs_buffered, job_logs_older!.

source
JuliaHub.haslastFunction
JuliaHub.haslast(::AbstractJobLogsBuffer) -> Bool

Determines whether the job log buffer has the last message of the job logs. Note that if the job has not finished, this will always be false, since the job may produce additional logs.

See also: hasfirst, job_logs_buffered.

source
JuliaHub.interrupt!Function
JuliaHub.interrupt!(::AbstractJobLogsBuffer; wait::Bool=true)

Can be use to interrupt the asynchronous log streaming task. If the log buffer is not streaming, this function is a no-op.

Note that the effect of JuliaHub.interrupt! may not be immediate and the function will block until the task has stopped. wait = false can be passed to make interrupt! return immediately, but in that case the buffer may stream for a little while longer.

source
JuliaHub.job_filesFunction
JuliaHub.job_files(job::Job, [filetype::Symbol]) -> Vector{JobFile}

Return the list of inputs and/or output files associated job.

The optional filetype argument should be one of :input, :source, :result or :project, and can be used to filter the file list down to either just job input files (such as the appbundle or Julia environment files), or output files (such as the one uploaded via RESULTS_FILE).

Note: job_file(job) is equivalent to job.files, and the latter is preferred. This function is primarily meant to be used when filtering by file type.

See also: Job.

source
JuliaHub.job_fileFunction
JuliaHub.job_file(job::Job, type::Symbol, filename::AbstractString) -> JobFile | Nothing

Searches for a job output file of a specified type and with the specific filename for job job, or nothing if the file was not found.

type should be one of the standard job file types. See JobFile and job_files for more information.

source
JuliaHub.download_job_fileFunction
JuliaHub.download_job_file(file::JobFile, path::AbstractString; [auth]) -> String
JuliaHub.download_job_file(file::JobFile, io::IO; [auth])

Downloads a JobFile to a local path. Alternative, writeable stream object can be passed as the second argument to write the contents directly into the stream.

When a local path is passed, it returns the path (which can be useful when calling the function as e.g. JuliaHub.download_job_file(file, tempname()))). When an IO object is passed, it returns nothing.

For example, to download a file into a temporary file:

julia> file = JuliaHub.job_file(JuliaHub.job("jr-eezd3arpcj"), :result, "outdir.tar.gz")
JuliaHub.JobFile outdir.tar.gz (jr-eezd3arpcj, :result, 632143 bytes)
sha2_256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Uploaded: 2023-03-15T07:59:29.473898+00:00

julia> tmp = tempname()
"/tmp/jl_nE3uvkZwvC"

julia> JuliaHub.download_job_file(file, tmp)
"/tmp/jl_BmHgj8rQXe"

julia> bytes2hex(open(sha2_256, tmp))
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

Alternatively, you can also download the file into a writable IO stream, such as IOBuffer:

julia> buffer = IOBuffer();

julia> JuliaHub.download_job_file(file, buffer)

julia> bytes2hex(sha2_256(take!(buffer)))
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

See also: Job, JobFile.

source
JuliaHub.JobType
struct Job

Represents a single job submitted to JuliaHub. Objects have the following properties:

  • id :: String: the unique, automatically generated ID of the job
  • alias :: String: a non-unique, but descriptive alias for the job (often set by e.g. applications)
  • status :: JobStatus: a string-like JobStatus object storing the state of the job
  • env :: Dict: a dictionary of environment variables that were set when the job was submitted
  • results :: String: the output value set via ENV["RESULTS"] (an empty string if it was not explicitly set)
  • files :: Vector{JobFiles}: a list of JobFile objects, representing the input and output files of the job (see: job_files, job_file, download_job_file).
  • hostname :: Union{String, Nothing}: for jobs that expose a port over HTTP, this will be set to the hostname of the job (nothing otherwise; see: the relevant section in the manual)

See also: job, jobs.

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.JobStatusType
struct JobStatus

Type of the .status field of a Job object, representing the current state of the job. Should be one of: Submitted, Running, Failed, Stopped, Completed.

In practice, the .status field should be treated as string and only used in string comparisons.

See also: isdone.

julia> job = JuliaHub.job("jr-novcmdtiz6")
JuliaHub.Job: jr-novcmdtiz6 (Completed)
 submitted: 2023-03-15T07:56:50.974+00:00
 started:   2023-03-15T07:56:51.251+00:00
 finished:  2023-03-15T07:56:59.000+00:00
 files:
  - code.jl (input; 3 bytes)
  - code.jl (source; 3 bytes)
  - Project.toml (project; 244 bytes)
  - Manifest.toml (project; 9056 bytes)
 outputs: "{}"

julia> job.status == "Submitted"
false

julia> job.status == "Completed"
true
No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.JobFileType
struct JobFile

A reference to a job input or output file, with the following properties:

  • .name :: String: the name of the Job this file is attached to
  • .type :: Symbol: indicated the file type (see below)
  • .filename :: String: file name
  • .size :: Int: size of the file in bytes (reported to be zero in cases where the file contents is missing)
  • .hash :: Union{FileHash, Nothing}: a FileHash object containing the file hash, but may also be missing (nothing) in some cases, like when the file upload has not completed yet.

The file is uniquely identified by the (job, type, filename) triplet.

The type of the file should be one of:

  • :input: various job input files, which includes code, environment, and appbundle files.
  • :source: source
  • :project: Julia project environment files (e.g. Project.toml, Manifest.toml, Artifacts.toml)
  • :result: output results file, defined via the RESULTS_FILE environment variable in the job

Note that some files are duplicated under multiple categories.

See also: Job, job_files, job_file.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.FileHashType
struct FileHash

Stores a hash and the algorithm used to calcute it. The object has the following properties:

  • .algorithm :: Symbol: hash algorithm
  • .hash :: Vector{UInt8}: the hash as a sequence of bytes
No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.requestFunction
function request(
    job::Job,
    method::AbstractString,
    uripath::AbstractString,
    [body];
    [auth::Authentication],
    [extra_headers],
    kwargs...
) -> HTTP.Response

Performs an authenticated HTTP request against the HTTP server exposed by the job (with the authentication token of the currently authenticated user). The function is a thin wrapper around the HTTP.request function, constructing the correct URL and setting the authentication headers.

Arguments:

  • job::Job: JuliaHub job (from JuliaHub.job)

  • method::AbstractString: HTTP method (gets directly passed to HTTP.jl)

  • uripath::AbstractString: the path and query portion of the URL, which gets appended to the scheme and hostname port of the URL. Must start with a /.

  • body: gets passed as the body argument to HTTP.jl

Keyword arguments:

  • extra_headers: an iterable of extra HTTP headers, that gets concatenated with the list of necessary authentication headers and passed on to HTTP.request.

  • Additional keyword arguments must be valid HTTP.jl keyword arguments and will get directly passed to the HTTP.request function.

Note

See the manual section on jobs with exposed ports and the expose argument to submit_job.

source

Index