Dyad AI Models

The Dyad agent reaches language models through an LLM gateway that runs inside your JuliaHub installation, served under /llm on your JuliaHub hostname. The gateway is LiteLLM, so it can front most model providers: Azure OpenAI, Amazon Bedrock, Anthropic, Google Vertex AI, or any OpenAI-compatible endpoint, including models you host yourself.

You decide which models the agent may use and where requests go. The model configuration and the provider credentials both stay inside your cluster: credentials are read from Kubernetes Secrets you create, and the model configuration can be kept in one as well, so neither has to pass through JuliaHub.

Requires a Dyad entitlement

The gateway is deployed automatically when your license includes Dyad. If you would like to use Dyad AI and your license does not include it, contact JuliaHub support.

Until you configure models, the gateway runs with none, and the agent has no models to offer.

Configuring models

The configuration is LiteLLM proxy configuration in YAML. Its model_list defines the models offered to the agent:

model_list:
  - model_name: gpt-5.4
    litellm_params:
      model: azure/<deployment-name>
      api_base: https://<resource>.openai.azure.com
      api_version: <api-version>
      api_key: os.environ/AZURE_OPENAI_API_KEY
  • model_name is the name users see and select in the agent. Choose these names once, when you first configure a model: renaming a model later loses users' saved model selections.
  • litellm_params.model is the provider prefix and the provider's own model or deployment id. See the LiteLLM provider reference for the prefix and parameters each provider expects.
  • Credentials are never written into the configuration. Reference them as os.environ/<NAME> and supply the values from a Secret, as described in Supplying credentials.

Your configuration is merged over the gateway's built-in settings. No models are built in, so model_list is the complete list the agent offers. Any other top-level section you give (router_settings, litellm_settings, and so on) is merged key by key, so you only need to state what you want to change. The settings JuliaHub relies on, such as its usage-logging callback, are kept whatever you configure. See the LiteLLM configuration reference for the full set of options.

Examples

Amazon Bedrock, using an access key:

model_list:
  - model_name: claude-sonnet-4-5
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
      aws_region_name: us-east-1
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY

When aws_access_key_id and aws_secret_access_key are left out, the gateway uses the standard AWS credential chain, such as an IAM role granted to its service account through IAM Roles for Service Accounts or EKS Pod Identity. The gateway runs under the platform's shared service account (serviceAccount.name in the Helm values), so such a role is also available to every other platform service using that account. Scope its policy to the Bedrock models you intend to offer.

An OpenAI-compatible endpoint, such as an internal model gateway or a self-hosted model served by vLLM:

model_list:
  - model_name: internal-coder
    litellm_params:
      model: openai/<model-id-at-the-endpoint>
      api_base: https://llm.internal.example.com/v1
      api_key: os.environ/INTERNAL_LLM_API_KEY

Anthropic, directly:

model_list:
  - model_name: claude-opus-4-8
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY

If JuliaHub support has supplied a provider adapter for your environment, add the settings it comes with alongside your model_list in the same configuration.

Model details shown to the agent

For models LiteLLM already knows, such as the provider models in its model catalog, the agent learns each model's context window and reasoning support automatically. The exception is a model LiteLLM knows to reason but that is served through an OpenAI-compatible endpoint or a hosting service such as Fireworks AI, Together AI or OpenRouter: the gateway can't tell whether that endpoint accepts a reasoning-effort setting, so it guesses, and reports the model in the warnings of the model details endpoint until you confirm its reasoning support as described below. For a model LiteLLM cannot identify, such as one behind an OpenAI-compatible endpoint or an Azure deployment with a custom name, describe it under model_info.juliahub:

model_list:
  - model_name: internal-coder
    litellm_params:
      model: openai/<model-id-at-the-endpoint>
      api_base: https://llm.internal.example.com/v1
      api_key: os.environ/INTERNAL_LLM_API_KEY
    model_info:
      juliahub:
        display_name: Internal Coder
        context_window: 128000
        max_output_tokens: 16000

The gateway also accepts reasoning, supported_parameters and prompt_caching in this block. The block is checked when the gateway starts: if it has an unknown or misspelled key, or an invalid value, it is ignored as a whole, and the model falls back to the details LiteLLM can derive. The problem is reported in the gateway log and in the warnings of the model details endpoint, with a suggested correction for a misspelled key.

reasoning.supported_efforts lists the reasoning-effort levels users can choose for the model, from none, minimal, low, medium, high, xhigh and max. For a model that reasons but has no effort setting, or whose endpoint rejects one, use an empty list, and the agent sends no effort setting to that model:

    model_info:
      juliahub:
        reasoning:
          supported_efforts: []

Values set here take precedence over the model details the Dyad agent ships with.

Where the configuration goes

Replicated

When your license includes Dyad, the configuration screen has a Dyad AI Models section with three settings:

  • Model Configuration — paste the YAML configuration described above.
  • Model Configuration Secret — alternatively, the name of a Secret holding it, if you prefer to keep the configuration out of the configuration screen (see below). If both are set, Model Configuration is applied last and wins where they overlap.
  • Credentials Secret — the name of the Secret holding your provider credentials (see Supplying credentials).

Deploy the new configuration for it to take effect.

Helm

Set litellmsrvr.additionalConfig to the configuration, and name your credentials Secret in litellmsrvr.additionalConfigEnvSecrets:

litellmsrvr:
  additionalConfigEnvSecrets:
    - litellm-credentials
  additionalConfig:
    model_list:
      - model_name: gpt-5.4
        litellm_params:
          model: azure/<deployment-name>
          api_base: https://<resource>.openai.azure.com
          api_version: <api-version>
          api_key: os.environ/AZURE_OPENAI_API_KEY

additionalConfig must be a mapping of top-level configuration keys, such as model_list:. A bare list of models fails the deployment with an error saying so.

Keeping the configuration in a Secret

Model names and endpoints can be sensitive in their own right. To keep them out of your Helm values and the Replicated configuration screen, put the configuration in a file, for example litellm_config.yaml, and create a Secret from it in the JuliaHub namespace:

kubectl create secret generic litellm-models \
  --namespace <namespace> \
  --from-file=litellm_config.yaml

Then set Model Configuration Secret (Replicated) or litellmsrvr.additionalConfigSecret (Helm) to litellm-models. Every key in the Secret ending in .yaml, .yml or .json is read as configuration, in key-name order.

To change the configuration later, update the Secret and restart the gateway:

kubectl create secret generic litellm-models \
  --namespace <namespace> \
  --from-file=litellm_config.yaml \
  --dry-run=client -o yaml | kubectl apply -f -

Supplying credentials

Create a Secret in the JuliaHub namespace holding the API keys and tokens your providers need:

kubectl create secret generic litellm-credentials \
  --namespace <namespace> \
  --from-literal=AZURE_OPENAI_API_KEY='<key>'

Name it in Credentials Secret (Replicated) or litellmsrvr.additionalConfigEnvSecrets (Helm). Each key in the Secret becomes an environment variable in the gateway, which the configuration references as os.environ/<KEY>. For example, the key AZURE_OPENAI_API_KEY above is referenced as api_key: os.environ/AZURE_OPENAI_API_KEY.

The Secret may be created after installation: the gateway starts without it, and models that need its credentials are reported as unhealthy with a "Missing credentials" error until it exists. To rotate a credential, update the Secret and restart the gateway.

Applying Secret changes

The gateway reads its configuration and credentials only when it starts. Configuration entered in the Replicated configuration screen or in litellmsrvr.additionalConfig is applied on the next deploy. A change to a Secret you manage is not picked up until the gateway restarts:

kubectl rollout restart deployment/litellmsrvr --namespace <namespace>

If your cluster runs Stakater Reloader, the gateway is restarted automatically when any of the Secrets named in its configuration change.

Checking the configuration

Sign in to JuliaHub from Julia or Dyad, then use your JuliaHub access token to query the gateway's health endpoint:

curl https://<hostname>/llm/api/v1/health \
  -H "Authorization: Bearer <access-token>"

After signing in from Julia with JuliaHub.jl (JuliaHub.authenticate("<hostname>")), the token is the access_token in ~/.julia/servers/<hostname>/auth.toml.

Each configured model is listed under healthy_endpoints or, together with the error the provider returned, under unhealthy_endpoints. The check sends a small request to every model, so it uses a few tokens of each.

To see the models and the details the agent receives for each one, query https://<hostname>/llm/api/v1/juliahub/models the same way. Its warnings list problems with a model_info.juliahub block, and models whose reasoning support was guessed rather than confirmed (see Model details shown to the agent).

If the gateway does not start at all, for example because the configuration is not valid YAML, its logs report why:

kubectl logs deployment/litellmsrvr --namespace <namespace>