confit sources: bulk secret loading without the subprocess tax

The Provider Tax

When I built confit's provider system, the mental model was simple: a provider maps a URI scheme to a CLI command, and that command runs once per key. op://Vault/DB_PASSWORD runs op read once and returns a value. Clean.

The problem shows up at scale. If you have eight secrets in a [db] section all coming from the same Infisical project or 1Password environment, you're spawning eight subprocesses. Each one authenticates, makes a network call, and returns one value. It works, but it's slow and unnecessary — your secret manager already knows how to dump the whole thing at once.

The right model for bulk-export tools isn't "run per key" — it's "run once, cache the result, look up fields." That's what [sources] does.

Introducing [sources]

A source is a named load command that outputs KEY=VALUE pairs. You declare it in [sources] and reference individual fields with source-name://FIELD_NAME. confit runs the command the first time a field is requested and caches the result — every subsequent lookup is just a map access.

The shorthand form is just the load command as a string:

[vars]
stage = "dev"

[sources]
infisical = "infisical export --env={vars.stage} --format=dotenv"

[db]
password = "infisical://DB_PASSWORD"
host     = "infisical://DB_HOST"
port     = "infisical://DB_PORT"

[app]
api_key  = "infisical://API_KEY"
sentry   = "infisical://SENTRY_DSN"

All five values come from one infisical export call. The table form adds options like secret = true:

[sources.infisical]
load   = "infisical export --env={vars.stage} --format=dotenv"
secret = true

How It Works

When confit encounters infisical://DB_PASSWORD, it checks whether infisical is declared as a source. It is — so it runs the load command, parses the output, stores the result in SourceCache, and returns DB_PASSWORD's value. The next time it hits infisical://DB_HOST, the cache already has the parsed map — no second subprocess.

The load command output is parsed as dotenv format. All of these work:

FOO=bar
export FOO=bar
FOO="bar baz"
FOO='bar baz'
# comments and blank lines are ignored

This covers the output of infisical export --format=dotenv, op environment read, and most other secret manager export commands without any transformation.

The load command also supports {vars.*} interpolation, so you can parameterize by stage or any other variable:

[vars]
stage = "dev"

[sources]
mysrc = "my-secrets-cli export --env={vars.stage}"

One constraint worth knowing: {path} and {uri} are not available in source load templates. Sources load a bag, so there's no per-key path to substitute. If you need per-key dispatch (e.g. vault kv get secret/{path}), that's what providers are for.

A missing field is a hard error. infisical://NOPE where NOPE isn't in the output will fail loudly: Field 'NOPE' not found in source 'infisical'. No silent empty strings.

Secret Masking at the Source Level

Setting secret = true on a source marks every field from it as sensitive. confit show and confit resolve will display *** by default; --reveal shows real values; confit run always injects the real value into the process. The source-level secret flag is basically a blanket secret:// applied to everything that source loads.

You can also apply secret:// to individual fields when you don't want to mark the whole source as secret:

[sources]
config = "my-config-cli export --format=dotenv"

[app]
public_name = "config://APP_NAME"          # shown in plain text
db_password = "secret://config://DB_PASSWORD"  # masked

Both compose as expected: secret://source://FIELD marks that field as secret even when the source doesn't have secret = true.

env:// — The Built-in Source

There's a built-in source you don't need to declare: env://. It reads directly from the process environment with no subprocess:

[app]
debug   = "env://DEBUG"
db_host = "env://DATABASE_HOST"

An unset variable is a hard error: Environment variable 'DATABASE_HOST' is not set. This replaces the silent-empty-string behavior you'd get from ${DATABASE_HOST:-} in a shell script. If you're pulling from env, you mean it to be there — a missing var should be loud.

The main use case is bridging CI-injected environment variables into the confit config model without a separate declaration. Your CI sets DATABASE_HOST; your config references it as env://DATABASE_HOST; confit validates it exists on confit validate. No more wondering if a missing env var will silently blow something up downstream.

Sources vs Providers

The honest answer is: it depends on what the tool's interface looks like.

Use a source when your tool has a bulk-export mode that dumps all variables at once — Infisical, 1Password Environments, Doppler's doppler secrets download, most --format=dotenv commands. If you're referencing multiple keys from the same source in one config section, sources are almost always faster.

Use a provider when your tool fetches one secret at a time by path — Vault's vault kv get secret/{path}, AWS SSM get-parameter, individual 1Password item reads. These inherently need the per-key path to construct the right command, which is exactly what {path} in provider templates is for.

They compose fine. A real config might pull secrets from a 1Password environment via a source for bulk app config, while using a Terraform output provider for infrastructure values that genuinely need per-key dispatch:

[sources.openv]
load   = "op environment read {op.environments.{vars.stage}}"
secret = true

[providers.tf]
cmd = "terraform -chdir=iac/stages/{stage} output -raw {path}"

[db]
password = "openv://DB_PASSWORD"   # from 1Password, one subprocess total
host     = "tf://db_endpoint"      # from Terraform, per-key

1Password Environments Example

The 1Password Environments integration is the clearest example of why sources exist. op environment read <id> dumps the entire environment as KEY=VALUE pairs in one call. With the old provider approach, each secret reference triggered a full op environment read — which authenticates and makes a network call every time. A section with 10 secrets was 10 op invocations.

With sources it's one:

[vars]
stage = "dev"

[op.environments]
dev  = "env_abc123"
prod = "env_ghi789"

[sources.openv]
load   = "op environment read {op.environments.{vars.stage}}"
secret = true

[db]
password = "openv://DB_PASSWORD"
host     = "openv://DB_HOST"
url      = "postgres://app:{db.password}@{db.host}/mydb"

[app]
api_key  = "openv://API_KEY"
cdn      = "openv://CDN_DOMAIN"

Switching environments is one flag:

# dev (default)
confit run app -- node server.js

# production
confit --set stage=prod run app -- node server.js

One subprocess. All your secrets.