grind - v0.1.0

Grind

Grind is a durable background-job runtime for Gleam applications on Erlang/OTP and PostgreSQL. It retains typed job outcomes in PostgreSQL and runs handlers under OTP supervision.

Use Grind when work must outlive the request that created it and your application already fits Erlang/OTP and PostgreSQL. Typical work includes sending notifications, processing provider callbacks, and running delayed or retryable tasks. It also supports jobs admitted in the same database transaction as an application write, and an operator recovery path when an external effect is uncertain.

Grind is a fit for durable individual jobs with local per-node queue limits. It does not currently provide cron schedules, durable dependency workflows, batch coordination, or cluster-wide rate limits.

Capabilities

Install

Add Grind from Hex:

gleam add grind

The example below uses these direct dependencies. Commit gleam.toml and manifest.toml to retain the resolved versions.

[dependencies]
grind = ">= 0.1.0 and < 0.2.0"
gleam_stdlib = ">= 0.70.0 and < 2.0.0"
gleam_erlang = ">= 1.0.0 and < 2.0.0"
gleam_otp = ">= 1.0.0 and < 2.0.0"
gleam_json = ">= 3.1.0 and < 4.0.0"
gleam_time = ">= 1.0.0 and < 2.0.0"
exception = ">= 2.1.1 and < 3.0.0"
pog = ">= 4.1.0 and < 4.2.0"

The Erlang target requires Gleam 1.18 or later and PostgreSQL. The repository’s development shell uses OTP 29 and PostgreSQL 16.

Quick start

Define a worker, add Grind to the application’s supervision tree, and submit through its registered name. This example uses application-owned Email values and simulates delivery by returning a message ID. The application supervisor owns startup, restart, and shutdown.

import gleam/dynamic/decode
import gleam/erlang/process
import gleam/json
import gleam/otp/static_supervisor as supervisor
import gleam/time/duration
import grind
import grind/job
import grind/worker
import pog

pub type Email {
  Email(to: String, subject: String)
}

fn encode_email(email: Email) -> json.Json {
  json.object([
    #("to", json.string(email.to)),
    #("subject", json.string(email.subject)),
  ])
}

fn email_decoder() -> decode.Decoder(Email) {
  use to <- decode.field("to", decode.string)
  use subject <- decode.field("subject", decode.string)
  decode.success(Email(to:, subject:))
}

pub fn mailer() -> worker.Worker(Email, String, Nil) {
  worker.new(
    "mailer.send",
    input: worker.codec(worker.infallible(encode_email), email_decoder()),
    output: worker.codec(worker.infallible(json.string), decode.string),
    perform: fn(email) { Ok("msg:" <> email.to) },
  )
  |> worker.with_queue("mailers")
}

pub fn children(pool: pog.Config, name: process.Name(grind.Message)) {
  let config =
    grind.new(pool)
    |> grind.with_worker(mailer())
    |> grind.with_startup_migration
  supervisor.new(supervisor.OneForOne)
  |> supervisor.add(grind.supervised(config, name))
}

pub fn send(name: process.Name(grind.Message), email: Email) {
  let jobs = grind.named(name)
  let assert Ok(admission) = grind.submit(jobs, job.new(mailer(), email))
  grind.await(jobs, grind.handle(admission), within: duration.seconds(5))
}

The example opts into startup migration. In production, give DDL permission to the runtime only when that is your migration plan; otherwise call grind.migrate in a deployment step or use your migration tool. Choose one migration owner per schema.

How the runtime fits together

Each configured worker has typed codecs and a stable identity. The runtime starts a coordinator for each used queue; a coordinator claims work into local capacity and starts one temporary attempt actor per handler invocation. Each attempt is fenced by its lease and identity. A separate renewer uses a reserved database connection so ordinary pool saturation does not also stop lease renewal.

flowchart LR
  App[Application registers workers and submits jobs] --> Runtime[Grind OTP supervisor]
  Runtime --> Queue[Per-queue coordinator]
  Queue --> Attempt[Attempt actor]
  Attempt --> Handler[Linked application handler]
  Handler -->|application-owned effect| Provider[External system]
  Attempt -->|fenced outcome and receipt| Postgres[(PostgreSQL)]
  Queue --> Renewer[Lease renewer]
  Renewer -->|reserved connection| Postgres
  Runtime -->|best-effort events| Sinal[Sinal]

Queue concurrency is local to each node. Budget one dedicated renewal connection per used queue in addition to the main pool. The application owns external effects; a committed job outcome records Grind’s view of the handler result, not an atomic transaction with a provider.

Enqueue with an application write

grind.submit_in stages admission in the caller’s open READ COMMITTED transaction on the same database. The job commits or rolls back with the application write. Its reply is staged admission, not proof that the outer transaction committed.

let order = support.unique("order")
let assert Ok(grind.Inserted(handle)) =
  pog.transaction(db, fn(tx) {
    let assert Ok(_) =
      pog.query("INSERT INTO usage_orders (id) VALUES ($1)")
      |> pog.parameter(pog.text(order))
      |> pog.execute(tx)
    grind.submit_in(jobs, tx, job.new(receipt, 9) |> job.with_id(order))
  })

The full runnable transaction example also shows rollback behavior: transactions.gleam.

Recovery and operating limits

Learn more

✨ Search Document