Skip to main content

Ingest images and audio

By the end of this tutorial you will have a collection that ingests images and audio as readily as it ingests PDFs — a receipt photo and a meeting recording, both retrievable by question.

The platform reads PDFs, text and markdown natively. Everything else needs a converter: a rule that claims a media type and names who turns a file of that type into text. There are two kinds, and this tutorial builds one of each, on the job it suits:

ConverterClaimsBecause
Agentimage/*A multimodal model reads an image directly. Nothing to map — you point at an agent and you are done.
Toolaudio/*Speech-to-text APIs are not chat endpoints, so no model can call one. An http tool calls it directly.

Five steps:

  1. Create the OCR agent.
  2. Route images to it.
  3. Ingest an image.
  4. (Optional) OCR scanned PDFs too.
  5. Route audio to a transcription tool.

Every step is one API call, shown for all three clients. The ids in the responses are examples — copy the ones your own calls return.

Prerequisites​

  1. A credential, exported as NATURALI_TOKEN, with your client set up — the CLI, the SDK or plain curl:

    export NATURALI_TOKEN=nat_sk_...
    export NATURALI_API=https://api.naturali.ai/v1   # curl examples only

  2. A project with a provider whose model can see images. Any vision-capable model in the catalog will do. Create a provider builds one; arrive with both ids exported:

    export PROJECT=proj_V1StGXR8Z5jdHi6B
    export PROVIDER=aip_V1StGXR8Z5jdHi6B
    export NATURALI_PROJECT=$PROJECT   # lets the CLI omit --project-id

  3. A collection to ingest into. Search a library of PDFs builds one and teaches the upload call this tutorial reuses:

    export COLLECTION=kcol_V1StGXR8Z5jdHi6B

  4. A receipt.png in the working directory — any photo or screenshot with legible text — and, for step 5, a short meeting.mp3.

1. Create the OCR agent​

An agent converter is just an agent. The instructions carry all the weight: the model must transcribe, not summarize, because its answer becomes the document text verbatim.

naturali create-agent \
  --project-id "$PROJECT" \
  --provider-id "$PROVIDER" \
  --name ocr \
  --instructions 'Extract all text from the provided file verbatim. Return plain text only — no commentary, no summary, no markdown fences.'
{
  "id": "agent_V1StGXR8Z5jdHi6B",
  "project_id": "proj_V1StGXR8Z5jdHi6B",
  "name": "ocr",
  "status": "active"
}
export OCR_AGENT=agent_V1StGXR8Z5jdHi6B

2. Route images to it​

The converter claims image/* and names the agent. whole chunking keeps a short OCR result as one passage rather than splitting a receipt into fragments.

naturali create-knowledge-converter \
  --project-id "$PROJECT" \
  --content-type 'image/*' \
  --agent-id "$OCR_AGENT" \
  --chunk-strategy whole
{
  "id": "igr_V1StGXR8Z5jdHi6B",
  "project_id": "proj_V1StGXR8Z5jdHi6B",
  "content_type": "image/*",
  "agent_id": "agent_V1StGXR8Z5jdHi6B",
  "tool_id": null,
  "native_extraction": "first",
  "chunk_strategy": "whole"
}

A media type belongs to one converter: a second one claiming image/* is a 409 converter_content_type_taken, because two rules for the same type would make routing a coin toss.

3. Ingest an image​

The upload call is the one from step 2 of the PDF tutorial, unchanged. Nothing here names the converter or the agent — content_type: image/png is what routes the file, and the converter is resolved from it.

naturali create-knowledge-document \
  --project-id "$PROJECT" \
  --collection-id "$COLLECTION" \
  --filename receipt.png \
  --content-type image/png \
  --file "$(base64 -w0 receipt.png)"
{
  "id": "doc_V1StGXR8Z5jdHi6B",
  "filename": "receipt.png",
  "content_type": "image/png",
  "status": "pending",
  "error": null
}

Read the document until it is indexed (step 3 of the PDF tutorial), then query the collection for something only the image says — the OCR'd text is indexed like any other passage:

naturali query-knowledge-collection \
  --project-id "$PROJECT" \
  --collection-id "$COLLECTION" \
  --query 'total amount on the receipt' \
  --top-k 3
A converter run is a model call, and model calls can miss

status: failed with error: "CONVERTER_FAILED" usually means the model returned a non-answer rather than the text — the ordinary intermittency of any LLM call. Retry with POST /v1/projects/{project_id}/knowledge/collections/{collection_id}/documents/{document_id}:reingest. If it fails every time, the model behind your provider probably cannot see images.

4. (Optional) OCR scanned PDFs too​

A scanned PDF is application/pdf with no text layer, so native extraction finds nothing. Point a second converter at the same agent and it becomes the fallback for exactly those files — because a converter on a natively-readable type is consulted only when extraction comes up empty, born-digital PDFs keep taking the fast path and cost nothing extra.

naturali create-knowledge-converter \
  --project-id "$PROJECT" \
  --content-type application/pdf \
  --agent-id "$OCR_AGENT" \
  --chunk-strategy whole

To OCR every PDF regardless of its text layer, create it with native_extraction: "skip" instead.

5. Route audio to a transcription tool​

Speech-to-text endpoints take multipart/form-data and return JSON — they are not chat completions, so no agent can call one. That is what a tool converter is for.

First the tool. Three things make it work against a real speech-to-text API:

  • body_mode: "multipart" — the file field is decoded from base64 and attached as a real file part, instead of being sent as JSON, which such endpoints reject outright.
  • headers — the API key. Headers are write-only: accepted here, never returned by any read of the tool.
  • output_mapping — the endpoint answers with an object like { "text": "…", "duration": 4.2 }, and a converter must yield a bare string. This reduces the response to just the transcript.
naturali create-tool \
  --project-id "$PROJECT" \
  --name speech-to-text \
  --type http \
  --description 'Transcribes an audio file.' \
  --execute '{"url":"https://api.example-stt.com/v1/stt","method":"POST","body_mode":"multipart","headers":{"Authorization":"Bearer '"$STT_API_KEY"'"}}' \
  --output-mapping '{"var":"output.text"}' \
  --parameters '{"type":"object","properties":{"file":{"type":"object"},"language":{"type":"string"}}}'
export STT_TOOL=tool_V1StGXR8Z5jdHi6B

Now the converter. tool_id instead of agent_id is the only structural difference from step 2; preset_parameters pins arguments the caller never supplies, and a transcript is one long block of prose, so size chunking retrieves from it far better than whole would.

naturali create-knowledge-converter \
  --project-id "$PROJECT" \
  --content-type 'audio/*' \
  --tool-id "$STT_TOOL" \
  --preset-parameters '{"language":"en"}' \
  --chunk-strategy size \
  --chunk-size 1000 \
  --chunk-overlap 200

Ingest the recording exactly like the image — the call is identical but for the file and its media type:

naturali create-knowledge-document \
  --project-id "$PROJECT" \
  --collection-id "$COLLECTION" \
  --filename meeting.mp3 \
  --content-type audio/mpeg \
  --file "$(base64 -w0 meeting.mp3)"

Once it is indexed, query the collection for something that was only ever said out loud. The transcript is searchable next to the PDF pages and the receipt — one index, three media types, and the caller never named a converter for any of them.

What's next​

  • Knowledge — chunking, the ingestion lifecycle and the converter contract in full.
  • Tools — body_mode, output_mapping and write-only headers, beyond this one example.
  • Webhooks — a converter run is slower than a PDF parse, so subscribe to knowledge.document_ingested / knowledge.ingest_failed rather than polling.