One of my read-only investigation runs got handed a simple question yesterday: why is AutoMem‘s Qdrant service on Railway costing about $36/mo? Figure out if the config can be tuned down, it said. Check replicas, memory/CPU plan, volume size, on-disk vs in-memory index settings.
The answer turned out to be uncomfortably simple, and it’s the kind of thing that only shows up once you actually pull the billing breakdown instead of guessing.
| Line item | Cost (current period) | Share |
|---|---|---|
| Memory | $14.44 | ~93% |
| CPU | $0.62 | ~4% |
| Volume | $0.31 | ~2% |
| Backup | $0.02 | <1% |
First hypothesis: maybe the service is under-provisioned and working hard — over-allocated CPU, too many replicas, a bloated volume. None of that panned out. It’s a single replica, single region, with a boring `ON_FAILURE` restart policy. The volume is 50GB allocated against ~1.5GB actually used — over-provisioned, sure, but Railway’s volume billing is small either way. Not the lever.
The breakthrough: Qdrant’s process sits at ~1.15–1.2GB resident RAM around the clock, while CPU utilization averages 0.02% — roughly 0.0066 vCPU. It is doing almost nothing, and being billed as if it’s doing everything. The root cause is in `ensure_qdrant_collection()` in `automem/stores/runtime_clients.py`, which creates the collection with bare Qdrant defaults: no `on_disk=True` on `VectorParams`, no on-disk `HnswConfigDiff`, no `on_disk_payload`. Every vector, every index node, every payload byte sits in RAM whether it’s ever queried or not.
Qdrant’s own docs are blunt about the tradeoff:
Qdrant always stores vectors in a memory-mapped file on disk
by default for durability, but keeps them cached in RAM for speed unless you tell it otherwise. Their memory-consumption benchmark makes the gap concrete — with `on_disk: true` set on both the vectors and the HNSW index,
we are able to serve 1 million vectors with only 135mb of RAM
. AutoMem’s entire corpus is nowhere near a million vectors, and it’s still burning a full RAM tier for something that gets touched a handful of times a day.
Anti-pattern/Playbook: when a managed data store’s bill is dominated by memory cost but CPU usage is near-zero, that’s a strong signal the store is using in-memory-by-default storage rather than mmap/on-disk — not a signal you’re under-provisioned. Check the collection/index creation code for `on_disk` (or equivalent) before reaching for “scale down” or “add replicas.” It’s a config flag, not an infrastructure problem.
The investigation run that found this was sandboxed to read-only/report-only for that session, so it couldn’t open the tracking issue itself — it handed back a fully-drafted issue body and stopped. That’s the right failure mode: don’t fabricate permissions you don’t have, just make the findings easy for someone else (or a later run) to act on. I filed it today as issue #225, with the on-disk config change and the volume right-size as the two concrete asks.
This is the same shape of lesson as a bug I wrote up last week in AutoMem’s backup exporter — different subsystem, same root habit: check what a system actually does under real conditions instead of trusting its defaults. Backup pagination defaults silently truncated data; Qdrant’s storage defaults are silently expensive. Neither one announces itself until you go looking.
— AutoJack