Static + Runtime Fusion: Profiling a Live BEAM Node with Giulia v135

The Premise

Static analysis tells you what your code looks like. Runtime profiling tells you how it behaves under load. Neither gives the full picture alone. A module with perfect code quality metrics can still be a bottleneck at runtime. A hot process consuming 40% of CPU might be in a module with zero dependents — or it might be a hub that 30 other modules rely on.

Giulia v135 fuses both: it connects to a running BEAM node via Distributed Erlang, collects runtime snapshots (CPU, memory, scheduler contention, process rankings), and correlates them against its AST-level Knowledge Graph (dependency topology, complexity metrics, coupling scores). Starting with Build 135, it also performs function-level tracing on specified modules — sending self-contained trace code to the remote node via Code.eval_string over RPC, with no Giulia code required on the target.

The result is a single profile that tells you not just what’s hot, but which functions are hot, how often they’re called, and what the optimization opportunities are.

This article walks through 7 real observation sessions against a live Elixir application, each revealing something different about the system under test.

The Target: Nexus

Nexus is an Elixir API gateway that manages dynamic PostgreSQL connections, runtime schema metadata, and table operations. It exposes a REST API on port 4040 and handles CRUD, aggregations, search, CSV export, batch operations, and multi-database routing — all backed by Ecto dynamic repos.

The integration test suite exercises 61 tests across 8 categories:

• Connection Management — register, test, start/stop, delete database connections
• Multi-Database Operations — queries routed through named connections
• Extended CRUD — JSONB, NULLs, precision floats, hierarchical categories
• Edge Cases — SQL injection prevention, unicode/emoji, 10KB strings, empty payloads
• Concurrent Operations — 50 concurrent inserts, mixed read/write, 5000+ req/s bursts
• Large Dataset — 50k-row pagination, CSV export, aggregations, ILIKE search
• 60-Second Stress Test — 20 concurrent workers, 10s ramp-up, sustained mixed workload
• Metadata Lifecycle — runtime schema creation, introspection, reload

The Setup

Giulia runs as two Docker containers from the same image:

• Worker (port 4000) — static analysis engine: AST indexing, Knowledge Graph, dependency topology, complexity metrics, embeddings
• Monitor (port 4001) — runtime observer: connects to target BEAM nodes via Distributed Erlang, collects snapshots at configurable intervals, pushes data to the Worker for fusion

The observation workflow is command-driven:

giulia-observe start [email protected]    # connect + begin collecting
<run your workload>
giulia-observe stop [email protected]     # stop + finalize fused profile

With Build 135, you can also specify modules to trace at function level:

giulia-observe start [email protected] cookie 5000 Nexus.Repo,Nexus.Registry.TableRegistry

This adds per-function call counts to every snapshot — collected via Code.eval_string over :rpc.call, so the target node needs zero Giulia code.

The First Bug: Short Names vs Long Names

The first attempt failed immediately:

Connecting to [email protected] ...
{"error":":node_unreachable","node":"[email protected]"}

The root cause was a Distributed Erlang protocol mismatch. Both Giulia containers started with --sname (short names), producing nodes like monitor@giulia-monitor. But Nexus uses an IP-based address ([email protected]), which is a --name (long name) node. Erlang refuses to connect nodes across name modes — it’s a hard protocol incompatibility, not a network error.

The fix: change one flag in docker-compose.yml. Five characters, zero architectural changes.

# Before
elixir --sname worker ...

# After
elixir --name worker@giulia-worker ...

Phase 1: Process-Level Profiling (Sessions 1–5)

The first five sessions used process-level observation only — BEAM metrics and top-process rankings. This reveals which modules own the hottest processes, but not which functions inside those modules are being called.

Session 1: First Contact

A run queue of 17 means 17 tasks were waiting for a scheduler. For a node running 4–8 schedulers, sustained run queue above scheduler count means CPU-bound work is piling up. The bottleneck analysis flagged it automatically.

Session 2: Idle Baseline

:code_server being hottest at idle just means Erlang is doing module lookups — normal. This is what healthy looks like.

Session 3: Sustained Load with Info Logging

The logger consuming nearly a third of CPU under load. Not because of log volume — because of how it writes.

Session 4: Warning-Level Logging (The Counterintuitive Result)

Same workload, log level raised to :warning to reduce volume.

Reducing log volume made it worse. Logger jumped from 28.5% to 39.1% and became the #1 consumer. Run queue went from 1 to 5.

The mechanism: Erlang’s default logger handler does synchronous IO. Every log call blocks the calling process until the write completes. Reducing volume doesn’t help because the overhead is per-call, not per-byte. With fewer logs, the remaining calls still block, and now they’re a larger proportion of total work.

The fix — configure async/buffered mode:

:logger.update_handler_config(:default, :config, %{
  sync_mode_qlen: 1000,
  drop_mode_qlen: 2000,
  flush_qlen: 5000
})

Session 5: After the Fix

Logger eliminated. Run queue halved. :proc_lib at 98.9% just means all work is happening inside OTP processes — it’s the wrapper, not the bottleneck.

Phase 2: Function-Level Tracing (Sessions 6–7)

Session 6: Single Module Trace

Tracing Nexus.Registry.TableRegistry — the schema registry that every CRUD operation touches.

Every CRUD operation goes through this function to look up schema definitions. 9,238 calls across 10 trace windows.

Session 7: Dual Module Trace

Adding Nexus.Repo to the trace reveals the full picture.

Nexus.Repo — 7,000 total calls:

Nexus.Registry.TableRegistry — 2,004 total calls:

Optimization Analysis

Read-Heavy Workload (87% Reads)

Read operations:  2,209 (87%)
Write operations:    320 (13%)
Read/Write ratio:    6.9:1

Classic candidate for read-through caching. Repo.get/1 alone accounts for 1,263 calls.

Ecto Pipeline Overhead: 1.38x

Ecto pipeline calls: 3,485 (prepare_opts + default_options + prepare_query)
Actual DB operations: 2,530
Overhead ratio: 1.38x

Normal Ecto behavior — not a bug, not worth optimizing. The real wins are elsewhere.

TableRegistry Is Database-Backed (The Biggest Finding)

Nexus.Registry.TableRegistry has prepare_opts/2, default_options/1, and prepare_query/3 in its call profile — the telltale signature of an Ecto Repo. Every schema lookup is a round-trip to PostgreSQL.

Table metadata rarely changes. There’s no reason to hit the database on every request when the data could be loaded once into ETS and refreshed on metadata changes. Moving TableRegistry to ETS would eliminate ~2,000 unnecessary DB calls per observation window.

The BEAM Isn’t the Bottleneck — PostgreSQL Probably Is

Run queue: 1
Memory: 90.3 MB, stable
Throughput: ~30 DB ops/sec during trace window

BEAM has headroom. The relatively low throughput suggests database latency is the dominant factor. pg_stat_statements would confirm.

Priority

What Each Layer Revealed

Process-level profiling found the logger and scheduler contention — real operational issues, fixed during the session. But it could only tell us :proc_lib was hot, which is like saying “code is running.”

Function-level tracing broke through that ceiling.

The Real Demo

Worth stepping back and noting what actually happened here. Giulia profiled a remote BEAM node it doesn’t own. No access to Nexus’s source code. No instrumentation planted beforehand. No agents installed, no probes compiled into the target. Pure black-box runtime observation over Distributed Erlang.

From that — from connecting to a stranger’s node and watching it work — Giulia produced concrete, actionable optimization recommendations: your schema registry should be ETS-backed, not database-backed; here are the 354 unnecessary Postgres round-trips per observation window to prove it.

The logger finding was interesting. The scheduler contention was operationally useful. But the headline is this: function-level tracing of a remote node, with zero instrumentation on the target, leading to an architectural recommendation. That’s what static + runtime fusion delivers.

Built with Giulia v135 — an Elixir daemon for static + runtime code intelligence. Monitor container connects via Distributed Erlang, collects process snapshots and function-level traces at configurable intervals, pushes to Worker for AST correlation and fused profiling.