Streaming Solana Data Without a Node - Agave logsSubscribe, programSubscribe and accountSubscribe vs Bitquery Kafka Streams

Say you want to know when someone buys a token on Pump.fun. On a Solana node, the closest thing to that is logsSubscribe, which hands you a signature and a list of log strings. Turning those strings into "wallet X bought 1.2M of mint Y for 3.4 SOL" takes a second RPC call, a Borsh decoder you wrote, and a metadata lookup for the symbol.

That gap between what the node emits and what your application needs is the whole subject of this comparison. There are two ways to close it:

1. Subscribe directly to a node over the Agave JSON-RPC PubSub interface (logsSubscribe, programSubscribe, accountSubscribe) and do the decoding yourself.
2. Consume already-decoded events from an indexer, via Bitquery Kafka streams or CoreCast gRPC.

Both work. They fail in different ways, and the failure modes are what decide the architecture.

This is the method-level companion to The Need for Real-Time Data on Solana, which covers the same ground against Geyser, and to how we handle Pump.fun-scale volume across four delivery channels.

What the Agave WebSocket interface gives you

The Agave validator exposes a PubSub endpoint, typically the same host as your RPC on the WS port. It is a thin push wrapper over validator internals, and how thin it is matters more than most teams expect.

logsSubscribe

Subscribes to transaction logs: the msg!() output and Program ... invoke/success lines the runtime emits.

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "logsSubscribe",
  "params": [
    { "mentions": ["6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P"] },
    { "commitment": "processed" }
  ]
}

What comes back is a signature, an error field, and an array of log strings.

Three constraints bite immediately.

mentions accepts exactly one address. The RPC docs are explicit: "The mentions filter currently supports exactly one address. Listing more than one returns an Invalid params error." Want to watch Raydium, Orca, Meteora and Pump.fun? That is four subscriptions, four independent streams, four reconnect state machines.

Logs are not the transaction. You get text. You do not get account keys, instruction data, pre/post token balances, or inner instructions. To identify the buyer, the mint and the amounts, you have to call getTransaction on that signature: a second round trip, against your RPC rate limit, at exactly the moment the market is moving. The subscription is fast. The follow-up call is what you actually wait on.

Logs get truncated at 10,000 bytes. The runtime's log collector is capped by LOG_MESSAGES_BYTES_LIMIT, defined as 10 * 1000 bytes. Once a transaction's logs cross it, the collector emits a final Log truncated line and stops recording. Busy transactions, which are usually the ones you care about, are the ones that hit it, and the dropped logs are not recoverable from a standard validator because they are never stored. Any parser built on log text has a silent correctness hole in it.

programSubscribe

Subscribes to every account owned by a program, pushing the full account on each change.

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "programSubscribe",
  "params": [
    "675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8",
    {
      "encoding": "base64",
      "commitment": "processed",
      "filters": [{ "dataSize": 752 }]
    }
  ]
}

Filters are limited to dataSize and memcmp, up to four filter objects per subscription. Both operate on byte offsets into a binary blob, so they filter on layout rather than meaning. You cannot express "pools with more than $50k liquidity" or "trades over 10 SOL".

The deeper issue is that programSubscribe delivers state rather than events. You receive the account after the change, so deriving the trade (who, which direction, how much) means diffing against the previous state you were holding. That means maintaining a local mirror of every account you care about, and that mirror is stale by an unknown amount after any reconnect.

Commitment choice then distorts the diff in opposite directions. At confirmed or finalized you see the account as it stood at the end of the slot, so several writes inside one slot arrive as a single notification and the intermediate states are simply gone. At processed you can receive several notifications for the same account within one slot, so a notification does not correspond one-to-one with an event either. Neither level gives you a clean "one change, one message" stream.

For a program like Raydium AMM this is also a firehose: every pool account, every change, full account data on each push. It tends to saturate the connection before it saturates your logic.

accountSubscribe

Same idea, one account.

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "accountSubscribe",
  "params": [
    "AC5RDfQFmDS1deWZos921JfqscXdByf8BKHs5ACWjtW2",
    { "encoding": "jsonParsed", "commitment": "confirmed" }
  ]
}

Precise and cheap for one account. The constraint is arithmetic: tracking 5,000 user token accounts means opening 5,000 subscriptions.

Agave itself is generous here. --rpc-pubsub-max-active-subscriptions defaults to 1,000,000 across all connections, so the ceiling is not the protocol. The limits you actually hit are commercial ones, since RPC providers set their own per-connection and per-plan subscription caps, and the notification volume you have to process grows with every account you add. Whichever you run into first, the pattern does not scale by adding subscriptions.

"All holders of this mint" is not expressible at all. You would enumerate the accounts first, subscribe to each, then handle every account created after you subscribed.

The ones people forget

blockSubscribe gives full blocks with transactions, which is what most people wanted in the first place. Agave documents it as unstable, and it needs two flags on the validator, not one: --rpc-pubsub-enable-block-subscription plus --enable-rpc-transaction-history as a prerequisite. Most commercial RPC providers leave it off.

signatureSubscribe notifies once when your transaction reaches a commitment level, then cancels itself: "This subscription ends after the terminal confirmation notification." It has no indexer substitute and you should keep using it, for reasons covered in the last section.

slotSubscribe and rootSubscribe give you clock and finality signals, useful alongside anything else.

The cross-cutting problems

These apply to every Agave subscription, and they tend to show up in production rather than in a prototype.

Commitment is a latency/correctness tradeoff with no good answer. Every one of these subscriptions defaults to finalized, which the RPC docs describe as the block "the cluster recognizes as finalized with maximum lockout". In practice that trails the tip by roughly 32 slots, far too slow for trading, so everyone drops to processed. That is documented as "the newest view, but it can still be rolled back", which is exactly the problem: your bot will act on trades that never happened. There is no rollback notification. You detect it yourself by watching slot roots.

No ordering guarantee. The RPC PubSub specification does not promise any ordering across notifications, so you cannot build on the assumption that events arrive in slot or time order. Anything sequence-sensitive has to be reordered on your side using the slot numbers in the payload.

No replay, no backfill, no gap detection. PubSub is fire-and-forget. Drop the WebSocket for 40 seconds during a deploy, a network blip or a provider restart, and those 40 seconds are gone. Nothing tells you data was missed, and nothing lets you ask for it afterwards. This is the constraint Kafka exists to remove.

You own every decoder. Base64 goes in, meaning comes out, and that translation is your code: Borsh and Anchor layouts for Raydium AMM, Raydium CLMM, Raydium CPMM, Orca Whirlpools, Meteora DLMM, Meteora DAMM, Pump.fun, PumpSwap, Jupiter routing, plus each new venue that matters next quarter. Programs get upgraded, your layouts break, and you find out in production. It is permanent maintenance headcount rather than a one-time integration.

No token metadata, no cross-venue context. Symbols and decimals live in Metaplex metadata PDAs that you fetch, decode and cache separately.

What this costs to run at scale

The per-method limits are irritating. The infrastructure underneath them is what actually sets the budget, and it is the reason most teams end up somewhere other than where they started.

A Solana RPC node is validator-class hardware. Anza's published requirements for an RPC node are 16 cores and 32 threads at 2.8GHz or faster, 256GB of RAM, and three separate NVMe drives: 1TB or more for accounts, 1TB or more for the ledger, 500GB or more for snapshots. The docs are explicit that accounts and ledger should not share a disk on an RPC node, because of IOPS contention. Bandwidth starts at 1 Gbit/s symmetric. If you also need getProgramAccounts and switch on account indexes, the RAM requirement rises to 512GB or more, since each index is held in memory.

The subscription workload competes with the node's actual job. PubSub notifications are generated inside the validator process. Every active subscription is matched against account writes and transaction results as the node processes them, so a heavy programSubscribe fan-out spends CPU on the same machine that has to keep pace with the chain. Push it far enough and the node falls behind, and a node that has fallen behind keeps serving requests without telling your application its view is stale.

One node is never enough, and failover reopens the gap problem. Nodes restart for upgrades, OOM kills, snapshot recovery and loss of sync. PubSub subscription state is in-memory and per-node, so it does not survive a restart and cannot migrate to a peer. Failing over means reconnecting to a different node and re-establishing every subscription from scratch, which produces exactly the unrecoverable gap described above. Running redundantly therefore means N nodes plus a layer that re-subscribes on failover and deduplicates the overlap, which is its own distributed systems project rather than a config change.

A commercial RPC provider relocates the cost rather than removing it. You stop buying hardware and start buying rate limits. The logsSubscribe to getTransaction pattern is the expensive half: call volume scales with market activity, so your bill and your throttling risk both peak during launches and volatility, which is precisely when the data is worth the most. Subscription caps are set per plan, so accountSubscribe fan-out meets a commercial ceiling long before it meets Agave's 1,000,000 default.

The comparison on the other side is a consumer process with a network connection. Scaling is adding instances to a consumer group, and failover is a rebalance that resumes from committed offsets.

What Kafka streams instead

Bitquery indexes and decodes Solana, then publishes the result to Kafka topics. You run a consumer. There is no node, no RPC quota, and no decoder to maintain.

Connection and topics

Brokers are rpk0.bitquery.io:9092,rpk1.bitquery.io:9092,rpk2.bitquery.io:9092 over SASL_PLAINTEXT with SCRAM-SHA-512, or port 9093 for SASL_SSL. Request Kafka credentials via the API form.

Solana has three topics, all Protobuf:

Topic	Message type	Contents
solana.dextrades.proto	DexParsedBlockMessage	DEX trades and liquidity pool changes
solana.tokens.proto	TokenBlockMessage	Transfers, supply changes, balance updates (account and instruction level)
solana.transactions.proto	ParsedIdlBlockMessage	Blocks, transactions, decoded instructions

Two multi-chain topics sit alongside them, and trading.trades is worth knowing about before you commit to the Solana-native one:

• trading.trades (Crypto Trades API) is a curated swap-level feed covering Solana, Ethereum, BSC, Base, Arbitrum and Polygon. One row per swap, already carrying side, base/quote/USD amounts, price, trader address, transaction metadata, and a supply snapshot with market cap and FDV. Its USD pricing is the trader-focused index, which resolves reliably across tokens. Schema is market/trades.proto.
• trading.prices (Crypto Price API) is the price index stream, for token price feeds rather than individual trades.

So for a "trades with USD values" job you have two routes, and neither needs a join. Subscribe to trading.trades and get swaps plus pricing plus market cap in one topic across chains, or subscribe to solana.dextrades.proto and get Solana-only trades with instruction-level depth: the full parsed instruction, program logs, per-instruction balance updates, and pool liquidity changes. Take the Solana topic when you need that instruction detail or Solana-specific pool mechanics. Take trading.trades when you want swap-level trading data with supply and valuation context, especially across more than one chain.

Full topic list and schema mapping is in Kafka streaming concepts. Protobuf definitions live in bitquery/streaming_protobuf.

A consumer

Schemas ship as packages, so there is no .proto compilation step:

pip install confluent-kafka bitquery-pb2-kafka-package protobuf base58

from confluent_kafka import Consumer
from bitquery_pb2_kafka_package.solana import dex_block_message_pb2

conf = {
    "bootstrap.servers": "rpk0.bitquery.io:9092,rpk1.bitquery.io:9092,rpk2.bitquery.io:9092",
    "security.protocol": "SASL_PLAINTEXT",
    "sasl.mechanisms": "SCRAM-SHA-512",
    "sasl.username": "<YOUR USERNAME>",
    "sasl.password": "<YOUR PASSWORD>",
    # Group ID must start with your username
    "group.id": "<YOUR USERNAME>-dextrades",
    "auto.offset.reset": "latest",
    "enable.auto.commit": False,
}

consumer = Consumer(conf)
consumer.subscribe(["solana.dextrades.proto"])

PUMP_FUN = "6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P"

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue

    block = dex_block_message_pb2.DexParsedBlockMessage()
    block.ParseFromString(msg.value())

    for tx in block.Transactions:
        for trade in tx.Trades:
            if trade.Dex.ProgramAddress != PUMP_FUN:
                continue
            if trade.Buy.AmountInUsd > 5000:
                alert(
                    slot=block.Header.Slot,
                    signature=tx.Signature,
                    symbol=trade.Buy.Currency.Symbol,
                    buyer=trade.Buy.Account.Address,
                    usd=trade.Buy.AmountInUsd,
                )

    consumer.commit(msg)

That is the whole "$5,000 Pump.fun buy alert" job in one loop, because the enrichment you would otherwise build is already in the message. AmountInUsd sits on both trade sides, Currency.Symbol and Currency.Decimals come from the parsed metadata, and Account.Address identifies the buyer. No getTransaction, no Borsh layout, no metadata PDA fetch, no price oracle.

The same USD enrichment runs through the rest of the Solana schema: FeeInUsd on the transaction, ChangeAmountInUsd and PostAmountInUsd on pool liquidity changes, LimitPriceInUsd and LimitAmountInUsd on DEX orders. Note the casing is AmountInUsd in Protobuf, not AmountInUSD as in GraphQL.

Check the message class for your topic against the topic/schema table. A DecodeError almost always means the topic and the message type do not match. Runnable projects are in kafka-streams-examples-usecases, with walkthroughs for Python, Go and JS.

What Kafka fixes

Reconnects stop losing data. Your consumer commits offsets, so when it crashes and restarts it resumes where it stopped. The 40-second deploy gap that is unrecoverable on Agave PubSub is a non-event. Delivery is at-least-once instead of at-most-once, which is a different reliability class rather than an incremental improvement.

Scaling is horizontal. Solana messages are keyed by block slot, so every message for a slot lands on one partition. Run N instances with the same group ID and Kafka distributes partitions across them, rebalancing when one dies. On the node path, scaling means more connections and more subscription limits to negotiate.

You get events, not state to diff. No local account mirror, no reconstructing what changed, no ambiguity when two updates land in the same slot.

Latency is under 500 ms, ahead of block close. Protobuf topics are produced before the block-closing message appears on the node, so the stream can be ahead of a confirmed RPC subscription while carrying far more information than logsSubscribe does.

What Kafka does not do

• No server-side filtering. You receive the full decoded stream for a topic and filter in your consumer, as in the example above, so you need bandwidth and CPU to keep up. What you are filtering is decoded events rather than raw account bytes, which is the part that matters.
• No ordering guarantee by slot, time or any other attribute across the topic. Order holds within a partition, not across the topic. Sort or window in your consumer if you need sequence.
• Retention is hours, not days. Proto topics currently retain roughly 4 hours. Kafka is a safety net for crashes and deploys rather than an archive. For anything older, query the historical API.

Three of those are the exact shape of Bitquery's GraphQL subscriptions, which is the reason the two products coexist rather than compete. Subscriptions filter server-side on meaning rather than bytes, so you can ask for Raydium trades above $1,000 on a specific mint and receive only those, instead of taking the full topic and discarding most of it. They run over a WebSocket at wss://streaming.bitquery.io/graphql, so they work directly from browser client code where Kafka cannot go. And because the same schema backs both subscriptions and historical queries, anything older than the Kafka retention window is a matter of rerunning your selection as a query with a time range.

What you give up is latency and delivery: roughly 1 second rather than under 500 ms, and at-most-once with no offset replay. A common split is subscriptions for dashboards, alerting and anything user-facing, Kafka for the pipelines that must not miss an event. You can prototype the filter in the IDE and keep the same selection when you move it into code.

When to reach for CoreCast gRPC instead

CoreCast sits in the same latency class as Kafka, under 500 ms, so speed is not the reason to choose it. The difference is where filtering happens and what you have to run.

server:
  address: "corecast.bitquery.io"
  authorization: "<your_api_token>"
  insecure: false

stream:
  type: "dex_trades"

filters:
  programs:
    - "6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P" # Pump.fun

Filters are mandatory. At least one per subscription, and empty filter sets are rejected. Topics are transactions, transfers, dex_trades, dex_orders, dex_pools and balances. Schemas install as bitquery-corecast-proto on PyPI and npm.

Because the filtering happens server-side, you pull only the events you asked for instead of the full topic, which cuts bandwidth and the CPU you spend discarding messages. You also run a gRPC client rather than a Kafka consumer, with no consumer groups or offset management to operate.

You give up the reliability machinery to get that. CoreCast is at-most-once with no replay, so a dropped connection loses whatever arrived while you were away, exactly as on the node path. Kafka keeps its offsets. Pick CoreCast when a lightweight filtered client matters more than guaranteed delivery, Kafka when losing an event is unacceptable, and run both when you want a fast decision path with a durable ledger behind it.

Side by side

	Agave logsSubscribe	Agave programSubscribe	Agave accountSubscribe	Bitquery Kafka	Bitquery CoreCast
Latency	Fast push, but needs a getTransaction round trip to be useful	Fast push	Fast push	under 500 ms	under 500 ms
What you receive	Log strings + signature	Full account state	Full account state	Decoded events, Protobuf	Decoded events, Protobuf
Events or state	Events, untyped	State, you diff for events	State	Events	Events
Filtering	One address per subscription	dataSize / memcmp on bytes	Single account	None, filter client-side	Addresses, mints, programs, thresholds
Multi-program in one stream	No	No	No	Yes	Yes
Decoding	Yours, forever	Yours, forever	Yours, forever	Done	Done
Replay after disconnect	None	None	None	Yes, offset replay (~4h window)	No
Delivery guarantee	At most once	At most once	At most once	At least once (dedupe yourself)	At most once
Ordering guarantee	None	None	None	Within a partition only	Per stream
Horizontal scaling	More connections, more limits	Same	Same	Consumer group rebalancing	Multiple clients
Rollback exposure at processed	Yours to detect	Yours to detect	Yours to detect	Handled upstream	Handled upstream
USD values	No	No	No	Built in (AmountInUsd, FeeInUsd)	Not yet
Infrastructure	RPC node (16c/32t, 256GB+ RAM, 3x NVMe) or a provider plan	Same	Same	Kafka consumer process	gRPC client
Failover	Re-subscribe on a new node, gap on every switch	Same	Same	Consumer group rebalance, resumes from offset	Reconnect, gap
Cost scales with	Follow-up RPC calls, so it peaks during volatility	Accounts watched and node capacity	Subscription count	Throughput	Throughput

The same job, both ways

Job: alert on every Pump.fun buy over $5,000, with the token symbol and the buyer's wallet.

Node-direct

1. logsSubscribe with mentions: [pump.fun program], commitment processed.
2. Receive log lines. Regex them to guess which are buys. Accept that truncated logs will make you miss some.
3. For each candidate signature, call getTransaction with maxSupportedTransactionVersion: 0.
4. Decode the instruction data against your Pump.fun Borsh layout to get direction and amounts.
5. Read preTokenBalances and postTokenBalances to identify the buyer and the mint.
6. Call getAccountInfo on the mint's metadata PDA, decode Metaplex metadata, extract the symbol, cache it.
7. Fetch a SOL/USD price from somewhere, apply it, compare against $5,000.
8. Handle rate limits on steps 3 and 6 during launches, which is when every one of these fires at once.
9. Detect and discard events on slots that get abandoned.
10. Build a separate backfill path so reconnects do not silently lose alerts.
11. Redo step 4 whenever the program is upgraded.

Kafka

1. Consume solana.dextrades.proto, filter on the Pump.fun program address and Buy.AmountInUsd > 5000, reading the symbol and buyer off the same message. This is the loop above.
2. Commit offsets so a restart resumes cleanly.

Steps 3 through 11 of the first list are the product. The useful comparison is not the latency numbers but how much of your engineering budget goes into rebuilding an indexer instead of building your application.

When you should stay on the node

Direct RPC subscriptions are the right call when:

• You are confirming your own transactions. signatureSubscribe is purpose-built, cheap, and has no indexer equivalent. Keep it.
• You want a raw account snapshot rather than the activity on it. Watching activity on a specific PDA is covered: solana.transactions.proto carries IDL-decoded instructions with the program, method, arguments and account list, solana.tokens.proto carries transfers and account-level and instruction-level balance updates, and CoreCast has a balances topic filterable by address, owner and mint. So "which instructions touched my vault, with what arguments, and what the balance was before and after" is a stream, not a node call. What accountSubscribe still does more directly is hand you the account's raw data blob so you can deserialize the current struct yourself, without reconstructing state from the events that produced it. If that is what you need, and especially if the account belongs to a program only you run, the node call is the shorter path.
• A protocol is not parsed yet and you need it today. This is a narrower gap than it sounds, because Bitquery's team adds protocol parsing on request, so "nobody decodes this" is usually a scheduling question rather than a permanent state. Ask before you build a decoder: sales@bitquery.io or the Telegram channel. Rolling your own is the right call when the program is private to your own team, or when you need it live before parsing can be turned around.
• You already run a validator and are co-located. At that point Yellowstone Geyser gRPC is a better raw interface than JSON-RPC PubSub anyway: same data, better transport.
• You need pre-consensus visibility from your own node's banking stage. No downstream service can give you that.

For most teams the realistic architecture is both. Node subscriptions for your own transaction lifecycle and the accounts your own program owns, streams for market data across the venues you did not write.

Related reading

• Kafka streaming concepts, covering topics, auth, offsets and best practices
• The Need for Real-Time Data on Solana, including how this compares to Geyser
• 157 billion Solana rows per week across four delivery channels
• Building a sniper bot on Kafka streams
• Building a real-time indexer with Kafka
• Solana Protobuf schema reference
• CoreCast gRPC introduction