Create a Custom Datafeed

Use a custom datafeed to bring an exchange, a market-data service or your own historical dataset into AQE. The datafeed does not place orders: it can work with PaperBroker or a separate execution adapter.

This guide follows the AQE 0.2.0 interfaces. Use the trait definitions from the same AQE revision as your strategy, particularly when upgrading an existing integration.

1. Implement the two traits

Create a Rust type that owns your client, connection state and subscription tasks. Implement aq_engine::core::broker::traits::{DataFeed, DataProvider} for that type. It must be Send + Sync.

Trait	Required methods	Responsibility
DataFeed	connect, disconnect, is_connected	Establish and release the transport; report its actual state.
DataProvider	get_ticker_info	Return an AQE Asset with the symbol, trading rules and precision.
DataProvider	get_history	Return a Polars DataFrame for the requested symbol, UTC date range and TimeFrame.
DataProvider	get_latest_bar, get_latest_quote	Return a Bar or Quote, including its source timestamp.
DataProvider	subscribe_to_data_stream, unsubscribe_from_data_stream	Manage candle callbacks for the requested symbols and timeframe.

The traits use native Rust async methods. connect and get_latest_quote explicitly require futures that are Send; an async fn implementation can satisfy this when its captured state is Send. Do not hold a non-Send lock guard across an .await.

A cancelled connection attempt must leave its resources owned by the adapter so disconnect can clean them up. Stop background tasks and release subscriptions on disconnect; avoid detached workers that keep delivering data afterwards.

Declare only supported capabilities

Override DataFeed::get_features() with a static slice of DataFeedFeature values. The default is an empty slice.

• History and current prices: HistoricalCandles, LatestCandle, LatestQuote.
• Candle delivery: CandleStream and, only when implemented, IntraCandleStream.
• Independent quotes: QuoteStream, alongside NativeBidAsk or SyntheticQuotes as appropriate.

Required trait methods still need implementations when a capability is absent. Return a clear BrokerError for an unsupported operation rather than fabricated data or a successful no-op. An adapter that only provides history can be useful for backtests without advertising live streaming.

Return complete instrument metadata

get_ticker_info returns an Asset, not just a symbol. Populate the instrument’s identity, asset type, exchange, trading permissions and these execution rules from the provider’s metadata:

Fields	What to provide
min_order_size, quantity_base	The minimum quantity and quantity precision recognised by AQE. Keep these consistent with the venue’s quantity rules.
max_order_size, max_total_size	Per-order and aggregate symbol limits respectively. Use None when the provider does not report a limit; do not confuse the two.
min_price_increment, price_base, contract_size	The price increment, price precision and contract multiplier when known. Do not assume every instrument uses shares or a multiplier of one.
trading_calendar	An Option<String> containing producer-recorded calendar JSON, including its source. Keep None for an unknown schedule; do not infer continuous trading from missing metadata.
fees	AssetFees for commissions and swap when supplied. Current commission metadata uses commission.entry and commission.exit, each with long/short values; swap uses swap.long and swap.short.

AssetFees::default() supplies empty fee settings; it is not proof that the venue charges no fees. The legacy entry and exit fee fields remain compatibility fallbacks. A separate execution venue can have different rules and costs from the data provider: validate the metadata against the execution account rather than assuming market data alone authorises an order.

2. Normalise bars and quotes

Map the provider’s records to AQE Bar values, using UTC timestamps and consistent symbols. Handle pagination, chronological ordering, duplicates and requested timeframes explicitly. Do not silently return a different timeframe or replace missing history with current prices.

Historical frames need these columns:

Column	Canonical representation
symbol	String
open, high, low, close, volume	f64
timestamp	UTC Unix milliseconds as i64, as produced by AQE’s default converter

Prefer the existing converter to constructing a frame manually. This helper assumes you have already decoded and validated the vendor response as Vec<Bar>:

use aq_engine::core::broker::{
    traits::DataProvider,
    types::{Bar, BarData, BrokerError},
};
use polars::prelude::DataFrame;

pub fn normalise_history<D: DataProvider>(
    provider: &D,
    bars: Vec<Bar>,
) -> Result<DataFrame, BrokerError> {
    provider
        .format_on_bar(BarData::Bars(bars))
        .map_err(BrokerError::DataFeedError)
}

Use a Polars version compatible with your AQE dependency. format_on_bar(BarData::Frame(...)) checks required column names; it does not fully validate types, ordering or market-data quality for you.

A native Quote includes symbol, bid, ask, sizes, optional last-trade values and a UTC timestamp. Use finite positive prices, ask >= bid and non-negative sizes. For a bar-only source, Quote::from_bar_close(&bar) creates a synthetic bid=ask quote without inventing order-book size. Advertise SyntheticQuotes, not NativeBidAsk, for that representation.

Include the new quote valuation field

Quote now also contains profit_model. Add this field to existing Rust struct literals, even if your provider cannot supply a valuation model. Its #[serde(default)] attribute keeps older serialised quotes readable; it does not fill in a missing field in Rust source code.

For a generic datafeed, use profit_model: None. This example assumes the incoming prices and timestamp have already been validated; zero sizes mean no reported size, not invented liquidity:

use aq_engine::core::broker::types::Quote;
use chrono::{DateTime, Utc};

pub fn quote_without_valuation(
    symbol: String,
    bid: f64,
    ask: f64,
    timestamp: DateTime<Utc>,
) -> Quote {
    Quote {
        profit_model: None,
        symbol,
        bid,
        ask,
        bid_size: 0.0,
        ask_size: 0.0,
        last: None,
        last_size: None,
        timestamp,
    }
}

Only attach a BrokerQuoteProfit model when the execution broker can supply valid account-currency coefficients for that account, symbol and quote. It records version, account_id, currency, symbol, as_of_us and separate buy_gain, buy_loss, sell_gain and sell_loss coefficients. The current version is 1; coefficients must be finite and positive. The observation time uses UTC Unix microseconds, whereas the Quote.timestamp field is a UTC DateTime.

These coefficients are broker-calibrated cash values per price unit and broker quantity unit, not a generic FX conversion or a realised-profit record. Preserve their account, currency, symbol and freshness; do not copy them between accounts or manufacture them from a bar close. Quote::from_bar_close deliberately leaves the model absent. This metadata does not replace the existing quote capability flags.

3. Implement subscriptions

subscribe_to_data_stream receives symbols, a TimeFrame, a DataStreamMode and an Arc<dyn Fn(Bar) + Send + Sync> callback. Respect the mode: CompletedBar delivers completed candles; Intrabar allows updates to the forming candle. Keep callback execution short and non-blocking.

For a polling provider, inspect the existing DataStreamManager and Yahoo adapter before writing another scheduler. A websocket provider may manage its own reader tasks, but must still honour the same callback and unsubscribe contract.

Quote streaming is optional. To advertise QuoteStream, override both subscribe_to_quote_stream and unsubscribe_from_quote_stream. Each subscription has its own ID: removing it must not stop another consumer’s quotes or candles. The default subscribe method returns an unsupported error. The quote API does not promise every exchange tick.

4. Connect the provider to AQE

Once your type implements both traits, pass an instance to UnifiedBroker. This reusable helper accepts any compliant provider; it does not implement the provider’s network API:

use aq_engine::core::broker::{
    paper_broker::PaperBroker,
    traits::{DataFeed, DataProvider},
    types::AccountType,
    UnifiedBroker,
};

pub fn backtest_with<D>(datafeed: D) -> UnifiedBroker<PaperBroker, D>
where
    D: DataFeed + DataProvider,
{
    let execution = PaperBroker::new(AccountType::Paper, 100_000.0, 1);
    UnifiedBroker::new_backtest(execution, datafeed)
}

For live or paper-live operation, use UnifiedBroker::new(execution, datafeed). Keep historical replay on new_backtest; its replay quotes come from consumed bars rather than a current market-price endpoint.

Implementing these traits enables use in Rust strategies. Adding an AQS connection form or provider choice requires a separate desktop integration.

Optional provider hooks

supports_historical_stream(symbol, timeframe) defaults to Ok(true). Override it for finite datasets so an unavailable symbol/timeframe can be skipped deliberately. Keep genuine network, authentication and data errors as errors; do not use this hook to hide a failed request or silently substitute another timeframe.

configure_live_session is an optional session-context hook with a no-op default. It does not make valuation metadata or cloud connectivity mandatory for a custom datafeed.

5. Validate before live use

1. Test invalid symbols, empty history, pagination, timestamp conversion and unsupported timeframes.
2. Verify OHLCV values and chronological order against a small known dataset.
3. Test completed-bar and intrabar delivery separately, including reconnects and duplicate messages.
4. Create two quote subscriptions, remove one and confirm the other remains active.
5. Disconnect during connection and streaming; verify all owned resources are released.
6. Run a deterministic PaperBroker backtest before connecting a live execution account.
7. Check every Asset and Quote struct literal after upgrading AQE. Test missing valuation metadata, stale or foreign-account models and synthetic quotes without treating unavailable values as zero profit.

Source reference

In your AQE checkout, start with:

• src/core/broker/mod.rs: DataFeed, DataProvider, DataStreamMode and UnifiedBroker.
• src/core/broker/types/mod.rs: Asset, Bar, Quote, BarData and BAR_DATAFRAME_COLUMNS.
• src/core/broker/features.rs: supported capability flags.
• src/core/broker/data_feeds/yahoo.rs: historical retrieval, normalisation and polling.
• src/core/broker/data_stream.rs: shared candle subscription management.
• live-sync-protocol/src/quote_profit.rs: BrokerQuoteProfit scope, coefficients and freshness checks.

Next: Create a custom broker or Brokers and Datafeeds.