Mapping the spread between two markets
This project examines the difference between day-ahead and real-time settlement prices in ERCOT. The aim is to organize historical price, load, weather, and congestion signals into a model that can estimate when and where the spread may widen.
The working process begins with aligning time-series data, studying recurring market patterns, and defining a baseline before testing more expressive modeling approaches.
The DART spread is useful because it condenses the difference between two related views of the same delivery hour. Day-ahead prices reflect expectations formed before real operating conditions are known, while real-time prices respond to the grid as it is actually dispatched. A widening spread can therefore point toward forecast error, congestion, outages, scarcity, or a combination of several conditions.
Rather than treating every interval as interchangeable, the analysis would preserve the sequence of events around each observation. The project would ask when spreads become predictable, which inputs are known early enough to be useful, and whether a model remains informative once simple time-of-day and recent-price behavior are accounted for.
From market context to a usable signal
The model would be evaluated across different seasons and operating conditions, with particular attention paid to volatility and unusual grid events. The goal is not only accuracy, but a result that can be interpreted and compared with known market behavior.
A single average error could hide the intervals that make the problem interesting. Results would therefore be separated by hour, season, settlement location, and spread magnitude. Directional accuracy would be reviewed alongside numerical error to distinguish a model that identifies market movement from one that merely predicts values close to zero.
The output should also be understandable enough to challenge. If a forecast changes sharply, the accompanying analysis should show whether recent prices, demand conditions, weather, or congestion-related variables drove the movement. That interpretability would make it easier to decide whether the signal reflects a plausible market condition or an artifact of the training data.
Building the modeling dataset
A useful training set would bring the day-ahead and real-time prices onto a consistent hourly timeline, then join them with load forecasts, actual demand, weather, outages, and congestion indicators. Each feature would be shifted carefully so that the model only sees information available at prediction time.
Exploratory work would focus on the shape of the spread by hour, season, and settlement point. Large positive and negative intervals are especially important because they can reveal conditions that disappear when performance is summarized by an average alone.
Data preparation would begin by resolving missing intervals, daylight-saving transitions, repeated timestamps, and changes in naming conventions. Price spikes would not be removed simply because they look unusual; they are part of the target behavior. Instead, the project would document them and use robust summaries so that ordinary patterns remain visible beside the tail events.
Candidate features could include recent day-ahead and real-time prices, lagged spreads, forecast load, actual load, wind and solar expectations, temperature, hour and weekday indicators, and available measures of transmission conditions. Each variable would be recorded with the time at which it becomes known, preventing information published after the forecast decision from entering the model.
Choosing a baseline and model
The first comparison would be a simple historical baseline based on recent spreads and time-of-day behavior. A more flexible model could then be evaluated on whether it adds meaningful information beyond those recurring patterns.
Validation would remain chronological and include entire blocks of unseen time. This avoids overstating performance through random train-test splits and makes the test conditions closer to the way a market forecast would actually be used.
The baseline might begin with the median spread for the same hour and recent comparable days, followed by a linear model that makes its assumptions easy to inspect. Tree-based boosting could then capture nonlinear interactions among demand, weather, renewable output, and recent pricing without requiring those relationships to be specified in advance.
Model selection would prioritize stability rather than the best score from one test window. Walk-forward evaluation across several seasons would show whether a configuration behaves consistently as the market changes. Performance during volatile weeks would be reported separately so a few quiet months cannot dominate the conclusion.
Interpreting the result
Evaluation would consider forecast error alongside directional accuracy and performance during high-volatility intervals. Breaking results down by hour and market condition would make it easier to see whether the model captures genuinely useful signals or simply predicts the most common outcome.
Feature attribution could help connect statistical performance back to market structure. If recent real-time prices dominate every forecast, the model may offer little beyond persistence. If congestion, load-forecast error, or renewable conditions become important during specific regimes, those patterns could support a more useful explanation of why the spread moves.
Calibration would matter as well. Forecasts grouped into increasingly large predicted spreads should show a corresponding increase in observed spread magnitude. A model that ranks intervals well but consistently overstates their size might still be useful, but it would need a different interpretation from a model intended to estimate exact prices.
Limitations and next steps
Price spreads can react to outages, congestion, scarcity, and other events that are difficult to represent with a small historical dataset. A later iteration could add richer grid-condition data, compare multiple settlement points, and test whether the approach remains stable as market behavior changes.
Publicly available inputs may not capture every constraint affecting real-time dispatch, and market rules can change over the life of the dataset. Those limits mean historical accuracy should not be treated as proof of future trading value. Transaction costs, timing, and the difference between published and actionable information would also need to be considered before moving beyond research.
The next version could begin with a small set of settlement points and a clearly defined forecast timestamp, then expand only after the leakage checks and baselines are reliable. Comparing the same framework across nodes would show whether the model learns general ERCOT conditions or location-specific behavior.
Conclusion
The project is intended as a practical study of how market structure, time-series validation, and interpretable modeling can work together. The final case study will focus as much on where the model fails as on the accuracy it achieves.
A useful outcome would be a model whose predictions can be traced back to recognizable system conditions and whose limitations remain visible. That would make the project more than an exercise in minimizing error: it would become a structured way to investigate how expectations and real operations diverge in ERCOT.