The trajectory balance method is a training technique used in trajectory optimization and reinforcement learning that balances the probability of generating a trajectory against the probability of its reverse, ensuring that a learned policy produces paths consistent with a target distribution. You input a target trajectory distribution (or reward function) and a policy parameterization, and the method returns a corrected policy that satisfies detailed balance, enabling accurate sampling or planning. This article explains the method’s background, the mathematical formulation, practical implementation steps, common pitfalls, and how it compares to alternative approaches.
Formula
The trajectory balance objective is defined as: for a trajectory τ = (s0, a0, s1, …, sT), the method enforces that the forward probability F(τ) = Z · R(τ) equals the backward probability B(τ), where Z is a learned normalization constant, R(τ) is the target reward (or unnormalized density), and B(τ) is the probability of generating the reverse trajectory. The loss is typically the squared log-ratio:
L(θ) = (log Fθ(τ) – log Bθ(τ))²,
where Fθ(τ) = ∏t πθ(at|st) · P(st+1|st, at) and Bθ(τ) = ∏t πθ(at|st+1) · P(st|st+1, at). Here, πθ is the policy, P is the environment transition probability, and Z is a scalar that normalizes the target distribution. In practice, the transition probabilities are often known or estimated, and the policy is trained to minimize this loss via stochastic gradient descent.
Worked Example
Consider a simple two-step trajectory with states s0, s1, s2. Let the target reward R(τ) = exp(β · sum of rewards) with β = 1 and per-step reward r = 1, so R(τ) = exp(1 · (1+1)) = e² ≈ 7.389. The forward policy is parameterized as π(a|s) = softmax(θ), and the backward policy is the same but reversed. For a uniform prior, the log partition function log Z is estimated as the log of the sum of exp(rewards) over all possible trajectories. For a single trajectory, log Z = log(e²) = 2. The forward log-probability is log F = log π(a0|s0) + log π(a1|s1) + log P(s1|s0,a0) + log P(s2|s1,a1). If we assume deterministic transitions (P=1) and initial policy π = 0.5 for each action, then log F = log 0.5 + log 0.5 = -1.386. The backward log-probability is similar: log B = log π(a0|s1) + log π(a1|s2) + log P(s1|s2,a1) + log P(s0|s1,a0) = -1.386. The trajectory balance loss is (log F – log B)² = 0, which is already balanced. With a non-uniform target, the loss would be non-zero and the policy would be updated to reduce it.
Assumptions
- The environment transition probabilities P(s’|s,a) are known or can be estimated; otherwise, they must be learned separately.
- The target distribution is defined by a reward function R(τ) that is non-negative and finite for all trajectories.
- The policy is fully parameterized and can represent any distribution over actions given a state.
- The method assumes that the reverse process is well-defined, which requires that the state space is reversible (i.e., for every transition, there is a possible reverse transition).
- The normalization constant Z is learned as a parameter, typically initialized to 1 and updated via the loss.
When to Use Trajectory Balance
Trajectory balance is particularly useful in settings where you want to sample from a target distribution over trajectories, such as in generative models of graphs, molecules, or planning problems. It is a variant of the more general “balance” conditions used in Markov chain Monte Carlo and in the field of stochastic thermodynamics. Unlike simpler methods that only match marginal distributions, trajectory balance ensures that the full path distribution is correct, which is crucial when the reward depends on the entire trajectory (e.g., cumulative reward in reinforcement learning). It is often used in combination with neural networks to learn policies for discrete action spaces, such as in the GFlowNet framework.
How to Choose the Reward and Temperature
The reward function should reflect the desirability of a trajectory, often set as the exponential of a negative energy or as a product of per-step rewards. The inverse temperature β controls the sharpness of the distribution: high β makes the policy focus on high-reward trajectories, while low β makes it more exploratory. In practice, you should choose β so that the effective number of trajectories (the entropy of the target distribution) is manageable. If β is too high, the loss may become numerically unstable due to large exponentials; if too low, the policy may ignore reward differences. A common heuristic is to start with β = 1 and adjust based on the variance of rewards.
Common Problems and Misconceptions
- Ignoring the normalization constant: Some implementations forget to learn Z, leading to a biased objective. Always include Z as a trainable parameter.
- Using a fixed backward policy: The backward policy must also be learned or known; otherwise, the balance condition cannot be satisfied.
- Assuming known transitions: In model-free settings, the transition probabilities are unknown, making the backward term intractable. In such cases, you must use a learned model or use a different variant like detailed balance.
- Misinterpreting the loss: The loss is not the difference in probabilities but the squared log-ratio; using the raw difference can lead to poor gradients.
- Believing it is the same as maximum entropy RL: While related, trajectory balance explicitly enforces reversibility, which maximum entropy RL does not necessarily do.
Comparison to Other Balance Methods
Trajectory balance is one of several balance conditions used in GFlowNets. The most basic is “flow matching,” which enforces that the incoming flow equals the outgoing flow at each state. “Detailed balance” enforces a local balance condition for each transition, which is stronger than flow matching but weaker than trajectory balance. Trajectory balance is the strongest condition, requiring that the entire forward and backward path probabilities match. In practice, trajectory balance often leads to lower variance in gradient estimates compared to detailed balance, but it can be harder to optimize because the loss is over the whole trajectory. For long trajectories, the product of probabilities can become very small, causing numerical underflow; this is often mitigated by working in log space.
Implementation Tips
- Work in log space to avoid underflow: compute log F and log B as sums of log-probabilities.
- Use a separate neural network for the backward policy if the environment is stochastic.
- Initialize Z as the log of the total number of trajectories if the reward is uniform; otherwise, start with a small value and let it adapt.
- Use gradient clipping to prevent large updates when the log-ratio is extreme.
- For environments with continuous actions, you may need to use a change of variables to compute the backward probability.
Frequently Asked Questions
What is the difference between trajectory balance and detailed balance?
Detailed balance enforces that the forward probability of a single transition equals the backward probability of that same transition, multiplied by a ratio of state flows. Trajectory balance enforces that the product of forward probabilities over an entire trajectory equals the product of backward probabilities, times a global constant. Trajectory balance is a stronger condition, and it directly ensures that the distribution over full paths is correct, whereas detailed balance only guarantees local consistency.
Can trajectory balance be used with model-free reinforcement learning?
Yes, but it requires estimating the transition probabilities or using a learned model. In model-free settings, you can use a variant called “sub-trajectory balance” that only considers a segment of the trajectory, or you can learn a dynamics model. The original formulation assumes known transitions, which is often the case in planning problems with a simulator.
How do I choose the number of steps in the trajectory?
The number of steps depends on the problem. In episodic tasks, it is the episode length. In generative modeling, it is the number of actions needed to construct an object (e.g., the number of atoms in a molecule). The trajectory must be long enough to capture the dependencies that affect the reward. If the trajectory is too short, you may miss important structure; if too long, the loss may become difficult to optimize.
What does the normalization constant Z represent?
Z is the partition function, which normalizes the target distribution so that the sum of probabilities over all trajectories equals 1. In trajectory balance, Z is learned as a scalar parameter. It represents the total flow of probability from the initial state to the terminal states. In practice, its log is often used as a trainable parameter to improve numerical stability.
Is trajectory balance only for discrete action spaces?
No, it can be used for continuous actions, but the computation of the backward probability becomes more involved because you need to account for the change of variables in the probability density. In continuous spaces, you typically use the Jacobian determinant of the transformation. Many implementations focus on discrete spaces because the products are simpler, but the principle applies broadly.
Why is the loss squared log-ratio instead of something else?
The squared log-ratio is used because it is symmetric and ensures that the gradient is zero when the balance condition is met. It also works well with stochastic gradient descent because the gradient of the squared log-ratio is well-behaved. Other losses, like the absolute difference, are less smooth and can lead to slower convergence.
The result from this calculator is an estimate based on the inputs and assumptions; for exact figures, consult the relevant technical documentation or your own experiments.
Related
Formula used
in p | Number of steps in trajectory | steps | 10 | 1 | 1000 in r | Target reward per step (constant) | - | 1 | 0 | 100 in beta | Inverse temperature (1/kT) | - | 1 | 0.01 | 10 out logZ | Log partition function (estimate) | - | log(sum(exp(beta * r) for each step)) | 2 out weight | Weight of target trajectory | - | exp(beta * r * p) / exp(logZ) | 4
Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work
Method: the standard loan formula for this calculation, fixed rate and monthly compounding unless stated, rounded as each result states. Not financial advice; no lender, rate or product is recommended. See formulas and assumptions, the disclaimer or report an error.
More in this section
- Secured Loan with Savings Account: How It Works$405.53
- Secured Loan Servicing: How It Works & What to Know$405.53
- Secured Loan With a Car Title? How It Works$405.53
- Secured Loan Interest Rates: How They Work$405.53
- Secured Loan Requirements: What Lenders Check$405.53
- Secured Loan Home Improvement: How It Works$405.53