Choosing a loss

A loss defines the target domain, fitting scale, and transformation applied by predict. Pass the loss as the third argument to fit_tree:

using LinearTrees

X = reshape(collect(0.0:5.0), :, 1)
y = [0, 0, 1, 2, 3, 5]
tree = fit_tree(X, y, Poisson(); max_depth = 1)
(score = score(tree, X), response = predict(tree, X))

(score = [0.6061358035703155, 0.6061358035703155, 0.6061358035703155, 0.6061358035703155, 0.6061358035703155, 0.6061358035703155], response = [1.8333333333333333, 1.8333333333333333, 1.8333333333333333, 1.8333333333333333, 1.8333333333333333, 1.8333333333333333])

score returns values on the loss scale, with score clipping enabled by default. predict applies the inverse link. They are equal for identity-link losses. Logistic scores are log odds. Log-link scores are log means. Softmax scores are class logits.

Choose a loss from the target and the quantity you want to estimate.

Huber(δ) changes from squared to linear loss at residual magnitude δ. Quantile(τ) requires 0 < τ < 1. NegBin(θ) requires θ > 0. Tweedie(ρ) supports powers 1 < ρ < 2.

validate_target checks finite values and each loss domain. Every class from 1 through K must occur when fitting Softmax(K).

Classification conventions

Use numeric targets in {0, 1} with Logistic. A prediction is the probability that the target equals 1.

The Tables and StatsAPI classifier accepts labels. It sorts observed labels and stores that order in fit.classes. For two classes, fit.classes[1] is the positive class. Probability column k always represents fit.classes[k].

Softmax(K) uses class K as the reference class. score returns K - 1 logits against that reference. predict returns all K probabilities in class-code order.

X = reshape([-2.0, -1.0, 1.0, 2.0, 0.0, 0.5], :, 1)
y = [1, 1, 2, 2, 3, 3]
tree = fit_tree(X, y, Softmax(3); max_depth = 1)
(size(score(tree, X)), size(predict(tree, X)))

((6, 2), (6, 3))

Softmax fitting uses the diagonal of the reference-logit Hessian. It omits cross-class Hessian terms. For Softmax(2), this diagonal is the exact logistic Hessian.

How losses are fitted

Most smooth losses fit each candidate model from gradients and Hessians on the raw score scale. The working response is -g/h, and the working weight is h. The method proposes one local Newton step. It does not solve every node's full nonlinear optimization problem.

The main derivatives are:

Here f is the score, p is the logistic inverse link, and μ = exp(f). gradhess! provides the complete loss-specific definitions. Ordinary row Hessians have a floor of 1e-6 before sample weights apply.

Huber fitting uses the residual weight min(1, δ / abs(f - y)), with weight one at zero residual. This gives a tangent quadratic majorizer and avoids the inflated working responses caused by zero tail curvature. The same rule applies to Loss(HuberLoss(δ)) with positive loss scale and to boosting. gradhess! still reports the exact gradient and floored true Hessian. BIC compares the working quadratic fits, so it remains a surrogate selection criterion for Huber rather than exact likelihood BIC. Choose δ on the response scale using independent validation data.

Quantile and MAD are non-smooth. They use iteratively reweighted least squares instead of a true Newton step. Split search computes one set of residual weights. After selection, the chosen node refits for niter iterations, which defaults to five. The result approximates the node's L1 objective.

Boosting freezes the non-smooth working weights once per round. Its base tree does not run the node refit. See Boosting for the ensemble algorithm.

Smooth-loss step safeguard

Small Hessians can produce an excessive Newton step, particularly for rare logistic outcomes. Huber trees minimize the convex loss along the proposed direction between zero and one full step. A selected constant Huber node retains its original Newton direction before this check, allowing the step to reach a robust location when one residual-weighted update would stop short. Logistic, binary softmax, and log-link trees keep a non-increasing full step or halve it until the actual weighted loss descends. They use a zero update if no tested scale succeeds. These checks include the same score and feature clamps used for prediction.

It ensures descent for that node update within floating-point accuracy. It does not find the exact maximum-likelihood node fit or calibrate held-out probabilities.

Deviance and truncation

deviance reports

\[2 \sum_i w_i\,\ell(y_i, f_i),\]

where f is the raw score. This convention includes constants present in the implemented point loss. It omits distribution constants absent from that loss. Compare deviances fitted with the same loss and data. Lower values indicate a better fit under that loss.

With truncate=true, scorebound limits fitted scores. Identity-link losses derive bounds from the target range. Logistic and softmax use [-10, 10]. Log-link losses use an asymmetric score range: the lower bound admits small positive rates down to the representable exponential range, and the upper bound pads the largest observed response while keeping its exponential finite. Positive initial means retain their scale even below 1e-6. An all-zero count target instead starts from a finite near-zero mean because its log-mean optimum is not finite. Set truncate=false when extrapolation beyond the training range is required.

LossFunctions.jl losses

Wrap a supported LossFunctions.jl loss with Loss. Use IdentityLink for distance losses. Use LogitLink for margin losses. Use LogLink for PoissonLoss. canonical_scale matches an equivalent built-in loss where one exists.

using LinearTrees
using LossFunctions

loss = Loss(LossFunctions.L2DistLoss(), IdentityLink())
tree = fit_tree(X, y, loss)

Add LossFunctions.jl to your project to use its loss types directly. Adapted L1 and quantile losses use the same IRLS path as MAD and Quantile. Adapted LogitMarginLoss values use the logistic step safeguard.