API reference

fit_tree(X, y, loss=MSE(); kwargs...) -> LinearTree

Fit a linear model tree to the rows of numeric matrix X and target vector y. Columns are features. Targets must satisfy the domain of loss; feature values must be finite. Use predict for response-scale predictions and score for scores on the loss scale.

Keywords

• weights=nothing: nonnegative frequency weights, with a positive total. The default gives every row weight one. Zero-weight rows are excluded.
• categorical=Int[]: columns containing positive integer category codes.
• rule=BIC(): node selection rule. See BIC and GainRule.
• split_search=ExactSearch(): numeric threshold search. BinnedSearch and HybridSearch offer approximate search for scalar-score losses.
• max_depth=12: maximum branching depth. Unsplit LIN nodes add no depth.
• min_fit=10: minimum total row weight for another model-selection step.
• min_leaf=5: minimum total row weight on each side of a split.
• min_sum_hessian=1.0: stop below this total Hessian, summed over score coordinates.
• max_lin_chain=10: maximum number of consecutive unsplit linear nodes.
• truncate=true: bound score accumulation and each node's feature extrapolation.
• truncation_factor=3: parameter passed to scorebound. Must be at least one.
• features=1:size(X, 2): feature columns available for fitting. Prediction still expects the original matrix layout.
• nthreads=Threads.nthreads(): maximum number of available Julia threads to use.
• niter=5: node-refit iterations for non-smooth losses such as MAD and Quantile. Smooth losses ignore this keyword.
• presort=nothing: optional Int32 matrix of stable sorted row indices, one column per original feature. Used by exact and local-bin search.

Nodes add score increments along a path. Model selection uses a quadratic surrogate for nonquadratic losses. Logistic updates also check true training loss before accepting a full step. No pruning pass follows growth.

For table inputs and stored encoders, use LinearTreeRegressorFit or LinearTreeClassifierFit through fit.

fit_boost(X, y, loss = MSE(); nrounds = 100, eta = 0.1, max_depth = 5,
          min_fit = 10, min_leaf = 5, min_sum_hessian = 1.0,
          lambda_slope = 1.0, lambda_intercept = 1.0, gamma = 0.0,
          subsample = 1.0, colsample = 1.0, rng = Random.default_rng(),
          weights = nothing, categorical = Int[], truncate = true,
          Xval = nothing, yval = nothing, wval = nothing, patience = 10,
          split_search = ExactSearch(), nthreads = Threads.nthreads()) -> LinearBoost

Second-order gradient boosting (Guryanov 2019) with the linear model tree as base learner. Each round computes the gradient and Hessian of loss at the ensemble score, freezes them into a Frozen target, and fits one tree with GainRule(lambda_slope, lambda_intercept, gamma), depth max_depth, and the row and column samples drawn from rng. The tree's raw score times eta is added to the ensemble. With Xval/yval, the validation deviance is recorded per round and fitting stops after patience rounds without improvement, keeping the trees up to the best round. Exact and node-local binned searches presort X once. HybridSearch prepares global bin IDs once from all positive-weight training rows, then selects the sampled rows' IDs each round. It supports numeric features and scalar losses. Sampled-out rows have weight zero for that tree only. history holds the per-round deviance and the ensemble's validated flag says whether it is the validation deviance, and so whether early stopping was in play.

For Quantile and MAD the frozen Hessian is the IRLS weight at the ensemble score, so a round is one IRLS step, not an exact L1 fit.

predict(tree_or_ensemble, X; nthreads=Threads.nthreads())

Prediction on the response scale, linkinv(tree_or_ensemble.loss, score).

predict(tree_or_ensemble, X; nthreads=Threads.nthreads())

Softmax override: n × K matrix of class probabilities, linkinv applied row by row to the K-1-vector score.

In-place Softmax prediction into an n × K matrix.

score(tree_or_ensemble, X; clip=true, nthreads=Threads.nthreads())

Raw path sum per row on the link scale. clip=false returns the additive unclipped sum. Rows split into nthreads contiguous blocks when there are at least PARALLEL_MIN_ROWS of them; each row writes only its own output slot, so the result matches the serial loop exactly.

score(tree_or_ensemble, X; clip=true, nthreads=Threads.nthreads())

Softmax override: n × (K-1) matrix of raw reference-class logits.

Fitted tree. lo, hi are the stored first-truncation bounds on the score scale. base is the SHAP empty-coalition value. truncate == false disables both clamps at prediction.

LinearBoost{T,V,L}

A gradient boosting ensemble of Frozen-loss linear model trees. The score is f0 + eta Σ_t score(trees[t], x), clamped to [lo, hi] (the base loss's scorebound on the training target) when truncate, then passed through the base loss's link. history[t] is the validation deviance after round t when validation data was given, else the training deviance; validated says which, so a reader can tell whether early stopping was in play. Build with fit_boost.

Number of trees in the ensemble.

One node of a linear model tree. Coefficients have type V, which is the feature type T for scalar-score losses and SVector{K-1,T} for softmax. Categorical split nodes keep a packed left-level mask in the tree's mask pool at words catstart:catstart+catwords-1. Numeric nodes have catwords == 0.

Per-node model kind. LIN nodes have one child and add no depth.

Constant node: one intercept, no split.

Single line across the whole node: a x + b. One child, no added depth.

Two constants, one per side of a threshold.

Broken line, continuous at the threshold: basis [x, 1, max(x - t, 0)].

Two independent lines, one per side of a threshold.

PILOT's BIC rule: n log(dev/n) + dof(kind) log(n) with dof per kind (con, lin, pcon, blin, plin). The defaults are the paper's (1, 2, 5, 5, 7).

Test-only rule: pick the lowest surrogate deviance among kinds, never stop on con.

GainRule(; lambda_slope = 1.0, lambda_intercept = 1.0, gamma = 0.0)

Boosting's selection rule (Guryanov 2019, eq. 12 and 13). Offers con, pcon, and plin. The regularised deviance of a fit is `Σ h (z − a x − b)²

• lambdaslope a² + lambdaintercept b²`, obtained by adding the two ridges

to the Gram sums in fit_lin and fit_con. A split's score is its children's regularised deviance plus gamma per coefficient coordinate, so a split is taken exactly when its gain over the parent's constant fit exceeds gamma. BIC's dof penalty and dmin floor play no part.

Policy for choosing the numeric thresholds evaluated during tree fitting.

ExactSearch()

Evaluate every eligible numeric threshold. This is the default split search.

BinnedSearch(; nbins=64, refine=true)

Approximate numeric split search for scalar-score losses. Build at most nbins equal-count bins within each node, keeping tied values together. Evaluate bin edges using sufficient statistics of the original feature values and responses. Weights retain their frequency meaning. Categorical search remains exact.

With refine=true, scan all eligible row boundaries in the two bins adjacent to the winning coarse split. No refinement runs when a constant or unsplit line wins. Nodes with at most nbins rows use exact search.

This can change the chosen model and predictions. It does not guarantee the exact optimum or an error bound. nbins must be at least two. Vector-score losses, including Softmax, require ExactSearch().

HybridSearch(; nbins=64)

Approximate numeric split search for scalar-score losses. Assign rows to at most nbins equal-count bins once per fit, keeping equal feature values in the same bin. Tree nodes scan these global bins and refine the best coarse boundary over its two adjacent occupied bins. Nodes with at most nbins rows use exact search.

The sufficient statistics use the original feature values and responses, and weights retain their frequency meaning. This can change the selected model and predictions; it does not guarantee the exact optimum or an error bound. Use ExactSearch() or BinnedSearch() for categorical features, and ExactSearch() for vector-score losses. nbins must lie in 2:65535. Boosting reuses bins from all positive-weight training rows across rounds.

A twice-differentiable or IRLS-approximated loss. Implement gradhess!, linkinv, initscore, deviance, scorebound, and validate_target.

Squared-error loss for a real-valued target. Identity link.

Huber loss with transition δ: quadratic within δ of the target, linear beyond it. Identity link.

Pinball (quantile) loss at quantile τ. Non-smooth: fit by IRLS. Identity link.

Mean absolute deviation loss. Non-smooth: fit by IRLS. Identity link.

Logistic (binomial cross-entropy) loss for a {0,1} target. Logit link.

Poisson deviance loss for a non-negative integer count target. Log link.

Negative binomial deviance loss with dispersion θ, for an overdispersed count target. Log link.

Gamma deviance loss for a positive real target. Log link.

Tweedie deviance loss with power ρ ∈ (1, 2), for a non-negative target with a point mass at zero. Log link.

Softmax(K)

K-class softmax with class K as the reference at logit zero. Diagonal Hessian. K is a type parameter so the coefficient type SVector{K-1,T} is known at compile time and every fit specialises on it.

Frozen{V}() <: Loss

Boosting's per-tree loss. Each target row is a pair (z0, h0) of working response and frozen Hessian at the ensemble score, both of type V, so one tree minimises Σ w h0 (f − z0)² / 2: the second-order objective of a boosting round with z0 = −g0 / h0. gradhess! returns g = (f − z0) h0 and h = h0, including exact zero components, the start score is zero, the score bounds are infinite so truncate = true keeps only the feature clamp, and the loss is smooth so no IRLS refit runs. A tree fitted with Frozen predicts its raw score; only a LinearBoost gives it a link.

gradhess!(g, h, loss, y, f)

Unweighted g = ∂ℓ/∂f and h = max(∂²ℓ/∂f², HMIN) per row. Frozen preserves its supplied Hessian, including zero. The caller applies frequency weights after any floor.

gradhess!(g, h, ::Softmax, y, f)

Per-row cross-entropy gradient and diagonal Hessian against the K-1 reference-class logits, with integer class labels y in 1:K.

linkinv(loss, s)

Map raw score s to the response scale: the identity for MSE, Huber, Quantile, and MAD, the logistic sigmoid for Logistic, exp for the count and rate losses, and class probabilities for Softmax.

initscore(::Softmax, y, w)

Log-odds of each non-reference class against the reference class K, from the weighted class frequencies.

deviance(loss, y, f, w)

Weighted 2 Σ w[i] pointloss(loss, y[i], f[i]), the model's reported deviance on the score scale f.

deviance(m::LinearTreeClassifierFit)

Training deviance against m's stored coded target: 0/1 (positive class classes[1]) for a two-class Logistic fit, the 1-based sorted-class index for a Softmax fit. score(m.tree, m.X) returns a plain Matrix for Softmax, so each row is repacked into the SVector pointloss needs.

scorebound(loss, y; truncation_factor=3)

(lo, hi) score-scale clamp bounds fit to the training target y, used when fit_tree's truncate is set. With half-width B = (max(y) - min(y)) / 2, truncation_factor pads each side by (truncation_factor - 1) * B, so the bounds are [min(y) - (truncation_factor - 1) B, max(y) + (truncation_factor - 1) B].

validate_target(loss, y)

Throw ArgumentError if y is not finite everywhere or does not satisfy loss's domain (e.g. {0,1} for Logistic, non-negative integers for Poisson); otherwise return nothing.

issmooth(loss)

true when loss has a well-defined Hessian everywhere and fits by Newton steps; false for the L1-type losses (Quantile, MAD), which fit by IRLS.

Adapter around a LossFunctions.SupervisedLoss. Margin losses receive targets in {0,1} and are re-expressed with t = 2y - 1. The pointwise loss, gradient, and hessian are multiplied by scale.

Score-space link between the tree's raw score and the value the inner loss expects.

Score-space link marking the tree's raw score as a logit, for a margin loss.

Score-space link marking the tree's raw score as a log mean, for PoissonLoss.

Newton-step scale that lines an inner loss up with the matching native Loss.

feature_importance(tree)

Surrogate-deviance drop per feature, normalised to sum to one. Zeros when no node has positive gain.

feature_importance(boost)

Positive split gains summed over every tree, normalised to sum to one.

coeftable(tree, x)

Intercept and per-feature slopes of the unclipped score at x. A feature clamped at x contributes zero slope and its piece value goes to the intercept. Categorical pieces are constants.

coeftable(boost, x)

Intercept and per-feature slopes of the unclipped ensemble score at x: f0 plus eta times the sum of each tree's coeftable.

shap(tree, X; nthreads=Threads.nthreads()) -> ShapResult

Path-dependent TreeSHAP values for every row of X against every feature of tree, on the unclipped score scale. Each row's SHAP values sum to score_row(tree, X, i, false) - result.base, the game's efficiency identity.

shap(boost, X; nthreads=Threads.nthreads()) -> ShapResult

Path-dependent TreeSHAP for a LinearBoost; see shap!.

shap!(values, clipped, tree, X; nthreads=Threads.nthreads())

In-place shap: write into values (n × p, or n × p × (K-1) for Softmax) and clipped (Vector{Bool}, length n), then return a ShapResult wrapping them.

shap!(values, clipped, boost, X; nthreads=Threads.nthreads())

In-place shap for an ensemble: eta times the sum of every tree's path-dependent SHAP values, base f0 + eta Σ expected_score(tree), so each row sums to score(boost, x; clip = false) − base. clipped[i] is true when the ensemble clamp changed row i. One PathPool per row block, reused across the ensemble's trees.

ShapResult{A,B,C}

values is n × p for scalar-score trees, n × p × (K-1) for Softmax. base is the game's empty-coalition value, expected_score(tree); fit_tree stores the same number in tree.base. clipped[i] is true when the score bound changed row i's prediction.

expected_score(tree) -> V

Cover-weighted game_value(tree, ·, ∅) over the whole tree: at each split the absent feature's piece is evaluated at the node's xmean on both children, weighted by cover(child) / cover(parent). Independent of x. fit_tree stores it in tree.base. shap/shap! recompute it from the nodes (O(nodes)) so a hand-built tree whose base field is stale still satisfies the efficiency identity.

LinearTreeRegressorFit{Tr}

A fit_tree result wrapped with its input encoder and training data as a StatsAPI.RegressionModel. Build with fit(LinearTreeRegressorFit, X, y; loss=MSE(), weights=nothing, unseen=:error, kwargs...), where X is a matrix or a Tables.jl table and kwargs forward to fit_tree.

LinearTreeClassifierFit{Tr,C}

A fit_tree result for a categorical target, wrapped as a StatsAPI.StatisticalModel. Build with fit(LinearTreeClassifierFit, X, y; weights=nothing, unseen=:error, kwargs...). classes = sort(unique(y)) fixes both the label-to-code map and the column order of predict's probability matrix. Two classes fit a single Logistic tree with classes[1] (the first sorted class) as the positive outcome; predict returns hcat(p1, 1 .- p1), so column k is still P(classes[k]). Three or more classes fit a Softmax tree with classes[k] at coordinate k.

LinearBoostRegressorFit{B}

A fit_boost result wrapped with its input encoder and training data as a StatsAPI.RegressionModel. Build with fit(LinearBoostRegressorFit, X, y; loss=MSE(), weights=nothing, unseen=:error, Xval=nothing, yval=nothing, wval=nothing, kwargs...); Xval may be a matrix or table and is encoded like X; kwargs forward to fit_boost.

LinearBoostClassifierFit{B,C}

A fit_boost result for a categorical target, with the same class conventions as LinearTreeClassifierFit: classes[1] is the Logistic positive outcome for two classes, and column k of predict is P(classes[k]). Build with fit(LinearBoostClassifierFit, X, y; weights=nothing, unseen=:error, Xval=nothing, yval=nothing, wval=nothing, kwargs...).

fit(LinearTreeRegressorFit, X, y; loss=MSE(), weights=nothing, unseen=:error, kwargs...)
fit(LinearTreeClassifierFit, X, y; weights=nothing, unseen=:error, kwargs...)

Fit a LinearTreeRegressorFit or LinearTreeClassifierFit from a matrix or Tables.jl table X and a target y. kwargs forward to fit_tree. unseen is :error (default) or :right, applied to a categorical level absent from training at predict time.

Number of rows passed to fit, zero-weight rows included.

dof(m)

Count of stored coefficients across every node: 1 for CON, 2 for LIN or PCON, 3 for BLIN, 4 for PLIN, times K-1 for a Softmax fit. A parameter count, not an effective degrees of freedom, and unrelated to BIC's per-kind selection penalty of the same name.

Training row weights, as stored (ones when fit was called with no weights).

Training residuals y - predict(tree, X), on the response scale.

LinearTreeRegressor

A model type for constructing a linear tree regressor, based on LinearTrees.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

LinearTreeRegressor = @load LinearTreeRegressor pkg=LinearTrees

Do model = LinearTreeRegressor() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in LinearTreeRegressor(loss=...).

LinearTreeRegressor fits a PILOT-style linear model tree for a continuous or count target, generalised to any twice-differentiable loss.

Hyperparameters

• loss::Loss = MSE(): the loss to fit; see MSE, Huber, Quantile, MAD, Poisson, NegBin, Gamma, Tweedie.
• rule::SelectionRule = BIC(): model-kind selection rule at each node.
• max_depth::Int = 12: maximum tree depth.
• min_samples_split::Int = 10: minimum total weight in a node to attempt a split.
• min_samples_leaf::Int = 5: minimum total weight required in each child.
• min_sum_hessian::Float64 = 1.0: minimum summed hessian in a node to attempt a split.
• max_lin_chain::Int = 10: maximum consecutive lin fits along one root-to-node path.
• truncate::Bool = true: clamp predictions to the training score range.
• truncation_factor::Float64 = 3.0: half-range multiplier for the truncation bounds.

Operations

• predict(mach, Xnew): return the predicted response for Xnew.

Fitted parameters

The fields of fitted_params(mach) are:

• tree: the fitted LinearTree.

Report

The fields of report(mach) are:

• feature_importances: split-gain feature importances, as in feature_importance.
• kind_counts: a count of fitted nodes by ModelKind.

See also LinearTreeClassifier.

LinearTreeClassifier

A model type for constructing a linear tree classifier, based on LinearTrees.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

LinearTreeClassifier = @load LinearTreeClassifier pkg=LinearTrees

Do model = LinearTreeClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in LinearTreeClassifier(rule=...).

LinearTreeClassifier fits a PILOT-style linear model tree for a finite (two- or many-class) target: a Logistic tree for two classes, a Softmax tree for three or more.

Hyperparameters

• rule::SelectionRule = BIC(): model-kind selection rule at each node.
• max_depth::Int = 12: maximum tree depth.
• min_samples_split::Int = 10: minimum total weight in a node to attempt a split.
• min_samples_leaf::Int = 5: minimum total weight required in each child.
• min_sum_hessian::Float64 = 1.0: minimum summed hessian in a node to attempt a split.
• max_lin_chain::Int = 10: maximum consecutive lin fits along one root-to-node path.
• truncate::Bool = true: clamp predictions to the training score range.
• truncation_factor::Float64 = 3.0: half-range multiplier for the truncation bounds.

Operations

• predict(mach, Xnew): return the predicted class distribution for Xnew, as a UnivariateFinite. Use predict_mode(mach, Xnew) for point predictions.

Fitted parameters

The fields of fitted_params(mach) are:

• tree: the fitted LinearTree.

Report

The fields of report(mach) are:

• feature_importances: split-gain feature importances, as in feature_importance.
• kind_counts: a count of fitted nodes by ModelKind.

See also LinearTreeRegressor.

LinearBoostRegressor

A model type for constructing a linear boost regressor, based on LinearTrees.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

LinearBoostRegressor = @load LinearBoostRegressor pkg=LinearTrees

Do model = LinearBoostRegressor() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in LinearBoostRegressor(loss=...).

LinearBoostRegressor fits a gradient-boosted ensemble of linear model trees for a continuous or count target. MLJ iteration controls nrounds; this model does not expose validation early stopping, for which use IteratedModel.

Hyperparameters

• loss::Loss = MSE(): scalar-response loss, such as MSE, Huber, Quantile, MAD, Poisson, NegBin, Gamma, or Tweedie.
• nrounds::Int = 100: number of boosting rounds.
• eta::Float64 = 0.1: shrinkage per round.
• max_depth::Int = 5: maximum base-tree depth.
• min_samples_split::Int = 10: minimum node weight for a split.
• min_samples_leaf::Int = 5: minimum child weight.
• min_sum_hessian::Float64 = 1.0: minimum node Hessian sum.
• lambda_slope::Float64 = 1.0: slope penalty.
• lambda_intercept::Float64 = 1.0: intercept penalty.
• gamma::Float64 = 0.0: split-gain penalty.
• subsample::Float64 = 1.0: row-sampling fraction.
• colsample::Float64 = 1.0: feature-sampling fraction.
• truncate::Bool = true: clamp predictions to the training score range.
• rng::Union{AbstractRNG,Integer} = default_rng(): random source; an integer starts a fresh Xoshiro on each fit.

Fitted parameters and report

fitted_params(mach).boost is the fitted LinearBoost. report(mach) holds feature_importances, nrounds, and per-round history; the history is also available through training_losses(mach).

See also LinearBoostClassifier.

LinearBoostClassifier

A model type for constructing a linear boost classifier, based on LinearTrees.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

LinearBoostClassifier = @load LinearBoostClassifier pkg=LinearTrees

Do model = LinearBoostClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in LinearBoostClassifier(split_search=...).

LinearBoostClassifier fits a gradient-boosted ensemble of linear model trees for a finite target. It uses Logistic loss for two classes and Softmax for three or more. MLJ iteration controls nrounds; this model does not expose validation early stopping, for which use IteratedModel.

Hyperparameters

• nrounds::Int = 100: number of boosting rounds.
• eta::Float64 = 0.1: shrinkage per round.
• max_depth::Int = 5: maximum base-tree depth.
• min_samples_split::Int = 10: minimum node weight for a split.
• min_samples_leaf::Int = 5: minimum child weight.
• min_sum_hessian::Float64 = 1.0: minimum node Hessian sum.
• lambda_slope::Float64 = 1.0: slope penalty.
• lambda_intercept::Float64 = 1.0: intercept penalty.
• gamma::Float64 = 0.0: split-gain penalty.
• subsample::Float64 = 1.0: row-sampling fraction.
• colsample::Float64 = 1.0: feature-sampling fraction.
• truncate::Bool = true: clamp predictions to the training score range.
• rng::Union{AbstractRNG,Integer} = default_rng(): random source; an integer starts a fresh Xoshiro on each fit.

Operations

predict(mach, Xnew) returns a UnivariateFinite; use predict_mode for point predictions.

Fitted parameters and report

fitted_params(mach).boost is the fitted LinearBoost. report(mach) holds feature_importances, nrounds, and per-round history; the history is also available through training_losses(mach).

See also LinearBoostRegressor.

A named view of a tree for AbstractTrees traversal and printing.

to_dict(tree)

JSON-friendly form: nodes as vectors of field dictionaries, loss as a name plus parameters. Three encodings keep every field JSON-safe:

• Every Real-valued field (threshold, lcoef, lintercept, rcoef, rintercept, xmin, xmax, cover, xmean, gain, lo, hi, base) goes through jsonnum: finite values become Float64, non-finite values become the string "Inf", "-Inf", or "NaN", and an SVector field maps elementwise into a Vector{Any}.
• Integer fields (feature, left, right, catstart, catwords, model) store as Int, restored to their struct field type (Int32 or ModelKind) by from_dict.
• catmasks (UInt64, may use the top bit) store as lowercase hex strings via string(m; base = 16), since a plain integer would overflow Int64 and lose exactness in a JSON parser; restored with parse(UInt64, s; base = 16).

from_dict(d) -> LinearTree

Inverse of to_dict. Throws ArgumentError for an unrecognised format version, element type, or loss name.