Machine Learning
See recent articles
Showing new listings for Friday, 11 September 2026
- [1] arXiv:2609.10767 [pdf, html, other]
-
Title: Weighted Empirical Risk Minimization for Machine Learning under Long-Range Dependence: Exact Pathwise Rates and Learning-Error GeometryComments: 42 pages, 5 figuresSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
We develop an exact almost-sure learning theory for smooth parametric models trained by regularly weighted empirical risk minimization on long-range dependent data. The training observations are generated from a fixed finite window of a stationary Gaussian sequence, and the sample weights are regularly varying. If the loss gradient at the population minimizer has Wiener-chaos rank $m$ and a nonzero low-frequency coefficient, then, in the long-memory interior regime, the finite-lag score reduces on the iterated-logarithm scale to a single weighted Hermite chaos. This yields an almost-sure Bahadur representation, an exact limsup law for the learned parameter, and, for $m\ge2$, the functional cluster set of the complete learning trajectory. The polynomial learning exponent is determined by the memory parameter and the chaos rank and is invariant under the admissible power weighting, whereas the sharp pathwise constant and cluster geometry depend on the weights. In the rank-one case, global optimization over the admissible power exponents shows that every optimizer is positive. Time-series prediction and classification examples illustrate the results.
- [2] arXiv:2609.11295 [pdf, html, other]
-
Title: A Hilbert-Valued Functional Decomposition Framework for Explaining Time-Dependent OutputsSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Feature-based explanations quantify features' influence on model predictions, but are primarily designed for scalar outputs. In many applications, however, outputs are functional or multivariate, such as time-dependent trajectories in demand forecasting. Consequently, existing approaches typically explain each output location independently, ignoring dependencies across the output components. We address this limitation by developing a unified framework for feature-based explanations of time-dependent outputs. Specifically, we generalize functional decomposition to Hilbert-valued prediction functions and extend an existing feature-based explanation framework to this setting. Our framework introduces kernel-based output representations that enable time-dependency-aware explanations at multiple levels of temporal granularity, including time-specific, time-resolved, and time-aggregated, while providing a unified view in which existing methods arise as special cases. We validate our framework on synthetic and real-world data, including intraday financial market volatility prediction and energy demand forecasting.
- [3] arXiv:2609.11524 [pdf, other]
-
Title: Risk-Averse Decision Making with Multi-Level Reliability GuaranteesSubjects: Machine Learning (stat.ML); Information Theory (cs.IT); Machine Learning (cs.LG)
Many applications in engineering, including wireless broadcasting, require designs that provide performance certificates at different target outage levels. This paper studies the problem of maximizing the weighted average of such certificates in the presence of uncertainty about the true system state. The problem is shown to be equivalent to an optimization over nested prediction sets, connecting to the literature on conformal prediction and extending prior art on single-level risk-averse decision making. Furthermore, we derive a dual formulation that decouples optimization across input values. Numerical experiments on a diversity-based wireless transmission system illustrate the cost of enforcing multi-level certificates with a single shared policy and trace the Pareto trade-off between multiple reliability levels.
- [4] arXiv:2609.11592 [pdf, html, other]
-
Title: A distribution-free certification framework for trustworthy crash-severity predictionSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Crash-severity models inform screening, dispatch and site prioritization, yet are deployed without a finite-sample statement of what one prediction means. Off-the-shelf guarantees fail here, because the features that make crash severity distinctive defeat them: the KABCO outcome is ordinal, the recorded label is a field assessment agreeing with medical severity about half the time, erring in a structured way, and deployment crosses jurisdictions and years calibration never saw. We develop a certification layer that wraps any severity model unmodified, with distribution-free guarantees using this structure: contiguous ordinal sets that read as "B or worse"; per-class validity for any pre-declared partition, with an oracle efficiency characterization; transfer of coverage to unobserved true severity through a declared reporting band, with a worst-case sharpness result; a one-sided certificate under deployment shift; and severity-weighted risk control. The guarantees compose with an attributable slack budget. The same analysis bounds what certification can achieve. A certified set's informativeness is governed by a functional of the true law that no base model can evade and that cannot be lower-bounded distribution-free; given a declared misreporting channel identified from record-linkage data, a nonvacuous lower bound on that floor becomes computable. On 5.2 million Texas records across seven base models spanning four decades, the layer attaches identical validity and certifies, on the vulnerable road users, a model-independent floor on set width that no base model beats, separating it from a remainder that stays bounded but distribution-free unidentifiable. The framework is released as an open-source package with theorem-level tests.
- [5] arXiv:2609.11606 [pdf, html, other]
-
Title: Identifiability of Nonnegative Tensor Decompositions via Positive ScatteringSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Combinatorics (math.CO); Statistics Theory (math.ST)
Identifiability of tensor decompositions is often established through linear-algebraic conditions on the factor families. For nonnegative decompositions, however, positivity provides additional information that is not captured by dimension and independence alone: nonnegative terms cannot cancel, and their supports constrain competing decompositions. We introduce a positive scattering term that quantifies this additional source of identifiability and combine it with the dimension budget underlying the Lovitz--Petrov generalization of Kruskal's theorem. For every subset of components, we obtain two sufficient conditions: a threshold of $2|S|-2$ guarantees minimality and nonnegative rank, while the stronger threshold $2|S|-1$ guarantees uniqueness among nonnegative decompositions of the same length. The key result is a positive splitting inequality for irreducible exchanges of nonnegative rank-one tensors, which combines the dimension constraint with support-induced geometric rigidity. Although the scattering term is defined through an optimization over intermediate factor spaces, we show that its mode costs are exactly $0$, $1$, or $+\infty$, yielding an exact activation characterization in terms of graph connectivity. The resulting criterion can strictly certify sparse nonnegative tensor decompositions beyond the reach of Kruskal and Lovitz--Petrov conditions, including examples for which those conditions fail even after reshaping. In the matrix case, the two criteria reduce respectively to full-rank factorization and two-sided separability.
- [6] arXiv:2609.11712 [pdf, html, other]
-
Title: Generalization Analysis of Distributed Kernel-based Robust Gradient Descent AlgorithmsComments: 40 pages, 4 figuresSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Operator Algebras (math.OA); Probability (math.PR)
In this paper, we investigate the generalization performance of distributed gradient descent algorithms in a reproducing kernel Hilbert space under a robust loss function $l_{\sigma}$. By exploiting the spectral characterization of gradient descent together with the intrinsic properties of robust loss functions, we establish optimal learning rates for the distributed kernel-based robust gradient descent (DKRGD) algorithm with an appropriately chosen scale parameter $\sigma$. The proposed parameter choice of $\sigma$ simultaneously alleviates the saturation phenomenon and guarantees statistical robustness. A key technical contribution is a novel error analysis that provides substantially sharper bounds for products of operators, thereby significantly relaxing existing restrictions on the maximum number of local machines while retaining optimal learning rates. Finally, we develop a communication-efficient strategy that further improves the convergence performance of DKRGD.
- [7] arXiv:2609.11807 [pdf, html, other]
-
Title: Near-Optimal Reinforcement Learning with Multi-Step Transition LookaheadSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
We study reinforcement learning (RL) with transition look-ahead, where the agent may observe which states would be visited upon playing any sequence of $\ell$ actions before deciding its course of action. Although look-ahead can substantially improve achievable performance, it is known that optimal planning with multi-step transition look-ahead is NP-hard, but this hardness was established using discount factors arbitrarily close to one. It was therefore unknown whether the problem remains hard for any discount factor, and whether near-optimal planning can nevertheless be performed efficiently. We resolve both questions. First, we show that for every fixed rational discount factor ($\gamma\in(0,1)$), exact planning remains NP-hard. Second, we introduce a randomized polynomial-time approximation scheme for every fixed look-ahead depth. We then extend our approach to unknown transitions and stochastic rewards using optimism and variance-adaptive confidence bounds. The resulting algorithm achieves cumulative regret whose leading term matches classical tabular discounted RL up to logarithmic factors. Thus, although exact planning with transition look-ahead is NP-hard, efficient near-optimal planning and learning remain possible.
- [8] arXiv:2609.11872 [pdf, html, other]
-
Title: Evaluating Time-Series Foundation Models and Multimodal Dietary Context for CGM ForecastingBowen Zhang, Hsiu-Wen Cheng, Hongyu Yang, Evie L. Shen, Joleen Vansomphone, Yuna Li, Kerry Zhou, Zitian Qu, Suning Zhao, Xiangning Deng, Hua Zhou, Jin J. ZhouSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Continuous glucose monitoring (CGM) provides high-frequency measurements of glucose dynamics and enables short-term glucose forecasting for diabetes management. Although time-series foundation models have shown strong general forecasting ability, their effectiveness for CGM prediction and the added value of multimodal dietary context remain unclear. We conduct a comprehensive empirical study using eight public CGM datasets spanning Type 1 diabetes, Type 2 diabetes, and non-diabetes populations. Under a unified protocol across multiple context lengths and prediction horizons, zero-shot foundation models did not consistently outperform strong task-specific baselines such as Elastic Net and PatchTST. In contrast, lightweight fine-tuning substantially improved forecasting performance. For example, fine-tuned Chronos-Bolt reduced RMSE by 6.5%-18.4% in the T1D cohort and by 8.6%-18.2% in the non-diabetes/T2D cohort, with comparable improvements in both in-distribution and out-of-distribution test settings. We further evaluate multimodal dietary context using CGMacros, which provides temporally aligned CGM signals, food images, and macronutrient records. A residual-based fusion framework reduced overall RMSE by approximately 3% and postprandial RMSE by approximately 15% relative to the CGM-only baseline. Moreover, Chronos-based CGM representations were more strongly correlated with observed postprandial glucose increments than representations from LSTM and CatBoost, even after those models incorporated additional dietary modalities, suggesting that pretrained temporal representations better preserve meal-induced excursion patterns. These findings show that foundation models require CGM-specific adaptation for reliable forecasting and that dietary context provides clinically meaningful signals beyond CGM alone, especially during postprandial periods.
- [9] arXiv:2609.11915 [pdf, html, other]
-
Title: Generative Marketing Mix Modeling: A Causal Inference Framework Linking GEO and GEM to Business ImpactSubjects: Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Econometrics (econ.EM); Methodology (stat.ME)
Generative artificial intelligence changes how firms reach customers, but standard marketing data do not record how often users see and notice a firm's name in generated answers. We develop Generative Marketing Mix Modeling (GMMM) to estimate the causal effects of Generative Engine Optimization (GEO) and Generative Engine Marketing (GEM). For GEO, GMMM combines repeated generated answers with question counts, shares of use across generative systems, and notice probabilities. For GEM, it combines records of sponsored placements with notice probabilities. GMMM compares expected business responses under alternative treatment sequences and establishes sufficient conditions for identifying the resulting effects. We investigate the empirical performance of the proposed method using simulated answers to product recommendation in English and Japanese.
New submissions (showing 9 of 9 entries)
- [10] arXiv:2608.14641 (cross-list from cs.AI) [pdf, html, other]
-
Title: Task- and Session-Level Model Routing: A Common-Interface Hybrid Evaluation of Four Open-Source Routers Across Four BenchmarksComments: 34 pages, 25 tablesSubjects: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
Agentic systems increasingly delegate model selection to a router, yet open-source routers are usually evaluated with different tasks, candidate pools, and execution protocols, limiting direct comparison. We present a common measurement protocol and hybrid evaluation of four router implementations across RouterBench, BFCL v4, tau2-bench, and WebArena. We evaluate 290 frozen tasks against a locked matrix of 2,610 candidate outcomes. Three routers emit constant or near-constant tier assignments; only vLLM Semantic Router varies materially with prompt content, and it has the highest observed success rate on none of the four benchmarks. Always-Mid matches Aurelio exactly on three benchmarks and within 0.003 on the fourth. For vLLM, task-level superiority tests detect no task-specific advantage over a share-matched content-blind allocation; equivalence is established only on WebArena at the protocol-declared five-percentage-point margin. The results show that, under these configurations and controls, observed gains track selected-tier composition more closely than demonstrated task-specific targeting. Fixed-tier baselines and selected-tier distributions are therefore necessary controls in router evaluation; the findings are scoped to these configurations, candidate pool, and frozen benchmark samples, not to routing paradigms in general.
- [11] arXiv:2609.10563 (cross-list from math.OC) [pdf, other]
-
Title: Supply Chain Analytics: A Data-Driven ApproachComments: Draft chapters / working book manuscript, 101 pagesSubjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Machine Learning (stat.ML)
Modern supply chain networks increasingly rely on real-time data to navigate structural uncertainties, market volatility, and operational disruptions. This manuscript bridges the gap between statistical data-driven learning and robust decision-making frameworks in logistics and operations management. We present a comprehensive, mathematically rigorous treatment of supply chain analytics, moving from empirical demand forecasting to optimal inventory and network control under uncertainty. Key topics explored include sample minimization, dynamic programming recursions for time-varying inventory replenishment, network fulfillment frameworks, and advanced distributionally robust optimization (DRO) via transport theory to hedge against rare events. By integrating predictive statistical models with prescriptive control algorithms, such as column generation for vehicle routing and non-homogeneous queueing regimes, this text provides the foundational tools necessary for designing resilient, data-driven automated systems. It serves as both a theoretical blueprint and an algorithmic guide for researchers and practitioners operating at the intersection of machine learning, mathematical optimization, and applied probability.
- [12] arXiv:2609.10611 (cross-list from cs.CR) [pdf, html, other]
-
Title: Black-Box Membership Inference via Word-Level Probability EstimationComments: 17 pages, 5 figures, and 17 tablesSubjects: Cryptography and Security (cs.CR); Machine Learning (cs.LG); Machine Learning (stat.ML)
Membership inference attacks (MIAs) have emerged as critical tools for auditing privacy risks in large language models (LLMs), aiming to determine whether a given text was included in a model's training corpus. However, most existing MIAs require access to per-token logits or probabilities, making them inapplicable in practice to proprietary LLMs that expose only textual continuations. To address this underexplored setting, we propose Word-level Probability MIA (WPMIA), a statistically principled MIA for strict black-box privacy auditing. WPMIA estimates word-level generation probabilities via Monte Carlo sampling with local kernel smoothing, then aggregates these estimates into a sequence-level likelihood estimator. Furthermore, WPMIA constructs the likelihood conditioned on different prefixes, thereby amplifying the distributional differences between members and non-members. We evaluate WPMIA across various open-source LLMs and find that it consistently outperforms existing black-box baselines. Importantly, we also evaluate WPMIA on modern proprietary LLMs, including GPT-5-Chat, Gemini-2.5-Flash, and Claude-4.5-Haiku, achieving an average TPR@5\%FPR of 42.0 across these models. These results offer a sound foundation for future research on strict black-box membership inference. Code is available at \href{this https URL}{this https URL}.
- [13] arXiv:2609.10729 (cross-list from quant-ph) [pdf, html, other]
-
Title: A Quantum-Inspired Dequantization Method for Diagonally Weighted Matrix Functions: Application to Learning with Optimized Random FeaturesComments: 18 pages, 1 figureSubjects: Quantum Physics (quant-ph); Machine Learning (cs.LG); Machine Learning (stat.ML)
Quantum-inspired classical algorithms have dequantized several quantum machine learning routines by replacing quantum linear-algebra subroutines with classical counterparts. However, the sampler based on quantum singular value transformation (QSVT) for learning with optimized random features is not covered by existing dequantization frameworks, because the matrix to be inverted is not itself available through sampling access. In this work, we develop a classical algorithm to address this type of quantum-advantage candidate. Our method samples heavy indices, reduces the transformation to a small principal block, and outputs a sparse classical representation with operator-norm guarantees. Applying this method dequantizes the sampler for optimized random features, giving a classical sampler with prescribed accuracy and polynomially related runtime. These results show that the factorization underlying a quantum block encoding can itself provide sufficient classical structure even when sampling-and-query access to the composite matrix is unavailable.
- [14] arXiv:2609.10737 (cross-list from cs.LG) [pdf, html, other]
-
Title: Conformal Calibration TransferComments: Accepted at the 43rd International Conference on Machine Learning (ICML 2026)Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Conformal prediction converts point predictions into set-valued predictions with coverage guarantees under exchangeability between calibration and deployment data. We study conformal calibration transfer, where this requirement fails because labeled calibration is available only in a source space, while prediction sets are needed in a target space linked to the source through unlabeled paired observations (e.g., paired modalities or sensor changes). We propose Transported Conformal Calibration (TCC): we transport labeled source calibration into the target space using the paired data, and then correct residual post-transport mismatch using only unlabeled target inputs. We instantiate this correction with two complementary methods: TCC-KS, which uses a label-free uncertainty surrogate to detect mismatch and adjust calibration conservatively, and weighted-TCC, which reweights transported calibration toward the target domain for improved efficiency when weights are stable. We provide finite-sample target-domain coverage guarantees that adapt to an observable measure of mismatch. Across CIFAR-100-C, Tiny-ImageNet-C, and SEN12MS, we show reliable target-domain coverage transfer without labeled target calibration data, with label-free diagnostics that predict when correction is needed.
- [15] arXiv:2609.10863 (cross-list from cs.LG) [pdf, html, other]
-
Title: Flow Duality and Source Geometry for Categorical GenerationSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Continuous and discrete flow matching are usually treated as separate constructions. This paper identifies a duality between them: projecting continuous convex-interpolant paths with one-hot targets through a position-wise argmax yields discrete convex-interpolant paths. The result requires source laws with appropriate coordinate symmetry and boundary regularity, and it makes the continuous source distribution an explicit design choice for categorical generation. We derive the induced discrete interpolation behavior for Gaussian, bounded-uniform, and centered negative-exponential sources, showing that different source geometries lead to qualitatively different transition timing and vocabulary-size dependence. Small visual diagnostics and a short language-modeling pilot suggest that these source-design effects can also appear in learned transports and early generative quality.
- [16] arXiv:2609.10879 (cross-list from cs.LG) [pdf, html, other]
-
Title: Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and SymmetryComments: 102 pagesSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Recent work has identified incremental learning in shallow networks trained on single-index and multi-index models. However, existing analyses often rely on simplifying settings, such as small initialization, correlation loss, or layer-wise training. These choices reduce neuron interactions and leave some feature learning dynamics under standard initialization unexplored. We study training dynamics for polynomial-width two-layer networks learning orthogonal multi-index targets under standard initialization using polynomially many samples. We first prove that incremental learning still occurs: the loss decreases sequentially according to the Hermite expansion of the target, with lower-order components learned before higher-order components recover the individual target directions. In this standard initialization regime, training also shows a competitive reallocation of parameter mass: after the total mass fits the target mean and stabilizes, mass shifts into the target subspace and then concentrates on aligned neurons. Our theoretical analysis uses slightly modified gradient flow, while vanilla gradient descent empirically exhibits the same qualitative dynamics. Technically, we introduce a symmetry-based finite-width approximation via symmetrized networks, rather than comparing directly with an infinite-width limit. This yields better control of approximation errors and may be of independent interest.
- [17] arXiv:2609.10886 (cross-list from cs.LG) [pdf, html, other]
-
Title: Relatively Smart II: Tractable or Semi-Supervised Instance-Optimal LearningSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
We continue the study of relatively smart learning, introduced by Dughmi and Pour (2026), which asks a supervised learner to compete, marginal by marginal, with every distribution-fixed error guarantee soundly certifiable from unlabeled data. They showed that the One-Inclusion Graph (OIG) learner is relatively smart with a quadratic sample-complexity blowup, and that no relatively smart learner can do better, leaving open whether ERM or another natural or tractable learner achieves comparable guarantees. They also left open whether the blowup can be restricted to unlabeled data.
Our firs results shows that ERM---and in fact any proper consistent learner---is relatively smart for binary classification in the distribution-free setting. We show that a small certifiable error with $m$ samples implies a similarly small error on the uniform distribution over a random sample of size $O(m^2)$, yielding a cover of size at most $2^{m+1}$ on that sample. This suffices to control the error of proper consistent learners with $O(m^2)$ samples.
We then show that semi-supervised relatively smart learning is information-theoretically possible with a quadratic blowup only in unlabeled sample complexity and no blowup in labeled sample complexity. The learner uses a natural generalization of OIG to a leave-most-out transductive problem, where labels of part of a finite pool are revealed and the remaining labels are predicted.
Finally, this label efficiency comes at a cost in simplicity and tractability. If the hypothesis class is accessed only through an agnostic ERM oracle, any semi-supervised relatively smart learner with substantially sub-quadratic labeled-sample blowup requires super-polynomially many oracle calls. This holds even when the marginal is given explicitly, and thus also yields an intractability result for distribution-fixed learning that may be of independent interest. - [18] arXiv:2609.10928 (cross-list from cs.LG) [pdf, html, other]
-
Title: AUC Maximization from Biased Positive-unlabeled Data with ConfidenceAtsutoshi Kumagai, Tomoharu Iwata, Hiroshi Takahashi, Taishi Nishiyama, Kazuki Adachi, Yasuhiro FujiwaraComments: 31 pagesSubjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
Maximizing the area under the receiver operating characteristic curve (AUC) is a standard approach to imbalanced binary classification. Although positive and negative data are required for maximizing the AUC, negative data are often difficult to collect in some real-world applications due to privacy concerns or the need for specialized expertise to annotate them. Thus, AUC maximization from positive and unlabeled (PU) data has been attracting attention. Existing methods assume that labeled positive data are unbiased samples from the true positive distribution. However, this ideal assumption is often violated in practice. In this paper, we propose a method to maximize the AUC from biased PU data. To address the bias, our key idea is to exploit {\it confidence}, i.e., the probability that an instance is positive, associated with the small number of labeled positive data. We derive an estimator of the AUC risk using biased PU data with confidence, enabling AUC maximization under such bias. We further show that the rewritten AUC risk induces a Bayes-optimal AUC ranking even when the available confidence is any strictly increasing transformation of the true posterior probability. We experimentally show the effectiveness of our method on eight real-world datasets.
- [19] arXiv:2609.10976 (cross-list from cs.LG) [pdf, html, other]
-
Title: Phases in a class of associative memories via hidden neuronsComments: 43 pages, 5 figuresSubjects: Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)
Associative memory in the Hopfield network is attractor dynamics in a disordered many-body system, and higher-order and exponential extensions turn its retrieval update into softmax attention. The polynomial and exponential regimes have been analyzed by different methods, with no common architecture in which to ask what fixes the storage scale. In this paper we study the bipartite architecture of Krotov and Hopfield, which we call the class $H$, whose model is fixed by a Lagrangian for each layer, taking the hidden neurons as the order parameter of retrieval. At polynomial load the replica method yields the replica-symmetric phase diagrams and closed-form capacities, and the crosstalk moment is common to Ising and spherical visible neurons, so their differences come from the visible entropy. With a softmax hidden layer the load is exponential, and a copy representation maps the thermodynamics onto random-energy-model counting, with paramagnetic, condensed, and frozen phases. Heating destabilizes retrieval by quantized reassignments of attention, and typical Gaussian patterns remain metastable at every load. The regimes differ in their crosstalk statistics, central-limit at polynomial load and large-deviation at exponential load, and the class $H$ splits retrieval into two roles, the visible Lagrangian fixing stability and the hidden one the storage scale, two axes that may also guide the design of new Lagrangians.
- [20] arXiv:2609.10994 (cross-list from cs.LG) [pdf, html, other]
-
Title: Importance Weighting for Unlabeled-unlabeled Learning under Distribution ShiftAtsutoshi Kumagai, Tomoharu Iwata, Hiroshi Takahashi, Taishi Nishiyama, Kazuki Adachi, Yasuhiro FujiwaraComments: 19 pagesSubjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
Unlabeled-unlabeled (UU) learning allows us to learn a binary classifier from two sets of unlabeled data with different class-priors. It is a general framework because it includes a wide variety of supervised learning such as positive-unlabeled (PU) learning, noisy label learning, and similarity-based learning. Existing UU learning assumes that the test and training distributions have the same class-conditional densities. However, this assumption rarely holds in practice due to distribution shifts. This paper proposes a distribution shift adaptation method for UU learning that uses UU data in the training distribution and a few UU data in the test distribution. The proposed method is based on the importance weighting, which minimizes the test risk by using training data with estimated importance weights. Although existing importance weighting methods cannot handle UU data, we show that it can be done in a principled manner. Thanks to the generality of UU learning, our method can handle various learning problems such as PU and noisy label learning under distribution shift within a single framework while existing methods are usually tailored to a specific problem. Moreover, it does not require any assumption of the shift types such as covariate shift. We experimentally demonstrate the effectiveness of the proposed method with real-world datasets.
- [21] arXiv:2609.11073 (cross-list from math.OC) [pdf, html, other]
-
Title: Conformal-DRO: Distributionally Robust Optimization with Conformalized Ambiguity SetSubjects: Optimization and Control (math.OC); Machine Learning (stat.ML)
Data-driven distributionally robust optimization (DRO) typically treats the conditional outcome law as fixed and uses ambiguity sets to capture estimation error. This paper studies latent distributional heterogeneity, where each instance has an unobserved law but contributes only one observation, so uncertainty persists even if the mixture law is known. We propose Conformal-DRO, which uses nested conformal regions to construct an ambiguity set for the future latent law. Under exchangeability, the set covers this law with probability at least $1-\alpha$ in finite samples, without estimating underlying latent laws or their mixing mechanism. The conformal path induces a data-driven transport geometry, while $\alpha$ determines the radius. The worst-case problem reduces to a finite linear program over conformal shells and admits sparse adversarial solutions. The resulting robust value provides a finite-sample certificate for the selected decision's expected cost.
- [22] arXiv:2609.11132 (cross-list from cs.LG) [pdf, html, other]
-
Title: How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KLComments: 28 pages, 3 figures, 8 tablesSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Accurate posterior prediction need not require accurate approximation of Bayesian updates. We prove that an unbounded gap between the update maps can coexist with vanishing predictive KL for every fixed finite $K\ge2$ in a stationary symmetric Gaussian HMM. Exact Bayesian mixing and an explicit deterministic radial filter act on the same $K-1$ belief coordinates. As $q\to0^+$, their separation in centered logits in the worst case grows at least linearly in the natural confidence scale $L_K(q)$, while their categorical $D_{\mathrm{KL}}(\mathrm{exact}\|\mathrm{radial})$ vanishes at the same explicit witness. Along stationary HMM trajectories, the expected terminal KL between filtered posteriors also converges to zero at $H(q)=\lceil-\log(q)/c\rceil+1$. Typical blocks without switches drive both filters into a common confidence cone, where softmax curvature suppresses their disagreement; a single Gaussian maximal event controls adaptive noise. A sweep with equally spaced Gaussians over $K\in\{2,4,8\}$ illustrates the opposing trends, and binary controls at long horizons compare saturating and nonsaturating recurrences. The result isolates two missing links between internal update gaps and predictive cost: the contribution of separating states to expected loss and decoder sensitivity. Thus even an unbounded internal update gap does not by itself certify predictive failure. The construction is fixed in $K$ and does not provide a universal criterion for when compression is harmless or characterize when internal gaps must incur task loss.
- [23] arXiv:2609.11173 (cross-list from cs.LG) [pdf, html, other]
-
Title: Hierarchical Clustering Can Jointly Satisfy Richness, Consistency, and Scale InvarianceComments: 51 pages, 3 figuresSubjects: Machine Learning (cs.LG); Methodology (stat.ME); Machine Learning (stat.ML)
Despite its ubiquity, clustering lacks a universally accepted definition of what is a cluster. Kleinberg's Impossibility Theorem formalizes this difficulty by showing that no flat clustering method can simultaneously satisfy three natural axioms: scale invariance, richness, and consistency. In this paper, we ask whether this impossibility persists when the output is a hierarchy rather than a single partition. We show that, in contrast to the flat clustering setting, the hierarchical analog of these axioms are jointly satisfiable. In fact, there exist uncountably many hierarchical clustering methods satisfying these axioms, which we call admissible. We explicitly construct several admissible methods, including methods based on well-separated clusters and a non-binary version of single linkage. For certain pairs of admissible methods, the hierarchy produced by one always refines that produced by the other. This refinement relation defines a partial order on the class of admissible methods. This partially ordered set has no greatest element and contains uncountably many pairwise incompatible maximal elements, revealing substantial diversity among admissible methods. Nevertheless, this diversity is constrained: every admissible method contains a hierarchy of sufficiently well-separated clusters, and every finite collection of admissible methods shares such a nontrivial common backbone.
- [24] arXiv:2609.11310 (cross-list from cs.CV) [pdf, html, other]
-
Title: Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language ModelsGautam Rajendrakumar Gare, Siyi Li, Hewei Wang, Cesar Daniel Hernandez, Wei Zhao, Wolfgang M. Pauli, John Galeotti, Deva RamananSubjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Image and Video Processing (eess.IV); Machine Learning (stat.ML)
We address few-shot object detection with vision-language models (VLMs) in out-of-domain settings such as aerial, industrial, and medical imagery, using only ten annotated images for supervision. Existing adaptation methods are discrete prompt optimization and LoRA fine-tuning. We revisit a third option: soft prompting, where a small number of continuous prompt tokens are optimized while the pretrained backbone remains frozen.
We identify two key design choices. First, placing prompt tokens at the cross-modal boundary between visual and text tokens outperforms other placements (10.0 vs. 8.4 mAP). Second, initializing prompts from the empty space token outperforms semantic and random initialization.
With these choices, one to three learned tokens (7,168 parameters on average) match the best LoRA configuration on Roboflow20-VL (14.2 mAP, 10-shot) while training over 20,000x fewer parameters. Soft prompting remains harder to optimize, exhibiting higher variance across random seeds. Unlike LoRA, however, it causes no forgetting: the LoRA rank matching our accuracy reduces NaturalBench VQA accuracy by 35% relative, rising to 56% at the largest rank, whereas soft prompting leaves pretrained performance unchanged.
The learned tokens behave like prompts rather than weights. They transfer to a newer model without retraining (+0.8 mAP on Qwen3.5-9B) and can be verbalized into readable prompts competitive with prompt-search methods (matching DetPO and outperforming GEPA).
The approach also extends beyond detection. On RoboCasa manipulation tasks, the frozen $\pi_{0.5}$ vision-language-action policy benefits from soft prompting, matching the LoRA baseline on two of three tasks when tokens are placed at the gradient bottleneck. These results suggest modern VLMs already encode much of what is needed for specialized domains; the challenge is learning how to ask. - [25] arXiv:2609.11401 (cross-list from gr-qc) [pdf, html, other]
-
Title: Improving the Sensitivity of Gravitational Wave Detection with Weighted Conformal PredictionJournal-ref: Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 329:937-957, 2026Subjects: General Relativity and Quantum Cosmology (gr-qc); Machine Learning (cs.LG); Machine Learning (stat.ML)
In the last decade, kilometre-scale interferometric gravitational-wave detectors have observed hundreds of compact binary mergers, the majority of which are binary black holes. However, the data are noise-dominated, and multiple independent search algorithms (pipelines) are used to enhance sensitivity and improve robustness. Rather than the standard approach of selecting the most significant pipeline output, we combine the outputs from all pipelines using a conformal prediction-based framework to provide statistically rigorous confidence estimates for candidate events. While combining pipelines improves sensitivity and ranking robustness, it requires a principled statistical framework that remains valid as data properties evolve across observing runs. A key challenge is distribution shifts between simulated datasets used for training and calibration and the real, unlabelled, observations used for testing, which can invalidate coverage guarantees and bias confidence estimates. In this work, we address this challenge by incorporating likelihood-ratio reweighting into our conformal prediction framework to account for covariate shift. Using mock datasets containing simulated signals, we demonstrate that weighted conformal prediction restores well-calibrated coverage under covariate shift and increases the confidence of events near the detection threshold, recovering true signals that would otherwise be missed.
- [26] arXiv:2609.11521 (cross-list from cs.LG) [pdf, html, other]
-
Title: Generalized Score Matching for Parameter Estimation on Convex DomainsSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Maximum likelihood (ML) estimation is a principled and statistically efficient approach for learning probabilistic models. However, for unnormalized models, ML estimation requires evaluating the partition function and differentiating through it, which may not always be tractable. Score matching provides a practically viable alternative that circumvents this obstacle by fitting the score in a way that eliminates dependence on the normalizing constant. We derive the generalized score matching objective on a convex subset of $\mathbb{R}^{d}$ constructively starting from Minimum Probability Flow (MPF) learning, and show how classical score matching as well as domain-adapted variants for non-negative data arise naturally within the proposed framework. We show that the resulting objective is a {\it proper local scoring rule} of second-order, which provides the theoretical guarantee that the true density is recovered when the objective is minimized. Furthermore, for a model belonging to the exponential family, we establish convexity of the objective together with consistency of the finite-sample estimator under standard regularity conditions. Our derivation sheds new light on the scope and applicability of generalized score matching in various problem settings. We compare generalized score matching-based estimators on constrained domains, where the partition function is analytically intractable. We provide experimental results on parameter estimation for model densities belonging to the exponential family defined over convex subsets of $\mathbb{R}^{d}$, and a generative modeling use-case to demonstrate broader applicability of the proposed generalized score matching framework.
- [27] arXiv:2609.11648 (cross-list from cs.LG) [pdf, html, other]
-
Title: RDDMPI: Residual Denoising Diffusion Model for Probabilistic Multivariate Time Series ImputationSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Multivariate time series imputation (MTSI) aims to recover missing values in temporal data composed of multiple interdependent variables. This problem is central to real-world applications such as healthcare monitoring, traffic networks, and energy systems. Recent diffusion-based approaches have shown strong potential for probabilistic imputation by learning to generate missing values through iterative denoising. However, most existing approaches perform diffusion directly in the original data space, requiring the denoising network to simultaneously capture global structure, temporal dynamics, and stochastic variability. This makes the generative task unnecessarily complex, especially when modern deterministic imputers can already provide accurate initial reconstructions. To address this limitation, we propose RDDMPI, a conditional residual diffusion framework that operates directly in residual space. Instead of modeling the full missing signal directly, we reformulate probabilistic imputation as a baseline-residual decomposition, where a pretrained model captures the dominant signal and a diffusion process models the residual uncertainty. To better exploit deterministic guidance, \model{} conditions the reverse denoising process on both the baseline-completed signal and its latent representation, while a reliability-aware conditioning mechanism adaptively controls the influence of baseline information during residual generation. This formulation simplifies the diffusion learning objective, enabling it to focus on structured correction terms rather than reconstructing the full signal. Experiments on multiple benchmark datasets demonstrate that RDDMPI consistently improves both reconstruction accuracy and uncertainty quantification.
- [28] arXiv:2609.11696 (cross-list from physics.comp-ph) [pdf, html, other]
-
Title: Stress-Testing Dynamical and Generative Downscaling Using Subseasonal Extreme Precipitation ForecastsMauricio Lima, Marika Koukoula, Romain Pilon, Monika Feldmann, Erwan Koch, Daniela I.V. Domeisen, Tom BeuclerComments: 36 pages, 12 figures, 6 tables, submitted to JAMESSubjects: Computational Physics (physics.comp-ph); Geophysics (physics.geo-ph); Machine Learning (stat.ML)
Coarse spatial resolution limits the ability of subseasonal prediction models to resolve extreme precipitation. Downscaling with either dynamical or deep generative models can overcome this issue, but the comparative performance of these models for extremes across different atmospheric regimes remains poorly understood. In this work, we evaluate the Weather Research and Forecasting (WRF) model against a diffusion-based generative model by downscaling two physically distinct, extreme precipitation events up to lead times of 3 weeks. For a fair comparison with WRF, which can downscale boundary conditions from different driving models without model-specific training, the diffusion model is trained in an unpaired fashion. Both approaches improve upon the raw European Centre for Medium-Range Weather Forecasts forecasts, in comparison to fused rain gauge-radar observations in Switzerland (CombiPrecip), but exhibit regime-dependent strengths. WRF achieves the highest probabilistic skill for a multicell, non-stationary event. Conversely, the diffusion model is more consistent across different performance metrics for the two events, outperforming WRF in a more stationary supercell event. These results demonstrate that explicit dynamical modeling can add value for specific precipitation events for subseasonal lead times, and that generative downscaling adds value more broadly in different situations.
- [29] arXiv:2609.11749 (cross-list from math.OC) [pdf, html, other]
-
Title: Sparsity Regularized and Robust Mean Variance Portfolio Selection Under Ellipsoidal UncertaintySubjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Machine Learning (stat.ML)
We investigate mean-variance portfolio selection with an $\ell_0$-penalty to promote sparsity in asset allocations. Uncertainty in the mean return vector is incorporated through an ellipsoidal uncertainty set, yielding a robust sparse optimization framework. We characterize the structure of both local and global minimizers and exploit these properties in the risk minimization and return maximization formulations. Building on this structural insight, we develop a branch-and-bound algorithm tailored to the resulting robust sparse portfolio problems, together with a new pruning rule that can discard exponentially many candidate portfolios in a single step. Extensive computational experiments on real market data, together with comparisons against a mixed-integer second-order cone programming solver, demonstrate the effectiveness and competitiveness of the proposed approach.
- [30] arXiv:2609.11837 (cross-list from math.AP) [pdf, html, other]
-
Title: Quantitative Diffusive Limits for Singular Nonlocal TransportComments: 60 pages, 1 figureSubjects: Analysis of PDEs (math.AP); Machine Learning (stat.ML)
We study the nonlocal continuity equation \[ \partial_t\mu_b =\operatorname{div}\!\left( \mu_b\nabla\log\bigl((I-b^2\Delta)^{-1}\mu_b\bigr) \right) \] on a closed connected Riemannian manifold. For smooth strictly positive initial data, we prove that as $b \to 0$, its global solution converges to heat flow $\mu(t)$ at the sharp, uniform-in-time rate \[ \sup_{t\ge0}\|\mu_b(t)-\mu(t)\|_{L^1}\le Cb^2. \] The key estimate is the uniform dissipation of a $b$-weighted higher-order resolvent energy, which yields exponential relaxation despite the absence of a Wasserstein gradient-flow structure.
On the circle, we also analyze the corresponding deterministic $N$-particle dynamics. A weak--strong modulated energy argument gives \[ \mathbb E\!\left[ \sup_{t\ge0}W_1(\mu_b^N(t),\mu_b(t)) \right] \le C(Nb)^{-1/2} \] for iid initialization. Consequently, the choice $b\asymp N^{-1/5}$ approximates heat flow uniformly in time at rate $N^{-2/5}$. - [31] arXiv:2609.11918 (cross-list from cs.LG) [pdf, html, other]
-
Title: General Quantification of Covariate and Concept ShiftsComments: 38 pages, 9 figures, accepted at the 43rd International Conference on Machine Learning (ICML 2026)Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
Generalization under distribution shift remains a core challenge in modern machine learning, yet existing learning bound theory is limited to narrow, idealized settings and is non-estimable from samples. In this paper, we bridge the gap between theory and practical applications. We first show that existing definition of concept shift breaks when the source and target supports mismatch. Leveraging entropic optimal transport, we propose a key notion: $\gamma^{*}\!$-concept shifts, and derive a general error bound unifying covariate and $\gamma^{*}\!$-concept shifts, which applies to broad loss functions, label spaces, and stochastic labeling. We further develop estimators for these shifts with concentration guarantees, and the DataShifts algorithm, which can quantify distribution shifts and estimate the error bound in most applications - a rigorous and general tool for analyzing learning error under distribution shift.
Cross submissions (showing 22 of 22 entries)
- [32] arXiv:2501.12299 (replaced) [pdf, html, other]
-
Title: Sublinear Variational Optimization of Gaussian Mixture Models with Millions to Billions of ParametersComments: Published in Journal of Machine Learning Research, see this https URLJournal-ref: S. Salwig, T. Kahlke, F. Hirschberger, D. Forster, J. L\"ucke, Journal of Machine Learning Research, 27(167):1-70, 2026Subjects: Machine Learning (stat.ML); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
Gaussian Mixture Models (GMMs) range among the most frequently used models in machine learning. However, training large, general GMMs becomes computationally prohibitive for data sets that have many data points $N$ of high-dimensionality $D$. For GMMs with arbitrary covariances, we here derive a highly efficient variational approximation, which is then integrated with mixtures of factor analyzers (MFAs). For GMMs with $C$ components, our proposed algorithm substantially reduces runtime complexity from $\mathcal{O}(NCD^2)$ per iteration to a complexity scaling linearly with $D$ and sublinearly with $NC$. In numerical experiments, we first validate that the complexity reduction results in a sublinear scaling for the entire GMM optimization process. Second, we show on large-scale benchmarks that the sublinear algorithm results in speed-ups of an order-of-magnitude compared to the state-of-the-art. Third, as a proof of concept, we finally train GMMs with over 10 billion parameters on about 100 million images, observing training times of less than nine hours on a single state-of-the-art CPU. Finally, and fourth, we demonstrate the effectiveness of large-scale GMMs on the task of zero-shot image denoising, where sublinear training results in state-of-the-art denoising times while competitive denoising performance is maintained.
- [33] arXiv:2502.07891 (replaced) [pdf, html, other]
-
Title: The observational partial order of causal structures with latent variablesComments: 48 pages, 30 figures; changes following referee reviews. The most significant changes are the text in Sections 3.4 and 7.2, fixes to the statements of Propositions 7 and 8 and to the proof of Lemma 10, and the removal of the previous Proposition 3Journal-ref: Journal of Causal Inference,vol.14, no.1, 2026, pp.20250009Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Quantum Physics (quant-ph)
For two causal structures with the same set of visible variables, one is said to observationally dominate the other if the set of distributions over the visible variables realizable by the first contains the set of distributions over the visible variables realizable by the second. Knowing such dominance relations is useful for adjudicating between these structures given observational data. Here, we consider the problem of determining the partial order of equivalence classes of causal structures with latent variables relative to observational dominance. We provide a complete characterization of the dominance order in the case of three visible variables, and a partial characterization in the case of four visible variables. Our techniques also help to identify which observational equivalence classes have a set of realizable distributions that is characterized by nontrivial inequality constraints, analogous to Bell inequalities and instrumental inequalities. We find evidence that as one increases the number of visible variables, the equivalence classes satisfying nontrivial inequality constraints become ubiquitous. (Because such classes are the ones for which there can be a difference in the distributions that are quantumly and classically realizable, this implies that the potential for quantum-classical gaps is also ubiquitous.) Furthermore, we find evidence that constraint-based causal discovery algorithms that rely solely on conditional independence constraints have a significantly weaker distinguishing power among observational equivalence classes than algorithms that go beyond these (i.e., algorithms that also leverage nested Markov constraints and inequality constraints).
- [34] arXiv:2506.19695 (replaced) [pdf, html, other]
-
Title: Near-optimal estimates for the $\ell^p$-Lipschitz constants of deep random ReLU neural networksSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Probability (math.PR)
This paper studies the $\ell^p$-Lipschitz constants of ReLU neural networks $\Phi: \mathbb{R}^d \to \mathbb{R}$ with random parameters for $p \in [1,\infty]$. The distribution of the weights follows a variant of the He initialization. In the case of zero-bias networks, we derive high probability upper and lower bounds for wide networks that differ at most by a factor that is logarithmic in the network's depth. Remarkably, the behavior of the $\ell^p$-Lipschitz constant varies significantly between the regimes $ p \in [1,2) $ and $ p \in [2,\infty] $. For $p \in [2,\infty]$, the $\ell^p$-Lipschitz constant behaves similarly to $\Vert g\Vert_{p'}$, where $g \in \mathbb{R}^d$ is a $d$-dimensional standard Gaussian vector and $1/p + 1/p' = 1$. In contrast, for $p \in [1,2)$, the $\ell^p$-Lipschitz constant aligns more closely to $\Vert g \Vert_{2}$. We extend our analysis to networks with possibly non-zero biases drawn from arbitrary symmetric distributions. In this case, we obtain high probability upper and lower bounds that differ at most by a factor that is logarithmic in the network's width and linear in its depth.
- [35] arXiv:2509.25741 (replaced) [pdf, html, other]
-
Title: Test time training enhances in-context learning of nonlinear functionsComments: Under review at NeurIPS 2026. 44 pages, 2 figures, appendix included; revised synthetic experiment, corrected mistakes, and added background sectionSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction to adapt to the test data. While TTT has demonstrated considerable empirical success, its theoretical underpinnings remain limited, particularly for nonlinear models. In this paper, we investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time. We analyze this framework in the setting of single-index models, where the feature vector is drawn from a hidden low-dimensional subspace. For single-layer transformers trained with gradient-based algorithms and adopting TTT, we establish an upper bound on the prediction risk. Our theory reveals that TTT enables the single-layer transformers to adapt to both the feature vector and the link function, which vary across tasks. This creates a sharp contrast with ICL alone, which is theoretically difficult to adapt to shifts in the link function. Moreover, we provide the convergence rate with respect to the data length, showing the predictive error can be driven arbitrarily close to the noise level as the context size and the network width grow.
- [36] arXiv:2512.06956 (replaced) [pdf, html, other]
-
Title: Statistical analysis of Inverse Entropy-regularized Reinforcement LearningComments: 44 pagesSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Statistics Theory (math.ST)
Inverse reinforcement learning aims to infer the reward function that explains expert behavior observed through trajectories of state--action pairs. A long-standing difficulty in classical IRL is the non-uniqueness of the recovered reward: many reward functions can induce the same optimal policy, rendering the inverse problem ill-posed. In this paper, we develop a statistical framework for Inverse Entropy-regularized Reinforcement Learning that resolves this ambiguity by combining entropy regularization with a least-squares reconstruction of the reward from the soft Bellman residual. This combination yields a unique and well-defined so-called least-squares reward consistent with the expert policy. We model the expert demonstrations as a Markov chain with the invariant distribution defined by an unknown expert policy $\pi^\star$ and estimate the policy by a penalized maximum-likelihood procedure over a class of conditional distributions on the action space. We establish high-probability bounds for the excess Kullback--Leibler divergence between the estimated policy and the expert policy, accounting for statistical complexity through covering numbers of the policy class. These results lead to non-asymptotic minimax optimal convergence rates for the least-squares reward function, revealing the interplay between smoothing (entropy regularization), model complexity, and sample size. Our analysis bridges the gap between behavior cloning, inverse reinforcement learning, and modern statistical learning theory.
- [37] arXiv:2603.06851 (replaced) [pdf, html, other]
-
Title: Bilateral Trade Under Heavy-Tailed Valuations: Minimax Regret without a Variance BoundComments: 29 pages. v4: title changed (v3: Minimax Regret with Infinite Variance); abstract and introduction reframed around the feedback-interface message; adds a formal parametric two-point lower bound and corollaries on the price of adaptivity; corrections to the lower-bound construction and epoch assembly; related work expandedSubjects: Machine Learning (stat.ML); Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG)
In contextual bilateral trade under full feedback, the posted price does not affect which valuations are observed. We show that in this model such action-independent feedback removes the polynomial adaptation penalty familiar from heavy-tailed bandits: fully parameter-free algorithms attain the oracle minimax $T$-exponents up to logarithmic factors, with no knowledge of the moment order $p \in (1,2)$ or its scale $\sigma_p$, and -- in the nonparametric case -- none of the effective Hölder smoothness $\beta \in (0,1]$. The statistic that makes model selection possible is a paired squared-loss difference, whose noise-square term cancels exactly, leaving noise damped by the candidate gap. The resulting bilateral-trade regret rates are new. Trader valuations have bounded conditional densities and heavy tails -- finite $p$-th moments for some $p \in (1,2)$, with possibly infinite variance. An epoch-based algorithm with truncated means achieves regret $\widetilde{O}(T^{(2-p)/p})$ in the parametric model and $\widetilde{O}(T^{1-2\beta(p-1)/(\beta p + d(p-1))})$ when the market value function is $\beta$-Hölder, with matching $\Omega(\cdot)$ lower bounds -- under a mild nondegeneracy condition -- via Assouad's method and a fixed-support mixture construction -- characterizing the minimax rate in $T$ up to logarithmic factors over the effective smoothness range $\beta \in (0,1]$, interpolating between the classical nonparametric rate at $p{=}2$ and the trivial linear rate as $p \to 1^+$. The enabling structural step extends the self-bounding property of Bachoc et al. (ICML 2025) from bounded to real-valued valuations: within our conditionally independent, conditionally centered noise model, bounded conditional densities and finite first moments suffice for the expected regret of any price $\pi$ to satisfy $\mathbb{E}[g(m,V,W) - g(\pi,V,W)] \le L|m-\pi|^2$ -- no second moment is needed.
- [38] arXiv:2603.21235 (replaced) [pdf, html, other]
-
Title: Domain Elastic Transform: Bayesian Function Registration for High-Dimensional Scientific DataComments: Accepted for publication in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). This version corresponds to the accepted manuscript. 18 pages, 8 figuresSubjects: Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
Nonrigid registration is conventionally divided into point set registration, which aligns sparse geometries, and image registration, which aligns continuous intensity fields on regular grids. This dichotomy is limiting for emerging scientific data such as spatial transcriptomics, where high-dimensional vector-valued functions, e.g., gene expression, are defined on irregular sparse manifolds. Researchers must therefore either sacrifice single-cell resolution through voxelization or ignore functional signals in favor of geometric alignment.
We propose Domain Elastic Transform (DET), a grid-free probabilistic framework that jointly aligns geometry and function. By treating data as functions on irregular domains, DET registers high-dimensional signals directly without binning. Within a generalized Bayesian formulation, domain deformation is modeled as elastic motion guided by a joint spatial-functional likelihood. DET is fully unsupervised and scalable through registration on sampled points followed by displacement interpolation.
We evaluate DET on MERFISH mouse-brain slices and Stereo-seq mouse-embryo atlases. On a 90-case MERFISH benchmark with severe perturbations and no prior initialization, DET achieved the strongest spatial overlap and topology among the evaluated pipelines, while an accelerated PASTE2 variant achieved the highest label-transfer ARI. In an atlas-scale MOSTA feasibility study without cross-stage ground truth, nonrigid refinement improved several within-pipeline anatomical-domain and boundary-consistency measures.
These results suggest that grid-free function registration complements point-set, image-based, and optimal-transport approaches for high-dimensional scientific data. The DET implementation is available at this https URL (since Mar, 2025). - [39] arXiv:2605.20145 (replaced) [pdf, html, other]
-
Title: Goal-Oriented Lower-Tail Calibration of Gaussian Processes for Bayesian OptimizationJournal-ref: Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 2026Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Methodology (stat.ME)
Gaussian process (GP) predictive distributions are commonly used in Bayesian optimization (BO) to guide the selection of evaluation points for expensive objective functions. The choice of kernel and hyperparameters has a strong influence on the exploration--exploitation trade-off. For minimization, sampling criteria such as expected improvement (EI) depend on both the probability mass below the current best value and the shape of the predictive distribution in this region. This article studies goal-oriented calibration of GP predictive distributions below a low threshold $t$ in the noiseless setting, for standard GP models with hyperparameters selected by maximum likelihood. We consider two complementary forms of calibration below $t$ for inputs distributed according to a reference measure $\mu$: occurrence calibration over the design space and thresholded $\mu$-calibration on sublevel sets of the form $\{x\in\mathbb{X}, f(x)\le t\}$. We propose tcGP, a post-hoc method that combines these two forms of calibration for GP predictive distributions below $t$. With fixed GP hyperparameters, the exact EI sampling criterion based on tcGP generates a sequence of evaluation points that is dense in the design space. Experiments on standard benchmarks show improved lower-tail calibration and BO performance relative to standard GP models and globally calibrated GP models.
- [40] arXiv:2606.25601 (replaced) [pdf, html, other]
-
Title: Statistically Valid Post-Training Hyperparameter Selection: From Tuning to GuaranteesSubjects: Machine Learning (stat.ML); Information Theory (cs.IT); Machine Learning (cs.LG); Statistics Theory (math.ST)
Post-training hyperparameter selection is a critical step in the deployment of modern artificial intelligence systems, given the need to tune degrees of freedom of pre-trained models such as inference-time parameters, implementation-level settings, and thresholds driving decision rules. Despite its practical importance, hyperparameter selection is typically performed using best-effort empirical methods such as grid search or Bayesian optimization, which provide no formal statistical guarantees on reliability or safety. This monograph, intended for an audience of signal processing and machine learning researchers, presents a unified statistical framework for reliable post-training hyperparameter selection, centered on the learn-then-test (LTT) paradigm. LTT formulates the hyperparameter selection problem as multiple hypothesis testing over a candidate set of hyperparameters. The framework enables the choice of hyperparameters that provably satisfy application-specific reliability requirements---such as bounds on average risk, quantile risk, or information-theoretic constraints---with explicit, finite-sample control of error probabilities. The supporting statistical machinery, namely p-values, e-values, and concentration inequalities, is developed from first principles.
- [41] arXiv:2310.10092 (replaced) [pdf, html, other]
-
Title: Label Differential Privacy via AggregationSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
This paper explores the use of linear aggregation to protect the privacy of sensitive training labels through the concept of \emph{label differential privacy} (label-DP) while maintaining regression task utility. Our key finding is that weighted linear aggregation of training instances with i.i.d. $N(0, 1)$ weights can achieve $(\varepsilon, \delta)$-label-DP with $m = O\left(n/(\log(1/\delta))\right)$. Unlike prior methods, our approach relies on the minimum linear regression loss rather than the minimum singular value of the data matrix, resulting in better practical bounds on real datasets.
We also examine real-world mechanisms involving disjoint sets or \textit{bags} of instances. We demonstrate that aggregating labels from sub-sampled disjoint $k$-sized bags using i.i.d. $N(0,1)$ weights achieves $(\varepsilon,\delta)$-label-DP with $k \geq \Omega\left(\left((1/\varepsilon)\log\left(1/\delta\right)\right)^2\right)$. In both scenarios, the optimal linear mse-regressor on the aggregated data approximates the original dataset's optimum with high probability, without needing additive label noise.
Furthermore, we show that adding $N(0,1)$ noise to any constant fraction of labels allows for similar label-DP guarantees when aggregating labels over random disjoint bags, while preserving the utility of Lipschitz-bounded neural mse-regression tasks. - [42] arXiv:2403.19448 (replaced) [pdf, html, other]
-
Title: Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy GradientsComments: 25 pages, 4 figures, to appear at SIAM Journal on OptimizationJournal-ref: SIAM Journal on Optimization, Vol. 35, Iss. 2 (2025), 10.1137/24M1653422Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Systems and Control (eess.SY); Numerical Analysis (math.NA); Machine Learning (stat.ML)
Kakade's natural policy gradient method has been studied extensively in recent years, showing linear convergence with and without regularization. We study another natural gradient method based on the Fisher information matrix of the state-action distributions which has received little attention from the theoretical side. Here, the state-action distributions follow the Fisher-Rao gradient flow inside the state-action polytope with respect to a linear potential. Therefore, we study Fisher-Rao gradient flows of linear programs more generally and show linear convergence with a rate that depends on the geometry of the linear program. Equivalently, this yields an estimate on the error induced by entropic regularization of the linear program which improves existing results. We extend these results and show sublinear convergence for perturbed Fisher-Rao gradient flows and natural gradient flows up to an approximation error. In particular, these general results cover the case of state-action natural policy gradients.
- [43] arXiv:2505.07298 (replaced) [pdf, html, other]
-
Title: Learning-Based Surrogate Method for Stochastic Optimization under Decision-Dependent Uncertainty with Adaptive Random DesignsSubjects: Optimization and Control (math.OC); Machine Learning (stat.ML)
We study stochastic programs in which the latent decision-dependent uncertainty is described via a nonparametric regression model. The major challenge is that, without convexity assumptions on either the cost function or the regression model, the resulting objective is both nonconvex and nonsmooth, and its first-order information is unavailable due to the unknown decision-dependent distribution. To address this issue, we construct a learning-based surrogate model that integrates simulation and statistical learning by embedding Jacobian estimates of the regression function, which are updated iteratively and interactively during the optimization procedure. We develop an adaptive random design that concentrates design points around the current iterate for Jacobian estimation and we show that the mean squared error of Jacobian estimates achieves a dimension-independent convergence rate. Building on this, we propose the learning-based stochastic prox-linear (L-SPL) algorithm with adaptive random design and establish its nonasymptotic convergence rates under various parameter settings. Numerical results demonstrate that L-SPL algorithm significantly improves sample efficiency and achieves substantially lower objective values compared to the state-of-the-art methods. More broadly, our method implies that the statistical design in an iterative learning-based optimization algorithm can be novelly tailored to the local information of the optimization procedure to sharpen estimates and enhance the convergence performance and sample efficiency of the resulting algorithm.
- [44] arXiv:2511.09902 (replaced) [pdf, html, other]
-
Title: Autonomous-Flow-Based GenerationSubjects: Machine Learning (cs.LG); Classical Analysis and ODEs (math.CA); Dynamical Systems (math.DS); Numerical Analysis (math.NA); Machine Learning (stat.ML)
We show that using autonomous-flow-based generation, one can universally approximate orientation-preserving diffeomorphisms defined on the cube by Neural ODEs with rate $\mathcal{O}(P^{-1/d})$ with $P$ parameters. On the other hand, we show that by using only a single autonomous flow, the class of Neural ODEs is nowhere dense on the cube in dimension $d \ge 2$ . Under a compact-support$_\mathrm{id}$ condition on $(0,1)^d$, we show that using autonomous-flow-based generation, one can universally approximate compactly supported$_\mathrm{id}$ diffeomorphisms on $(0,1)^d$ for any dimension with rate $\mathcal{O}((\frac{P}{\log P})^{-2/d})$ with $P$ parameters and for compactly supported$_\mathrm{id}$ homeomorphisms on $(0,1)^d$ in dimension $d \geq 5$ with rate $\mathcal{O}(P^{-1/(d+1)})$ with $P$ parameters and by a composition of at most $I_d$ autonomous Neural ODEs with the same support$_\mathrm{id}$, where $I_d$ depends only on the dimension. Moreover, we show that the class of single autonomous flows compactly supported$_\mathrm{id}$ on $(0,1)^d$ is meagre in the space of compactly supported$_\mathrm{id}$ homeomorphisms on $(0,1)^d$ for $d\ge 2$. By linearly lifting the domain into one higher dimension, we obtain a universal approximation result for Lipschitz functions compactly supported on $(0,1)^d$ with rate $\mathcal{O}(P^{-1/(d+1)})$ with $P$ parameters.
- [45] arXiv:2603.27137 (replaced) [pdf, html, other]
-
Title: A Mean Field Games Perspective on Evolutionary ClusteringComments: Accepted for publication in Applied Mathematical ModellingSubjects: Numerical Analysis (math.NA); Machine Learning (stat.ML)
We propose a control-theoretic framework for evolutionary clustering based on quasi-stationary Mean Field Games. Each cluster is represented by a probability density whose evolution is governed by a Fokker--Planck equation, while the associated velocity field is determined through a stationary Hamilton--Jacobi equation. The general formulation does not prescribe a finite-dimensional statistical shape for the component densities, although the number of components is fixed. In the Gaussian specialization, we show that suitable affine dynamics reproduce the mean and covariance trajectories generated by the classical Expectation--Maximization procedure. To improve temporal coherence in the presence of noise and temporary cluster overlaps, we introduce causal and non-causal time-averaged log-likelihood objectives. We also develop a fully density-based numerical implementation for non-Gaussian components. The proposed formulations are assessed on synthetic and real time-dependent datasets and compared with independent snapshot Expectation--Maximization and with the same method applied to temporally smoothed observations. In the two-dimensional benchmark, an established evolutionary \(k\)-means method is also included as an external dynamic-clustering baseline.
- [46] arXiv:2604.13130 (replaced) [pdf, html, other]
-
Title: Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin UpdatesSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
We study learning to learn through the lens of hyperparameter tuning. We propose the Langevin Gradient Descent Algorithm (LGD), which approximates the mean of the posterior distribution defined by the loss function and regularizer of a regression task with convex objective. For classification tasks, the LGD algorithm estimates the posterior probabilities of each class on the test set. We prove the existence of an optimal hyperparameter configuration for which the LGD algorithm achieves the Bayes' optimal solution for squared loss on regression tasks, and for which LGD closely approximates the posterior probabilities for well-specified classification tasks. Subsequently, we study generalization guarantees on meta learning optimal hyperparameters for the LGD algorithm from a given set of tasks in the data-driven setting. For a number of parameters $d$ and hyperparameter dimension $h$, we show a pseudo-dimension bound of $O(dh)$, up to logarithmic terms under mild assumptions on LGD. This matches the dependence of the bounds on number of parameters obtained in prior work for linear regression using the elastic net, which only allows for $h=2$ hyperparameters, and extends their bounds to regression on convex loss. Compared to bounds on regularized logistic regression that allow for only $h=1$ hyperparameter, our bounds improve greatly on the dependence on samples per task at the cost of worse dependence on the number of parameters by accounting for hardware-aware procedures. Finally, we show empirical evidence of the success of LGD and the meta learning procedure for few-shot learning on linear and logistic regression using synthetically created datasets.
- [47] arXiv:2605.18530 (replaced) [pdf, html, other]
-
Title: Continuous Diffusion Scales Competitively with Discrete Diffusion for LanguageZhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John ThickstunSubjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML)
While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only $20\times$ compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of $22.1$ among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.
- [48] arXiv:2607.24041 (replaced) [pdf, html, other]
-
Title: The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized RegressionSubjects: Statistics Theory (math.ST); Machine Learning (cs.LG); Machine Learning (stat.ML)
Over-parameterized linear regression has been widely studied over the last decade. However, most existing works assume that the covariates are independent and that their covariance matrices are non-degenerate. In this paper, we relax both assumptions and derive deterministic equivalents for the prediction risk in a vanishing-ridge regime. We show that degeneracy of the covariance matrices and dependence can lead to multiple descent, and characterize where the corresponding peaks can occur. Our proofs use a novel graph representation of the variance profile. We show that maximum matchings and the Dulmage--Mendelsohn decomposition of the associated bipartite graph identify the configurations at which the variance becomes singular.
- [49] arXiv:2608.13060 (replaced) [pdf, html, other]
-
Title: VALG: An Agentic System for ML Theory ResearchSubjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
Machine learning theory studies learning procedures through mathematical setups in which the data model, training protocol, oracle access, loss, metric, and randomness define the phenomenon that a theorem is meant to explain. Solving an open problem therefore requires the problem formulation, theorem target, and proof mechanism to be developed in concert. Researchers formulate hypotheses, test them through preliminary theoretical or empirical analysis, and refine both assumptions and proofs. We investigate whether this process can be organized as an autonomous agentic workflow for ML theory research. We develop VALG, an agentic system that combines multi-level Verification, Adaptive formulation of Learning-theory problems, and Graph-structured proof development. Within each source-relative theorem branch, VALG maintains a fixed mathematical specification, checks the theorem-level composition of a typed proof-dependency graph, and constructs and reviews local proofs in dependency order. When a proof attempt fails, VALG identifies whether the obstruction lies in a derivation, the proof structure, or the theorem formulation and routes the next attempt accordingly. Formulation-level obstructions initiate an explicitly related variant or relaxation, preserving the mathematical relation between the resulting theorem and the source problem. We evaluate VALG on nine subproblems from five COLT 2026 open problems. Two runs produce internally finalized theorem candidates that match the scope of their source briefs; the remaining seven yield restricted-method results, special cases, or conditional theorems. These case studies show how VALG keeps source-scope matches, relaxations, conditional results, and blocked attempts mathematically distinct. VALG is open source at this https URL.
- [50] arXiv:2608.24007 (replaced) [pdf, html, other]
-
Title: Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPsSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has focused on the emergence of a global low-dimensional representation. We show that this picture is incomplete. In regression problems with clustered data, we demonstrate that multilayer perceptrons (MLPs) naturally develop monosemantic specialized neurons: individual neurons become strongly aligned with a specific predictive feature relevant to a particular region of the input space. Rather than learning a single global low-dimensional representation, MLPs learn a collection of local low-dimensional representations. We show that this ability to specialize gives MLPs a provable data-efficiency advantage over feature-learning methods based on a global low-dimensional representation.
