Symmetry and Other Structures: Topics in Nonparametric Regression

In regression we measure covariate samples X_i and response variables Y_i and seek to estimate the conditional mean, or regression function, f(x) = E(Y_i ∣ X_i = x) within some hypothesis class F of possible regression functions. Estimation of a nonparametric regression function f ∶ [0, 1]^d → R becomes increasingly difficult as the dimension of the covariate space increases. Specifically, the number of samples n required to ensure that sup_{f∈F} E(∥f_n - f∥2) ≤ η for a fixed η > 0 grows exponentially in d for all estimators fˆ_n in many usual problem contexts. To overcome this, we often seek to find structure within the function f we can exploit. The two main structures that have been previously studied are Sparsity, where f depends only on s < d of the covariates’ coordinates, and Additivity, where f is a sum of functions on each of the covariates’ coordinates independently. In this thesis we present a new framework for lower dimensional structures that can capture much of the same benefits but now in non-linear situations. This framework exploits the structure of the level sets of the regression function. If we know the level set [x]_f = {u ∈ [0, 1]^d ∶ f(x) = f(u)} then we can average any sufficiently local estimator fn over this level set to obtain an estimator with a pointwise risk that is bounded in terms of d − dim[x]_f rather than d. In general it is hard to learn [x]_f ; however, even a simple plug in estimator (i.e., that uses the level sets of some estimate pn of f) can be shown to achieve these faster rates (up to log factors) under some strong conditions on f and demonstrate remarkable performance improvements in some finite sample experiments. Instead of placing conditions on f, this thesis gives sufficient conditions on structured hypothe- sis classes for the level sets [x]_f under which we can achieve the same fast rates of convergence. This key idea is to generalise sparsity, multi-index models, and other non-linear lower dimensional patterns in f in terms of transformations of the covariate space: Ψ_f ⊆ {Bijections ψ∶X→X} (1) for which f ○ ψ = f for all ψ ∈ Ψ_f . Functions that satisfy this property have Symmetry: in the same way that we can translate a sparse function along the unimportant axes we can transform f using Ψ_f without effect. Any such f must have level sets [x]_f that contain the sets {ψ(x) ∶ ψ ∈ Ψ_f }, which we can seek to exploit. We show that an oracle that knows any symmetry group Ψ_f of f performs with worst case L_2 risk decay that depends on a smaller dimension s ≤ d, rather than the dimension d of the original covariate space. The regression function f can have many possible symmetry groups, but when f is continuous there is always a unique maximal symmetry group Ψ_f within the classes considered. We then show that we can use an M-estimator of this maximal Ψ_f in order to achieve the dimension reduction adaptively under some technical conditions on Ψ_f and the hypothesis class of symmetries considered. We then show that we can give an inferential framework for this maximal Ψ, giving both hypothesis tests for: H_0(Ψ)∶f○ψ=f, for all ψ∈Ψ for any transformation group Ψ, as well as confidence regions for the maximal Ψ in the hypothesis class of possible Ψ we consider. This also leads to another estimator of Ψ that has the computational advantage of not estimating f directly. We demonstrate the performance of these tools both on synthetic datasets and on real data from both the Earth’s magnetic field and temporospatial locations of sunspots.

Name dc:type.qualificationname

Doctor of Philosophy (PhD)

Level dc:type.qualificationlevel

Doctoral

Grantor dc:publisher.institution

University of Cambridge

Year dc:date.issued

2024

Author dc:creator

• Christie, Louis

Advisor dc:contributor.advisor

• Aston, John

• Nonparametric Regression
• Invariant Models
• Dimension Reduction

Licence

• https://apollo8-f-pro.lib.cam.ac.uk/bitstreams/d8d3ff0e-4d41-4468-b6c6-f956edc7f28f/download
• http://purl.org/NET/rdflicense/allrightsreserved

Language dc:language

eng

DOI dc:identifier.doi

https://doi.org/10.17863/CAM.115448

OAI identifier oai:identifier

oai:www.repository.cam.ac.uk:1810/379315