Research

Pre-Trained LLMs for Personalization Applications

Another captivating application of Large Language Models (LLMs) lies in harnessing the vast knowledge they've accumulated for content matching tasks like recommendations, personalization, nudging, and more. In this context, we view LLMs as pre-trained networks providing relevant embeddings for the content items, which becomes especially useful in tackling cross-modal tasks that are well-known to be challenging due to the vastly different natures and representations of the inputs. While traditional approaches typically necessitate individual systems for each modality, an innovative alternative emerges: obtaining a text description for each input and employing an LLM to embed these standardized representations into a shared latent space, where matching and comparison takes place.

For a demonstration of this idea in more detail, check out the paper Zero-Shot Recommendations with Pre-Trained Large Language Models for Multimodal Nudging, where we explore how this concept can be used for content matching across diverse domains.

Policy Entropy and Action Selection

In practical applications, the focus often extends beyond the agent's capacity to learn, and encompasses the behavior it exhibits during the learning process. This becomes critical in contexts featuring a multitude of available actions, as the challenge of exploring the state--action space becomes unfeasible. Moreover, when combined with practical challenges, such as non-stationarity of the action space or non-determinicity of the feedback, this issue poses a significant obstacle, particularly in the domains of recommendation in the fields of healthcare, nudging, and behavioral change, where the amount of agent--user interactions are fundamentally limited.

Remarkably, one of the primary factors shaping the agent's behavior is the choice of the learning algorithm. Specifically, Policy Optimization and Q-Learning yield fundamentally distinct behaviors, even though both ultimately converge to the same optimal policy. This intriguing and somewhat unexpected finding appears to remain somewhat obscured within the broader research community. This ongoing investigation delves deeper into this phenomenon; further insights can be found in the paper Examining Policy Entropy of Reinforcement Learning Agents for Personalization Tasks.

Reinforcement Learning for Hyperpersonalization

Reinforcement learning for hyperpersonalization is an emerging area becoming prominent in such critical applications as individualized healthcare and behavioral intervention where the conventional methods for personalization (e.g. A/B testing, collaborative filtering, recommender system) are not sufficient due their inability to learn the latent features and appropriately differentiate the input data. While there exist approaches that are better suited to handle hyperpersonalization tasks, they are typically benchmarked on simple simulated environments and do not suggest generalization guarantees to real-life applications. To complicate the matter further, applications dealing with human feedback are notoriously messy since human interactions are restricted and the behavioral feedback is non-deterministic by nature, which drastically complicates the feasibility of assessing the agent's performance.

To address such settings, I designed synthetic environments that showcase the underlying complexity of hyperpersonalization tasks and develop novel approaches that can recognize and overcome these challenges. This is an actively developing direction and you can read about it in the paper On the Unreasonable Efficiency of State Space Clustering in Personalization Tasks or check out my Reinforcement Learning Personalization Challenge.

Policy Evaluation and Ranking

In real-life reinforcement learning one often has to estimate the agent's performance to assess the state of the training process. This is usually achieved through one of the policy evaluation methods—techniques that utilize the logged agent-environment interactions to estimate the value of the policy. There has been a lot of development in the policy evaluation field recently with a variety of methods that take into account the distribution of the historical data, the structure of the reward signal, and the available information about the environment. Despite these advances, the existing methods require large amounts of data to provide a reasonable estimate, which renders them unreliable in critical real-life applications where the agent-environment interactions are limited due to either ethical, practical, or security considerations. In such cases, one may instead utilize a policy ranking approach---a technique that orders a set of agents with respect to the values of their policies. While typically being simpler than the evaluation, policy ranking is still a nontrivial task that requires careful consideration.

I've designed a policy ranking method that is intended to work in situations where the historical data is restricted in terms of both the volume and distribution. Read more about it in the paper Offline Policy Comparison under Limited Historical Agent-Environment Interactions. 

Neural Architecture Design and Approximation Theory

The choice of architecture for a neural network is nontrivial and typically relies on a set of heuristic rules empirically established by the early practitioners. However, in novel real-world applications, the existing conventional rules, established for the determined benchmark datasets, don't always apply. This often leaves the user in the undesirable position of relying on a trial-and-error approach to find an appropriate architecture. Such an approach is not only unfeasible in critical applications but is also mathematically unsatisfactory to any sensible scientist. After all, the architecture indicates the parameterization of the learning target and thus the architecture selection can be approached from the perspective of approximation theory.

This is how I was introduced to deep learning, and I have since established several promising connections between these fields, including a technique for constructing efficient and compact shallow networks based on the provided training data. For more detail, refer to my papers Neural Network Integral Representations with the ReLU Activation Function and Greedy Shallow Networks: An Approach for Constructing and Training Neural Networks and check out my Shallow Network Approximation Challenge.

Smoothing-Based Optimization

Optimization is an essential and intricate component of the training of a neural network. Even though there are lots of heuristics for the choice of an optimization algorithm and its hyperparameters, the training process is far from straightforward. Most notably, the loss minimization problem is typically plagued by the nonconvexity of the loss landscape, which often causes the optimizer to get stuck in a local minimum. If the attained error value is not satisfactory, the conventional approach is to fine-tune the hyperparameters and repeat the whole minimization process, which is a tedious process with no guaranteed outcome. An alternative approach is to employ an optimization technique that can, to some extent, alleviate the challenge of nonconvexity. I'm really excited about the idea of smoothing-based optimization where the minimization is performed on the smoothed surrogate rather than on the original target function. As a result, the optimizer can navigate the smoothed loss landscape more reliably and converge faster, both of which are vital advantages in practical applications.

I have designed a blackbox smoothing-based optimizer that demonstrates outstanding performance on a wide range of nonconvex functions. This direction is in active development in terms of both theoretical analysis and practical implementation. Find more information in the papers An Adaptive Stochastic Gradient-Free Approach for High-Dimensional Blackbox Optimization and Gaussian Smoothing Gradient Descent for Minimizing High-Dimensional Non-Convex Functions.

Quasi-Random Low-Discrepancy Sampling

Quasi-random sampling is an essential component of many stochastic computational methods---from the simplicity of numerical integration to the complexity of behavior of a reinforcement learning agent---with Monte Carlo sampling being the most conventional and widely-employed technique. However, in critical real-life applications where the sampling space is high-dimensional and the user is limited in the amount of samples that can be drawn, the distribution of the Monte Carlo points might not be uniform enough, which, in turn, would distort the predicted outcome. In such situations, one would want to utilize a technique that provides samples with low discrepancy, thus guaranteeing appropriate distribution over the sampling space. Despite a seemingly simple and flexible formulation, generating quasi-random low-discrepancy samples is an extremely challenging problem in a general case. For instance, a particular version of this problem---picking uniformly distributed points on a 2-sphere---is closely related to Smale's seventh problem and is still unresolved. While this direction is currently on hold due to shifted priorities, I'm still quite interested in the problem. See the project repository for our progress so far.

Greedy Algorithms for Reduced Order Modeling

Reduced order modeling is an umbrella term for reducing the computational complexity of mathematical models in numerical simulations. It is crucial in such applications as aero- and hydrodynamics, structural mechanics, and design optimization. Examples of reduced order modeling methods include proper orthogonal decomposition, nonlinear manifold algorithms, and reduced basis methods, with the latter being of most interest to me. Reduced basis methods iteratively construct a sequence of nested subspaces that approximate a given manifold, with the most prominent methods being the Empirical Interpolation Method and the Orthogonal Greedy Algorithm.

During my brief stay at the Oden Institute, I understood the engineering need for more flexible reduced basis methods. To address this challenge I have designed and analysed the Natural Greedy Algorithm---a reduced basis method that iteratively constructs an approximating subspace to the given manifold with respect to any chosen norm by utilizing the norming functionals of the space. In addition to being computationally simple, it turns out that the NGA coincides with the OGA in the case of 2-norm and with the EIM in the case of infinity-norm, and in general acts as a generalization between these two methods, depending on the selected norm. Read more in my paper The Natural Greedy Algorithm for Reduced Bases in Banach Spaces.

Greedy Algorithms with Adjustable Computational Complexity

While greedy algorithms are widely used in a variety of applications, the choice of which algorithm to employ often boils down to the available computational budget. Specifically, one of two algorithms are most often used: the Pure Greedy Algorithm (also known as the Matching Pursuit) or the Orthogonal Greedy Algorithm (also known as the Orthogonal Matching Pursuit). The PGA is a computationally simple iterative approach that on each iteration calculates a projection onto one-dimensional space, while the OGA projects onto a space of ever-increasing dimensionality, thus becoming more computationally expensive with each iteration. Due to the practical demand, it is of interest to develop a class of greedy algorithms that adapts to the available computational budget of a particular setting and thus provides a freedom of choice to the practitioner. This is an ongoing project with my collaborators from the High-Dimensional Approximation and Applications Laboratory and Lomonosov Moscow State University.

Greedy Algorithms with Imprecise Calculations

In practice, realization of many numerical algorithms is plagued with numerical instability due to their dependence on precise calculations. A conventional workaround is to employ algorithms that are robust to various forms of computational errors. I've researched this direction in the context of greedy algorithms with a particular focus on the Chebyshev Greedy Algorithm---a generalization of the Orthogonal Greedy Algorithm to the Banach space setting---in the areas of sparse approximation and convex optimization. This topic was the main direction of my PhD research and I published 3 papers on it: Biorthogonal Greedy Algorithms in Convex Optimization, On the Generalized Approximate Weak Chebyshev Greedy Algorithm, and On the Approximate Weak Chebyshev Greedy Algorithm in Uniformly Smooth Banach Spaces.