Unlock the Magic of Bayesian Optimization: 6 Easy Steps to Boost Efficiency and Achieve Amazing Results with Your Black-Box Functions

Bayesian optimization is a powerful technique used for optimizing black-box functions, especially when function evaluations are expensive or have noise. It works well when the search space is continuous, and the function is expensive to evaluate. In this tutorial, we will introduce Bayesian optimization, discuss its components, and provide a step-by-step example.

Bayesian Optimization

Overview of Bayesian Optimization

Bayesian optimization works by maintaining a probabilistic model of the unknown function and updating the model using the observations from the evaluations of the function. It uses this model to determine the next point to evaluate based on a balance between exploration (searching areas of the search space with high uncertainty) and exploitation (searching areas with high predicted values).

Components of Bayesian Optimization:

There are two main components of Bayesian optimization:

1. Surrogate model:

This model approximates the unknown function. Gaussian Process (GP) regression is commonly used as a surrogate model in Bayesian optimization because it provides a distribution over functions with a simple closed-form update rule.

2. Acquisition function:

This function helps to decide the next point to query in the search space. The acquisition function balances exploration and exploitation. Popular acquisition functions include Expected Improvement (EI), Probability of Improvement (PI), and Upper Confidence Bound (UCB).

Step-by-Step Example

For this example, let’s assume we have an expensive black-box function f(x) that we want to optimize, and the search space is bounded by the interval [-2, 2]. We will use a Gaussian Process as our surrogate model and Expected Improvement as our acquisition function.

Step 1: Set up the problem and import necessary libraries

import numpy as np
import matplotlib.pyplot as plt
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, ConstantKernel

Step 2: Define the objective function (unknown to the algorithm)

def f(x):
    return -1 * np.exp(-0.5 * x**2) * np.sin(5 * x)

Step 3: Define the acquisition function (Expected Improvement)

def expected_improvement(X, model, current_best):
    mean, std = model.predict(X, return_std=True)
    std = np.clip(std, 1e-9, None)  # Avoid division by zero
    Z = (mean - current_best) / std
    return std * (Z * norm.cdf(Z) + norm.pdf(Z))

Step 4: Initialize the Gaussian Process and sample initial points

kernel = ConstantKernel(1.0) * RBF(length_scale=1.0)
gp = GaussianProcessRegressor(kernel=kernel, alpha=1e-6, normalize_y=True)

n_init = 3
X_init = np.random.uniform(-2, 2, n_init).reshape(-1, 1)
Y_init = f(X_init)

Step 5: Optimize the function using Bayesian Optimization

n_iterations = 20

X = X_init
Y = Y_init
current_best = np.min(Y)

for _ in range(n_iterations):
    gp.fit(X, Y)
    x_new = np.argmax(expected_improvement(np.arange(-2, 2, 0.01).reshape(-1, 1), gp, current_best))
    x_new = x_new * 0.01 - 2
    y_new = f(x_new)
    X = np.vstack([X, [x_new]])
    Y = np.vstack([Y, [y_new]])
    current_best = np.min(Y)

optimized_x = X[np.argmin(Y)]

Now, optimized_x contains the x-value that optimizes the objective function according to the Bayesian optimization algorithm. You can visualize the optimization process as follows.

Step 6: Visualize the Bayesian Optimization Process

def plot_gp(gp, X, Y, X_new=None, Y_new=None, true_function=None):
    x_plot = np.linspace(-2, 2, 1000).reshape(-1, 1)
    mean, std = gp.predict(x_plot, return_std=True)
    plt.figure(figsize=(12, 6))
    if true_function:
        plt.plot(x_plot, true_function(x_plot), label='True function', linestyle='--', color='black')
    plt.plot(X, Y, 'kx', markersize=10, label='Observations')
    plt.plot(x_plot, mean, label='GP mean', color='C0')
    plt.fill_between(x_plot.ravel(), mean - 1.96 * std, mean + 1.96 * std, alpha=0.2, color='C0')
    if X_new is not None and Y_new is not None:
        plt.plot(X_new, Y_new, 'ro', markersize=10, label='New observation')

# Example visualization after the optimization
plot_gp(gp, X, Y, true_function=f)

In this example, you will see a plot with the true function in a dashed line, the GP mean in blue, the uncertainty as a shaded region, and the observations as black crosses. After the optimization process is completed, you should see that the Bayesian optimization has found the minimum of the function. The more iterations you perform, the more accurate the estimation will be. However, keep in mind that each iteration requires the evaluation of the expensive black-box function.

This tutorial provides a basic understanding of Bayesian optimization and a simple example. In practice, you may need to adjust the Gaussian Process kernel, the acquisition function, and the optimization algorithm for the acquisition function to suit your specific problem.


Hello! I'm a Quantum Computing Scientist based in Silicon Valley with a strong background in software engineering. My blog is dedicated to sharing the tools and trends I come across in my research and development work, as well as fun everyday anecdotes.

Articles: 22

Leave a Reply