貝葉斯方法是非?；A(chǔ)且重要的方法，在前文中斷斷續(xù)續(xù)也有所介紹，感興趣的話可以自行移步閱讀即可：

《數(shù)學(xué)之美番外篇：平凡而又神奇的貝葉斯方法》

《貝葉斯深度學(xué)習(xí)——基于PyMC3的變分推理》

《模型優(yōu)化調(diào)參利器貝葉斯優(yōu)化bayesian-optimization實(shí)踐》

在《模型優(yōu)化調(diào)參利器貝葉斯優(yōu)化bayesian-optimization實(shí)踐》 一文中，我們基于bayesian-optimization庫來實(shí)現(xiàn)了貝葉斯優(yōu)化實(shí)踐，本文同樣是要應(yīng)用實(shí)踐貝葉斯優(yōu)化方法，只不過這里我們使用的是skopt模塊來完成對(duì)應(yīng)的實(shí)踐的。

對(duì)于目標(biāo)函數(shù)f:

noise_level = 0.1def f(x, noise_level=noise_level):return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2))\+ np.random.randn() * noise_level

可以先繪制f的邊界輪廓，如下；

x = np.linspace(-2, 2, 400).reshape(-1, 1)
fx = [f(x_i, noise_level=0.0) for x_i in x]
plt.plot(x, fx, "r--", label="True (unknown)")
plt.fill(np.concatenate([x, x[::-1]]),np.concatenate(([fx_i - 1.9600 * noise_level for fx_i in fx],[fx_i + 1.9600 * noise_level for fx_i in fx[::-1]])),alpha=.45, fc="g", ec="None")
plt.legend()
plt.title("Function Contours")
plt.show()

結(jié)果如下所示：

貝葉斯優(yōu)化是建立在高斯過程之上的，如果每個(gè)函數(shù)評(píng)估都很昂貴，例如，當(dāng)參數(shù)是神經(jīng)網(wǎng)絡(luò)的超參數(shù)且函數(shù)評(píng)估是十倍的平均交叉驗(yàn)證分?jǐn)?shù)時(shí)，則使用標(biāo)準(zhǔn)優(yōu)化例程優(yōu)化超參數(shù)將永遠(yuǎn)花費(fèi)！
其思想是使用高斯過程來近似函數(shù)。換句話說，假定函數(shù)值遵循多元高斯分布。函數(shù)值的協(xié)方差由參數(shù)之間的GP核給出。然后，利用捕獲函數(shù)在高斯先驗(yàn)下選擇下一個(gè)待評(píng)估參數(shù)，使得評(píng)估速度更快。

from skopt import gp_minimizeres = gp_minimize(f,                  # the function to minimize[(-2.0, 2.0)],      # the bounds on each dimension of xacq_func="EI",      # the acquisition functionn_calls=15,         # the number of evaluations of fn_random_starts=5,  # the number of random initialization pointsnoise=0.1**2,       # the noise level (optional)random_state=1234)   # the random seed

計(jì)算過程輸出如下所示：

    fun: -1.0079192525206238func_vals: array([ 0.03716044,  0.00673852,  0.63515442, -0.16042062,  0.10695907,-0.24436726, -0.58630532,  0.05238726, -1.00791925, -0.98466748,-0.86259916,  0.18102445, -0.10782771,  0.00815673, -0.79756401])models: [GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775)]random_state: RandomState(MT19937) at 0x1BC23E3DDB0space: Space([Real(low=-2.0, high=2.0, prior='uniform', transform='normalize')])specs: {'args': {'model_queue_size': None, 'n_jobs': 1, 'kappa': 1.96, 'xi': 0.01, 'n_restarts_optimizer': 5, 'n_points': 10000, 'callback': None, 'verbose': False, 'random_state': RandomState(MT19937) at 0x1BC23E3DDB0, 'y0': None, 'x0': None, 'acq_optimizer': 'auto', 'acq_func': 'EI', 'initial_point_generator': 'random', 'n_initial_points': 10, 'n_random_starts': 5, 'n_calls': 15, 'base_estimator': GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5),n_restarts_optimizer=2, noise=0.010000000000000002,normalize_y=True, random_state=822569775), 'dimensions': Space([Real(low=-2.0, high=2.0, prior='uniform', transform='normalize')]), 'func': <function f at 0x000001BBC7401E18>}, 'function': 'base_minimize'}x: [-0.3551841563751006]x_iters: [[-0.009345334109402526], [1.2713537644662787], [0.4484475787090836], [1.0854396754496047], [1.4426790855107496], [0.9579248468740365], [-0.4515808656827538], [-0.6859481043867404], [-0.3551841563751006], [-0.29315378760492994], [-0.3209941584981484], [-2.0], [2.0], [-1.3373741960111043], [-0.24784229191660678]]

同樣可以對(duì)收斂的過程進(jìn)行可視化：

from skopt.plots import plot_convergence
plot_convergence(res)

結(jié)果如下所示：

接下來可以進(jìn)一步檢查可視化：1、擬合gp模型到原始函數(shù)的近似? 2、確定下一個(gè)要查詢點(diǎn)的采集值

接下來繪制5個(gè)隨機(jī)點(diǎn)下的五個(gè)迭代：

def f_wo_noise(x):return f(x, noise_level=0)for n_iter in range(5):# Plot true function.plt.subplot(5, 2, 2*n_iter+1)if n_iter == 0:show_legend = Trueelse:show_legend = Falseax = plot_gaussian_process(res, n_calls=n_iter,objective=f_wo_noise,noise_level=noise_level,show_legend=show_legend, show_title=False,show_next_point=False, show_acq_func=False)ax.set_ylabel("")ax.set_xlabel("")# Plot EI(x)plt.subplot(5, 2, 2*n_iter+2)ax = plot_gaussian_process(res, n_calls=n_iter,show_legend=show_legend, show_title=False,show_mu=False, show_acq_func=True,show_observations=False,show_next_point=True)ax.set_ylabel("")ax.set_xlabel("")plt.show()

可視化結(jié)果如下所示：

第一列表示：1、真實(shí)的的函數(shù)、高斯過程模型對(duì)原函數(shù)的逼近、GP逼近的確定程度。
第二列顯示每個(gè)代理模型擬合后的采集函數(shù)值。我們可能不選擇全局最小值，而是根據(jù)用于最小化捕獲函數(shù)的最小值選擇局部最小值。在更接近之前在處計(jì)算的點(diǎn)處，方差下降為零。最后，隨著點(diǎn)數(shù)的增加，GP模型更接近實(shí)際函數(shù)。最后幾個(gè)點(diǎn)聚集在最小值附近，因?yàn)镚P無法通過進(jìn)一步探索獲得更多信息：