Problem Statement

The challenge for the automated feedback-driven workbench controller is to design a set of experiments to obtain accurate peak rates for a set of test points, and in particular for test points selected to approximate a response surface efficiently.

Response surface mapping is expensive. Algorithm 1 presents the overall benchmarking approach that is used by the workbench controller to map a response surface, and Table 2 summarizes some relevant notation. The overall approach consists of an outer loop that iterates over selected samples from $\langle F_1, \ldots, F_n \rangle$, where $F_1, \ldots, F_n$ is a subset of factors in the larger $\langle \vec{W}, \vec{R}, \vec{C}\rangle$ space (Step 2). The inner loop (Step 3) finds the peak rate $\lambda ^*$ for each sample by generating a series of test loads for the sample. For each test load $\lambda$, the controller must choose the runlength $r$ or observation interval, and the number of independent trials $t$ to obtain a response time measure under load $\lambda$.

The goal of the automated feedback-driven controller is to address the following problems.

1. Find Peak Rate (§3). For a given sample from the outer loop of Algorithm 1, minimize the benchmarking cost for finding the peak rate $\lambda ^*$ subject to a target confidence level $c$ and target accuracy $a$ (defined below). Determining the NFSOPS rating of an NFS filer is one instance of this problem.

2. Map Response Surface (§4). Minimize the total benchmarking cost to map a response surface for all $\langle F_1, \ldots, F_n \rangle$ samples in the outer loop of Algorithm 1.

Minimizing benchmarking cost involves choosing values carefully for the runlength $r$, the number of trials $t$, and test loads $\lambda$ so that the controller converges quickly to the peak rate. Sections 3 and 4 present algorithms that the controller uses to address these problems.