Binary search results

Results of the binary search:

full_merged_dataset.csv is the dataset of the monitored data of all requests that were run. Each experiment is identified by the subject prompt, the number of sent requests per second, and the time at which the experiment was run. The source_file column includes this information and denotates the experiment to which each request belongs to.

experiment_stats.csv has the previous dataset results aggregated and condensed per experiment executed.

One may notice that Gemma-2-9b experiments can be seen in the data. This is because we have run experiments using Gemma, but the results were not satisfactory enough to be processed within our time constraints.

Experiment extended description:

To validate the effectiveness of binary search as a strategy for identifying the operational limits of the system, we first conducted an auxiliary experiment to determine a suitable initial upper bound for the search. The objective was to identify a load level likely to exceed the system's operational limits, while remaining close enough to ensure efficient convergence of the search. This initial estimate serves as a required input parameter for the binary search algorithm.

To establish this bound, we selected a simple prompt—specifically, "What is the capital of France?" designed to produce minimal input, output, and reasoning token counts, as well as low perplexity, without being trivial. This choice was based on the assumption that prompts with greater token complexity and perplexity are more resource-intensive, and thus lead to lower sustainable request rates under fixed quality constraints. We then gradually increased the request rate linearly using only this prompt until the system began to violate the predefined performance thresholds. The request rate at which this occurred, 10 req/s, became the upper bound of the subsequent binary search procedure.

The binary search algorithm starts by setting the lower bound (n=0), the upper bound (N=10) and the halt threshold (t=0.5). The experiment starts with r=(N+n)/2 requests per second being sent, which would initially be r=5. After the iteration ends, a statistical evaluation of significant changes is applied to the first and second half of the iteration. The evaluation will determine if the iteration has been sustainable or not. If it was sustainable, then we need to increase the number of requests per second, otherwise, we need to reduce it. To do so, we updates the values of n=r (if sustainable) or N=r (if non-sustainable), and repeat with these newfound values. The objective is to find the greatest possible sustainable r. t determines the end condition of the experiment; as n and N update to seek the greatest r, if N-n ≤ t then the experiment stops, assigning r=n if the last iteration was non-sustainable, and r=N otherwise. If at this point r=n=0 or r=N=10 (i.e. there has been no sustainable iteration or all iterations have been sustainable), then the experiment is resumed for t=0.1; if this happens again, continue after adjusting the value of t=t/3. 

It is possible to know beforehand how many iterations an experiment will have by following a formula (with the exceptions for when all/no iterations are sustainable):

#iterations=Ceil{log₂[(N-n)/t)]}

For example, for the default values of N=10, n=0 and t=0.5, the number of iterations will be 5.