Multiple Blockchain Consensus Performance Evaluation

NANCY project has received funding from the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme under Grant Agreement No 101096456. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of theEuropean Union or the SNS JU. Neither the European Union nor the granting authority can be held responsible for them. Environment setup The data creation process includes a combination of internally generated experimental data and simulated workloads designed to emulate real-world blockchain environments. The primary focus was to analyse the latency and throughput characteristics of various blockchain systems and consensus protocols, specifically the ones mentioned in Table1: Table 1: Systems Under Test (SUTs) Platform Version Consensus Protocol Hyperledger Fabric v2.5 Raft (CFT) Hyperledger Fabric v3.0 SBFT Hyperledger Besu v25.0 QBFT, IBFT2.0 These decentralised networks were deployed in a controlled environment that allowed to monitor, record, and extract key performance metrics under different configurations and workloads. The data collection focused on network latency, block propagation times, transaction finality and overall system responsiveness under various stress levels. The experiment was conducted on blockchain solutions that are appropriate for industrial use cases. The tool that was used for conducting the tests was Hyperledger Caliper [7], an open-source benchmarking tool designed to evaluate the performance of blockchain networks. It allows users to run predefined or custom test workloads on various blockchain platforms and generates detailed reports on key performance metrics such as transaction latency, throughput (TPS), and success/failure rates. The characteristics of the workstation that was used for the benchmarking process is described in Table 2. Table 2: Workstation specifications Component Specification Operating System Ubuntu 24.04.2 LTS Motherboard Gigabyte Z790 AORUS ELITE AX CPU Intel® Core™ i7-14700KF (20-core: 8 P-cores + 12 E-cores, up to 5.5 GHz) Memory 64 GiB DDR5 RAM (2×32 GiB @ 5600 MHz) Benchmark configuration Figure1 is an example of a benchmark configuration file that was used for the tests. Figure 1: Benchmark config file Each key parameter corresponds to a specific value parameter, with the key - value pairs explained Table 3. Table 3: Benchmark configuration explanation Parameter Description test.workers Configuration for how transaction load is generated test.workers.number Specifies the number of worker processes to use for executing the workload test.rounds Array of objects, each describing the settings of a round test.rounds[i].label A short name of the rounds, usually corresponding to the types of submitted TX test.rounds[i].txDuration The length of the round in seconds during which Caliper will submit TXs test.rounds[i].rateControl The object describing the rate controller to use for the round test.rounds[].rateControl.type The rate control strategy. fixed-rate means transactions are sent at a steady, constant rate. test.rounds[].rateControl.opts.tps TPS to send. Here, Caliper will attempt to submit 100 transactions per second throughout the 60- second test test.rounds[i].workload The object describing the workload module used for the round test.rounds[i].workload.module The path to the benchmark workload module implementation that will construct the TXs to submit A configuration with the above structure will define a benchmark run that consists of multiple rounds. Each round is associated with a rate controller that is responsible for the scheduling of transactions (TXs) and a workload module that will generate the actual content of the scheduled TXs. The benchmarking included various transaction rates, ranging from 5 to 452 TPS in fixed steps (Table 4). Each test round lasted 500 transactions, with 1 to 10 local worker threads generating load. The same application workload was used across all tests. The standard FabCar smart contract was used, simulating realistic asset creation scenarios. The controller of choice was fixed rate to simulate a stable transaction load and to mimic a high but stable demand of transaction executions. The fixed-rate controller sends transactions at a constant, pre-defined rate (i.e., a fixed number of TPS). Table 4: Benchmark round parameters Parameter Description Benchmarking tool Hyperledger Calieper v0.6 Transaction Rates (TPS) 5 to 452 (in fixed steps of 3) Test duration 500 transactions per round Load Generation 1-10 local workers Workload createCar and queryCar transaction for the FabCar smart contrac Network configuration Varying number of orderers/validators depending on protocol Workload generation setup A sample of the workload module used for the testing is provided in Figure 2 Figure 2: Workload module Workload modules are implemented as Node.JS [8] modules that expose a certain Application Programming Interface (API). There are no further restrictions on the implementation, allowing developers to implement arbitrary logic (using further arbitrary components). These modules are loaded through factory functions, just like other pluggable modules in Hyperledger Caliper. Accordingly, a workload module implementation must export a single factory function, named “createWorkloadModule”. The factory function must return an instance that implements the [WorkloadModuleInterface]. The “submitTransaction” function is the backbone of the workload generation. The worker process calls this function every time the rate controller enables the next TX and is its responsibility to submit the TX through the connector API. Key Performance Metrics Hyperledger Caliper provides a detailed set of performance metrics after each benchmarking test, helping you evaluate how well a blockchain system performs under load. Table 8 summarizes the key metrics Caliper returns and their corresponding descriptions. Table 8: Network performance metrics Metric Description Success Number of transactions that were successfully submitted and confirmed on the blockchai Failed Number of transactions that were submitted but failed due to errors like timeouts, endorsement issues, or network problem Send Rate (TPS) The average number of transactions sent per second during the test. Throughput (TPS) The number of successful transactions per second (i.e., how many were confirmed on-chain). Latency (min/avg/max) Time taken for a transaction to go from submission to confirmation. Caliper reports the minimum, average, and maximum latency values. References: [1] “Hyperledger Fabric.“ [Online] [2] “The Raft Consensus Algorithm.” [Online] [3] Y. Manevich, Y. Tock, and H. Meir, “Hyperledger Fabric v3: Delivering Smart Byzantine Fault Tolerant consensus” Sep. 2024. [Online] [4] “Besu” [Online] [5] “Configure QBFT consensus” Apr. 2025. [Online] [6] “Configure IBFT 2.0 consensus” Apr 2025. [Online] [7] “Caliper” [Online] [8] “Node.js - Run JavaScript Everywhere” [Online]