Benchmarking Overview
In this document, we explain the different benchmarks created to measure the performance of various components of our system. Each section outlines the functionality being benchmarked, why it is important, and what we aim to achieve through the benchmarking process.
Neural Network Training
The Neural Network Training benchmark measures how efficiently the neural network can be trained on a set of input data over multiple epochs. Training a neural network involves updating its internal weights to minimize the error in predictions. The training process is resource-intensive, particularly with larger datasets and deeper networks, so it's important to monitor and optimize its performance.
Purpose:
- To evaluate the time and resource usage during the training process.
- To identify potential bottlenecks when scaling the neural network for larger datasets.
- To ensure that the training loop can handle different configurations efficiently.
Neural Network Prediction
The Neural Network Prediction benchmark tests the speed and efficiency of the network when making predictions. Prediction, or inference, is the process of using a trained neural network to generate an output from a given input. In production environments, particularly in real-time systems, prediction speed is crucial.
Purpose:
-
To evaluate the performance of the neural network's inference.
-
To ensure that the system can make fast predictions with minimal latency.
-
To monitor how well the network scales when processing inputs of varying sizes.
DAG Vertex Addition
This benchmark measures the performance of adding vertices to the Directed Acyclic Graph (DAG) structure. The DAG is essential in managing and organizing transactions in our system, and vertex addition plays a critical role in expanding the graph as new transactions are processed.
Purpose:
- To test the efficiency of vertex insertion within the DAG.
- To monitor how well the system handles multiple vertex additions in parallel.
- To optimize the process to avoid bottlenecks in transaction processing.
DAG Iterative Topological Sort
The Topological Sort benchmark tests the efficiency of sorting the vertices in the DAG based on their dependencies. This is a key operation in many graph-based systems, especially when organizing tasks or transactions that need to follow a specific order.
Purpose:
- To measure how quickly the system can perform topological sorting on a growing set of vertices.
- To identify potential bottlenecks when the graph grows large.
- To ensure that the system can process dependent transactions in the correct order.
DAG Neural Network Prediction
This benchmark measures the time it takes to make predictions using a neural network in conjunction with the DAG. It evaluates how efficiently the system can utilize the neural network to perform predictions based on data associated with the DAG's vertices.
Purpose:
- To test the integration of neural networks with the DAG.
- To measure the efficiency of making predictions on data tied to the DAG structure.
- To optimize the flow between graph processing and neural network inference.
Virtual Machine Contract Execution
This benchmark measures the performance of executing smart contracts in the virtual machine (VM). The virtual machine plays a crucial role in running contracts deployed on the platform, and the speed at which these contracts are executed impacts the overall system's responsiveness.
Purpose:
- To evaluate the efficiency of executing different types of smart contracts.
- To ensure that the virtual machine can handle contract execution under varying loads.
- To optimize the VM for faster execution times, especially for contracts that involve complex logic or multiple dependencies.
