What is Bristlecone?

Bristlecone provides tools for testing database performance. There are two main tools included in the package.

  • Evaluator generates mixed loads of inserts, updates, deletes, and selects. Output can be built-in graphics, HTML, XML, or CSV.
  • Benchmark runs performance test cases with systematically varying parameters. Output can be HTML or CSV.

The Bristlecone tools were designed for comparative evaluations of database clusters. They include built-in support for threading, ability to generate new tests quickly with simple configuration file changes, and the ability to do systematic tests across different database implementations.

Bristlecone is written in Java. It has mostly run on Linux but is easily adaptable to other platforms. Bristlecone has been tested against the following database types: PostgreSQL, MySQL, and HSQLDB (Hypersonic). We have also run it extensively against Tungsten clusters as well as  Continuent uni/cluster.

Portable Scale-Out Benchmarks

Scale-out is an architectural approach for database systems that increases availability and performance by spreading multiple copies of data across hosts connected by a network. Such clusters can be built using native database replication or middleware replication such as Sequoia provides. More ambitious designs include pluggable replication as implemented by the GORDA Project.

Bristlecone comes with scenarios that constitute a "portable scale-out benchmark" to compare performance of database clusters that use different scale-out approaches. Scale-out architectures vary in their capabilities with respect to the following:

  • Read scaling -- throughput, response time, and scaling
  • Write scaling -- throughput, response time, and scaling
  • Latency -- Delay on reads and writes induced by proxies; latency between replicated databases
  • Transactions -- throughput and number of aborts or deadlocks under different transaction models

The behavior of particular architectures and machine configurations can vary greatly as you change things like the number of clients, number of hosts, width and length of data selected, data types, number of operations per transactions, or as you switch between full transactions and auto-commit operations.