Overview of the TPC Benchmark C

Overview of the TPC Benchmark C:
The Order-Entry Benchmark

By Francois Raab, Walt Kohler, Amitabh Shah

The Transaction Processing Performance Council (TPC) is about to approve the third in its series of benchmarks which measure the performance and price/performance of transaction processing systems. Like TPC-A, the TPC's first benchmark, the new TPC Benchmark C, or TPC-C, is an on-line transaction processing (OLTP) benchmark. However, TPC-C is different and more complex than TPC-A because of its multiple transaction types, more complex database, and overall execution structure. TPC-C is based on a workload presented to the TPC two years ago and refined by a subcommittee of 16 members representing a cross section of the industry.

The goal of TPC benchmarks is to define a set of functional requirements that can be run on any transaction processing system, regardless of hardware or operating system. It is then up to the test sponsor to submit proof (in the form of a full disclosure report) that they have met all the requirements. This methodology allows any vendor, using "proprietary" or "open" systems, to implement the TPC benchmark and guarantees to end-users that they will see an apples-to-apples comparison. This is a dramatic departure from most other benchmarks where test sponsors are limited to comparing machines that run on just one operating system or benchmarks that execute the same set of software instructions.

TPC benchmarks also differ from other benchmarks in that TPC benchmarks are modeled after actual production applications and environments rather than stand-alone computer tests which may not evaluate key performance factors like user interface, communications, disk I/Os, data storage, and backup and recovery. The difficulty in designing TPC benchmarks lies in reducing the diversity of operations found in a production application, while retaining its essential performance characteristics, namely, the level of system utilization and the complexity of its operations. A large number of functions have to be performed to manage a production system. Since many of these functions are proportionally small in terms of system resource utilization or in terms of frequency of execution, they are not of primary interest for performance analysis. Although these functions are vital for a production system, within the context of a standard benchmark, they would merely create excessive diversity and expense and are, therefore, omitted.

The Benchmark Model

As an OLTP system benchmark, TPC-C simulates a complete environment where a population of terminal operators executes transactions against a database. The benchmark is centered around the principal activities (transactions) of an order-entry environment. These transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses. However, it should be stressed that it is not the intent of TPC-C to specify how to best implement an Order-Entry system. While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any particular business segment, but, rather, represents any industry that must manage, sell, or distribute a product or service.

In the TPC-C business model, a wholesale parts supplier (called the Company below) operates out of a number of warehouses and their associated sales districts. The TPC benchmark is designed to scale just as the Company expands and new warehouses are created. However, certain consistent requirements must be maintained as the benchmark is scaled. Each warehouse in the TPC- C model must supply ten sales districts, and each district serves three thousand customers. An operator from a sales district can select, at any time, one of the five operations or transactions offered by the Company's order-entry system. Like the transactions themselves, the frequency of the individual transactions are modeled after realistic scenarios.

The most frequent transaction consists of entering a new order which, on average, is comprised of ten different items. Each warehouse tries to maintain stock for the 100,000 items in the Company's catalog and fill orders from that stock. However, in reality, one warehouse will probably not have all the parts required to fill every order. Therefore, TPC-C requires that close to ten percent of all orders must be supplied by another warehouse of the Company. Another frequent transaction consists in recording a payment received from a customer. Less frequently, operators will request the status of a previously placed order, process a batch of ten orders for delivery, or query the system for potential supply shortages by examining the level of stock at the local warehouse. A total of five types of transactions, then, are used to model this business activity. The performance metric reported by TPC-C measures the number of orders that can be fully processed per minute and is expressed in tpm-C.

TPC-C was designed to carry over many of the characteristics of TPC-A, the TPC's standard version of DebitCredit. Therefore, TPC-C includes all the components of a basic OLTP benchmark. To make the benchmark applicable to systems of varying computing powers, TPC-C implementations must scale both the number of terminals and the size of the database proportionally to the computing power of the measured system. To test whether the measured system is a fully production-ready system with sufficient recovery capabilities, the database must provide what are defined as the ACID properties: atomicity, consistency, isolation, and durability. To facilitate independent verification of the benchmark results, the test sponsor must release, in a full disclosure report, all information necessary to reproduce the reported performance. This performance, which measures the throughput of the system, must be reported along with the total system cost. The total system cost is a close approximation of the true cost of the vendor-supplied portion of the system to the end-user. It includes the cost of all hardware and software components; maintenance costs over 5 years; and sufficient storage capacity to hold the data generated over a period of 180 eight-hour days of operation at the reported throughput.

More OLTP Features and Complexity

With the coming approval of TPC-C, the TPC will have two OLTP benchmarks, TPC-A and TPC-C. The TPC will continue to support and publish results on TPC-A, its first OLTP benchmark. TPC-A simulates all the major functions of a complete OLTP system and has now been accepted by the industry as a valid means of comparing systems.

With the introduction of TPC-C, the TPC is providing the industry with an additional, more complex OLTP measurement tool. TPC-C involves a mix of five concurrent transactions of different types and complexity either executed on-line or queued for deferred execution. This is one of the most substantial extensions of the basic OLTP benchmarking model as new components of the measured system are being stressed by having multiple transactions of different natures compete for system resources. Another substantial extension is the increased complexity of its database structure. The new database is comprised of nine types of records with a wide range of record and population sizes. As a result, there is greater diversity in the data manipulated by each of the five transactions. The data entered by operators in TPC-C include some of the basic characteristics of real-life data input. For example, operators may enter an invalid item number, forcing the transaction to be cancelled.

In moving toward modeling more realistic environments, TPC-C reduces the number of artificial limitations commonly found in other benchmarks. For example, to promote the use of fully-functional terminals or work-stations and screen management software, TPC-C requires all terminal inputs and displays to be usable by real-life operators. To that end, all screens must be formatted using labeled input and output fields, as specified, and must provide all the common screen manipulation features, including moving forward or backward through the input fields and entering numbers in right justified fields. In another area, any physical database design technique that can be used to improve the performance of a real-life application, such as partitioning or replication of data, is allowed in TPC-C. The use of database records by the transactions has been carefully defined to preclude test sponsors from gaining unrealistic advantages from any of these techniques.

A Measure of Business Throughput

The throughput of TPC-C is a direct result of the level of activity at the terminals. Each warehouse has ten terminals and all five transactions are available at each terminal. A remote terminal emulator (RTE) is used to maintain the required mix of transactions over the performance measurement period. This mix represents the complete business processing of an order as it is entered, paid for, checked, and delivered. More specifically, the required mix is defined to produce an equal number of New-Order and Payment transactions and to produce one Delivery transaction, one Order-Status transaction, and one Stock-Level transaction for every ten New-Order transactions.

The tpm-C metric is the number of New-Order transactions executed per minute. Given the required mix and the wide range of complexity and types among the transactions, this metric more closely simulates a complete business activity, not just one or two transactions or computer operations. For this reason, the tpm-C metric is considered to be a measure of business throughput.

The RTE is also used to measure the response time of each transaction and to simulate keying times and think times. The keying time represents the time spent entering data at the terminal and the think time represents the time spent, by the operator, to read the result of the transaction at the terminal before requesting another transaction. Each transaction has a minimum keying time and a minimum think time. In addition, the response time of each transaction must be below a required threshold. These thresholds have been defined to give predominance to New-Order as the performance limiting transaction.

A New Yardstick

Users of benchmark information and results, whether they be members of the press, market researchers, or commercial users, want to be assured that the benchmark results they see are valid measures of performance. To meet that demand, the TPC has designed its benchmarks to simulate and test systems with all the necessary production-oriented features, including backup and recovery features. In addition, the TPC requires complete documentation of the benchmark run (the full disclosure report). These reports are available to any user and pass through the TPC's own internal review process. All these requirements help to ensure that users of TPC benchmark results will see valid, objective measures of performance.

TPC-C follows the TPC's benchmarking philosophy and methodology in all the above respects, but it also includes new elements and more complex requirements. TPC-C's performance measurement metric, tpm-C, does not just measure a few basic computer or database transactions, but measures how many complete business operations can be processed per minute. This new benchmark should give users a more extensive, more complex yardstick for measuring OLTP system performance.

 

from:http://www.tpc.org/tpcc/detail.asp