Performance Testing Core Activity: Criteria-- PerfTestPlus

Core Performance Testing Principle:

Criteria

Overview

Performance acceptance criteria are typically derived from five perspectives: system users, business owners of the system, technologies, compliance expectations, and the project team.

Determining the desired performance characteristics for a system from particular perspectives is only the first part of determining the overall performance acceptance criteria for that system. After examining the desired performance characteristics from limited perspectives, you must resolve those characteristics against one another. For example, end-users may desire sub-second response time for every transaction; business stakeholders may want the system to support millions of users; and compliance criteria may mandate strict security policies.

Individually, any of these characteristics may be achievable, but collectively they may not be possible due to time, technology, and/or budget constraints. Finding a way to achieve these desired characteristics presents team-wide challenges that should be addressed proactively, not reactively after conflicts become apparent through testing.

Terminology

Performance Acceptance Criteria – Performance testing objectives refer to data that is collected through the process of performance testing and that is anticipated to have value in determining or improving the quality
Performance Testing Objectives – Performance testing objectives refer to data that is collected through the process of performance testing and that is anticipated to have value in determining or improving the quality of the product. However, these objectives are not necessarily quantitative or directly related to a performance requirement, goal, or stated quality of service (QoS) specification.
Performance Objectives – Performance objectives are usually specified in terms of response times, throughput (transactions per second), and resource utilization levels. Resource utilization levels include the amount of processor capacity, memory, disk I/O, and network I/O that your application consumes.
Performance Requirements – Performance requirements are those criteria which are absolutely non-negotiable due to contractual obligations, service level agreements, or non-negotiable business needs. Any performance criteria that will not unquestionably lead to a decision to delay a release until the criteria pass is not absolutely required, and therefore, not a requirement.
Performance Goal – Performance goals are those criteria that are desired for release, but may be negotiable under certain circumstances. For instance, if a response time goal of three seconds is set for a particular transaction, but the actual response time is determined to be 3.3 seconds, it is likely that the stakeholders will choose to release the application and defer performance tuning of that transaction for a future release.
Performance Targets – Performance targets are the desired values for a resource of interest under a particular set of conditions, usually specified in terms of response times, throughput, and resource utilization levels. Resource utilization levels include the amount of processor capacity, memory, disk I/O, and network I/O that your application consumes.
Performance Thresholds – Performance thresholds represent the maximum acceptable value for a resource of interest, usually specified in terms of response times, throughput (transactions per second), and resource utilization levels. Resource utilization levels include the amount of processor capacity, memory, disk input/output (I/O), and network I/O that your application consumes.
Performance Budgets or Allocations – Performance budgets are your constraints. Performance budgets specify the amount of resources that you can use for specific scenarios and operations and still be successful.