Measuring Browser Performance: Understanding issues in benchmarking and performa

 

 

 

 

Measuring Browser Performance 

Understanding issues in benchmarking and performance analysis 

 

 

Published March 2009 

 

 

 

For the latest information, see http://www.microsoft.com/ie8 

Version 1.0 

 

 

Abstract  

This document will explain various browser components and how each can impact performance 

when benchmarking. The abilities and pitfalls of various benchmarking tools will be compared, 

and ways to design tests that avoid these issues will be discussed. Also included is an overview 

of instructions on how to set up a benchmarking environment to conduct some of the testing 

processes discussed here. The information contained in this document represents the current 

view of Microsoft Corp. on the issues discussed as of the date of 

publication. Because Microsoft must respond to changing market 

conditions, it should not be interpreted to be a commitment on the 

part of Microsoft, and Microsoft cannot guarantee the accuracy of 

any information presented after the date of publication. 

This guide is for informational purposes only. MICROSOFT MAKES 

NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. 

Complying with all applicable copyright laws is the responsibility of 

the user. Without limiting the rights under copyright, no part of this 

document may be reproduced, stored in or introduced into a retrieval 

system, or transmitted in any form, by any means (electronic, 

mechanical, photocopying, recording or otherwise), or for any 

purpose, without the express written permission of Microsoft. 

Microsoft may have patents, patent applications, trademarks, 

copyrights or other intellectual property rights covering subject 

matter in this document. Except as expressly provided in any written 

license agreement from Microsoft, the furnishing of this document 

does not give you any license to these patents, trademarks, 

copyrights or other intellectual property. 

Unless otherwise noted, the example companies, organizations, 

products, domain names, e-mail addresses, logos, people, places 

and events depicted herein are fictitious, and no association with 

any real company, organization, product, domain name, e-mail 

address, logo, person, place or event is intended or should be 

inferred. 

Microsoft, Windows, SmartScreen, InPrivate and Internet Explorer 

are either registered trademarks or trademarks of Microsoft Corp. in 

the United States and/or other countries. The names of actual 

companies and products mentioned herein may be the trademarks 

of their respective owners. 

© 2009 Microsoft Corp. All rights reserved. Measuring Browser Performance    3 

Table of Contents 

Measuring browser performance ............................................................................................................ 4 

Testing for the real world  ......................................................................................................................... 4 

Testing tools  ............................................................................................................................................ 5 

Caching behavior  .................................................................................................................................... 5 

Measurement overhead and resource competition  ................................................................................. 6 

Connectivity and network device latency  ................................................................................................ 6 

Dealing with Internet-based data ............................................................................................................ 6 

Browser targeted content  ........................................................................................................................ 7 

Machine Configurations .......................................................................................................................... 7 

Extensibility issues  .................................................................................................................................. 8 

Inconsistent definitions  ............................................................................................................................ 9 

Measuring for yourself  ............................................................................................................................. 9 

Step 1. Choose testing targets  ................................................................................................ 10 

Step 2. Setting up the system  .................................................................................................. 10 

Step 3. Test in sequence  ......................................................................................................... 10 

Step 4. Warm start testing  ....................................................................................................... 10 

Step 5. Do the benchmarking  .................................................................................................. 10 

Step 6. Repeat  ......................................................................................................................... 11 

Putting it all together ............................................................................................................................. 11 

Appendix A: List of Sites Tested  ........................................................................................................... 13 

 

 

   Measuring Browser Performance    4 

Measuring browser performance 

Users arguably spend more time using their web browser than any other application on their machine, 

so browser vendors must work hard to make their products stable, robust and fast. Benchmarking 

browser speed is easier said than done. This document will explain various browser components and 

how each can impact performance when benchmarking. Benchmarking tools will be compared, and we 

will present ways to design tests that avoid these issues. Lastly, this whitepaper includes an overview of 

how to set up a benchmarking environment that will enable you to conduct some of the testing 

processes discussed here. To go directly to the step-by-step instructions on how to conduct a 

benchmarking test, jump ahead to the section titled “Measuring for Yourself.” 

In the Internet Explorer testing lab, we run daily and monthly benchmarking passes in our controlled 

testing environment using a tiered testing model to catch regressions in the product. This process helps 

ensure that we are able to get a regular and consistent view of performance, as well as providing the 

ability to scale and test tens of thousands of websites. The daily testing covers approximately 25 

websites, such as those listed in Appendix A, although the specific list varies from day to day. Websites 

used in daily testing are chosen for a variety of reasons, but one important factor included in selection 

criteria is that the list incorporates a wide variety of websites, including international websites, in order 

to be representative of users on the Internet today. For the Internet Explorer 8 performance video we 

have published, we chose the Top 25 websites in the world as measured by the Global Internet 

Information provider comScore to reflect the sites users are most likely to visit on the web. 

Testing for the real world 

It is difficult to measure the performance of a browser by using “live” websites. There are many 

variables such as network latency, network congestion, and website traffic levels for “live” website tests 

that can affect measurements. To address this, artificial lab tests (benchmarks) are created to measure 

specific aspects of browser performance in a controlled setting.  These benchmarks necessarily 

characterize only a narrow set of the browser functions in a very constrained way.  End users, however, 

do not operate in a controlled environment, nor do they perform only a subset of their browsing tasks. 

When end users use their browser, they perform complete tasks, such as reading a news story, getting 

directions or online shopping. 

Browsers are composed of several subsystems, each of which plays a role in overall performance. End 

users don’t just run JavaScript or load test pages, so testing “micro-benchmarks” does not translate 

directly to the end user experience and can be a misleading indicator of overall performance. End users 

are more properly aligned with tests that measure overall “page load times” which represent the 

completion of a task, not isolated operations. 

Think about a real world task like going to the grocery store. The task is filled with many smaller 

functions, each critical to the task’s overall completion, but speed in one step does not correctly indicate 

the entire process will be fast. For example, imagine two neighbors who have the same grocery list. If 

one shopper leaves for the store in a station wagon at the same time his neighbor leaves in a sports car, 

it doesn’t mean the neighbor in the faster car will finish shopping first. Once at the store, the neighbor 

may be less efficient at navigating the store aisles and lose time finding items on his list. At checkout, Measuring Browser Performance    5 

one shopper may use cash, while the other pays by check, which also takes longer. As this example 

illustrates, measuring only one portion of the process can easily offer misleading direction on the overall 

performance. 

For the real world user, performance criteria are based on how quickly a page loads, not the 

performance of any individual subsystem. For that reason, this paper will focus on how to measure 

complete webpage load times and discuss the related complexities.  We will discuss the testing tools 

and their limitations, the affects of caching, the impact of measurement overhead and resource 

competition, and the variance introduced by network connectivity and other resource limitations. We 

will describe each of these and look at how and why they affect performance testing relative to the goal 

of measuring page load times.  

Testing tools 

Many benchmarking tests cite results from tools such as SunSpider, Celtic Kane, V8 and iBench. Each is 

designed, at least in some way, to test only a portion of the end-user scenario. SunSpider primarily tests 

JavaScript performance and also some other popular functions used on websites today. Other popular 

benchmarks, Celtic Kane and V8, are also geared at testing only the JavaScript engine. Meanwhile, the 

iBench suite relies on a browser reporting when it’s finished loading a page. Since browsers differ on 

when they define a page is done loading, this test also yields unreliable results.  

Caching behavior 

Caching, a common design feature used by browsers to optimize the users online experience and can 

impact benchmark testing. Browsers are designed to cache (store) content locally in order to avoid 

repeatedly requesting the same content. Many company networks and Internet Service Providers (ISPs) 

similarly exploit caching to greatly reduce the amount of content repeatedly requested over their 

network transmission lines, so users do not suffer delays when browsing a site with multiple pages 

containing the same content, such as company logos or other common graphics. 

However, this timesaving design can introduce factors that incorrectly impact benchmarking results. For 

example, if a test calls for opening 10 tabs to 10 different websites in one browser, and then to open the 

same 10 tabs in a second browser, the results could wrongfully indicate that the second browser was 

faster, when in fact the difference was due primarily to the content being stored by a nearby server 

when the first browser requested the pages. 

In the Internet Explorer lab: We visit each site prior to starting any site test. “Preloading” the 

cache prior to a test helps ensure systems are at a known base before starting. 

For you to try: Because it’s nearly impossible to avoid all network-level caching devices, one 

solution is to visit each Web site you plan to test—with each browser you plan to test—before 

starting the actual test. This “pre-caching” step helps ensure that each browser is being tested 

more equally.  Measuring Browser Performance    6 

Measurement overhead and resource competition 

Another benchmarking problem is that the process of measuring consumes resources and can impact 

the application you are trying to measure. This process is known as the “observer effect.” For example, 

adding debug code to an application may cause the program to execute more slowly than normal. There 

is a related condition, known often as a “Heisenbug,” in which the act of changing the code to measure 

it (or isolate a condition for bug testing) ends up altering the effect of variables or other system 

activities. 

Because there is no way to completely remove the measurement process load overhead, the goal is to 

find ways to minimize it. A popular approach is to use tests that focus on specific scenarios. The theory is 

that, while still having some performance impact, smaller tests yield fewer extraneous activities.  

Perhaps the largest issue in benchmarking is resource competition. Any given system only has so many 

resources, and they cannot all be available to every application at the same time. A process cannot start 

until resources are available, which could result in inconsistent benchmarking results, as resources are 

constantly in flux. Other applications running on the machine are competing for resources, so efforts 

should be taken to isolate or reduce them. 

In the Internet Explorer lab: We disable all other unnecessary applications and services.  

For you to try: To help reach a valid configuration, users should disable other applications during 

testing. 

Connectivity and network device latency 

Another issue is that while network connectivity is constantly increasing, network resources are not 

unlimited. Several things, like local area congestion, trunk line congestion and peering congestion, can 

impact network resource availability. Applications running at the network layer (such as proxy servers, 

load balancers, firewalls, and so on) can also have resource contention issues that impact benchmarking 

data.  

In the Internet Explorer lab: We use a DTAP in our testing to mitigate many of these issues with 

our testing network.  

For you to try: Because there is no way to completely isolate or remove these issues, the best 

way to address this is to identify common load periods and test under similar load levels each 

time. 

Dealing with Internet-based data  

Testing on Internet-based (“live”) Web servers can introduce factors that impact performance data 

observations. Simply put, what the server is doing and how the application is designed can impact the 

benchmarking process. Fortunately, many application issues can be isolated or removed entirely. Server 

load, however, can’t be dealt with in the same way. Measuring Browser Performance    7 

In the Internet Explorer lab: We perform several rounds of tests over time and remove outliers 

to get a statistical average. 

For you to try: The best solution to counteract these issues is to conduct several tests over a 

period of time and ensure consistency in network layer resources during those tests.  

Browser targeted content 

Many websites perform browser detection in order to provide users with the best possible experience 

when using their chosen browser. This can result in a website behaving differently when viewed with 

one browser versus another, but it may also cause the website to serve varied content depending on the 

client browser. That variation can lead to benchmarking errors and irregularities when content viewed in 

one browser cannot be directly compared to another. Slight differences in CSS or layout designs are 

generally acceptable—and often unavoidable—but large differences should be avoided. It is important 

to visually review each website prior to benchmark testing to verify the website isn’t drastically different 

when viewed with different browsers. 

In the Internet Explorer lab: We review each website on the list in each browser we are testing 

prior to starting a test pass. Any websites which appear drastically different are removed from 

the test list to help ensure a more consistent apples-to-apples comparison. 

For you to try: Make a list of websites to be used in benchmark testing. Visit each one using 

each browser to be tested and visually inspect them to ensure no large differences between the 

appearance or layout of each website.  

Machine Configurations 

Just as with humans, no two machines are exactly alike. As a case in point, there are several dozen 

machines in the Internet Explorer performance lab running performance tests every hour of every day. 

To maximize our lab’s flexibility, an attempt was made to create a set of “identical” machines which 

could be used interchangeably to produce a consistent set of performance data. Those machines bore 

consecutive serials numbers and were from the same assembly line, and all their component parts were 

“identical.” 

To understand this, it is important to remember that hardware vendors obtain parts from a broad range 

of suppliers in an effort to ensure sufficient supplies as well as manage costs. One common component, 

for example Ethernet ports, may be sourced from as many as three suppliers. As long as each product 

meets the proper engineering specifications, those parts can be used interchangeably—an Ethernet port 

is an Ethernet port, much like a Cat 5 cable can be interchanged with any other Cat 5 cable. As long as 

both meet the prescribed engineering standards, they should function in the same way. The problem is 

that minor differences such as device drivers can introduce performance issues of their own. Since there 

is no easy and scalable way to ensure all system components are truly identical, the best course is to 

minimize comparing test results from different machines.  Measuring Browser Performance    8 

For these reasons, despite all the efforts to have duplicate systems, data collected on the machines in 

the Internet Explorer performance lab has been sufficiently varied that it is best not to compare 

performance results from two different machines.   

In the Internet Explorer lab: Prior to starting each benchmarking test cycle, machines are re-

imaged to help ensure the most pristine state possible. 

For you to try: Set up a machine for your benchmarking tests, free of other applications and 

default settings that may impact performance. Use that same machine for all testing to ensure 

consistency. 

Extensibility issues 

The issue of extensibility, or add-ons, is yet another potentially confounding element when 

benchmarking. Though browser add-ons offer users a powerful way to get more from their browsing 

experience, they can also introduce performance issues, making it difficult to distinguish an “original” 

browser issue from one introduced by the third-party code. Mike Shaver, Chief Evangelist at Mozilla, 

noted this issue in an interview: “People using Firefox sometimes load it up with a pile of extensions and 

then saying, ‘Performance isn’t what it used to be’ or ‘I get this one weird crash.’”

1

 

To account for this, it is important to test both “clean” installations without any add-ons, as well as 

some with popular add-ons enabled (such as Adobe Flash, Microsoft Silverlight®, and Windows Media® 

Player). Comparing the results of those two data sets should help give some visibility into the impact of 

third-party elements. 

To disable add-ons for Internet Explorer 8, select Manage add-ons from the Tools menu. In the Manage 

add-ons dialog box, ensure that All add-ons are visible, and then disable all add-ons (this can also be 

done at a command prompt by typing iexplore.exe -extoff). Alternatively, can be run in “Safe 

Mode,” in which the browser is launched without any add-ons enabled. This mode can be found by 

going to Accessories > System Tools. Some browsers offer a form of safe mode operation, making it 

easier to benchmark without add-ons across different browsers.  

Browser  To Enable Safe Mode 

Internet Explorer  Start > All Programs > Accessories > System Tools > Internet Explorer (No 

Add-ons) 

Firefox  Start > Mozilla Firefox > Mozilla Firefox (Safe Mode) 

 

In the Internet Explorer lab: We install and test with the most common add-ons (Flash, 

Silverlight and Windows Media Player) necessary for common functionality on popular websites 

 

1

 Scott Swigart and Sean Campbell. How Software Is Built. 20 March 2008. 

http://howsoftwareisbuilt.com/2008/03/20/interview-with-mike-shaver-chief-evangelist-mozilla/  Measuring Browser Performance    9 

to ensure a real world user experience. We also test with no add-ons installed in order to get 

side-by-side measurement data. 

For you to try: Take an inventory of the add-ons installed on the browsers and ensure they are 

the same; then disable or remove any add-ons which are not available (or not equal) to those 

available for other browsers you plan to test. In addition, tests should be run with no add-ons to 

get comparison benchmarking data. 

Inconsistent definitions 

Another factor that impacts benchmarking is having a consistent baseline of what it means to complete 

a task. In car and horse racing, the end is clearthere is a finish line that all contestants must cross to 

finish the race. Browser benchmarking needs a similar way to define when a webpage is done loading. 

The problem is that there is no standard which dictates when a browser can or should report that it is 

“done” loading a page—each browser operates in a different way, so relying on this marker could 

produce highly variable results.  

The best approach is to utilize browser progress indicators and validate those indicators against when 

the webpage is actually loaded and ready for use. For example, when testing how quickly a particular 

webpage loads, look for a specific object or try to interact with the page while it is loading for the first 

time. If the webpage appears to be loaded and is interactive before the progress indicators complete, it 

is reasonable to ignore the indicators and use the page appearance for measurements. If the webpage is 

not ready for use, then the progress indicators should be sufficient to make an initial assessment of how 

quickly a page is downloading across various browsers.  

In the Internet Explorer lab: To avoid the inconsistent results of using the “Done” status 

notification, we primarily use visual cues to determine when a page is complete. For sites on 

which specific visual cues aren’t sufficient, we use a combination of visual cues and the ability to 

interact with content on a page to determine if a page is “done.” Some site designs utilize AJAX 

or similar technologies where determining “done” is even more complex, so in those cases it is 

more deterministic to use a visual cue. 

For you to try: Avoid using any browser self reporting status information and rely on your own 

experiences and responses. If you are able to interact with the page and it appears ready to use, 

it is acceptable to use that timing data. In cases where the site design is overly simplified (for 

example, where only a few objects are on a page) and it would be difficult to measure when the 

page is ready for use, you may find it easier to look for a visual property on the page and count 

the page as complete when that item is loaded. 

Measuring for yourself 

Armed with an understanding of the issues involved with benchmarking and ways to avoid those pitfalls 

in testing, the next step is to build a benchmarking test bed.  Measuring Browser Performance    10 

In the Internet Explorer lab: We use a video recording system to monitor all tests, which 

enables us to slow down the video for a frame-by-frame replay to capture extremely accurate 

performance times and read measurement cues. 

For you to try: While it is impractical to provide a blueprint to test as the Internet Explorer lab 

does, these steps will provide a basic environment for completing a suitable evaluation. 

Step 1. Choose testing targets 

Vary testing selection. Having a list of popular Web sites is important, but if the sites are too similar in 

content and functionality, you will not fully test the browser’s subsystems. Having a broad and mixed set 

of testing targets helps ensure you test all of the different browser subsystems rather than just one as 

many micro-benchmarks like SunSpider do. The list of sites we used is included in Appendix A. 

For each testing target, ensure user interaction steps are clear and understood before continuing. 

During the actual testing process, it is necessary to understand the testing target well enough to 

determine whether the site is usable; that is, that a user can navigate and interact with the site. If the 

test is going to be based on specific visual cue items, ensure those items are well known and 

recognizable.  

Step 2. Setting up the system 

Clear the cache. It is vital to ensure each browser is in a clean state before testing starts. Each browser 

has a slightly different way to clear the cache. For Internet Explorer, go the Safety menu and select 

Delete Browsing History. Since this feature is designed to delete several items in one menu, ensure that 

only the Temporary Internet Files option is selected. This will ensure each browser is in a common state. 

Stop other applications. To do this, close other open applications and look at applications running in the 

system tray. Where possible, close anything in the system tray. Launch Windows Task Manager (by 

pressing CTRL+SHIFT+ESC), highlight any tasks in the Applications tab (including the browser being 

tested) and select End Task. 

Step 3. Test in sequence 

Select one browser. It is important not to test side-by-side: that is, having both browsers running at the 

same time and accessing the target website. Doing so can impact one or both browsers and skew 

results.  

Step 4. Warm-start testing 

Load the browser. Launch one browser and visit the target website(s) selected for testing to prepare the 

cache and get various browser components loaded. This ensures there are no extra delays or 

performance impacts introduced by loading pieces. Wait to warm-start the other browser(s) being 

benchmarked until the testing cycle on the previous browser is complete. 

Step 5. Do the benchmarking 

Set up a video recording device. Capturing accurate timing information is critical for performing a 

detailed browser performance comparison. Using an external video recording device, either by 

capturing the PC video output or by using a standalone camcorder, avoids any impact or system Measuring Browser Performance    11 

overhead from affecting benchmarking results and enables you to slow down footage later to compare 

results. The Internet Explorer team captured video directly from the PC output to document benchmark 

timings. Set up the device in a location with a clear view of the screen and ensure the framing of the 

screen is cropped to fill the whole frame. Ensuring the video image is as large and clear as possible will 

help you identify even small objects as they load on the screen. Once the video capture is complete, you 

can replay the video by using media player software to slow down the images to get granular timing 

information. The Internet Explorer team uses media player software capable of viewing data frame-by-

frame2

, providing the ability to obtain timing data at the accuracy of approximately 1/30th

 of a second. 

To enable frame-by-frame playback using Windows Media Player, select the View menu, then 

Enhancements, then Play Speed Settings. This will display a controller which allows you to advance by 

single frames. Other media player software applications also support frame-by-frame playback so you 

may be able to use your preferred software in place of Windows Media Player. 

Another option for timing: get a stopwatch. If you are unable to set up a video recording device, it is 

possible to use a stopwatch to capture timing data. However, the stopwatch method introduces the 

potential for timing delays and inaccuracies, meaning the results are much less reliable and so should be 

avoided if at all possible. The stopwatch method creates a small delay, known as the “Personal Equation” 

(PE), which is the reaction time between the eye first noticing an event and the hand clicking the 

stopwatch. The PE for most people is around 0.4 seconds, so that amount should be removed from each 

start and stop measurement. 

Start the testing. Using your list of websites, enter the URL into the browser’s Address bar. Prepare the 

stopwatch and start timing when hitting ENTER. To avoid inconsistency in how each browser reports 

when a webpage is complete, record “done” times as the moment the visual cue appears or when the 

user is able to interact with the site. For example, if the site has a text entry box, “done” would mean 

when the user can enter text.  

Get multiple measurements. Since both internal and external factors can impact time measurement, it is 

important to conduct 10 tests for each Web site on the list. Once the list is complete, remove the top 

and bottom 2 samples and average the remaining numbers. 

Step 6. Repeat 

Test other browsers. Repeat Steps 4 and 5 for each browser being tested. 

Test again later. To avoid any issues with time of day network traffic and utilization impacts, all tests 

should be repeated at another time of day. Ideally, one set of tests would be done during normal 

working hours and the second approximately 12 hours later. Normal working hours should correspond 

to the target user base for the given website.  

Putting it all together 

As this analysis has illustrated, there are many factors involved in benchmarking and many steps 

involved in attaining valid results. This document should provide the information needed to perform 

benchmarking tests and isolate extraneous factors to allow for a true analysis of browser performance. 

 

2

 NTSC video records at 30 frames per second (fps). Measuring Browser Performance    12 

Having an understanding of the limitations of popular benchmarking tools will help everyone 

understand why it is important to view performance on a holistic level, looking at the bigger picture of 

page load times, which account for all browser subsystem elements. 

   Measuring Browser Performance    13 

Appendix A: Results from Browser Performance Test 

For the Internet Explorer 8 performance video we published, we chose the Top 25 websites in the world 

as measured by comScore to reflect the sites users are most likely to visit on the web. The following 

table shows results from the benchmark testing process for those 25 websites tested on the following 

browsers: Google Chrome, Mozilla Firefox, and Internet Explorer 8.  

As noted earlier, Microsoft chooses approximately 25 websites for daily testing, and tens of thousands 

on a monthly basis. Any list of websites to be used for benchmarking must contain a variety of websites, 

including international websites, to help ensure a complete picture of performance as users would 

experience on the Internet.  

Summary of Browser Load Times (in seconds) 

Rank  Web Site  Chrome 1.0  Firefox 3.05  Internet Explorer 8 

1  google.com  0.28  0.22  0.20 

2  yahoo.com  2.15  1.30  1.32 

3  live.com  3.48  3.42  3.40 

4  msn.com  0.55  0.97  0.83 

5  youtube.com  3.07  2.80  2.55 

6  microsoft.com  3.83  3.47  3.75 

7  wikipedia.org  1.22  1.35  1.88 

8  blogger.com  2.07  2.88  1.52 

9  facebook.com  2.12  2.08  1.98 

10  qq.com  6.82  7.88  7.33 

11  baidu.com  1.40  1.47  1.50 

12  myspace.com  2.13  5.53  3.68 

13  wordpress.com  0.95  1.58  1.45 

14  ebay.com  4.06  4.43  3.42 

15  sina.com.cn  5.48  6.37  8.03 

16  mozilla.com  1.28  1.53  1.27 

17  adobe.com  9.50  9.37  8.85 

18  aol.com  3.02  2.57  2.32 

19  amazon.com  2.12  3.35  2.82 

20  apple.com  2.52  3.07  1.63 

21  soso.com  3.25  2.64  2.07 

22  xunlei.com  8.60  9.97  8.70 

23  163.com  14.87  14.75  15.02 

24  google.cn  1.38  0.85  1.05 

25  ask.com  2.17  1.77  1.73 

 Measuring Browser Performance    14 

In line with the testing guidelines outlined in this paper we have used the browser “Done” indicator for 

timing when the page when the page is completely loaded at that point. For pages which continue to 

load and change after the “Done” indication we have used common visual cues to generate the timings. 

Timing is started when the Go button is pressed. These timings were captured in January 2009; because 

Internet content is always changing you may get different timings when you run these tests.