Scaling Site Xby Bryan Che (bryanche@arsdigita.com) Submitted on: 2001-02-28
ArsDigita : ArsDigita Systems Journal : One article
Following is a detailed description of how ArsDigita arrived at these
findings:
For load testing, ArsDigita exported a copy of Site X's production
database into the scaling lab's e450. It then setup the e450 and Netras
with Site X's code base and produced a mirror copy of Site X's
production environment for load testing.
ArsDigita wrote an e-Load script, called Index+3, to test this path.
This path consisted of Site X's index page, and then three additional
pages in the site:
ArsDigita used a databank to vary values for things such as originating
cities when running Index+3. This allowed testing using a variety of
data.
The initial run used the production setup of three Netras. Next,
ArsDigita tried a different configuration with a separate image server
and reduced main servers:
Table 1 Test Configuration with Separate Image Server
This did not seem to produce any significantly different results from
the previous configuration.
Then, ArsDigita gave each server instance its own Netra:
Table 2 Test Configuration with Six Netras
ArsDigita continued to add Netras until it reached a total of nine
Netras:
Table 3 Test Configuration with Nine Netras
The following table summarizes the results of these tests:
Table 4 Summary of Index+3 Results
In these tests, pages one and two (/index.adp and /page2.adp) always
performed within eight seconds before the Web site began to exhibit
errors. Page three, /page3.adp, however, was extremely slow; it
consistently took too long to load early on in the testing ramp-ups.
Page four, /page4.adp, was also slow, although it performed within
acceptable limits when tested with nine Netras.
The following graphs illustrate the page latencies from these test runs:
Figure 1 Page Performance Vs Users (3 Netras)
Figure 2 Page Performance Vs Users (6 Netras)
Figure 3 Page Performance Vs Users (9 Netras)
The following graph illustrate Site X's overall performance versus users
for index+3 testing:
Figure 4 Round One Index+3 Performance Vs. Users
One striking characteristic of these performance versus users results is
that they are linear. A typical performance versus users graph should
take the following form:
Figure 5 Typical Performance Versus Users Graph
A Web site's performance should remain flat until it hits a bottleneck
in the system. Once the site encounters a bottleneck, it will trade
performance for users-the more users the site handles, the slower it
performs.
Site X's performance-versus-users graphs are not flat anywhere.
Therefore, index+3 testing is encountering a fundamental bottleneck as
soon as testing begins. The reason this bottleneck shows up so quickly
was likely due to the testing pace being too fast as the virtual users
did not have any delay between page clicks. In other words, this test
pushed the system as much as possible even with only one user.
A typical throughput curve for a Web site will take the following form:
Figure 6 Typical Statistics Vs Time Graph
As long as a Web site's throughput is increasing as the number of
concurrent users increases, it has not hit a major bottleneck. But,
once the site's throughput levels off, the site has hit a bottleneck as
it is no longer able to achieve further throughput despite the increased
number of users.
Site X's throughput graphs took the following form:
Figure 7 Index+3 Transactions/Second - 9 Netras
This graph shows that Site X hit its scalability limit almost
immediately after testing began. However, increasing Netras increased
overall throughput. This, like the performance-versus-users graphs,
indicates that the virtual user pace for Round One testing was too fast.
In addition, ArsDigita decided to modify its testing strategy for Round
Two. First, ArsDigita would introduce delays between virtual user
clicks in order to slow down the testing process. This would enable
ArsDigita to more easily identify at what points scalability bottlenecks
manifested themselves.
ArsDigita would also add additional pages for testing during Round Two.
This would serve a couple purposes: it would allow ArsDigita to test the
scalability and performance of new XML pages written for the COMPANY Y
integration, and it would allow ArsDigita to make a more complete
assessment of Site X's overall scalability.
ArsDigita used the following configuration for this test:
Table 5 Test Configuration for /page3.adp test
Note that ArsDigita added a ten-second delay between virtual user
page-clicks to slow down the testing pace.
The results of this load test are illustrated in the following graphs:
Figure 8 /page3.adp Performance Vs Users
Figure 9 page3.adp Pages/Second Throughput
As these two charts illustrate, page3.adp performed quite well following
optimization so that on a single Netra running one AOLserver instance,
it could scale to 250 virtual users and achieve a steady-state
throughput of about 36 pages/second. These results were quite
acceptable, and so ArsDigita next turned to tuning the other slow page
in the Index+3 script.
After ArsDigita finished tuning /page4.adp, the page performed
acceptably up to around 40 users and scaled up to 90 users. Its average
throughput also increased fivefold to 5.56 pages/second. ArsDigita
deemed this figures acceptable as this page performs several expensive
computations.
ArsDigita used the following setup for evaluating this page:
Table 6 Configuration for /page4.adp test
The following graphs illustrate the improvements in this page:
Figure 10 Comparison of /page4.adp Performance Vs. Users
Figure 11 Comparison of /page4.adp Throughput: Pages/Second
Table 7 Comparison of Index+3 Test Setup for Round One Vs. Round
Two
Note that ArsDigita ran its Round Two Index+3 test with no virtual user
delays in order to compare with Round One results.
The following graphs show the Index+3 results following ArsDigita's
optimizations:
Figure 12 Comparison of Index+3 Results: Performance Vs. Users
Figure 13 Comparison of Index+3 Results: Throughput
As these graphs show, ArsDigita's tuning produced significantly improved
results while running on less hardware than in Round One testing.
Table 8 Configuration for xml-1 test
From this analysis, ArsDigita found that xml-1 scaled well up to about
45 users and an average throughput of about 37.1 pages/second for one
AOLserver on a single Netra:
Figure 14 xml-1 Performance Vs. Users
Figure 15 xml-1 Throughput: Pages/Second
Based on these results, ArsDigita concluded that xml-1 scaled
adequately.
Table 9 Configuration for xml-2 test
From this test, ArsDigita found that xml-2 scaled acceptably to about 90
users and a steady-state throughput of about 20 pages/second on one
AOLserver running on a single Netra.
Figure 16 xml-2 Performance Vs. Users
Figure 17 xml-2 Throughput: Pages/Second
Based on these results, ArsDigita concluded that xml-2 scaled
adequately.
To mimic projected traffic patterns following the joint Site
X-COMPANY Y launch, ArsDigita wrote additional scripts and
classified each script as either background or foreground.
Background scripts would consist mostly of XML bulk-download and
administration activity, whereas foreground scripts would consist of
pages that users would frequently visit.
ArsDigita further divided foreground scripts into two types: XML scripts
and consumer-site scripts. Because Site X was expecting that the
XML-generating pages would receive far more traffic than the regular
Site X pages, ArsDigita decided to deploy foreground scripts at a 5:1
XML:consumer-site ratio.
ArsDigita's e-Load license supports up to 500 virtual users. Therefore,
ArsDigita deployed its users using the background, XML-foreground, and
consumer-site foreground scripts as follows:
Table 10 Site-Wide Test Background Users
Table 11 Site-Wide Test Consumer-Site Foreground Users
Table 12 Site-Wide Test XML Foreground Users
All of these scripts were one page-scripts, except for the trace-*
scripts. These trace scripts performed administration functions and
also involved logging into the Site X site.
The configuration ArsDigita used for testing was as follows:
Table 13 Site-Wide Test Configuration
*The Cache did not actually run during this test.
During this test, the site achieved an overall throughput of about 20
pages/second-exceeding its scalability requirements. Despite these
figures, though, virtual users started encountering server errors such
as connection resets after a load of about 130 users. Upon analyzing
the test results, ArsDigita pinpointed these errors being due to Netras
running out of physical memory. During load testing, the AOLserver
processes on the four Main and XML Netras eventually consumed more
memory than the 256MB of physical RAM available to them. Once this
happened, the Netras' performance suffered, and the site began to
experience errors. ArsDigita noted this memory problem but did not
address it until the next round of testing.
One other problem that ArsDigita found during this test was that at
higher loads, many of the XML bulk-download pages were failing.
ArsDigita suspected this was because many of these pages didn't have
access to the database handles they required. Thus, to attend to this
problem, ArsDigita decided to dedicate an AOLserver specifically for
serving XML bulk-downloads in its next test.
Following are graphs highlighting some of the site-wide test results:
Figure 18 Main Foreground: Performance Vs. Users
Figure 19 XML Foreground: Performance Vs. Users
Note that xml-1's performance grew too slow beyond 150 users (when Netra
memory ran out), and the other XML foreground pages' performance became
unacceptable at around 250 users. The main foreground pages, however,
performed adequately throughout testing.
The next three graphs illustrate how errors started occurring once Netra
physical memory ran out:
Figure 20 Main Foreground Errors Vs. Users
Figure 21 XML Foreground Errors Vs. Users
Figure 22 XML Background Error Rate Vs. Users
Figure 23 Netra Memory Use Vs. Users
The point where each Netra crosses over the 25000.00 line is
approximately where it uses up its 256 MB of RAM. The two Netras that
do not cross over the 256 MB threshold are the image and cache servers.
ArsDigita was not able to test against a full complement of the nine
Netras that Site X planned on using for its site because four of the
Netras Site X planned on using with the e4500 were still deployed on the
existing e450-based site. Nevertheless, if Site X could pass
scalability requirements with five Netras, then it would certainly be
able to do so with nine.
Table 14 Round Three Background Users
Table 15 Round Three Consumer-Site Foreground Users
Table 16 Round Three XML Foreground Users
For its first test, ArsDigita setup the site as follows:
Table 17 Configuration for First Round-Three, Site-Wide Test
Note that ArsDigita, at Site X's request, reduced the virtual user delay
between clicks from ten seconds to five seconds.
One other thing that ArsDigita noticed during this test was that the XML
bulk-download, image, and cache Netras were not working hard at all.
ArsDigita decided that for its remaining tests in Round Three, it would
re-configure its Netras so that these three functions would reside on
one server:
Table 18 Configuration for Remaining Round-Three Site-Wide Tests
With this configuration in place, ArsDigita proceeded to work on
isolating the cause for the problematic AOLserver memory growth. After
multiple rounds of testing and analysis, ArsDigita was able to trace the
primary source of memory growth to the page, xml-2.
Once ArsDigita had determined that spawning threads was causing
AOLserver to grow in size, it installed a custom AOLserver module,
dqd_threadpool. With this module, ArsDigita created a pool of threads
dedicated to the reactive query launched by xml-2. AOLserver would
automatically create all the threads for this pool upon startup and keep
them for the life of the server.
Now, whenever xml-2 ran its reactive query, it would grab existing
threads from the thread pool rather than create new threads. If all the
threads within the thread pool were already being used by other
requests, then the reactive query would enter a queue to await the next
free thread.
Using this pool put a bound on the number of threads that xml-2 could
use, and it thus capped the memory size to which AOLserver could grow.
After setting up dqd_threadpool, Arsdigita tested xml-2 with a pool size
of ten without seeing any memory growth problems. It, therefore, setup
one final site-wide test for Site X to see how the site would scale with
the new e4500 in place and memory issues resolved.
Table 19 Final Site-Wide Test Configuration
During this test, the final site scaled acceptably to 500 users with a
throughput of about 49 pages/second. These results, on less hardware
than the production site, exceed the scalability requirements of 6.5
pages/second by nearly an order of magnitude.
The Netras did not run out of memory throughout this test, indicating
that using dqd_threadpool had largely solved the AOLserver memory
problem. Furthermore, the database and consumer-site Netras were not
using their CPU's to full capacity. The XML servers were using their
entire CPU capacity, but this did not hurt Site X's ability to meet
scalability or performance requirements.
A couple potential problems do reveal themselves in this test. First,
xml-13 performed poorly after about 300 concurrent users. The site
itself has already about reached its throughput bottleneck at this
point. Secondly, after about 200 users, the consumer-site pages started
experiencing errors, primarily in the form of connection resets or
timeouts. These problems occurred far above the required throughput but
may be areas which should be investigated in the future.
Following are some graphs highlighting the results from this test:
Figure 24 Final Site-Wide Test Overall Throughput: Pages/Second
Figure 25 Main Foreground: Performance Vs. Users
Figure 26 XML Foreground: Performance Vs. Users
Figure 27 Main Foreground: Error Rate Vs. Users
Figure 28 XML Foreground: Error Rate Vs. Users
Figure 29 XML Background: Performance Vs. Users
Figure 30 Trace Background: Performance Vs. Users
Figure 31 Netras Memory Vs. Users
The long-term test's scripts ran for a total of 1,469,016 iterations
with 5,152 total errors for a 0.35% error rate. Thus, for all intents
and purposes, the long-term test ran without any problems.
ArsDigita noted the memory on each Netra at the beginning and end of
the stability test. The results are as follows:
Table 20 Long-Term Stability Test Memory Usage
Based upon these results, Site X proved fairly stable. But, under
constant load, the AOLservers do slowly grow in memory usage-indicating,
perhaps, some kind of small memory leak. ArsDigita recommends that Site
X do the following with its production setup to deal with this memory
growth:
Following these steps should help ensure that Site X runs stably over
time.
As a result of this work, Site X currently far exceeds its scalability
requirement of 6.5 pages/second while running on less hardware than the
actual production site.
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
