WEBVTT
00:00:00.000 --> 00:00:02.370
>> Hi, I'm Donavon Brown,
00:00:02.370 --> 00:00:04.350
Principal DevOps
Manager for Microsoft.
00:00:04.350 --> 00:00:06.190
Load testing is
a great way to find
00:00:06.190 --> 00:00:08.590
issues in your software
before your users do.
00:00:08.590 --> 00:00:11.320
Make sure as your application
grows in popularity,
00:00:11.320 --> 00:00:13.165
it is ready for
the additional load.
00:00:13.165 --> 00:00:15.385
There are several ways
using Azure,
00:00:15.385 --> 00:00:16.930
Visual Studio Team Services,
00:00:16.930 --> 00:00:19.615
and Visual Studio, to get
started with load testing.
00:00:19.615 --> 00:00:20.890
Let's jump right in and let
00:00:20.890 --> 00:00:22.555
me show you how you get started.
00:00:22.555 --> 00:00:25.710
If your application is already
being hosted in Azure,
00:00:25.710 --> 00:00:27.030
the quickest way to get started
00:00:27.030 --> 00:00:28.935
is just to go to
that app service.
00:00:28.935 --> 00:00:30.810
On the app service blade itself,
00:00:30.810 --> 00:00:33.040
is a way for you to start
a performance test.
00:00:33.040 --> 00:00:36.015
Just scroll all the way down
to the developer tools,
00:00:36.015 --> 00:00:38.730
and here you're going to
see Performance Test.
00:00:38.730 --> 00:00:40.200
When you click on
"Performance Test",
00:00:40.200 --> 00:00:41.790
it's going to give you
the ability to create
00:00:41.790 --> 00:00:43.470
a very quick and easy
performance test
00:00:43.470 --> 00:00:45.060
right here in the Azure Portal.
00:00:45.060 --> 00:00:47.640
To get started, you just click
on "New", and from here,
00:00:47.640 --> 00:00:50.430
you get to choose the URL
for which you want to test.
00:00:50.430 --> 00:00:53.040
This happens to be a URL
for a web service.
00:00:53.040 --> 00:00:54.960
But if I already hit
this URL directly,
00:00:54.960 --> 00:00:56.760
what you are going to
actually see is a web page.
00:00:56.760 --> 00:00:57.910
That's not what I want.
00:00:57.910 --> 00:01:01.200
What I want to do is drilled
into the API.people,
00:01:01.200 --> 00:01:02.490
and when I click on this,
00:01:02.490 --> 00:01:03.810
you're going to see
that it's a web service
00:01:03.810 --> 00:01:05.550
bringing back some XML results.
00:01:05.550 --> 00:01:07.335
That's what I actually
want to load test.
00:01:07.335 --> 00:01:08.850
I'm simply going to copy that,
00:01:08.850 --> 00:01:10.980
and then paste that
as the URL that I
00:01:10.980 --> 00:01:13.440
want to go and to
do a get against.
00:01:13.440 --> 00:01:16.615
When that is done, I'm then
able to give the test a name,
00:01:16.615 --> 00:01:19.950
decide where in the world
that I want to run the test,
00:01:19.950 --> 00:01:22.230
and then also what the
load that I want to
00:01:22.230 --> 00:01:24.570
run on this particular test,
and for how long.
00:01:24.570 --> 00:01:26.520
I've already done this,
and what you'll see
00:01:26.520 --> 00:01:28.745
is you'll get the test results
just like this one.
00:01:28.745 --> 00:01:31.145
These are the results of
a load test running with
00:01:31.145 --> 00:01:34.455
250 concurrent users
against that same endpoint.
00:01:34.455 --> 00:01:36.180
You can see the response time,
00:01:36.180 --> 00:01:38.370
you can see how many failed
requests that we had,
00:01:38.370 --> 00:01:40.035
and get a quick overview on,
00:01:40.035 --> 00:01:41.655
what's the health
of my application?
00:01:41.655 --> 00:01:42.810
And will I be able to
00:01:42.810 --> 00:01:44.850
sustain the load that
we're looking for?
00:01:44.850 --> 00:01:46.080
Also down here at the bottom,
00:01:46.080 --> 00:01:48.500
you'll be able to see
the CPU time and memory usage to
00:01:48.500 --> 00:01:51.305
see if the scale unit that you
have is big enough or not,
00:01:51.305 --> 00:01:53.340
and if not, you can decide
to scale it up right
00:01:53.340 --> 00:01:54.390
now and make sure that
00:01:54.390 --> 00:01:55.770
you're going to be
ready for the load.
00:01:55.770 --> 00:01:57.555
This information is not only
00:01:57.555 --> 00:01:59.790
surfaced here in
the Azure Portal,
00:01:59.790 --> 00:02:02.685
it's also serviced inside a
Visual Studio Team Services.
00:02:02.685 --> 00:02:04.245
If I have to click
on this link here,
00:02:04.245 --> 00:02:05.895
I'll be taking to
the account that's
00:02:05.895 --> 00:02:07.800
backing this particular
application.
00:02:07.800 --> 00:02:11.495
From here, I can simply
click on my load test,
00:02:11.495 --> 00:02:13.770
so I'll go to "Test",
and then from here,
00:02:13.770 --> 00:02:15.090
I'm going to click on load test,
00:02:15.090 --> 00:02:15.705
and I'll be able to see
00:02:15.705 --> 00:02:17.130
all the load tests
that I'm running
00:02:17.130 --> 00:02:20.235
including the load test that
come from the Azure Portal.
00:02:20.235 --> 00:02:22.680
If you remember the name
of that particular test,
00:02:22.680 --> 00:02:25.050
which I believe is this one here,
00:02:25.050 --> 00:02:28.140
we can now see the same results
that we saw inside of
00:02:28.140 --> 00:02:29.430
the Azure Portal directly from
00:02:29.430 --> 00:02:31.740
inside a Visual
Studio Team Services.
00:02:31.740 --> 00:02:33.120
If I were to open this up,
00:02:33.120 --> 00:02:34.260
you'll get very similar charts.
00:02:34.260 --> 00:02:36.750
So, let me know again the
average response time,
00:02:36.750 --> 00:02:38.880
the user load, how
long the test ran,
00:02:38.880 --> 00:02:40.420
and if there were
any errors or not.
00:02:40.420 --> 00:02:42.360
It also shows me
my slowest request,
00:02:42.360 --> 00:02:43.410
of course we only made one,
00:02:43.410 --> 00:02:45.030
so that will be
the slowest request.
00:02:45.030 --> 00:02:46.500
Now you can also create
00:02:46.500 --> 00:02:48.360
that exact same test
from here but you'll
00:02:48.360 --> 00:02:49.920
have more control if you do it
00:02:49.920 --> 00:02:51.915
from inside a Visual
Studio Team Services.
00:02:51.915 --> 00:02:53.940
For example, not
only can I create
00:02:53.940 --> 00:02:57.540
the same type of
User-Base Test with the URL,
00:02:57.540 --> 00:02:59.890
but I can also use
a Visual Studio Test,
00:02:59.890 --> 00:03:02.370
which will learn how to
create here in just a moment.
00:03:02.370 --> 00:03:05.370
I can use a archive file
from Fiddler,
00:03:05.370 --> 00:03:07.380
if you've been using that
to track your traffic.
00:03:07.380 --> 00:03:08.460
Or I can even use
00:03:08.460 --> 00:03:10.485
a Apache Jmeter, if you're
using that instead.
00:03:10.485 --> 00:03:12.270
But let's duplicate
the same test that we
00:03:12.270 --> 00:03:13.910
created just a moment ago and see
00:03:13.910 --> 00:03:15.570
that we actually have
more power when we do
00:03:15.570 --> 00:03:17.595
it inside a Visual
Studio Team Services.
00:03:17.595 --> 00:03:18.600
We're just going to
go ahead and give
00:03:18.600 --> 00:03:20.250
this a temporary name,
00:03:20.250 --> 00:03:22.050
and I can add additional URLs
00:03:22.050 --> 00:03:23.280
which I could not do before.
00:03:23.280 --> 00:03:24.465
It was only one URL,
00:03:24.465 --> 00:03:26.040
and I could only do a get.
00:03:26.040 --> 00:03:28.200
But here inside
a Visual Studio Team services,
00:03:28.200 --> 00:03:30.495
I can do a get, post, put,
00:03:30.495 --> 00:03:32.595
delete, all of
the different types of
00:03:32.595 --> 00:03:34.110
methods that I'd want to perform
00:03:34.110 --> 00:03:35.560
against my particular endpoint.
00:03:35.560 --> 00:03:37.380
If I were to choose
"Post", for example,
00:03:37.380 --> 00:03:38.820
I can come down here and give it
00:03:38.820 --> 00:03:40.430
a content-type and actually
00:03:40.430 --> 00:03:42.075
give in the information
that I'd want.
00:03:42.075 --> 00:03:44.295
In addition to that, I
can click on "Settings",
00:03:44.295 --> 00:03:46.095
which allows me to
set the duration,
00:03:46.095 --> 00:03:48.000
also change the load pattern
00:03:48.000 --> 00:03:49.755
that I want to apply
to my application.
00:03:49.755 --> 00:03:52.630
Do I want all the 250 people
to hit it all at once?
00:03:52.630 --> 00:03:54.480
Or do I want to
slowly build up to
00:03:54.480 --> 00:03:57.750
250 as the application starts
to become more popular?
00:03:57.750 --> 00:03:59.175
I'll do constant here for now.
00:03:59.175 --> 00:04:01.515
I can choose the number
of people, the warm up,
00:04:01.515 --> 00:04:03.600
and I can also control
the browser mix which I
00:04:03.600 --> 00:04:06.045
could not control from
inside of the Azure Portal.
00:04:06.045 --> 00:04:08.070
Another thing that I
can do is choose from
00:04:08.070 --> 00:04:10.185
a much larger list of regions.
00:04:10.185 --> 00:04:11.250
As you'll notice, this list is
00:04:11.250 --> 00:04:12.300
much larger than the list that
00:04:12.300 --> 00:04:14.340
we had inside of
the Azure Portal.
00:04:14.340 --> 00:04:15.600
If you've already
taken the time to set
00:04:15.600 --> 00:04:16.950
up your own physical rig,
00:04:16.950 --> 00:04:19.070
you could actually use
that rig here as well.
00:04:19.070 --> 00:04:21.450
But the advantages
of using the Cloud,
00:04:21.450 --> 00:04:23.410
really make creating your own rig
00:04:23.410 --> 00:04:25.020
a thing that we don't do anymore.
00:04:25.020 --> 00:04:27.255
Now, you get to use the power
of the Cloud to provision
00:04:27.255 --> 00:04:28.860
all the resources necessary to
00:04:28.860 --> 00:04:30.665
generate an enormous
amount of load,
00:04:30.665 --> 00:04:32.870
and then when you're done, get
rid of all those machines.
00:04:32.870 --> 00:04:35.160
Plus you're having those
physical machines yourself.
00:04:35.160 --> 00:04:37.050
Once you're done with that,
you're going to have a test
00:04:37.050 --> 00:04:38.820
very similar to
what we did before,
00:04:38.820 --> 00:04:40.260
and your results are going
to look just like they
00:04:40.260 --> 00:04:42.915
did when they came
from the Azure Portal.
00:04:42.915 --> 00:04:44.670
Here I can open up
one of my runs.
00:04:44.670 --> 00:04:46.620
Again, I can see
my average response,
00:04:46.620 --> 00:04:47.865
the number of users I used,
00:04:47.865 --> 00:04:49.770
how long I ran it,
and any errors that I
00:04:49.770 --> 00:04:53.115
had that failed to
that particular test.
00:04:53.115 --> 00:04:55.455
Now, you may have noticed that,
00:04:55.455 --> 00:04:56.700
as we move along,
00:04:56.700 --> 00:04:58.080
we're getting
more and more control
00:04:58.080 --> 00:04:59.535
over the load test that we run.
00:04:59.535 --> 00:05:02.070
There's one other way that you
can create your load test.
00:05:02.070 --> 00:05:04.875
One of them is directly inside
a Visual Studio itself.
00:05:04.875 --> 00:05:07.380
There is a solution type that
you can create where you
00:05:07.380 --> 00:05:10.365
can author the test
that you want to run.
00:05:10.365 --> 00:05:12.095
For it to open up this web test,
00:05:12.095 --> 00:05:14.155
you can actually see
the request that I made,
00:05:14.155 --> 00:05:16.090
and let me bring up
the properties for you.
00:05:16.090 --> 00:05:17.610
When you bring up the properties,
00:05:17.610 --> 00:05:19.470
you'll see that there's
so much more control.
00:05:19.470 --> 00:05:21.540
Not only do I have the
control over the method,
00:05:21.540 --> 00:05:22.980
but even the version of
00:05:22.980 --> 00:05:25.035
the transport protocol
that I want to use,
00:05:25.035 --> 00:05:27.000
the URL that I want
to actually hit,
00:05:27.000 --> 00:05:29.250
and I can even control
the headers that get sent.
00:05:29.250 --> 00:05:31.500
If I want to enable
Gzip for example,
00:05:31.500 --> 00:05:32.820
or if I want to make
sure that it sends me
00:05:32.820 --> 00:05:34.710
back XML vs Json,
00:05:34.710 --> 00:05:36.420
I get to control
all of the headers,
00:05:36.420 --> 00:05:37.905
which are really important.
00:05:37.905 --> 00:05:40.470
Another thing that's really
important is your ability to
00:05:40.470 --> 00:05:43.680
add validation to the request
and the response.
00:05:43.680 --> 00:05:45.690
In the previous test,
we were sending
00:05:45.690 --> 00:05:47.415
the request and if we
were getting a 200,
00:05:47.415 --> 00:05:48.865
we assumed that
everything was good.
00:05:48.865 --> 00:05:50.710
But what if I got
a 200 but the data
00:05:50.710 --> 00:05:53.380
inside of it was not the data
that I was expecting?
00:05:53.380 --> 00:05:55.000
We were not able to test that
00:05:55.000 --> 00:05:56.680
with the other two
methods but here,
00:05:56.680 --> 00:05:58.315
I can add a validation rule.
00:05:58.315 --> 00:05:59.710
Because I know I'm
going to be getting
00:05:59.710 --> 00:06:01.390
XML back, I can say,
00:06:01.390 --> 00:06:04.600
lets go and require that
there be a first name tag.
00:06:04.600 --> 00:06:06.520
Then that first name,
let's say for example,
00:06:06.520 --> 00:06:08.680
Jessica is the value that
I want inside of it.
00:06:08.680 --> 00:06:11.080
I can use these tags and
00:06:11.080 --> 00:06:12.970
these validation rules
to be able to verify
00:06:12.970 --> 00:06:15.100
that I actually got
the data that I wanted to.
00:06:15.100 --> 00:06:16.420
If we run it at load,
00:06:16.420 --> 00:06:17.890
weird things don't
start to happen.
00:06:17.890 --> 00:06:19.090
So, again, we're starting to get
00:06:19.090 --> 00:06:20.930
more and more power and control
00:06:20.930 --> 00:06:22.450
over the load tests
that we're running
00:06:22.450 --> 00:06:24.355
as we get closer to
a Visual Studio.
00:06:24.355 --> 00:06:26.305
Once you have your test authored,
00:06:26.305 --> 00:06:28.960
you can go off and
create a load test.
00:06:28.960 --> 00:06:30.790
A load test allows you to control
00:06:30.790 --> 00:06:32.425
what computers are used,
00:06:32.425 --> 00:06:33.760
what addresses are hit,
00:06:33.760 --> 00:06:35.770
what browser mix that you want,
00:06:35.770 --> 00:06:37.555
what network mix that
you want as well.
00:06:37.555 --> 00:06:39.160
If you wanted to pretend
00:06:39.160 --> 00:06:40.210
that some people were coming from
00:06:40.210 --> 00:06:43.930
a phone using 3G or 4G versus
coming from a landline,
00:06:43.930 --> 00:06:45.100
you can do that to make sure that
00:06:45.100 --> 00:06:47.260
the responses worked the
way that you want to.
00:06:47.260 --> 00:06:49.090
You also have the ability to set
00:06:49.090 --> 00:06:51.250
thresholds which are
really important.
00:06:51.250 --> 00:06:52.765
Response times are great,
00:06:52.765 --> 00:06:53.930
but what if I wanted to check
00:06:53.930 --> 00:06:56.020
other metrics on my application?
00:06:56.020 --> 00:06:57.580
Here under the 'Load
Test" settings,
00:06:57.580 --> 00:06:59.590
I'm able to drill into
any item that I want
00:06:59.590 --> 00:07:01.805
to and actually set a threshold.
00:07:01.805 --> 00:07:04.090
A threshold is a value
that I want to
00:07:04.090 --> 00:07:06.490
be watched during
the execution of my test,
00:07:06.490 --> 00:07:08.995
and if those values
are ever eclipsed,
00:07:08.995 --> 00:07:10.510
I'm able to then be notified
00:07:10.510 --> 00:07:11.845
that at this point in your test,
00:07:11.845 --> 00:07:13.450
we had a threshold violation.
00:07:13.450 --> 00:07:16.350
You weren't able to sustain
as many simultaneous users,
00:07:16.350 --> 00:07:17.905
maybe your response time went up,
00:07:17.905 --> 00:07:18.985
your error accounts went up,
00:07:18.985 --> 00:07:21.280
but you are now in
complete control of what's
00:07:21.280 --> 00:07:22.750
important to you and what needs
00:07:22.750 --> 00:07:24.535
to be monitored during your test.
00:07:24.535 --> 00:07:25.900
Once this is done,
00:07:25.900 --> 00:07:27.980
you then execute the tests
like we did before,
00:07:27.980 --> 00:07:29.080
and we're going to be utilizing
00:07:29.080 --> 00:07:30.700
the resources that are in Azure,
00:07:30.700 --> 00:07:32.185
to generate the load for you.
00:07:32.185 --> 00:07:34.030
If you want, again,
you can actually use
00:07:34.030 --> 00:07:35.950
your own laptop for a small load,
00:07:35.950 --> 00:07:37.270
or set up your own rig,
00:07:37.270 --> 00:07:38.800
but I would encourage
you to make sure,
00:07:38.800 --> 00:07:39.850
you really don't want to use
00:07:39.850 --> 00:07:41.440
the stuff that's in the Cloud
because it's really,
00:07:41.440 --> 00:07:43.120
really nice, and it
makes life so much
00:07:43.120 --> 00:07:45.460
easier than you have in
the provision a rig yourself.
00:07:45.460 --> 00:07:47.125
Once the test is done,
00:07:47.125 --> 00:07:48.340
you'll be able to see all of the
00:07:48.340 --> 00:07:49.720
results like we did before.
00:07:49.720 --> 00:07:52.090
But we get much more
detailed results when
00:07:52.090 --> 00:07:54.655
we actually run them
inside a Visual Studio.
00:07:54.655 --> 00:07:56.050
Using this full testing,
00:07:56.050 --> 00:07:57.100
I can see the thresholds that
00:07:57.100 --> 00:07:58.870
were violated, and
as you can see,
00:07:58.870 --> 00:08:00.040
I set a threshold to see what
00:08:00.040 --> 00:08:01.920
my requests per second
were going to be,
00:08:01.920 --> 00:08:03.970
and if they ever
dropped below 100,
00:08:03.970 --> 00:08:05.170
I wanted to be alerted,
00:08:05.170 --> 00:08:06.580
and at one point during my test,
00:08:06.580 --> 00:08:08.740
they dropped to 83.26,
00:08:08.740 --> 00:08:10.705
which let me know
that at some point,
00:08:10.705 --> 00:08:12.610
my infrastructure was not able to
00:08:12.610 --> 00:08:15.310
sustain the same number of
users that I wanted to,
00:08:15.310 --> 00:08:16.720
and I might want
to go investigate
00:08:16.720 --> 00:08:18.265
this and figure out
what's going on.
00:08:18.265 --> 00:08:20.140
Any threshold violations
that might have
00:08:20.140 --> 00:08:23.770
happened would have been shown
for me here automatically.
00:08:23.770 --> 00:08:25.900
I can also go back in and
00:08:25.900 --> 00:08:27.160
look at the performance results,
00:08:27.160 --> 00:08:28.870
download the details,
and see all of
00:08:28.870 --> 00:08:30.040
the information
that was happening
00:08:30.040 --> 00:08:31.450
during the execution of my test.
00:08:31.450 --> 00:08:33.265
Now, load test is great
00:08:33.265 --> 00:08:35.560
as a one of that you might
want to go off and run,
00:08:35.560 --> 00:08:36.850
but being a DevOps guy,
00:08:36.850 --> 00:08:38.410
I want to figure out
how I can do this all
00:08:38.410 --> 00:08:40.405
the time throughout
my entire pipeline.
00:08:40.405 --> 00:08:42.990
What's really nice is using
Visual Studio Team Services,
00:08:42.990 --> 00:08:45.100
you can take the information
that I just showed you,
00:08:45.100 --> 00:08:46.240
and incorporate it into
00:08:46.240 --> 00:08:47.995
your build in
your release pipeline.
00:08:47.995 --> 00:08:50.500
I'm going to take you to
an existing released pipeline,
00:08:50.500 --> 00:08:52.720
and I'm going to show you
the task that you can use to
00:08:52.720 --> 00:08:55.405
run these tests during
your actual release.
00:08:55.405 --> 00:08:57.340
In doing so, you no
longer have to stop
00:08:57.340 --> 00:08:59.200
what you're doing and
manually run them.
00:08:59.200 --> 00:09:00.910
Your release is going
to tell you if you've
00:09:00.910 --> 00:09:02.770
made any detrimental changes
00:09:02.770 --> 00:09:04.540
to the performance
of your application
00:09:04.540 --> 00:09:06.925
while you're deploying your code
through your pipeline.
00:09:06.925 --> 00:09:08.380
A lot of people do this during
00:09:08.380 --> 00:09:10.000
their QA or
00:09:10.000 --> 00:09:11.870
their staging environments
which should look very,
00:09:11.870 --> 00:09:14.135
very similar to
their production environment.
00:09:14.135 --> 00:09:15.490
In doing so, they're able to
00:09:15.490 --> 00:09:17.260
determine if the changes
that they've made,
00:09:17.260 --> 00:09:18.595
have had a detrimental effect
00:09:18.595 --> 00:09:21.130
on their actual performance.
00:09:21.130 --> 00:09:23.740
As you can see here on
my release pipeline,
00:09:23.740 --> 00:09:27.025
if I go to the QA environment
and look at the task here,
00:09:27.025 --> 00:09:28.225
I actually happen to be running
00:09:28.225 --> 00:09:30.970
a performance test as
part of my release.
00:09:30.970 --> 00:09:32.860
There's other tasks
that I can use as well.
00:09:32.860 --> 00:09:36.550
So, if I come here and I
simply search for load test,
00:09:36.550 --> 00:09:39.520
you will see that there are
several test tasks that I
00:09:39.520 --> 00:09:42.930
can use to run load test as
part of my release pipeline,
00:09:42.930 --> 00:09:43.990
allowing me to make sure
00:09:43.990 --> 00:09:45.070
that none of my code
has actually may
00:09:45.070 --> 00:09:48.435
had a detrimental impact
on my performance.
00:09:48.435 --> 00:09:51.015
Before I go, I want to
encourage you to visit
00:09:51.015 --> 00:09:55.040
docs.microsoft.com to learn
more about load testing.