Category Archives: Performance techie

“Why are you writing this Q & A?”
“Because people have questions, which are unanswered about ‘Bits of performance testing’ . ”
“Like?”
“The use of drawings and perfornance test it self.”
“So I can send in questions?””
“Sure on this page. Why not?”

“Hey, I am the one who is supposed to ask questions.”

Plan
“You used a lot of pictures. Is this not a waste of time?”
“In my blog post I drew a picture about the customer journey. This led to the conclusion that the Wifi network should be split in a private and public ones. This is not a waste of time.”
“I agree. But there were other pictures, which did not have that big impact.”
“That is right. But the drawing of picture takes minutes. Implementing the wrong performance scripts takes days.”

“You used SFDIPOT instead of another heuristic., FIBLOTS”
I am aware that this heuristic exists. I also know that a performance tester made it. And I just forgot it.
The reason I chose to pick SFDIPOT is to use this heuristic in another context. I learn by taking small detours. What if I do it in another way? My main reason was that I wanted to write about performance test in another way.
This kept my spirit high and extended my Circle of Comfort. That is a comfortable thought.”

“Your story about the performance test is sometimes difficult to follow. Why did you write a nonlinear story?”
“Testing is an activity which is unpredictable. I can find bugs on the strangest moments. This can trigger other ideas.
In this blog post I tried to describe what is going on in my head.”
“How can you sell this to your boss?”
“Just ask her or him, whether business cases are also written in a linear way?”

“I miss disk storage That is an important resource to watch.”
“You are right. I missed that one.”
“How do you know that this is a good load profile?”
“Some data have to be gathered. Think about log files and analytic stuff. The challenge is not to confuse frequency with resource usage.”
“Would you please explain that?”
“If 1000 users look to a simple web page, then almost no resources are used. If 1 user asks all articles in stock, then a lot will happen. The database is bring queried and much data is moved over the network. So look for resource usage.”

“So I only have to focus on user actions with heavy resource usage?”
“That is only possible with a very simple web site. Sometimes web sites or programs on the backend do not remove their garbage.”
“What do you mean with garbage?”

“Suppose you have a cinema web site. The purchasing department wants to know what kind of drinks and snacks are ordered in advance. Suppose all the results of the queries are stored. It might be interesting for their department and it should be stored on one of their systems. But not in a module of the web site.

Another thing is to simulate the customer. Use the customer journey. A customer does not only buys cards, snacks and drinks. She or he will also collect them. This lead to click paths, how does the user maneuver through the system: which screens will be visited and which options are used?”

“So if you have a lot of data, what do you do?”
“The challenge is to find patterns. Joe Common is more like this:”
“Why did you not draw the payment page?”
“It is important. I left it out because of the page size. My plan was to sketch out a rough journey. Just wait a few pics.”

“I suppose you left out snacks out for the same reason. Smart ”
“Cheers”

“Why does he go one screen back?”
“That is hard to say. You can make some assumptions or hypotheses: the price was too high. Or the wrong movie was selected. You can always ask some questions to validate the assumptions.”

“Hey there is no deal. What’s wrong?”
“Most of the yearly cinema visitor is looking around or scouting. Were there any changes in the web site? Which movies can I see? ”

“What is going on here?”
“A children birthday party.”
“That looks like a lot of fun.”

“What are you showing me?”
“This is the number of screens and locations which were visited, and their corresponding numbers.”
“The bars in the graph are getting lower. Is this not frustating for marketing? ”
“Actually this is normal. This is a sales funnel.”

*It is like commercials: not everyone buys what they see on the television. They do not buy everything they see in the shop. You actually want people to do things which bring profit. The Call To Action or CTA for this web site is buy tickets, drinks, and snacks.”

“Okay Mr. Monologue. I’ve got a few questions.
How did you make that funnel?*
*Like this.*

*Why do you call this an extended sales funnel? Your CTA is buying tickets, drinks and snacks.*
*But there are two moments to buy drinks and snacks: in advance and on location.*

“If I had put the CTA in the first funnel picture, it would be cut off right under. By delaying this action I got a better view on the situation.

The second CTA is successful, if the first CTA.is good. John Common is not willing to buy extra drinks and snacks after a slow performance of the web site.”

*Are you writing that models can confuse people?*
*Yes, they do. Let me give another example.*
*Be my writer.*
*if you go twice to a cinema, you go twice through the sales funnel.

*So you should take repeat customers in account. *

For some businesses good customer contact is vital. In ‘Delivering Happiness: A Path to Profits, Passion, and Purpose’ it is called creating a moment of Wow.”

“And if things go wrong?”
“Just watch social media. Several companies have web care teams who actively look on the internet and engage with grumpy customers.”

“I would be interested in the numbers. So how would it look like?”

“Wait. This concept looks better than the draft one.”

“Are these estimated numbers for all visitors?”
“That is a good question. It is only for the website visitors, who also ordered drinks or snacks in advance. So I have to add a another set of numbers for the website visitors, who did not order drinks or snacks in advance and the visitors who pass the web site.”

“What is no showup?”
“That is a situation, when a customer does not get, what she or he paid for. He or she is ill. She or he found another group of friends and skipped the cinema.”

“This is strange: there are people who go to the shop without the ticket.”
“This is a common group mistake. A group of people at the cinema asking each other: “Who bought the tickets?””

“Why did you show the draft version?”
“I do not jump to solutions. I want to share my way of thinking. My thoughts and my pictures.”

“But those reversed trees with numbers is not a good starting point for scripting, I assume.”

“You are completely right. All that number crunching can take the attention away. If we have 3525 visitors coming in, then this must be translated in scripts somehow. We have to look at the chances.”
“But this is a cinema web site. It is not a casino.”

“Just have a look.”

“What does 70 % at Movie mean?”
“This means, that there is a chance of 70 % that a visitor goes from the Select Movie Window to Select Ticket Movie. You can call these click paths.”

“You are basically stating that the software has to roll a dice to determine the next step.”
“Yes. There is no single ideal path through the system. I already showed pictures of customer journeys.”

“Wait a sec.”
“Waiting.”
“The web site can easily be tested using a standard performance test, but a shop with real people is difficult. Do we need to test them? ”
“What is the system under test?”
“The web site and the shop.”
“Can you explain it to me?”
“The software supporting the web site and the shop are connected.”

“What does hold you back?”
“How can you test it?”
“I have not much experience with this particular situation. My suggestions would be: make a special interface for the shop just for performance test, mock the users in the shop, or use actual human beings. The latter option I call hybrid testing.”

“”Do you have experience with hybrid testing?”
“Yes, I have. For a web site we had to do a performance test. The visitors were simulated by the software. The web masters were real people including your interviewee. The actions of the web masters were too complicated to script, so we did testing manually. For more info there is a Dutch presentation, A performance test with a tail.”

“What is a good way to determine whether the performance is good enough?”
“A lot of people want a fixed number like a certain response time. Like the application will respond within 6 seconds. I’ve got an example.
In a performance test percentile is used: at least 90 % of the users have a response time of 6 seconds or less, if they do a specific action. This reduces discussions like:
“40 % has a response time of 5.3 seconds. That is quite good.”
“50 percentile has a response time of 6 seconds and the limit is 80 percentile.

Another thing to consider is to determine the worst realistic conditions for the system under test.”
Preparation
“Can automated tests be used with performance tests?”
“I assume you are talking about automated functional tests.”
“Yes.”
“I once got this request and it does not solve the problem. Is your manager writing a business case, a roadmap, and items for the back log for the same release at the same moment? I guess not. Every artefact has his own purpose.
But I wrote a blog post about the combination of automated tests and performance test.”

“When can the scripting start?”
“It is good to have a testable version before the final one.”
“But performance issues might still be in there. Also the interface can still change.”
“Change is always a companion of a tester. What I mean is that a tester should focus on things which will not change on long term. The technical basic components will probably not change. Test data is always needed. Data about the usage of the system is quite stable. Etc. ”

“There are always discussions about the test environment:
“We just put a part of the database in the test environment. “”
“Yes, a copy of the production environment is very expensive.
And yes, a performance test on a reduced environment can give a false impression.
I have a blog post about a huge test environment: Do you really need it?”

Execution or performance
“What would you advise during a performance test?”
“Have a hot line. If things go out of control, make sure everyone knows who to call or mail. Or your favourite communication style.”

Report
“So what should a test report contain?”
“In the report it should be described whether the performance criteria have been met. Percentile graphs are really cool. It is all about a special point in the graph:”

“I would like to see some real life graphs.”
“Sure be my guest. They are on slides 14 and 15 of the presentation, A performance test with a tail .”

“What about the resources?”
“That is a good question. This should be covered. A nice pic would be great, but that is something for another time.”

“Thanks for answering.”
“Thanks for questioning. And thanks for reading.”

The story so far: 3 blog posts ago I introduced a web site which offered the new option to buy drinks and snacks in advance for a cinema visit. This is an imaginary web site. Why should you need an extra drink? For the desert scene in Monsters Unlimited. Also imaginary stuff.

1 blog post ago an anxious project leader dropped by for a performance test for the next day. Somehow it ended up as a cliff hanger for … [Some dots] SFDIPOT in performance test.
Now I can continue to write about my standard talk for project leaders about performance tests. Let me add some flavour. Stirred, not shaken.

SFDIPOT is basically used to determine new angles. Up till now I have not used it for performance test. I could make a rough picture about the system. Why not? Visual testing anyone?

This is Structure. What are the building blocks?

The customer uses a web site to buy tickets. There is an extra module to handle orders of drinks and snacks. So an extra flow is added.
Decades ago I would be happy with this picture. But I know now that I would miss important information.

Now I have a look to the Function or what the system is expected to do.

“Hey, what’s that?”
“The marketing department is keeping a close eye on revenue.”
“I understand that they want to increase it, but this can impact the performance of the system.”
“You are right, so analysis takes place outside this system.”
“Okay.”

“But the counters are inside the system.”
“Sure enough.”

Another way to look at functions is to visualise it.

”Okay I do understand the pictures, but what is that rectangle with the heart?”
”It is the toilet.”
”But it has nothing to do with web site.”
“But it can impact the cinema visit. (: P)
This picture offers another perspective. You might call a visual user journey. ”

Data is all the information which exchanged in the system.
“You probably have a link to the purchasing department.”
”Yep and this one is also handled outside the website.”

”If I zoom in, I see tickets, drinks and snacks going this way. It’s first just raw data.
I miss some money. ”
”That is no problem. A supplier takes care of the payments. ”
”Ok.
What about all those snacks and drinks? ”
”Those are bar codes and numbers.”

Interfaces are ways to interact with the system. So …
”I am curious whether the system can handle streaming film trailers.”
”Sure no problem. The web site has two servers: one is completely dedicated for streaming. ”

”Do you have free WiFi?”
”Sure, but …”
”What happens, if a lot of visitors start to watch a movie trailer on their mobile?”
”Wait. You are telling me that my internal network could be slower. I did not think about this option. We need to take measures like a separate network for visitors. ”
”Yes.”

”How did you figure this out?”
”I was thinking very fast.
Like . ”
”Could you please slow down?”
“# $ €
|:| |:|”
”Man, you think fast. Let me see: # stands for numbers, €$ for money and |:| |:| for … ”
”For movie. ”
”Okay. I know a lot of symbols, but this one is new. ”
”I just figured that one out.
In the past films were on celluloid rolls. There was picture and some holes to transport the roll. : are the holes. You know like this. ”

”I still not get it.
How did this image popped up? ”
”I looked to my user journey.
So my customer has picked up a ticket, drink, and snack? What can she or he do?

This.”
[Hand moves to a pocket and takes out a smartphone.]
“Wow.”
“That’s why it is so important to use multiple senses to plan a proper test.
This movement [hand collecting phone] is muscle memory. ”

Now we have a look at the platforms or where is the system used?

“No problem. The web site and the connected systems work on one platform. It is used frequently. Known technology.”
“But the browsers can be used on smartphones and tablets with different browsers and different operating systems.”
“This is also known technology.”

Operations is about how the system is or will be used. This could lead to the following plausible talk.

“After our last talk I talked with the marketers. They noticed the following patterns:

Most tickets bought in advance are sold at least one day in advance.

Most tickets bought in advance are sold for the weekend.

The week patterns are almost the most same in the month except for the first week. “

“First week. Can you explain it?”
“Salary is paid in the last week of the month. People pay more for entertainment, if there is more money on their bank accounts. ”

“Of course there are exceptions on the months. December and the summer months are way higher because of the holidays. Watching a movie is a nice way to spend holiday hours.
There are of course still unexpected and expected mega ticket selling movies.”

“Do you also take discounted movie tickets in account?”
“Yes, this lead to some increases. About 5 to 10 percent.”

“I would suggest to use the December holiday as the standard load profile. This way the system can handle peaks. To be sure a percentage of 10 % is added.

“Is your customer also interested, when the system breaks down?”
“We already took precautions: the modules for the tickets, drinks, and snacks are placed in the cloud and their resources are automatically scaled.”

“So everything is covered.”
“It looks good. It is important to know which factors have a major impact on the performance like CPU or chip, RAM or memory and use of network.”

“Of course you are interested in the change of use of the resources.

“You go to the ticket counter. You get a ticket for the movie and vouchers for snacks and drinks.

If I look to the user journey I miss a step between the ticket counter and the shop counter?”

“Wait the lines are wrong, it should be like this:”
“The ticket counter should have access to the drinks and snacks ordered.

“Okay I update the pic later.”

“If I go with 5 excited kids, I end up with 6 movie tickets, 6 snack vouchers and 6 drink vouchers. Double the number of drinks for Monsters Unlimited with the desert scene.”

That is a lot of printing.
So I have 24 cards. This is bigger than the set of business cards I end with after a conference. And the set of followers on Social Media. Todo cursive.

So the network can handle the extra information for the printers.”
“Yes, it is insignificant number of bytes.”

How much time does it take to print ’em all?”
“0.5 sec a ticket.”
“So I have to wait 12 seconds.
Wait 180 people go to a movie. That makes 4 x 180 pieces of paper in worst case (1 movie ticket, 2 drinks, and 1snack person). Only the printing will take 4 x 180 x 0.5, which makes 360 seconds. That is exactly 1 hour.
Of course people will not arrive after each other, but like this. If I add the number of voucher for drinks and snacks, I get …”

“Could you please stop?”
“Did I do some miscalculation?”
“No. I remembered something else. It took us months to determine the right number of ticket counters. My customer had to increase the number of counters. In this case it would be by 4. That is not pretty.

This is out of my scope. I am only responsible for IT.”
“Printers are also IT. The system is not a single web site. Look at the pictures.

You might consider to reduce the number of vouchers. There is enough room on the paper for other things.”

The following item of interest is about the relationship between the system and Time. Yes finally.

A few months ago there was a question about ambiguous requirements. As a performance test coordinator I often heard: “It is fast enough.”

My solution was to perform a performance test on the old and the new system and compare the results.

Sometimes I was lucky and I had some real acceptance criteria. Like 95 % of the response time of function X is under 20 seconds. And a completely filled web site. The same content might give the same response. Or the other way around: almost no content is almost no delay.

“Now let me look at the response times. In the current system these are the response time for the tickets page, the payment module, and the ticket counter.
That payment module is swift and it is outsourced.
For the drink and snack page and the shop counter the times are added. Notice, that there is night between the day and the cinema visit.

I can add the numbers for one movie. Then I get these numbers Fir two movies on an evening there is reassuring low response time.

“Let me guess your system is fast enough to handle a blockbuster evening.”
“You bet.”
“I don’t take bets I can lose.

But what about this?
Snacks and drinks ordered in advance are collected by the cinema people.
There is probably an acceptance criterion that snacks and drinks can be collected by the customer within 2 minutes after collecting the ticket.
So if I come with 5 kids collecting 6 snacks and 12 drinks, someone has to be quite fast. But there might be more people needed who collect their drinks and snacks.”

“That will be a problem. But I am only responsible for IT.”
“What about the people who have to work it. And your customer who depends on order drinks and snacks in advance page?”

“One apple juice please.”
“Huh. Oh yeah, you won the bet.”
“No, I am thirsty after all that talking.”
“You ain’t seen the desert scene of Monsters Unlimited.”

[Small update: there is a Q&A a few blog posts later about this very blog post. You can even join the party.]

A colleague once told about a strange request. Now let me imagine the whole situation.

“Could you perform a performance test tomorrow?”
“If this is an unknown web site it takes 4 weeks: 1 week for test planning, 1 week for scripting, 1 week for execution of the scripts, and 1 week for test rest reporting.
If there are a few minor changes, it takes 2 weeks. If the test scripts and test data do not have to be changed, it will be one week.”

Now imagine the face of the project leader. Poor fellow.

Now I change the request of the project leader.
“I have a web site which must be tested on performance. I am exasperated.
I have a need for participation.
Would you please perform a performance test on my web site tomorrow?”

I could add a little more detail.
“The web host provider needs a test report of the web site, that the performance is good. Within 3 days the web site must go live.”

This is a decent request and it is almost impossible to say: No.
Really? I have to fit at least a week of testing in 1 day. It is fair to say that this is not possible. It is not a reasonable request, so it is better to decline.

Then the dance can start. Let me first focus on the need of the project leader.
“I notice that you do not like my answer. Am I right?”
“Yes, you are right.”
“And you have now a need for clarity?”
[after a slow nod]
“I have a need for cooperation.
I am willing to cooperate, but a test plan takes one week.”

Now the dance is focused on the performance tester. It is her or his turn to explain. Almost.

“You extended a cinema web site with an option to order drinks and snacks in advance. It is just one page.”
“Can you not have the computer just click on the page to see what happens? That’s what computers are good at.”

“Would you please let me explain?”
“Yes. Sure.”
“If I would go with 2 friends I would buy three drinks and 3 snacks. Unless it is Monsters Unlimited with the desert scene.
It takes at most 15 clicks for 1 visit.
But there more than 3 visitors for the movie.

Now many people can look to the movie?”
“180. “
“What is the average number of tickets in 1 order?”
“I do not know.”
“Let’s assume it is 2. Then 90 different people will go to the drinks and snack page. So I have 90 sessions for ordering.“

“I can check with marketing and we can go.”
“But those 90 orders are not placed on the same moment.”

At that moment the project leader would look for another performance tester willing to help him.

My colleague had lost a customer. In this case there is no problem.

What happened?
A lot of talking.
“Would you please be more specific?”
“Sure.”

There were needs and feelings mentioned. The performance test coordinator focused on the project leader. And he explained a lot. A lot of talking. But no project though.

It is also possible to focus first on the test coordinator instead of the project leader.

Ready? Set. Speak.

“How can I help you?”
“I talked to my customer. She agreed to postpone the go live date of drink and snack page in the cinema web site.”
“The last time I had not enough time to explain why a performance test takes this long.
I was disgruntled.
I have a need to be understood.
Would you please let me explain, why the performance test takes time?”

More than 5 years ago in my real life I talked with my boss about a performance test. I had calculated that the investment in a performance would be earned back in 3 months. According to some financial standards this is quite good. But budget was too tight. My boss just asked, whether I had signed anything. This was not the case, so he advised to stop.

Other things that happened
A week ago a fellow TestBash speaker wrote a personal story about her needs which were not fulfilled. Poor fellow. There was a lot of emotion in the post. So she quit. And found joy back in little things.

Now you might doubt whether people would really read this story. But there were. Reactions came in. People who were touched by her story. Feelings and needs can be a big + or PLUS in communication.

For the people who are really struggling:

Describe the situation in an objective way.

Go to the feelings inventory.And determine your feeling (watch the flags for other languages available)

There are other ways to handle difficult situations. One of my most read blog posts last year was about being fired. I unconsciously used stoicism. My need was clear: a need for support my family. But I had no bad feelings on that moment. I neglected the negative ones.