| Author |
Message |
Kaj S. Laursen
Guest
|
 Posted:
Thu Sep 01, 2005 4:16 pm Post subject:
Testing performance of storage |
|
|
Hi
I generally find it hard to find data about the performance of storage - it
seems that this is something that vendors perfer not to advertise, except
when they advertise something like a number of IO/s that are obviously when
it's to/from cache.
Some do have tests that show the performance in things like MMB2 score, but
in often in configurations that make it very hard to compare the results
between brands. Very few have tests with tools like IOmeter, JetStress or
SQLIO. Why is that - is it because the tests I mention are not the right
tools for performance tests? Is it because in general there is not much
difference in the performance if it's a comparable number of disks at the
same rpms?
What do people do - is it your experience that manufactures will supply
performance tests if requested? A couple of friends and I did some tests on
some storage (Nexsan and Equallogic), but it's just hard to find something
to compare it to.
Regards,
Kaj Laursen |
|
| Back to top |
|
 |
Faeandar
Guest
|
| Posted:
Fri Sep 02, 2005 12:16 am Post subject:
Re: Testing performance of storage |
|
|
On Thu, 1 Sep 2005 16:26:15 +0200, "Kaj S. Laursen" <kaj@hih.dk>
wrote:
| Quote: | Hi
I generally find it hard to find data about the performance of storage - it
seems that this is something that vendors perfer not to advertise, except
when they advertise something like a number of IO/s that are obviously when
it's to/from cache.
Some do have tests that show the performance in things like MMB2 score, but
in often in configurations that make it very hard to compare the results
between brands. Very few have tests with tools like IOmeter, JetStress or
SQLIO. Why is that - is it because the tests I mention are not the right
tools for performance tests? Is it because in general there is not much
difference in the performance if it's a comparable number of disks at the
same rpms?
What do people do - is it your experience that manufactures will supply
performance tests if requested? A couple of friends and I did some tests on
some storage (Nexsan and Equallogic), but it's just hard to find something
to compare it to.
Regards,
Kaj Laursen
|
Vendors may not post it on their website but they should supply them
to you if you ask. I've yet to run across a vendor who will not. If
they won't do it you probably don't want their stuff anyway.
But if you want to test it yourself you can ask the vendor for an eval
unit and run whatever tests you like. Or if you can't get an eval
unit at your site ask for time in their performance or test lab.
Any vendor worth anything will get you the info you ask for.
~F |
|
| Back to top |
|
|
SANweaver
Guest
|
| Posted:
Fri Sep 02, 2005 11:21 pm Post subject:
Re: Testing performance of storage |
|
|
Kaj,
I have done LOTS of benchmarking for all kinds of storage and the one
of the main tools I use is free and its the same one you see lots of
vendors post with and run at tradeshows.
http://www.iometer.org
You noted it above...give it a try!
Most vendors will talk about MB/s, IOPS, etc. You can get these and
much more yourself.
Most vendors do testing with iometer or similiar...if they DON'T have a
report for you, then as the other chap mentioned...maybe they are
hiding the results!
Cheers, Bill |
|
| Back to top |
|
|
Kaj S. Laursen
Guest
|
| Posted:
Mon Sep 05, 2005 8:15 am Post subject:
Re: Testing performance of storage |
|
|
Hi
"SANweaver" <sanweaver@gmail.com> wrote in message
news:1125685271.100581.23160@g44g2000cwa.googlegroups.com...
| Quote: | Kaj,
I have done LOTS of benchmarking for all kinds of storage and the one
of the main tools I use is free and its the same one you see lots of
vendors post with and run at tradeshows.
http://www.iometer.org
|
I'm happy that you say that IOmeter is the right tool to use. That has also
been my impression, and what I have been told by at least one vendor.
In my experience IOmeter quickly gives an impression of the performance. But
I'm confused regarding what is a "realistic" test using IOmeter. I know it
will depend strongly on the application running on the storage.
We used a configuration with 8kb blocks, 67% reads, 33% writes, 100% random.
But now one vendor says that is unrealistic - he recommends something like
larger blocks, say 64 kb blocks, 80% reads, 20% writes, and the only 20%
random.
I'm now running performance monitor on some servers to see what my use of
the storage is - that gives me an idea of the distribution of reads/writes.
But I have no idea how to check how much is random IO - so I would like some
input on that. What is your experience, and how can I check my applications?
I'm also not convinced that changing the setup in IOmeter will change much.
Of course that other setup will give more IOPS and larger throughput, but
will it really change the general impression of the performance of the
storage? Will the "ranking" of the storage change - won't all of it gain
about the same percentage?
Regards,
Kaj Laursen |
|
| Back to top |
|
|
HVB
Guest
|
| Posted:
Mon Sep 05, 2005 4:16 pm Post subject:
Re: Testing performance of storage |
|
|
On Mon, 5 Sep 2005 10:14:54 +0200, "Kaj S. Laursen" wrote:
| Quote: | In my experience IOmeter quickly gives an impression of the performance. But
I'm confused regarding what is a "realistic" test using IOmeter. I know it
will depend strongly on the application running on the storage.
We used a configuration with 8kb blocks, 67% reads, 33% writes, 100% random.
But now one vendor says that is unrealistic - he recommends something like
larger blocks, say 64 kb blocks, 80% reads, 20% writes, and the only 20%
random.
|
Kaj,
Be careful that you're not benchmarking simply for the sake of
benchmarking. Many people get hung up on making sure they are buying
the absolute fastest storage(!) on the planet that they forget about
their application requirements.
Your IOmeter configuration above *might* be representative of your
application, or it could be way off - you don't know. Unless the
vendor you mentioned knows your app, I would suggest that they know
what workload best suits their architecture and that's what the
numbers they supplied you with are.
Most storage devices can be configured to support particular workloads
in order to provide the best possible performance. You have to know
what the workload is before you can make any choices based upon it. If
you guess and you get it wrong, you can kill the performance of any
storage device.
For instance:
- OLTP vs Batch
- Sequential operations vs Random
- Small block i/o vs large block i/o
- Average load vs peak load
These are just a starting point. If your app is a database and you
have a DBA, go talk to them. They'll have a pretty good idea of what
traffic they get in the database and will be able to help you build a
profile. Ask what performance they need - don't wait for them to ask
what you can deliver.
Once you have built this profile, then you can apply this to IOmeter
testing and get results that will be approaching something meaningful.
A good vendor will help you configure and tune the storage to get the
best out of it.
HVB. |
|
| Back to top |
|
|
Spindle
Guest
|
| Posted:
Mon Sep 05, 2005 4:16 pm Post subject:
Re: Testing performance of storage |
|
|
| Quote: |
Some do have tests that show the performance in things like MMB2 score, but
in often in configurations that make it very hard to compare the results
between brands. Very few have tests with tools like IOmeter, JetStress or
SQLIO. Why is that - is it because the tests I mention are not the right
tools for performance tests? Is it because in general there is not much
difference in the performance if it's a comparable number of disks at the
same rpms?
What do people do - is it your experience that manufactures will supply
performance tests if requested? A couple of friends and I did some tests on
some storage (Nexsan and Equallogic), but it's just hard to find something
to compare it to.
|
The tools you mention are good, but, as you found out, it's difficult
to measure performance in a way that's easy to compare with other
configurations.
You could check the Storage Performance Council and the Transaction
Performance Council (URLs below). Those results are vendors prepared,
but go through a peer review from other vendors.
Perhaps some configurations and some applications loads are close to
what you have or want.
http://www.storageperformance.org/home
http://www.tpc.org/ |
|
| Back to top |
|
|
Kaj S. Laursen
Guest
|
| Posted:
Tue Sep 06, 2005 8:16 am Post subject:
Re: Testing performance of storage |
|
|
"Spindle" <arthedge@gmail.com> wrote in message
news:1125922657.249874.94330@g44g2000cwa.googlegroups.com...
| Quote: | The tools you mention are good, but, as you found out, it's difficult
to measure performance in a way that's easy to compare with other
configurations.
You could check the Storage Performance Council and the Transaction
Performance Council (URLs below). Those results are vendors prepared,
but go through a peer review from other vendors.
Perhaps some configurations and some applications loads are close to
what you have or want.
http://www.storageperformance.org/home
|
But there are not a lot of systems tested - and those that are tested are
much bigger than what I need. And since the test specification and/or tool
is not public, I can't myself do any tests to compare.
| Quote: | http://www.tpc.org/
|
Arent most of those tests server/cpu limited? The examples I checked it
seems they add a shitload of drives to make sure that the disksystems are
not the limiting factor?
Regards,
Kaj |
|
| Back to top |
|
|
Kaj S. Laursen
Guest
|
| Posted:
Tue Sep 06, 2005 8:16 am Post subject:
Re: Testing performance of storage |
|
|
"HVB" <devnull@127.0.0.1> wrote in message
news:tajoh1h5esflppnh980cp263srdur8sigf@4ax.com...
| Quote: | On Mon, 5 Sep 2005 10:14:54 +0200, "Kaj S. Laursen" wrote:
Kaj,
Be careful that you're not benchmarking simply for the sake of
benchmarking. Many people get hung up on making sure they are buying
the absolute fastest storage(!) on the planet that they forget about
their application requirements.
|
I'm not going to buy the fastest storage on the planet, I just want to make
sure that I do not buy something with lousy performance :-)
Maybe I should tell that my experience is with low-end disk-systems
(Internal disks in Raid, external raid systems like Fibrenetix, and a bit of
experience with my friends MSA1000 system). My experience is that some RAID
systems have pretty lousy implementation and performance, so my one reason
for testing is to make sure that I don't buy something that will dissapoint.
It's not that I want the fastest (for the money), I just don't want the
slowest.
My other reason for testing is that some vendors make claims that I just
don't understand. So I would like to see for myself, and compare to
something I know - like the performance of a MSA1000.
| Quote: | Your IOmeter configuration above *might* be representative of your
application, or it could be way off - you don't know. Unless the
vendor you mentioned knows your app, I would suggest that they know
what workload best suits their architecture and that's what the
numbers they supplied you with are.
|
When talking SAN, how can anybody know the precise workload? When I buy a
SAN, there will be exchange databases, SQL databases, virtual harddisks, and
maybe some system drives for systems that boot from the SAN. And it will
change over time.
So I don't have any specific numbers that I have to achieve. I just want to
get a general impression of the performance.
Regards,
Kaj Laursen |
|
| Back to top |
|
|
HVB
Guest
|
| Posted:
Tue Sep 06, 2005 2:59 pm Post subject:
Re: Testing performance of storage |
|
|
On Tue, 6 Sep 2005 09:44:47 +0200, "Kaj S. Laursen" wrote:
| Quote: | http://www.tpc.org/
Arent most of those tests server/cpu limited? The examples I checked it
seems they add a shitload of drives to make sure that the disksystems are
not the limiting factor?
|
The problem with these benchmarks is that the vendor wants to appear
to have a very high performance storage device.
You get high performance by throwing lots of spindles at it - as you
clearly know. All this tells you is that the benchmarks are not
representative of real world configurations.
Unfortunately not everybody understands/appreciates this, so some
people end up buying based on faulty information.
HVB. |
|
| Back to top |
|
|
HVB
Guest
|
| Posted:
Tue Sep 06, 2005 3:49 pm Post subject:
Re: Testing performance of storage |
|
|
On Tue, 6 Sep 2005 10:05:35 +0200, "Kaj S. Laursen" wrote:
(much snippage throughout)
| Quote: | My experience is that some RAID
systems have pretty lousy implementation and performance, so my one reason
for testing is to make sure that I don't buy something that will dissapoint.
|
I don't wish to throw stones here, but how do you know that they
weren't just configured badly? Without a detailed understanding of
how the storage device works internally it's possible to configure it
in such a way that will negatively impact performance.
This is where a good vendor can really help you out - they'll work
with you to understand your apps and relate that to how the storage
devices works to give you the best overall performance.
| Quote: | It's not that I want the fastest (for the money), I just don't want the
slowest.
|
You want the least worst. ;-)
In this situation you're better off determining what you need deliver
from your apps (front end) and then figure out what you need to get
from the storage (back end) to provide that.
| Quote: | When talking SAN, how can anybody know the precise workload?
|
By understanding the application workload.
| Quote: | When I buy a
SAN, there will be exchange databases, SQL databases, virtual harddisks, and
maybe some system drives for systems that boot from the SAN. And it will
change over time.
|
Ok - you have a starting point. Ideally, you need to build a profile
of what i/o characteristics your apps are likely to have. I
appreciate that I new systems it's a chicken and egg scenario, so
sometimes you have to make assumptions. Make sure you document those
assumptions and why you've made them - just in case someone questions
it later.
| Quote: | So I don't have any specific numbers that I have to achieve. I just want to
get a general impression of the performance.
|
This is where storage sizing becomes an art form.
Here's a (very) simple example... let's take your Exchange server.
Bear in mind this is 'back of an envelope' sizing! If you want me to
do this for real, I'll have to send you an invoice. :-)
STORAGE SIZING 101
How many users?
How big will you allow each mailbox to become?
What's the average message size?
How many messages will each user receive per day?
How many messages will each user send per day?
You can add more questions to the list above and try to make your
sizing more accurate - I've just taken these as a starting point.
Let's put some numbers against these:
Users = 100
Max mailbox size = 300 MB
Average message = 57 KB
Rx Messages per day per user = 50
Tx Messages per day per user = 14
Just from this small amount of information you can say that:
Max storage = 30,000 MB (~30GB -- 100 users * 300 MB mailbox)
Total daily messages = 6400 (50 Rx + 14 Tx * 100 users)
Daily growth rate = 364,800 KB (6400 mess * 57 KB avg mess. size)
Average messages per hour = 267 (6400 / 24)
Average workload per hour = 15,219 KB
Most environments will have a peak during the working day, so you
should probably skew your figures to allow for this and you should
consider how many users are likely to be concurrently accessing the
server - I've seen plenty of environments with huge numbers of users,
but very low concurrency - size for the right workload.
You can see that in this environment, we'll need the storage to
provide around 16MB per hour of throughput (<4.5 KB/sec) - that's not
to be a problem for practically any array on the market today. But
this is a small system - use your own numbers and you'll get realistic
expectations of what you are facing.
TIP - always round your calculations UP, err on the side of caution.
If you can really make the users stick to their 300 MB mailbox limits,
you'll have enough storage for approx 78 days of email with 30 GB of
capacity. You can decide if this is realistic for your environment.
You also need to provide storage for other items like log files, group
folders and additional data, so in this example, you'd probably be
best doubling your calculated mailbox size as a start (60GB).
Email is usually completely random in i/o profile, so you want some
storage that performs well in a random environment - arrays with huge
amounts of cache probably aren't going to help you here.
Now do the same for your other systems.
For my example, I'll assume that you have two SQL databases and a File
Server and that their profile is identical to the Exchange example
above. You should not make this assumption for your environment -
work out the numbers.
Add all your performance numbers together:
4.5 KB/sec * 4 server = 18 KB/sec
In this example, you need your storage device to deliver a consistent
performance of 18 KB/sec. This is still very low.
You are going to have peaks in this workload, so this number should be
the absolute minimum that you can accept. You should calculate your
peak workload for each server and base the performance requirements on
those numbers - or maybe slightly below if you think users will accept
some performance degradation at busy periods.
I hope this has given you something to work from - and you can see why
I said that a DBA can really help you out. They'll know things like
average record size and whether people are mostly writing to the
database or reading from it - which really makes this task easier.
Good luck!
HVB. |
|
| Back to top |
|
|
Faeandar
Guest
|
| Posted:
Wed Sep 07, 2005 6:12 am Post subject:
Re: Testing performance of storage |
|
|
On Mon, 5 Sep 2005 10:14:54 +0200, "Kaj S. Laursen" <kaj@hih.dk>
wrote:
| Quote: |
We used a configuration with 8kb blocks, 67% reads, 33% writes, 100% random.
But now one vendor says that is unrealistic - he recommends something like
larger blocks, say 64 kb blocks, 80% reads, 20% writes, and the only 20%
random.
|
Of course that's what the vendors want you to test because it's an
easier test to pass. Large block IO and mostly reads (that can be
cached).
I think you're original parameters are just fine, except maybe 16k
instead of 8. Though that depends alot on the file systems you use
too.
Also, you mentioned how you can estimate the workload of a SAN given
all the various applications you're talking about. Answer: over time
it's all random. Cache and spindles can help alleviate that issue
though.
~F |
|
| Back to top |
|
|
Kaj S. Laursen
Guest
|
| Posted:
Wed Sep 07, 2005 3:33 pm Post subject:
Re: Testing performance of storage |
|
|
"HVB" <devnull@127.0.0.1> wrote in message
news:gvpqh1h0sf23177eok00mv2lg779i5qhae@4ax.com...
| Quote: | On Tue, 6 Sep 2005 09:44:47 +0200, "Kaj S. Laursen" wrote:
The problem with these benchmarks is that the vendor wants to appear
to have a very high performance storage device.
You get high performance by throwing lots of spindles at it - as you
clearly know.
|
Yes. My question is sort of "Once you reach a certain level of quality of
the storage (*1), is there really any significant difference in the
performance of the storage, or is its then just a question of how many
spindels you have? And the difference between the storage is then in the
features they offer.
(*1) And I don't quite know where that quality level is, but my quess is
that's its not that high/expensive a level.
Regards,
Kaj Laursen |
|
| Back to top |
|
|
Chatz
Guest
|
| Posted:
Thu Sep 08, 2005 3:46 pm Post subject:
Re: Testing performance of storage |
|
|
There are vendors out there producing specialised hardware for
particular workloads. For example, Data Direct Network RAIDS are
extremely good at streaming reads and writes, consistent latency etc,
but poor at IOPS. So you typically see this storage in post-production
houses for video storage, not for NAS.
Other vendors will do much better with IOPS, but can be all over the
place in terms of latency. So good for NAS, but no good for real-time
video work.
Know your application is my advice, since 99% of benchmark tools do not
do what your application does. Trace it and have a look, every platform
has a tool for doing this. What most platforms lack however is a tool
to look at what the filesystem and O/S then do to that I/O.
My experience is that many application writers either have little idea
what impact different I/O will have, and even less of an idea of what
their application actually does until you show them.
David |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in