| Author |
Message |
Priyesh
Guest
|
 Posted:
Tue Oct 12, 2004 9:10 pm Post subject:
Optimal write/read bytes |
|
|
Is there an optimal size for file writes where performance would be maximum?
I ask this question in the context of having to write bytes to a file, one
at a time. Was wondering if i buffered them in memory to some threshold
before i do an I/O write, it would give me better performance? Or is it all
taken care of in the underlying file system implementation where disk writes
wont be made unless data in queue reaches a certain threshold?
Thanks in advance. |
|
| Back to top |
|
 |
Pat [MSFT]
Guest
|
| Posted:
Wed Oct 13, 2004 1:01 am Post subject:
Re: Optimal write/read bytes |
|
|
It depends on how you open the file. If the file is opened with buffering
enabled, then you can write whatever size you want. If you open the file
with FILE_FLAG_NOBUFFERING, you must write on sector boundaries (512).
I would suggest buffering and writing in larger blocks. There are a number
of reasons for this, among them:
1) minimize the number of User<-->Kernel transitions. These can really burn
cycles and you will get significantly better cache coherency (L1/L2) if you
do whatever you need in a loop and write out as few times as possible.
2) For every call to WriteFile, you will execute some number of
instructions. Highly performant code tries to minimize the number of
instructions, so calling WriteFile a lot (or fprint) will unnecessarily
increase the total amount of code executed to perform the task (writing to a
memory buffer is pretty cheap).
WriteFile():
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/writefile.asp
Pat
"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:ekX$77HsEHA.3876@TK2MSFTNGP15.phx.gbl...
| Quote: | Is there an optimal size for file writes where performance would be
maximum? I ask this question in the context of having to write bytes to a
file, one at a time. Was wondering if i buffered them in memory to some
threshold before i do an I/O write, it would give me better performance?
Or is it all taken care of in the underlying file system implementation
where disk writes wont be made unless data in queue reaches a certain
threshold?
Thanks in advance.
|
|
|
| Back to top |
|
|
Priyesh
Guest
|
| Posted:
Wed Oct 13, 2004 2:28 am Post subject:
Re: Optimal write/read bytes |
|
|
Thanks for your reply.
What would be an example of an application or type of application that would
benefit from FILE_FLAG_NOBUFFERING?
"Pat [MSFT]" <patfilot@online.microsoft.com> wrote in message
news:%23W7$s6JsEHA.2684@TK2MSFTNGP12.phx.gbl...
| Quote: | It depends on how you open the file. If the file is opened with buffering
enabled, then you can write whatever size you want. If you open the file
with FILE_FLAG_NOBUFFERING, you must write on sector boundaries (512).
I would suggest buffering and writing in larger blocks. There are a
number of reasons for this, among them:
1) minimize the number of User<-->Kernel transitions. These can really
burn cycles and you will get significantly better cache coherency (L1/L2)
if you do whatever you need in a loop and write out as few times as
possible.
2) For every call to WriteFile, you will execute some number of
instructions. Highly performant code tries to minimize the number of
instructions, so calling WriteFile a lot (or fprint) will unnecessarily
increase the total amount of code executed to perform the task (writing to
a memory buffer is pretty cheap).
WriteFile():
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/writefile.asp
Pat
"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:ekX$77HsEHA.3876@TK2MSFTNGP15.phx.gbl...
Is there an optimal size for file writes where performance would be
maximum? I ask this question in the context of having to write bytes to a
file, one at a time. Was wondering if i buffered them in memory to some
threshold before i do an I/O write, it would give me better performance?
Or is it all taken care of in the underlying file system implementation
where disk writes wont be made unless data in queue reaches a certain
threshold?
Thanks in advance.
|
|
|
| Back to top |
|
|
Pat [MSFT]
Guest
|
| Posted:
Wed Oct 13, 2004 9:07 pm Post subject:
Re: Optimal write/read bytes |
|
|
If you wanted to avoid double buffering, i.e. you had your own caching
mechanism and did not want the OS to also cache the same items. Or, you may
want to guarantee that whatever data you had was written to disk to minimize
loss in the event of catastrophic failures and were willing to accept the
performance penalty. For example, MSMQ will write to disk with buffering
disabled if you are sending transacted messages.
Pat
"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:uE%237utKsEHA.2300@TK2MSFTNGP09.phx.gbl...
| Quote: | Thanks for your reply.
What would be an example of an application or type of application that
would benefit from FILE_FLAG_NOBUFFERING?
"Pat [MSFT]" <patfilot@online.microsoft.com> wrote in message
news:%23W7$s6JsEHA.2684@TK2MSFTNGP12.phx.gbl...
It depends on how you open the file. If the file is opened with
buffering enabled, then you can write whatever size you want. If you
open the file with FILE_FLAG_NOBUFFERING, you must write on sector
boundaries (512).
I would suggest buffering and writing in larger blocks. There are a
number of reasons for this, among them:
1) minimize the number of User<-->Kernel transitions. These can really
burn cycles and you will get significantly better cache coherency (L1/L2)
if you do whatever you need in a loop and write out as few times as
possible.
2) For every call to WriteFile, you will execute some number of
instructions. Highly performant code tries to minimize the number of
instructions, so calling WriteFile a lot (or fprint) will unnecessarily
increase the total amount of code executed to perform the task (writing
to a memory buffer is pretty cheap).
WriteFile():
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/writefile.asp
Pat
"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:ekX$77HsEHA.3876@TK2MSFTNGP15.phx.gbl...
Is there an optimal size for file writes where performance would be
maximum? I ask this question in the context of having to write bytes to
a file, one at a time. Was wondering if i buffered them in memory to
some threshold before i do an I/O write, it would give me better
performance? Or is it all taken care of in the underlying file system
implementation where disk writes wont be made unless data in queue
reaches a certain threshold?
Thanks in advance.
|
|
|
| Back to top |
|
|
Alex Nichol
Guest
|
| Posted:
Wed Oct 13, 2004 10:32 pm Post subject:
Re: Optimal write/read bytes |
|
|
Priyesh wrote:
| Quote: | Is there an optimal size for file writes where performance would be maximum?
I ask this question in the context of having to write bytes to a file, one
at a time. Was wondering if i buffered them in memory to some threshold
before i do an I/O write, it would give me better performance? Or is it all
taken care of in the underlying file system implementation where disk writes
wont be made unless data in queue reaches a certain threshold?
|
It is really more a matter for the compiler run-time for the language
you use. Normally that will optimise; if designing a binary file, using
a buffer of 4 kb is usually best because that matches the cluster size
used most of the time in XP (almost always in NTFS, on partitions less
than 8GB in FAT 32)
--
Alex Nichol MS MVP (Windows Technologies)
Bournemouth, U.K. Alexn@mvps.D8E8L.org (remove the D8 bit) |
|
| Back to top |
|
|
Priyesh
Guest
|
| Posted:
Wed Oct 13, 2004 11:42 pm Post subject:
Re: Optimal write/read bytes |
|
|
Thanks for the explanations.
"Pat [MSFT]" <patfilot@online.microsoft.com> wrote in message
news:ejWC2cUsEHA.2788@TK2MSFTNGP09.phx.gbl...
| Quote: | If you wanted to avoid double buffering, i.e. you had your own caching
mechanism and did not want the OS to also cache the same items. Or, you
may want to guarantee that whatever data you had was written to disk to
minimize loss in the event of catastrophic failures and were willing to
accept the performance penalty. For example, MSMQ will write to disk with
buffering disabled if you are sending transacted messages.
Pat
"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:uE%237utKsEHA.2300@TK2MSFTNGP09.phx.gbl...
Thanks for your reply.
What would be an example of an application or type of application that
would benefit from FILE_FLAG_NOBUFFERING?
"Pat [MSFT]" <patfilot@online.microsoft.com> wrote in message
news:%23W7$s6JsEHA.2684@TK2MSFTNGP12.phx.gbl...
It depends on how you open the file. If the file is opened with
buffering enabled, then you can write whatever size you want. If you
open the file with FILE_FLAG_NOBUFFERING, you must write on sector
boundaries (512).
I would suggest buffering and writing in larger blocks. There are a
number of reasons for this, among them:
1) minimize the number of User<-->Kernel transitions. These can really
burn cycles and you will get significantly better cache coherency
(L1/L2) if you do whatever you need in a loop and write out as few times
as possible.
2) For every call to WriteFile, you will execute some number of
instructions. Highly performant code tries to minimize the number of
instructions, so calling WriteFile a lot (or fprint) will unnecessarily
increase the total amount of code executed to perform the task (writing
to a memory buffer is pretty cheap).
WriteFile():
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/writefile.asp
Pat
"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:ekX$77HsEHA.3876@TK2MSFTNGP15.phx.gbl...
Is there an optimal size for file writes where performance would be
maximum? I ask this question in the context of having to write bytes to
a file, one at a time. Was wondering if i buffered them in memory to
some threshold before i do an I/O write, it would give me better
performance? Or is it all taken care of in the underlying file system
implementation where disk writes wont be made unless data in queue
reaches a certain threshold?
Thanks in advance.
|
|
|
| Back to top |
|
|
Priyesh
Guest
|
| Posted:
Wed Oct 13, 2004 11:49 pm Post subject:
Re: Optimal write/read bytes |
|
|
Thanks for your reply.
Yes, this is a binary file with bit packed bytes as data and a variable
length header.
The target environment domain for this application will be computers running
windows OS, anywhere from 95 to XP and higher.
Do you think i need to do a performance analysis with variable length write
buffers? Could you recommend any
reading in this regard?
Thanks in Advance.
"Alex Nichol" <alexn.mvpdts@ntlworld.com> wrote in message
news:62tqm014j1e72oiitsp5o9pscg721vmvpb@4ax.com...
| Quote: | Priyesh wrote:
Is there an optimal size for file writes where performance would be
maximum?
I ask this question in the context of having to write bytes to a file, one
at a time. Was wondering if i buffered them in memory to some threshold
before i do an I/O write, it would give me better performance? Or is it
all
taken care of in the underlying file system implementation where disk
writes
wont be made unless data in queue reaches a certain threshold?
It is really more a matter for the compiler run-time for the language
you use. Normally that will optimise; if designing a binary file, using
a buffer of 4 kb is usually best because that matches the cluster size
used most of the time in XP (almost always in NTFS, on partitions less
than 8GB in FAT 32)
--
Alex Nichol MS MVP (Windows Technologies)
Bournemouth, U.K. Alexn@mvps.D8E8L.org (remove the D8 bit) |
|
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in