CASTalk.com

Is there an optimal size for file writes where performance would be
maximum? I ask this question in the context of having to write bytes to a
file, one at a time. Was wondering if i buffered them in memory to some
threshold before i do an I/O write, it would give me better performance?
Or is it all taken care of in the underlying file system implementation
where disk writes wont be made unless data in queue reaches a certain
threshold?

Thanks in advance.

It depends on how you open the file. If the file is opened with buffering
enabled, then you can write whatever size you want. If you open the file
with FILE_FLAG_NOBUFFERING, you must write on sector boundaries (512).

I would suggest buffering and writing in larger blocks. There are a
number of reasons for this, among them:
1) minimize the number of User<-->Kernel transitions. These can really
burn cycles and you will get significantly better cache coherency (L1/L2)
if you do whatever you need in a loop and write out as few times as
possible.
2) For every call to WriteFile, you will execute some number of
instructions. Highly performant code tries to minimize the number of
instructions, so calling WriteFile a lot (or fprint) will unnecessarily
increase the total amount of code executed to perform the task (writing to
a memory buffer is pretty cheap).

WriteFile():
http://msdn.microsoft.com/library/defau ... tefile.asp

Pat

"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:ekX$77HsEHA.3876@TK2MSFTNGP15.phx.gbl...
Is there an optimal size for file writes where performance would be
maximum? I ask this question in the context of having to write bytes to a
file, one at a time. Was wondering if i buffered them in memory to some
threshold before i do an I/O write, it would give me better performance?
Or is it all taken care of in the underlying file system implementation
where disk writes wont be made unless data in queue reaches a certain
threshold?

Thanks in advance.

Thanks for your reply.
What would be an example of an application or type of application that
would benefit from FILE_FLAG_NOBUFFERING?



"Pat [MSFT]" <patfilot@online.microsoft.com> wrote in message
news:%23W7$s6JsEHA.2684@TK2MSFTNGP12.phx.gbl...
It depends on how you open the file. If the file is opened with
buffering enabled, then you can write whatever size you want. If you
open the file with FILE_FLAG_NOBUFFERING, you must write on sector
boundaries (512).

I would suggest buffering and writing in larger blocks. There are a
number of reasons for this, among them:
1) minimize the number of User<-->Kernel transitions. These can really
burn cycles and you will get significantly better cache coherency (L1/L2)
if you do whatever you need in a loop and write out as few times as
possible.
2) For every call to WriteFile, you will execute some number of
instructions. Highly performant code tries to minimize the number of
instructions, so calling WriteFile a lot (or fprint) will unnecessarily
increase the total amount of code executed to perform the task (writing
to a memory buffer is pretty cheap).

WriteFile():
http://msdn.microsoft.com/library/defau ... tefile.asp

Pat

"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:ekX$77HsEHA.3876@TK2MSFTNGP15.phx.gbl...
Is there an optimal size for file writes where performance would be
maximum? I ask this question in the context of having to write bytes to
a file, one at a time. Was wondering if i buffered them in memory to
some threshold before i do an I/O write, it would give me better
performance? Or is it all taken care of in the underlying file system
implementation where disk writes wont be made unless data in queue
reaches a certain threshold?

Thanks in advance.

Is there an optimal size for file writes where performance would be maximum?
I ask this question in the context of having to write bytes to a file, one
at a time. Was wondering if i buffered them in memory to some threshold
before i do an I/O write, it would give me better performance? Or is it all
taken care of in the underlying file system implementation where disk writes
wont be made unless data in queue reaches a certain threshold?

If you wanted to avoid double buffering, i.e. you had your own caching
mechanism and did not want the OS to also cache the same items. Or, you
may want to guarantee that whatever data you had was written to disk to
minimize loss in the event of catastrophic failures and were willing to
accept the performance penalty. For example, MSMQ will write to disk with
buffering disabled if you are sending transacted messages.

Pat



"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:uE%237utKsEHA.2300@TK2MSFTNGP09.phx.gbl...
Thanks for your reply.
What would be an example of an application or type of application that
would benefit from FILE_FLAG_NOBUFFERING?



"Pat [MSFT]" <patfilot@online.microsoft.com> wrote in message
news:%23W7$s6JsEHA.2684@TK2MSFTNGP12.phx.gbl...
It depends on how you open the file. If the file is opened with
buffering enabled, then you can write whatever size you want. If you
open the file with FILE_FLAG_NOBUFFERING, you must write on sector
boundaries (512).

I would suggest buffering and writing in larger blocks. There are a
number of reasons for this, among them:
1) minimize the number of User<-->Kernel transitions. These can really
burn cycles and you will get significantly better cache coherency
(L1/L2) if you do whatever you need in a loop and write out as few times
as possible.
2) For every call to WriteFile, you will execute some number of
instructions. Highly performant code tries to minimize the number of
instructions, so calling WriteFile a lot (or fprint) will unnecessarily
increase the total amount of code executed to perform the task (writing
to a memory buffer is pretty cheap).

WriteFile():
http://msdn.microsoft.com/library/defau ... tefile.asp

Pat

"Priyesh" <PriyeshPadmavilasom@cougarmtn.com> wrote in message
news:ekX$77HsEHA.3876@TK2MSFTNGP15.phx.gbl...
Is there an optimal size for file writes where performance would be
maximum? I ask this question in the context of having to write bytes to
a file, one at a time. Was wondering if i buffered them in memory to
some threshold before i do an I/O write, it would give me better
performance? Or is it all taken care of in the underlying file system
implementation where disk writes wont be made unless data in queue
reaches a certain threshold?

Thanks in advance.

Priyesh wrote:

Is there an optimal size for file writes where performance would be
maximum?
I ask this question in the context of having to write bytes to a file, one
at a time. Was wondering if i buffered them in memory to some threshold
before i do an I/O write, it would give me better performance? Or is it
all
taken care of in the underlying file system implementation where disk
writes
wont be made unless data in queue reaches a certain threshold?

It is really more a matter for the compiler run-time for the language
you use. Normally that will optimise; if designing a binary file, using
a buffer of 4 kb is usually best because that matches the cluster size
used most of the time in XP (almost always in NTFS, on partitions less
than 8GB in FAT 32)


--
Alex Nichol MS MVP (Windows Technologies)
Bournemouth, U.K. Alexn@mvps.D8E8L.org (remove the D8 bit)