| Author |
Message |
Nick Maclaren
Guest
|
 Posted:
Sun Oct 02, 2005 10:51 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <5pCdnWbp1rsoKaLeRVnyuw@pipex.net>,
Steve at fivetrees <steve@NOSPAMTAfivetrees.com> wrote:
| Quote: | "Nick Maclaren" <nmm1@cus.cam.ac.uk> wrote in message
news:dhobjl$e2v$1@gemini.csx.cam.ac.uk...
extreme_pedant_mode=ON
Not so. In Fortran 77, it was possible to allocate all memory statically,
but it isn't generally possible. It can't be done in Fortran 90 or C.
even_more_extreme_pedant_mode=ON
It *can* be done in C - by avoiding malloc ;).
even_more_extreme_pedant_mode=OFF
|
I will skip the mode nesting, as things are getting ridiculous :-)
Problem one: in C, there is no memory management beyond stack scoped
except by using malloc. Any algorithm that requires more than that
can't be done if you don't use malloc.
Problem two: there are many library facilities that use malloc either
explicitly or implicitly - including almost all I/O - you would have
to avoid them, too.
Regards,
Nick Maclaren. |
|
| Back to top |
|
 |
Nick Maclaren
Guest
|
| Posted:
Sun Oct 02, 2005 10:56 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <slrndjvdc8.5la.basile-news@ours.starynkevitch.net>,
Basile Starynkevitch [news] <basile-news@starynkevitch.net> wrote:
| Quote: | On 2005-10-02, Nick Maclaren <nmm1@cus.cam.ac.uk> wrote:
The whole area of memory management schemes is sadly neglected, and
much of garbage collection work is based on unrealistic assumptions,
so much of what is currently believed isn't so.
While I tend to agree with the above position, I'm not sure at all to
understand what you mean (you might be thinking the opposite of my
guess).
Do you mean that some GC or memory management is based on unrealistic
assumption, in the sense that other or better GC (I'm thinking of
generational copying ones vs referenc counting) techniques should be
more widely used, or do you mean that GC never really works and that
on the contrary programs should only use static (à la Fortran77) or
manual (à la C++) memory management?
|
No. What I mean is that most garbage collection work ignores such
practical problems as diagnosing and handling memory exhaustion,
efficiency requirements, and trapping user errors before they fill
up the memory with junk. Even when it DOES consider such problems,
it often assumes that the user has the option of choosing garbage
collection algorithms and needs to sole only one problem at once.
| Quote: | I think that old Appel's paper "garbage collection can be faster than
stack allocation" does provide valuable insight, even if the actual
figures may not be true today (because of cache issues, etc..)
|
They never were realistic. Yes, good garbage collection can be
faster than bad stack allocation. So what else is new?
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Sun Oct 02, 2005 11:00 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <KyR%e.41367$VI6.15199@fe1.news.blueyonder.co.uk>,
David Hopwood <david.nospam.hopwood@blueyonder.co.uk> wrote:
| Quote: | Nick Maclaren wrote:
Also, of the memory management schemes that are lumped under the term of
'heap' ones, there are several that have deterministic behaviour and do
not suffer from fragmentation. The simplest one is if all allocations
are of the same size, and chains are used instead of arrays.
The whole area of memory management schemes is sadly neglected, and
much of garbage collection work is based on unrealistic assumptions,
so much of what is currently believed isn't so.
That's a worthlessly vague statement. As far as I can see, the problem isn't
lack of research; it's lack of application of that research in the most
commonly used systems.
|
On the contrary - one of the main reasons that it hasn't been applied
is the impracticality of the research. That was the case in 1965,
and is the case today. A fairly good rule when you find that real
engineers (and there are some real software engineers, even today,
and used to be more) are ignoring the wonderful results that come
out of 'scientific' research is to ask whether the research is
actually addressing the whole of the problem that the engineers
are faced with.
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Hank Oredson
Guest
|
| Posted:
Mon Oct 03, 2005 12:15 am Post subject:
Re: Regarding calculation of free memory |
|
|
"Nick Maclaren" <nmm1@cus.cam.ac.uk> wrote in message
news:dhp6mi$7u3$1@gemini.csx.cam.ac.uk...
| Quote: | In article <5pCdnWbp1rsoKaLeRVnyuw@pipex.net>,
Steve at fivetrees <steve@NOSPAMTAfivetrees.com> wrote:
"Nick Maclaren" <nmm1@cus.cam.ac.uk> wrote in message
news:dhobjl$e2v$1@gemini.csx.cam.ac.uk...
extreme_pedant_mode=ON
Not so. In Fortran 77, it was possible to allocate all memory
statically,
but it isn't generally possible. It can't be done in Fortran 90 or C.
even_more_extreme_pedant_mode=ON
It *can* be done in C - by avoiding malloc ;).
even_more_extreme_pedant_mode=OFF
I will skip the mode nesting, as things are getting ridiculous :-)
Problem one: in C, there is no memory management beyond stack scoped
except by using malloc. Any algorithm that requires more than that
can't be done if you don't use malloc.
|
sbrk()?
| Quote: | Problem two: there are many library facilities that use malloc either
explicitly or implicitly - including almost all I/O - you would have
to avoid them, too.
|
Provide your own malloc() so the I/O that needs it will get it
from your own memory pool. Since you are doing the I/O,
you can provide optimal buffering.
--
... Hank
http://home.earthlink.net/~horedson
http://home.earthlink.net/~w0rli |
|
| Back to top |
|
|
David Hopwood
Guest
|
| Posted:
Mon Oct 03, 2005 12:15 am Post subject:
Re: Regarding calculation of free memory |
|
|
Nick Maclaren wrote:
| Quote: | David Hopwood <david.nospam.hopwood@blueyonder.co.uk> wrote:
Nick Maclaren wrote:
Also, of the memory management schemes that are lumped under the term of
'heap' ones, there are several that have deterministic behaviour and do
not suffer from fragmentation. The simplest one is if all allocations
are of the same size, and chains are used instead of arrays.
The whole area of memory management schemes is sadly neglected, and
much of garbage collection work is based on unrealistic assumptions,
so much of what is currently believed isn't so.
That's a worthlessly vague statement. As far as I can see, the problem isn't
lack of research; it's lack of application of that research in the most
commonly used systems.
On the contrary - one of the main reasons that it hasn't been applied
is the impracticality of the research. That was the case in 1965,
and is the case today. A fairly good rule when you find that real
engineers (and there are some real software engineers, even today,
and used to be more) are ignoring the wonderful results that come
out of 'scientific' research is to ask whether the research is
actually addressing the whole of the problem that the engineers
are faced with.
|
Does every research paper on memory management describe something practical? No.
Does the research that has been done on memory management collectively address
the problems that arise in real systems in a practical way, for those engineers
that are prepared to do a bit of searching to find the papers relevant to those
problems? Yes, it does. If you think otherwise, then be specific about the
problems you think are not addressed.
--
David Hopwood <david.nospam.hopwood@blueyonder.co.uk> |
|
| Back to top |
|
|
Steve at fivetrees
Guest
|
| Posted:
Mon Oct 03, 2005 12:15 am Post subject:
Re: Regarding calculation of free memory |
|
|
"Nick Maclaren" <nmm1@cus.cam.ac.uk> wrote in message
news:dhp6mi$7u3$1@gemini.csx.cam.ac.uk...
| Quote: |
Problem two: there are many library facilities that use malloc either
explicitly or implicitly - including almost all I/O - you would have
to avoid them, too.
|
I'm posting this from c.a.e - embedded (or OS-less) is my main area. One
tends not to need such I/O functions therein.
You're right, though - it certainly pays to be aware of the platform
requirements of any such library function.
Steve
http://www.fivetrees.com |
|
| Back to top |
|
|
Sander Vesik
Guest
|
| Posted:
Mon Oct 03, 2005 10:38 am Post subject:
Re: Regarding calculation of free memory |
|
|
In comp.arch Nick Maclaren <nmm1@cus.cam.ac.uk> wrote:
| Quote: |
No. Replacing malloc is not supported in C, and often breaks
the nastier vendor and third-party libraries. The reason is that
the suite of malloc-related functions is not independent, and they
often use non-standard interfaces.
|
There are still systems where you can reasonably do so - shipping
more than one possible malloc to use is a good hint for example.
| Quote: |
Yes, I have done it, over many years - but it isn't something that
I recommend to users. Or can :-(
Regards,
Nick Maclaren.
|
--
Sander
+++ Out of cheese error +++ |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Mon Oct 03, 2005 1:16 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <aNmdnfsCnNylr93enZ2dnUVZ8qmdnZ2d@pipex.net>,
Steve at fivetrees <steve@NOSPAMTAfivetrees.com> wrote:
| Quote: | "Nick Maclaren" <nmm1@cus.cam.ac.uk> wrote in message
news:dhp6mi$7u3$1@gemini.csx.cam.ac.uk...
Problem two: there are many library facilities that use malloc either
explicitly or implicitly - including almost all I/O - you would have
to avoid them, too.
I'm posting this from c.a.e - embedded (or OS-less) is my main area. One
tends not to need such I/O functions therein.
|
And still less the internationalisation and wide character
facilities :-)
My experience of that area is that the worst problem is finding out
when a compiler has chosen to implement some basic facility (e.g.
a mathematical function) in a way that drags in completely
unrelated junk from the library. But I have been usually trying
to used hosted compilers in an embedded fashion - I would expect
compilers designed for embedded use to be better.
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Mon Oct 03, 2005 1:22 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <h1Y%e.49486$iW5.46400@fe3.news.blueyonder.co.uk>,
David Hopwood <david.nospam.hopwood@blueyonder.co.uk> wrote:
| Quote: |
Does every research paper on memory management describe something practical? No.
Does the research that has been done on memory management collectively address
the problems that arise in real systems in a practical way, for those engineers
that are prepared to do a bit of searching to find the papers relevant to those
problems? Yes, it does. If you think otherwise, then be specific about the
problems you think are not addressed.
|
I have, many times. You ignored the issues every time.
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Mon Oct 03, 2005 1:31 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <fmY%e.6314$oc.4151@newsread2.news.pas.earthlink.net>,
Hank Oredson <horedson@earthlink.net> wrote:
| Quote: |
Problem one: in C, there is no memory management beyond stack scoped
except by using malloc. Any algorithm that requires more than that
can't be done if you don't use malloc.
sbrk()?
|
Not in C - in fact, I can't think of any standard that it IS in,
because it makes such extremely horrible assumptions about the
implementation.
| Quote: | Problem two: there are many library facilities that use malloc either
explicitly or implicitly - including almost all I/O - you would have
to avoid them, too.
Provide your own malloc() so the I/O that needs it will get it
from your own memory pool. Since you are doing the I/O,
you can provide optimal buffering.
|
No. Replacing malloc is not supported in C, and often breaks
the nastier vendor and third-party libraries. The reason is that
the suite of malloc-related functions is not independent, and they
often use non-standard interfaces.
Yes, I have done it, over many years - but it isn't something that
I recommend to users. Or can :-(
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Mon Oct 03, 2005 3:51 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <1128335888.327425@haldjas.folklore.ee>,
Sander Vesik <sander@haldjas.folklore.ee> writes:
|> In comp.arch Nick Maclaren <nmm1@cus.cam.ac.uk> wrote:
|> >
|> > No. Replacing malloc is not supported in C, and often breaks
|> > the nastier vendor and third-party libraries. The reason is that
|> > the suite of malloc-related functions is not independent, and they
|> > often use non-standard interfaces.
|>
|> There are still systems where you can reasonably do so - shipping
|> more than one possible malloc to use is a good hint for example.
It's a hint that the VENDOR can do it - but not that the application
programmer can. Yes, an experienced hacker can, which was the basis
of my remark that I can, but I can't advise users to.
Also, are you aware of how often third-party libraries demand a
particular one of those mallocs, either in their release notes or
because they fall over with others?
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
David Gay
Guest
|
| Posted:
Mon Oct 03, 2005 4:15 pm Post subject:
Re: Regarding calculation of free memory |
|
|
nmm1@cus.cam.ac.uk (Nick Maclaren) writes:
| Quote: | In article <1128335888.327425@haldjas.folklore.ee>,
Sander Vesik <sander@haldjas.folklore.ee> writes:
|> In comp.arch Nick Maclaren <nmm1@cus.cam.ac.uk> wrote:
|
|> > No. Replacing malloc is not supported in C, and often breaks
|> > the nastier vendor and third-party libraries. The reason is that
|> > the suite of malloc-related functions is not independent, and they
|> > often use non-standard interfaces.
|
|> There are still systems where you can reasonably do so - shipping
|> more than one possible malloc to use is a good hint for example.
It's a hint that the VENDOR can do it - but not that the application
programmer can. Yes, an experienced hacker can, which was the basis
of my remark that I can, but I can't advise users to.
|
Ahhh, now I know why I've never had any problems replacing malloc on Linux,
Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?)
--
David Gay
dgay@acm.org |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Mon Oct 03, 2005 9:50 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <s71ek7237zp.fsf@beryl.CS.Berkeley.EDU>,
David Gay <dgay@beryl.CS.Berkeley.EDU> writes:
|>
|> > |> > No. Replacing malloc is not supported in C, and often breaks
|> > |> > the nastier vendor and third-party libraries. The reason is that
|> > |> > the suite of malloc-related functions is not independent, and they
|> > |> > often use non-standard interfaces.
|> > |>
|> > |> There are still systems where you can reasonably do so - shipping
|> > |> more than one possible malloc to use is a good hint for example.
|> >
|> > It's a hint that the VENDOR can do it - but not that the application
|> > programmer can. Yes, an experienced hacker can, which was the basis
|> > of my remark that I can, but I can't advise users to.
|>
|> Ahhh, now I know why I've never had any problems replacing malloc on Linux,
|> Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?)
Or perhaps you just have a slightly narrower range of experience
than I do?
I have seen failures on half a dozen Unices, including Solaris
and Linux. Of course, as I said, it also depends on what other
software you are using in your application.
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
David Gay
Guest
|
| Posted:
Mon Oct 03, 2005 10:00 pm Post subject:
Re: Regarding calculation of free memory |
|
|
nmm1@cus.cam.ac.uk (Nick Maclaren) writes:
| Quote: | In article <s71ek7237zp.fsf@beryl.CS.Berkeley.EDU>,
David Gay <dgay@beryl.CS.Berkeley.EDU> writes:
|
|> > |> > No. Replacing malloc is not supported in C, and often breaks
|> > |> > the nastier vendor and third-party libraries. The reason is that
|> > |> > the suite of malloc-related functions is not independent, and they
|> > |> > often use non-standard interfaces.
|> > |
|> > |> There are still systems where you can reasonably do so - shipping
|> > |> more than one possible malloc to use is a good hint for example.
|
|> > It's a hint that the VENDOR can do it - but not that the application
|> > programmer can. Yes, an experienced hacker can, which was the basis
|> > of my remark that I can, but I can't advise users to.
|
|> Ahhh, now I know why I've never had any problems replacing malloc on Linux,
|> Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?)
Or perhaps you just have a slightly narrower range of experience
than I do?
I have seen failures on half a dozen Unices, including Solaris
and Linux. Of course, as I said, it also depends on what other
software you are using in your application.
|
Well, frankly, for the Linux case, you can go and look at the source code
to check if there's any extra magic entry points or other dependencies. For
the others, it definitely pays to check linker output files to make sure
you didn't, e.g., accidentally end up with two copies of malloc. Yes,
you'll have to repeat such checks if you change libc version - that's a
price you have to pay. And if you're going to say "but I've run into
problems", I'd like to get a specific description, rather than some
nebulous "It happened to me somewhere, sometime, somehow".
--
David Gay
dgay@acm.org |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Mon Oct 03, 2005 11:09 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <s71wtkuilia.fsf@beryl.CS.Berkeley.EDU>,
David Gay <dgay@beryl.CS.Berkeley.EDU> wrote:
| Quote: |
|
|> > It's a hint that the VENDOR can do it - but not that the application
|> > programmer can. Yes, an experienced hacker can, which was the basis
|> > of my remark that I can, but I can't advise users to.
|
|> Ahhh, now I know why I've never had any problems replacing malloc on Linux,
|> Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?)
Or perhaps you just have a slightly narrower range of experience
than I do?
I have seen failures on half a dozen Unices, including Solaris
and Linux. Of course, as I said, it also depends on what other
software you are using in your application.
Well, frankly, for the Linux case, you can go and look at the source code
to check if there's any extra magic entry points or other dependencies. For
the others, it definitely pays to check linker output files to make sure
you didn't, e.g., accidentally end up with two copies of malloc. Yes,
you'll have to repeat such checks if you change libc version - that's a
price you have to pay. And if you're going to say "but I've run into
problems", I'd like to get a specific description, rather than some
nebulous "It happened to me somewhere, sometime, somehow".
|
As I said, this is (usually) within the ability of an experienced hacker.
Do you SERIOUSLY think that it is reasonable for me to tell a Fortran
programmer to find out what version of malloc a system is running
(when he did not install it) and to inspect both the source code of
that malloc and its replacement?
As far as Solaris etc. goes, you first have to find out which version
(or versions) of malloc the application is using, INCLUDING those
called by dynamic libraries. That is unusually hard under Solaris,
because ldd doesn't do what you apparently thinks that it does, and
a fair amount of reverse engineering of the behaviour of the linker
and loader is needed. Then you have to extract the relevant symbols,
see which library is using what, and decide if there is a clash.
Oh, and for non-standard symbols, you may have to reverse engineer
the binary to see what they do.
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in