| Author |
Message |
Nick Maclaren
Guest
|
 Posted:
Tue Oct 04, 2005 2:50 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <434243de$0$11074$e4fe514c@news.xs4all.nl>,
Casper H.S. Dik <Casper.Dik@Sun.COM> writes:
|>
|> >Ahhh, now I know why I've never had any problems replacing malloc on Linux,
|> >Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?)
|>
|> You can certainly do so in Solaris (where we do ship far too many:
|> libc malloc, BSD malloc, SysV -lmalloc, mmap -lmapmalloc, -lmtmalloc,
|> watchmalloc (for debugging) and libumem).
|>
|> And you can substitute your own as long as you make sure you
|> implement *all* routines our malloc libraries commonly export
|> (mallocs often define some of their functions in terms of the others;
|> but some implement foo() in terms of bar() and others vice versa and
|> this tend to lead to nasty problems if you don't roll a complete implementation)
cd /usr/lib
for i in *alloc*.so
do
echo +++ $i +++
nm $i | grep GLOB | grep -v UNDEF | sed 's/^.*|//' | grep -v '^_' | \
sort -u | xargs echo
done
+++ libbsdmalloc.so +++
SUNW_1.1 free malloc realloc
+++ libmalloc.so +++
SUNW_1.1 calloc free malloc realloc
+++ libmapmalloc.so +++
SUNW_0.7 SUNW_1.1 SUNWprivate_1.1 calloc cfree free mallinfo malloc
mallopt memalign realloc valloc
+++ libmtmalloc.so +++
SUNW_1.1 SUNW_1.1.1 SUNWprivate_1.1 calloc free malloc mallocctl
memalign realloc valloc
I can't remember which system it was that had an undocumented call
that I had to reverse engineer from the pseudo-assembler.
Regards,
Nick Maclaren. |
|
| Back to top |
|
 |
Michael N. Moran
Guest
|
| Posted:
Tue Oct 04, 2005 4:15 pm Post subject:
Re: Regarding calculation of free memory |
|
|
Nick Maclaren wrote:
| Quote: | In article <5pCdnWbp1rsoKaLeRVnyuw@pipex.net>,
Steve at fivetrees <steve@NOSPAMTAfivetrees.com> wrote:
"Nick Maclaren" <nmm1@cus.cam.ac.uk> wrote in message
news:dhobjl$e2v$1@gemini.csx.cam.ac.uk...
Not so. In Fortran 77, it was possible to allocate all memory statically,
but it isn't generally possible. It can't be done in Fortran 90 or C.
even_more_extreme_pedant_mode=ON
It *can* be done in C - by avoiding malloc ;).
Problem one: in C, there is no memory management beyond stack scoped
except by using malloc. Any algorithm that requires more than that
can't be done if you don't use malloc.
|
Well ... there *is* static allocation. Then subystems can their own
private memory allocations.
| Quote: | Problem two: there are many library facilities that use malloc either
explicitly or implicitly - including almost all I/O - you would have
to avoid them, too.
|
And, indeed, in systems that must operate indefinitely, one must
not use libraries with such dependencies because any global general
purpose heap (malloc/free) will fragment unless all blocks are the
same size.
Frequently, the same systems that have such requirements also have
limited memory resources and so using a global malloc with as single
fixed size block is unacceptable because of the wasted memory in
small allocations.
As a side note, its not the malloc that fragments heaps, its the
continuous malloc/free cycle over time. For this reason, systems
that must operate continuously for an indefinite period of time
*can* use malloc/new at initialization ... as long as they don't
use free/delete ...
--
Michael N. Moran (h) 770 516 7918
5009 Old Field Ct. (c) 678 521 5460
Kennesaw, GA, USA 30144 http://mnmoran.org
"So often times it happens, that we live our lives in chains
and we never even know we have the key."
The Eagles, "Already Gone"
The Beatles were wrong: 1 & 1 & 1 is 1 |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Tue Oct 04, 2005 4:15 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <slrndk4ool.et5.dave@fly.srk.fer.hr>,
Drazen Kacar <dave@fly.srk.fer.hr> writes:
|>
|> However, I'm confused with the mention of "reverse engineering" in the
|> above paragraph. For Solaris, at least, things one needs to do are fairly
|> well documented[1]. It's also possible to automate the process, so one
|> doesn't have to do it again and again by hand.
|>
|> Could you give me an example of something that required reverse
|> engineering?
One simple example is including two files that logically look like
the following:
extern int a = 1;
extern int b = 1;
and:
extern int a = 1;
extern int c = 1;
In the context of external references to a, b and c, which will
you get (a) searched for and (b) overridden?
There are a zillion other examples, and I have seen at least a
couple of dozen such issues arise in real programs.
The point is that Solaris, like all or almost all modern systems,
does not publish a specification of its linker and loader. It
provides a guide on how to use it, which is not the same thing
at all. Solaris 2.6 and before had some SERIOUS bugs in this area,
but it was very hard to get them reported because there wasn't any
document to say what should happen!
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
FredK
Guest
|
| Posted:
Tue Oct 04, 2005 4:15 pm Post subject:
Re: Regarding calculation of free memory |
|
|
"Nick Maclaren" <nmm1@cus.cam.ac.uk> wrote in message
news:dhtsd0$dsi$1@gemini.csx.cam.ac.uk...
| Quote: |
In article <slrndk4ool.et5.dave@fly.srk.fer.hr>,
Drazen Kacar <dave@fly.srk.fer.hr> writes:
|
|> However, I'm confused with the mention of "reverse engineering" in the
|> above paragraph. For Solaris, at least, things one needs to do are
fairly
|> well documented[1]. It's also possible to automate the process, so one
|> doesn't have to do it again and again by hand.
|
|> Could you give me an example of something that required reverse
|> engineering?
One simple example is including two files that logically look like
the following:
extern int a = 1;
extern int b = 1;
and:
extern int a = 1;
extern int c = 1;
In the context of external references to a, b and c, which will
you get (a) searched for and (b) overridden?
There are a zillion other examples, and I have seen at least a
couple of dozen such issues arise in real programs.
The point is that Solaris, like all or almost all modern systems,
does not publish a specification of its linker and loader. It
provides a guide on how to use it, which is not the same thing
at all. Solaris 2.6 and before had some SERIOUS bugs in this area,
but it was very hard to get them reported because there wasn't any
document to say what should happen!
|
To bring up an antique that gets it right, the VMS linker would
complain that there were multiple definitions and refuse to produce
an executable. Which from time to time generates complaints
from people porting from UNIX. Of course, they apparently don't
realize that what they are producing on UNIX may not be correct,
or only correct by accident.
But then again, I also have problems with the general UNIX/C
case sensitivity - having too many times ported code that had
routines with the same name differing only by case, and where
code randomly called the wrong one (usually in an error path
where nobody bothered to debug it - heck, even in a POSIX
based conformance test suite!). |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Tue Oct 04, 2005 4:15 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <buv0f.13759$e07.10785@news.cpqcorp.net>,
"FredK" <fred.nospam@nospam.dec.com> writes:
|>
|> To bring up an antique that gets it right, the VMS linker would
|> complain that there were multiple definitions and refuse to produce
|> an executable. Which from time to time generates complaints
|> from people porting from UNIX. Of course, they apparently don't
|> realize that what they are producing on UNIX may not be correct,
|> or only correct by accident.
Without a precise specification, I would say that the very concept
of correctness is absent. The best that can be hoped for is that
it will do what the user intended it to do - if, indeed, the user
had a definite intent in the matter. I have too often had people
refuse to fix such inconsistencies because they were not prepared
to make changes to code that they did not understand the intent of.
But the same problem can arise in 'correct' code, where some
of those linkages are weak externals or weak definitions. Fortran
COMMON is a particularly fruitful source of confusion, as it is
not quite a definition, not quite a reference and neither weak
nor strong. Mix that with other types of symbol, and try to work
out what should happen ....
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Casper H.S. Dik
Guest
|
| Posted:
Tue Oct 04, 2005 4:15 pm Post subject:
Re: Regarding calculation of free memory |
|
|
"FredK" <fred.nospam@nospam.dec.com> writes:
| Quote: | To bring up an antique that gets it right, the VMS linker would
complain that there were multiple definitions and refuse to produce
an executable. Which from time to time generates complaints
from people porting from UNIX. Of course, they apparently don't
realize that what they are producing on UNIX may not be correct,
or only correct by accident.
|
Same for Solaris, but not when linking these objects dynamically
where there are also matters as scoping (is the object visible or not)
and ordering are important (you are allowed to replace certain functions
in the C library, after all).
a.c:
b.c:
ld: fatal: symbol `a' is multiply-defined:
(file a.o type=OBJT; file b.o type=OBJT);
ld: fatal: File processing errors. No output written to a.out
| Quote: | But then again, I also have problems with the general UNIX/C
case sensitivity - having too many times ported code that had
routines with the same name differing only by case, and where
code randomly called the wrong one (usually in an error path
where nobody bothered to debug it - heck, even in a POSIX
based conformance test suite!).
|
That's not just an issue with case sensitivity but a similar
situations can arise with other naming schemes. (And this
one source of confusion is easily recmoved using a coding
style which specifies how to use case.
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth. |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Tue Oct 04, 2005 4:15 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <43429f8b$0$11074$e4fe514c@news.xs4all.nl>,
Casper H.S. Dik <Casper.Dik@Sun.COM> writes:
|>
|> Same for Solaris, but not when linking these objects dynamically
|> where there are also matters as scoping (is the object visible or not)
|> and ordering are important (you are allowed to replace certain functions
|> in the C library, after all).
Precisely. And similar remarks apply to every modern Unix I have
looked at, including Linux.
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
David Gay
Guest
|
| Posted:
Tue Oct 04, 2005 9:30 pm Post subject:
Re: Regarding calculation of free memory |
|
|
"Hank Oredson" <horedson@earthlink.net> writes:
| Quote: | "Michael N. Moran" <mike@mnmoran.org> wrote in message
Of course you *can* call it from C, but you can't expect
it to be there on all systems, especially "bare metal"
embedded systems.
The programmer must know her tools. If she is writing a
banking backroom system, she will not be interested in
porting it to a Blackberry, or an access point.
|
Though some access points (WRT54G) already have sbrk(), and I'm sure it's
not long until Blackberries have it too ;-)
--
David Gay
dgay@acm.org |
|
| Back to top |
|
|
Drazen Kacar
Guest
|
| Posted:
Tue Oct 04, 2005 10:26 pm Post subject:
Re: Regarding calculation of free memory |
|
|
Nick Maclaren wrote:
| Quote: | In article <1128388888.617990@haldjas.folklore.ee>,
Sander Vesik <sander@haldjas.folklore.ee> wrote:
Ahem. Sorry, Nick, but at the very least in case of Solaris, it is very
easy to find out what exactly it is that the dynamic linker does and
comapre the results of different runs. This is certainly not advanced
hacker level.
Oh, really? It is (relatively) easy to find out what is loaded,
though even that needs fairly advanced knowledge. But finding out
which library and which symbol in that library is used to satisfy
another symbol in another library is not so easy. I had a problem
recently that came down to this. Nasty.
|
It isn't that hard. Try this:
env LD_DEBUG=bindings ls >/dev/null
Works on Linux, too.
--
.-. .-. Yes, I am an agent of Satan, but my duties are largely
(_ \ / _) ceremonial.
|
| dave@fly.srk.fer.hr |
|
| Back to top |
|
|
FredK
Guest
|
| Posted:
Wed Oct 05, 2005 12:15 am Post subject:
Re: Regarding calculation of free memory |
|
|
"Casper H.S. Dik" <Casper.Dik@Sun.COM> wrote in message
news:43429f8b$0$11074$e4fe514c@news.xs4all.nl...
| Quote: | "FredK" <fred.nospam@nospam.dec.com> writes:
To bring up an antique that gets it right, the VMS linker would
complain that there were multiple definitions and refuse to produce
an executable. Which from time to time generates complaints
from people porting from UNIX. Of course, they apparently don't
realize that what they are producing on UNIX may not be correct,
or only correct by accident.
Same for Solaris, but not when linking these objects dynamically
where there are also matters as scoping (is the object visible or not)
and ordering are important (you are allowed to replace certain functions
in the C library, after all).
a.c:
b.c:
ld: fatal: symbol `a' is multiply-defined:
(file a.o type=OBJT; file b.o type=OBJT);
ld: fatal: File processing errors. No output written to a.out
|
By dynamic linking, I assume some sort of runtime activation of a
DLL? Not exactly sure what is meant.
| Quote: | But then again, I also have problems with the general UNIX/C
case sensitivity - having too many times ported code that had
routines with the same name differing only by case, and where
code randomly called the wrong one (usually in an error path
where nobody bothered to debug it - heck, even in a POSIX
based conformance test suite!).
That's not just an issue with case sensitivity but a similar
situations can arise with other naming schemes. (And this
one source of confusion is easily recmoved using a coding
style which specifies how to use case.
|
I've yet to find UNIX code that sticks to a convention, and that is
remotely safe from boneheaded mistakes. Probably 80% or more
of the code I've seen doesn't use C prototypes, which would
have at least pointed out they wanted Delete() instead of delete() -
each of which had different parameters (for an example). |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Wed Oct 05, 2005 12:15 am Post subject:
Re: Regarding calculation of free memory |
|
|
In article <LcD0f.13856$tj7.3768@news.cpqcorp.net>,
FredK <fred.nospam@nospam.dec.com> wrote:
| Quote: |
I've yet to find UNIX code that sticks to a convention, and that is
remotely safe from boneheaded mistakes. Probably 80% or more
of the code I've seen doesn't use C prototypes, which would
have at least pointed out they wanted Delete() instead of delete() -
each of which had different parameters (for an example).
|
Well, you haven't seen mine, then :-)
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Hank Oredson
Guest
|
| Posted:
Wed Oct 05, 2005 7:52 am Post subject:
Re: Regarding calculation of free memory |
|
|
"David Gay" <dgay@beryl.CS.Berkeley.EDU> wrote in message
news:s71slvhi6sl.fsf@beryl.CS.Berkeley.EDU...
| Quote: |
"Hank Oredson" <horedson@earthlink.net> writes:
"Michael N. Moran" <mike@mnmoran.org> wrote in message
Of course you *can* call it from C, but you can't expect
it to be there on all systems, especially "bare metal"
embedded systems.
The programmer must know her tools. If she is writing a
banking backroom system, she will not be interested in
porting it to a Blackberry, or an access point.
Though some access points (WRT54G) already have sbrk(), and I'm sure it's
not long until Blackberries have it too ;-)
|
Yes, I would imagine my wristwatch (were I to own such a thing)
might soon implement the whole of the C library and allow me to
run my own code via WiFi or some such ;-)
--
... Hank
http://home.earthlink.net/~horedson
http://home.earthlink.net/~w0rli |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Wed Oct 05, 2005 1:36 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <slrndk60bs.ias.dave@fly.srk.fer.hr>,
Drazen Kacar <dave@fly.srk.fer.hr> wrote:
| Quote: |
Oh, really? It is (relatively) easy to find out what is loaded,
though even that needs fairly advanced knowledge. But finding out
which library and which symbol in that library is used to satisfy
another symbol in another library is not so easy. I had a problem
recently that came down to this. Nasty.
It isn't that hard. Try this:
env LD_DEBUG=bindings ls >/dev/null
Works on Linux, too.
|
Sigh. This is getting tedious, so this will be my last response.
Most of these 'solutions' work, in that they do something that MAY
help. But the precise behaviour of the system and those facilities
is rarely, if ever, documented and they are more likely to do something
that appears to do what the user expects (or is needed) but actually
is not. This very often means that a a program checks out perfectly,
but then misbehaves in real use.
For example, that facility doesn't provide all of the information
that may be needed to answer the nastier questions I have described
and, anyway, is data dependent. So that you have to check it every
time you run your program to be certain it is doing what you expect.
That is ridiculous. For example, consider:
franklin-2$LD_DEBUG=bindings ls /dev/null 2>&1 | egrep 'calling|open'
13010: calling .init (from sorted order): /usr/lib/libc.so.1
13010: calling .init (done): /usr/lib/libc.so.1
13010: calling .fini: /usr/lib/libc.so.1
franklin-2$
versus:
franklin-2$LD_DEBUG=bindings ls -l /dev/null 2>&1 | egrep 'calling|open'
12852: calling .init (from sorted order): /usr/lib/libc.so.1
12852: calling .init (done): /usr/lib/libc.so.1
12852: binding file=/usr/lib/libc.so.1 to file=/usr/lib/libc.so.1: symbol `_open64'
12852: binding file=/usr/lib/libc.so.1 to file=/usr/lib/libc.so.1: symbol `_open'
12852: calling .fini: /usr/lib/libc.so.1
franklin-2$
To the best of my knowledge, there is no decently encapsulated tool
for modern Unices that will tell you the dependencies of an executable
in enough detail to predict exactly how the binding will work. So, if
you need to check that an executable is safe when used on another
system (or even by someone else!), you are stuffed. You would clearly
be surprised how often I see binding bugs in vendors' own software
run on vendors' own systems, caused by a 'transparent' upgrade, and
the main reason that it is so common is that it can't practically be
checked.
Real software engineering, back in the days before it was called that,
was about writing programs to be used by other people and often in
unknown environments. I am aware that this is not a concern of home
hackers and similar (including most computer scientists).
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Wed Oct 05, 2005 1:38 pm Post subject:
Re: Regarding calculation of free memory |
|
|
In article <zdH0f.2971$4h2.1733@newsread3.news.pas.earthlink.net>,
Hank Oredson <horedson@earthlink.net> wrote:
| Quote: | "David Gay" <dgay@beryl.CS.Berkeley.EDU> wrote in message
news:s71slvhi6sl.fsf@beryl.CS.Berkeley.EDU...
Though some access points (WRT54G) already have sbrk(), and I'm sure it's
not long until Blackberries have it too ;-)
Yes, I would imagine my wristwatch (were I to own such a thing)
might soon implement the whole of the C library and allow me to
run my own code via WiFi or some such ;-)
|
If they code to that standard, it is more likely to be someone else's
code run without your permission :-)
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Drazen Kacar
Guest
|
| Posted:
Wed Oct 05, 2005 2:51 pm Post subject:
Re: Regarding calculation of free memory |
|
|
Nick Maclaren wrote:
| Quote: | Sigh. This is getting tedious, so this will be my last response.
|
If you wish. I was under impression that the problem was "finding out
which library and which symbol in that library is used to satisfy another
symbol in another library is not so easy". In present tense.
| Quote: | Most of these 'solutions' work, in that they do something that MAY
help.
|
I wasn't offering a 'solution'. I was simply pointing you to the debugging
facility.
| Quote: | But the precise behaviour of the system and those facilities is rarely,
if ever, documented
|
What do you want? Something that can be run through a formal verification
process?
| Quote: | For example, that facility doesn't provide all of the information
that may be needed to answer the nastier questions I have described
and, anyway, is data dependent.
|
So try with "LD_DEBUG=bindings,detail". You'll get a few more numbers.
| Quote: | So that you have to check it every time you run your program to be
certain it is doing what you expect. That is ridiculous.
|
That's how ELF works. If you change the run-time environment, your ELF
objects might change the behavior. Even formal verification can't help
with that.
| Quote: | For example, consider:
franklin-2$LD_DEBUG=bindings ls /dev/null 2>&1 | egrep 'calling|open'
13010: calling .init (from sorted order): /usr/lib/libc.so.1
13010: calling .init (done): /usr/lib/libc.so.1
13010: calling .fini: /usr/lib/libc.so.1
franklin-2$
versus:
franklin-2$LD_DEBUG=bindings ls -l /dev/null 2>&1 | egrep 'calling|open'
12852: calling .init (from sorted order): /usr/lib/libc.so.1
12852: calling .init (done): /usr/lib/libc.so.1
12852: binding file=/usr/lib/libc.so.1 to file=/usr/lib/libc.so.1: symbol `_open64'
12852: binding file=/usr/lib/libc.so.1 to file=/usr/lib/libc.so.1: symbol `_open'
12852: calling .fini: /usr/lib/libc.so.1
franklin-2$
|
So? I don't see a problem.
| Quote: | To the best of my knowledge, there is no decently encapsulated tool
for modern Unices that will tell you the dependencies of an executable
in enough detail to predict exactly how the binding will work.
|
Not even on AIX? IIRC, AIX performs bindings at link time, so it should be
deterministic. ELF systems perform bindings at run-time, so it depends on
the run-time configuration. Some ELF systems have a way to perform
bindings at link time, but I'm not going to advise anyone to use that.
| Quote: | So, if you need to check that an executable is safe when used on
another system (or even by someone else!), you are stuffed. You would
clearly be surprised how often I see binding bugs in vendors' own
software run on vendors' own systems, caused by a 'transparent'
upgrade, and the main reason that it is so common is that it can't
practically be checked.
|
No, I wouldn't be surprised. I never said I liked ELF. Why do you think I
would?
--
.-. .-. Yes, I am an agent of Satan, but my duties are largely
(_ \ / _) ceremonial.
|
| dave@fly.srk.fer.hr |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in