| Author |
Message |
J. Clarke
Guest
|
 Posted:
Mon Dec 06, 2004 1:08 pm Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
Peter wrote:
| Quote: | Your calculations just don't match the real world, but what the hell :)
Out of 2200 or so 146GB FC drives, we replace 2-5 every week. This is
quite
a bit more than your one-in-136 a year.
That could be because:
1. Your hard drives work harder
2. Power on Hours per year are higher
|
One would hope that for enterprise-quality drives the design hours per year
would be 8766.
| Quote: | 3. Ambient temperature is higher
4. Drives are getting older
than specified for MTBF measurement.
|
--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net) |
|
| Back to top |
|
 |
Jesper Monsted
Guest
|
| Posted:
Mon Dec 06, 2004 9:23 pm Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
"J. Clarke" <jclarke@nospam.invalid> wrote in
news:cp096u01tfm@news2.newsguy.com:
| Quote: | Jesper Monsted wrote:
"Maxim S. Shatskih" <maxim@storagecraft.com> wrote in
news:coq0t2$rs7$1 @gavrilo.mtu.ru:
No, this only means that each year 1 of 136 disks will fail :)
Your calculations just don't match the real world, but what the hell
:)
Out of 2200 or so 146GB FC drives, we replace 2-5 every week. This is
quite a bit more than your one-in-136 a year.
(a) what is the MTBF on those drives
|
Not quite sure, since it's a mix of whatever EMC had on the shelf at the
time. There are quite a few seagates (rated at 1,400,000 hours and
1,200,000 hours, depending on the model) and Ultrastars (which i can't find
the MTBF for, but assume it's about the same).
| Quote: | (b) if the failure rate is significantly greather than the rated MTBF
would suggest then have you tried to find out what is killing them?
|
Firmware. They are being replaced as soon as anything looks like it's going
to fail. Soft errors, S.M.A.R.T. readouts etc is all taken into
consideration and drives preemptively replaced.
They still managed to fail two in the same raidset, b0rking 18 TB of
datawarehouse, though.
--
/Jesper Monsted |
|
| Back to top |
|
|
Jesper Monsted
Guest
|
| Posted:
Mon Dec 06, 2004 9:30 pm Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
"Peter" <peterfoxghost@yahoo.ca> wrote in news:31i5k1F3asuueU1
@individual.net:
| Quote: | Your calculations just don't match the real world, but what the hell :)
Out of 2200 or so 146GB FC drives, we replace 2-5 every week. This is
quite
a bit more than your one-in-136 a year.
That could be because:
1. Your hard drives work harder
|
Sitting in a enterprise class storage system, they might, but that
shouldn't fail them that often.
| Quote: | 2. Power on Hours per year are higher
|
24/7, just like they were designed for (i hope... who ever turns off
servers?)
| Quote: | 3. Ambient temperature is higher
|
Properly cooled datacenter at about 22 degrees C. Exhaust air from the box
isn't even warm to the touch.
| Quote: | 4. Drives are getting older
than specified for MTBF measurement.
|
About a year and a half now, but this has been the same since we turned on
the things.
As i noted in a different post, we (or rather, EMC) replace them as soon as
they act funny in any way, which probably means our definition of
"failure" is a bit different than the disc manufacturers...
I was told EMC scrapped a 100k unit shipment of ultrastars at one point -
there's a bad RMA for some support droid at IBM :)
--
/Jesper Monsted |
|
| Back to top |
|
|
Folkert Rienstra
Guest
|
| Posted:
Mon Dec 06, 2004 9:40 pm Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
"Peter" <peterfoxghost@yahoo.ca> wrote in message news:31i5k1F3asuueU1@individual.net
| Quote: | Your calculations just don't match the real world, but what the hell :)
Out of 2200 or so 146GB FC drives, we replace 2-5 every week. This is
quite a bit more than your one-in-136 a year.
That could be because:
1. Your hard drives work harder
|
They are FC drives.
| Quote: | 2. Power on Hours per year are higher
|
They are FC drives.
| Quote: | 3. Ambient temperature is higher
|
They are FC drives.
| Quote: | 4. Drives are getting older
|
Or at 146GB are still very young.
| Quote: | than specified for MTBF measurement.
|
146GB drives, older than 5 years? . |
|
| Back to top |
|
|
Thor Lancelot Simon
Guest
|
| Posted:
Mon Dec 06, 2004 11:17 pm Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
In article <31i5k1F3asuueU1@individual.net>,
Peter <peterfoxghost@yahoo.ca> wrote:
| Quote: | Your calculations just don't match the real world, but what the hell :)
Out of 2200 or so 146GB FC drives, we replace 2-5 every week. This is
quite
a bit more than your one-in-136 a year.
That could be because:
1. Your hard drives work harder
2. Power on Hours per year are higher
3. Ambient temperature is higher
4. Drives are getting older
than specified for MTBF measurement.
|
Or it could be because unless you pay for an application-specific MTBF
guarantee (which you will probably be contractually bound to not
disclose), what you're working with is a measurement so fundamentally
tied to marketing purposes that it's basically not useful for any
technical purpose at all.
You could reasonably think of those 1,200,000 hour MTBF numbers as
being generated like this:
"We want to quote a million-hour MTBF. How many drives to we have to
run for a month to quote that? We only have a month left in the
development cycle before we have to have the marketing materials ready
for product announcement."
"Hm. I see it's a little under 2000 drives. Well, let's pull 2,000 of the
first production run (that pass the initial QA test, which will exclude DOA
units and units with the kind of obvious mechanical problems that will
kill them early; this sort of QA is often _reduced_ for later production
runs) and put them in a room for 1000 hours. If only one or two fail,
we'll claim a 1,000,000 hour MTBF and nobody will be able to sue us."
Of course very few units actually fail in the first month of use, in
the "enterprise drive" space where even moderately rigorous QA is done
on the production line. The whole key to the fraud is in manipulating
the total length of the MTBF test so that it *never hits* the actual
point in time when wear on any component might cause that component to
have a significant likelihood of causing a unit failure.
The moral of the story is that if you run your disk drives for a month
and then throw them away, you can feel reasonably confident in trusting
manufacturer MTBF numbers. Otherwise, though... caveat emptor.
--
Thor Lancelot Simon tls@rek.tjls.com
Am I politic? Am I subtle? Am I a Machiavel?
-William Shakespeare |
|
| Back to top |
|
|
Peter
Guest
|
| Posted:
Tue Dec 07, 2004 12:22 am Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
| Quote: | That could be because:
1. Your hard drives work harder
Sitting in a enterprise class storage system, they might, but that
shouldn't fail them that often.
|
Seagate used to measure MTBF with "read/seak/write operation 20% of power-on
hours".
Some other data (from Seagate too) shows that by increasing disk load from
20% to 100%, MTBF drops to half. That would reduce effective MTBF from
1,200,000hr to 600,000hr.
| Quote: | 2. Power on Hours per year are higher
24/7, just like they were designed for (i hope... who ever turns off
servers?)
|
Seagate says 720 POH per month, but also 8,760 per year. That is quite close
to 24/7 but not exactly the same. Do hard drives benefit from a short
vacation?
| Quote: | 3. Ambient temperature is higher
Properly cooled datacenter at about 22 degrees C. Exhaust air from the box
isn't even warm to the touch.
|
There is a difference between warm to the touch and 22 degrees. Someone
would need to measure temperature inside of the EMC box.
Temperature kills MTBF. Again, you will loose additional 30% of MTBF for
temperature going from 25 to 34 degC.
Seagate did not say what temperature was used for MTBF testing. They only
said that recommended case temperature from 5 to 55 degC was not exceeded.
| Quote: | 4. Drives are getting older
than specified for MTBF measurement.
About a year and a half now, but this has been the same since we turned on
the things.
|
And that is interesting. Despite of a "bathtub" curve for a hard drive
failure rate, putting all drives in 1.5 year from manufacturing date, would
mean the best reliability period for them.
But, if a new drive is put in, initial failure rate would be much higher
(effective MTBF lower).
Again, Seagate said that drive had to be used for at least nine months, but
data was collected for a minimum period of six months. I will leave
interpretation of this procedure to you. |
|
| Back to top |
|
|
Peter
Guest
|
| Posted:
Tue Dec 07, 2004 12:45 am Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
| Quote: | You could reasonably think of those 1,200,000 hour MTBF numbers as
being generated like this:
"We want to quote a million-hour MTBF. How many drives to we have to
run for a month to quote that? We only have a month left in the
development cycle before we have to have the marketing materials ready
for product announcement."
"Hm. I see it's a little under 2000 drives. Well, let's pull 2,000 of
the
first production run (that pass the initial QA test, which will exclude
DOA
units and units with the kind of obvious mechanical problems that will
kill them early; this sort of QA is often _reduced_ for later production
runs) and put them in a room for 1000 hours. If only one or two fail,
we'll claim a 1,000,000 hour MTBF and nobody will be able to sue us."
|
They cannot provide proven reliability metrics with such a low number of
failures (one or two).
They would have to create conditions for much higher number of failures to
make scientifically sound claims. That would mean 1000 drives for 10,000
hours or 10,000 drives for 1,000 hours or other means (oven). Drive
operating constanly at close to 55 degC ambient temperature will have MTBF
only 25% of the MTBF for drive working in 25 degC conditions.
| Quote: | Of course very few units actually fail in the first month of use, in
the "enterprise drive" space where even moderately rigorous QA is done
on the production line. The whole key to the fraud is in manipulating
the total length of the MTBF test so that it *never hits* the actual
point in time when wear on any component might cause that component to
have a significant likelihood of causing a unit failure.
|
It is actually the other way round. They can disregard drives which fail
early, because they fail much more frequently at the beginning. Only when
drive approaches more than half of recommended life expectancy (5 years),
failure rate starts to slowly climb up again.
I guess, they pick the lowest failure rate period to calculate the
advertised MTBF. |
|
| Back to top |
|
|
J. Clarke
Guest
|
| Posted:
Tue Dec 07, 2004 9:17 am Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
Jesper Monsted wrote:
| Quote: | "Peter" <peterfoxghost@yahoo.ca> wrote in news:31i5k1F3asuueU1
@individual.net:
Your calculations just don't match the real world, but what the hell :)
Out of 2200 or so 146GB FC drives, we replace 2-5 every week. This is
quite
a bit more than your one-in-136 a year.
That could be because:
1. Your hard drives work harder
Sitting in a enterprise class storage system, they might, but that
shouldn't fail them that often.
2. Power on Hours per year are higher
24/7, just like they were designed for (i hope... who ever turns off
servers?)
3. Ambient temperature is higher
Properly cooled datacenter at about 22 degrees C. Exhaust air from the box
isn't even warm to the touch.
|
Which is a red flag. If it's not warm then it's probably not cooling
anything.
| Quote: | 4. Drives are getting older
than specified for MTBF measurement.
About a year and a half now, but this has been the same since we turned on
the things.
As i noted in a different post, we (or rather, EMC) replace them as soon
as they act funny in any way, which probably means our definition of
"failure" is a bit different than the disc manufacturers...
I was told EMC scrapped a 100k unit shipment of ultrastars at one point -
there's a bad RMA for some support droid at IBM :)
|
--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net) |
|
| Back to top |
|
|
Malcolm Weir
Guest
|
| Posted:
Tue Dec 07, 2004 10:27 am Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
On Sun, 05 Dec 2004 18:37:47 GMT, flux <support@fluxsoft.com> wrote:
| Quote: | In article <kl46r0pc08u3qrtma52thccv9rcslkepn8@4ax.com>,
Malcolm Weir <malc@gelt.org> wrote:
Yet, oddly, your opinions directly contradict those expressed by
Maxtor, Seagate, Hitachi, and Toshiba. So who did you talk to?
Do they?
Yes. So who did you talk to?
Maxtor and Western Digital, plus a few RAID manufacturers.
|
I don't believe you, because I know what Maxtor has to say.
And who are those "RAID manufacturers"? A few posts ago you hadn't
figured out what they did for a living!
[ Snip ]
| Quote: | Or.... can't find the paperwork/remember that they have a warranty...
Don't mean "and"?
No. I meant "or", which is why I wrote it.
But "and" is wrong.
No, it isn't. It's not what I meant, which is why I didn't write it,
It is what I meant.
|
Congratulations. Given that *you*, not *I*, introduced the whole
idea, your current position (which I expect to change soon) is:
You wrote: 'Don't mean "and"?",
I wrote "No."
You wrote: 'But "and" is wrong'.
Which makes no sense. Still, that's nothing new when it comes from
you!
| Quote: | Only because you're too stupid to understand that once the drive has
failed, there's going to be downtime. The issue, for people with more
of a clue than you, is whether you take advantage of the fact that
Sounds like a cat chasing its own tail.
|
Yes. Now go find some wool to play with, troll.
Malc. |
|
| Back to top |
|
|
Malcolm Weir
Guest
|
| Posted:
Tue Dec 07, 2004 10:31 am Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
On Sun, 05 Dec 2004 18:52:02 GMT, flux <support@fluxsoft.com> wrote:
| Quote: | In article <k156r09sr8156pkk6kjege1ugpnd19o1e1@4ax.com>,
Malcolm Weir <malc@gelt.org> wrote:
Well, let's see: what were you responding to:
Now, has it dawned on you that even the most rudimentary of network
servers has multiple NICs? Why do you think that is? Are server
manufacturers silly?
That's a very recent developlment. Even gigabit is relatively recent.
But my points above are still correct.
|
Nope. Gigabit is not relatively recent.
That's the point.
| Quote: | You want to tell that to people like Dell, because they seem to think
it worthwhile putting gigabit on their entry level Precision desktop!
Heck, even some of Dell's laptops have gigabit adapters!
This supports my assertion that gigabit is relatively recent.
|
Nope. It supports the assertion that gigabit is common, which you
claimed to untrue.
It says nothing about how long gigabit has been available, which is
what you seem to thing is terribly significant.
(And I'd note you probably still haven't grasped that a tiny query can
trigger huge amounts of IO!).
| Quote: | You make claims that are false, naive, and fail to provide anything to
substantiate your claims. And you clearly don't understand high-end
I take it the above is the evidence for your claims.
|
You want to claim that gigabit hasn't been around since 1999, and Dell
doesn't stick it in laptops and entry-level desktops, be my guest.
I'll then provide the cites, you'll look silly (well, sillier than you
currently do) and we'll all live happily ever after!
| Quote: | Me, I architect multi-terabyte HSMs.
I didn't know they were so complex that to require an "architect".
|
I'm not surprised you didn't know that.
They are.
Malc. |
|
| Back to top |
|
|
Malcolm Weir
Guest
|
| Posted:
Tue Dec 07, 2004 10:35 am Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
On Mon, 6 Dec 2004 14:22:26 -0500, "Peter" <peterfoxghost@yahoo.ca>
wrote:
[ Snip ]
| Quote: | 2. Power on Hours per year are higher
24/7, just like they were designed for (i hope... who ever turns off
servers?)
Seagate says 720 POH per month, but also 8,760 per year. That is quite close
to 24/7 but not exactly the same. Do hard drives benefit from a short
vacation?
|
No. That's actually a hit on most MTBF calculations, due to failures
that can only occur during device start-up. If you leave them on
24x365.25, there are no start-up events to worry about!
(Well, only the one, at the beginning...)
[ Snip ]
Malc. |
|
| Back to top |
|
|
flux
Guest
|
| Posted:
Tue Dec 07, 2004 11:27 am Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
In article <mqfar0193spg3c2n1l54ljtdm2iogjpcg0@4ax.com>,
Malcolm Weir <malc@gelt.org> wrote:
| Quote: | Nope. Gigabit is not relatively recent.
That's the point.
|
But it *is* recent. Actually, where's the evidence that is happening at
all.
| Quote: | Nope. It supports the assertion that gigabit is common, which you
claimed to untrue.
|
To be fair, that is so, but only indirectly. It is still possible for
not to be.
| Quote: | It says nothing about how long gigabit has been available, which is
what you seem to thing is terribly significant.
|
The time they introduced might.
| Quote: | (And I'd note you probably still haven't grasped that a tiny query can
trigger huge amounts of IO!).
|
So?
| Quote: | You want to claim that gigabit hasn't been around since 1999, and Dell
doesn't stick it in laptops and entry-level desktops, be my guest.
|
I didn't claim that, only that it was recent (if that).
| Quote: | I'll then provide the cites, you'll look silly (well, sillier than you
|
Let's see them.
| Quote: | Me, I architect multi-terabyte HSMs.
I didn't know they were so complex that to require an "architect".
I'm not surprised you didn't know that.
They are.
|
So you say. |
|
| Back to top |
|
|
J. Clarke
Guest
|
| Posted:
Tue Dec 07, 2004 7:33 pm Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
flux wrote:
| Quote: | In article <mqfar0193spg3c2n1l54ljtdm2iogjpcg0@4ax.com>,
Malcolm Weir <malc@gelt.org> wrote:
Nope. Gigabit is not relatively recent.
That's the point.
But it *is* recent. Actually, where's the evidence that is happening at
all.
|
You have an odd definition of "recent". In the computer industry 5 years is
an eternity. As to the "evidence" that it is "happening at all", Fry's has
gigabit NICs for $7.99 and five-port gigabit switches for $19.99. That
didn't happen because nobody was buying the stuff, that happened because
the production volume is enormous.
| Quote: | Nope. It supports the assertion that gigabit is common, which you
claimed to untrue.
To be fair, that is so, but only indirectly. It is still possible for
not to be.
|
Why is it so important to you that we believe that a common networking
technology, readily available, and becoming dirt cheap, is not common?
| Quote: | It says nothing about how long gigabit has been available, which is
what you seem to thing is terribly significant.
The time they introduced might.
|
The time who introduced what?
| Quote: | (And I'd note you probably still haven't grasped that a tiny query can
trigger huge amounts of IO!).
So?
|
Try to follow the argument.
| Quote: | You want to claim that gigabit hasn't been around since 1999, and Dell
doesn't stick it in laptops and entry-level desktops, be my guest.
I didn't claim that, only that it was recent (if that).
|
How is five years old "recent" in the computer industry?
| Quote: | I'll then provide the cites, you'll look silly (well, sillier than you
Let's see them.
Me, I architect multi-terabyte HSMs.
I didn't know they were so complex that to require an "architect".
I'm not surprised you didn't know that.
They are.
So you say.
|
So the computer industry says. People who put together and run systems like
that get paid quite a lot more than your average PC technician.
--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net) |
|
| Back to top |
|
|
Malcolm Weir
Guest
|
| Posted:
Tue Dec 07, 2004 8:36 pm Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
On Tue, 07 Dec 2004 06:27:59 GMT, flux <support@fluxsoft.com> wrote:
| Quote: | In article <mqfar0193spg3c2n1l54ljtdm2iogjpcg0@4ax.com>,
Malcolm Weir <malc@gelt.org> wrote:
Nope. Gigabit is not relatively recent.
That's the point.
But it *is* recent.
|
To you, maybe. But we've pretty much established you don't know what
you're talking about.
| Quote: | Actually, where's the evidence that is happening at
all.
|
Well, *somebody* thinks it worthwhile putting it in commodity
chipsets, which means it *is* happening, despite your ignorance.
| Quote: | Nope. It supports the assertion that gigabit is common, which you
claimed to untrue.
To be fair, that is so, but only indirectly. It is still possible for
not to be.
|
Are you not familiar with the word "support"? Unlike you, I'm not
claiming absolutes.
| Quote: | It says nothing about how long gigabit has been available, which is
what you seem to thing is terribly significant.
The time they introduced might.
|
Once again, 1999.
(Actually, it predated 1999. But that was when the standard was
done).
| Quote: | (And I'd note you probably still haven't grasped that a tiny query can
trigger huge amounts of IO!).
So?
|
So you don't *need* anything more than a 2400 baud phone line to
create a workload that involves more IO than you can get out of a SATA
drive. Which was what you seemed to think required gigabit ethernet!
| Quote: | You want to claim that gigabit hasn't been around since 1999, and Dell
doesn't stick it in laptops and entry-level desktops, be my guest.
I didn't claim that, only that it was recent (if that).
|
But 1999, kid, isn't recent. It's a (drive) lifetime ago.
| Quote: | I'll then provide the cites, you'll look silly (well, sillier than you
Let's see them.
|
OK. Go to www.dell.com. Look. Be enlightened.
Go to www.google.com. Enter "1000BaseT 1999". Look. Be enlightened.
Troll.
| Quote: | Me, I architect multi-terabyte HSMs.
I didn't know they were so complex that to require an "architect".
I'm not surprised you didn't know that.
They are.
So you say.
|
Indeed. However, unlike you, I do this for a living.
What do you do, again?
Oh, wait, you didn't answer last time, did you?
School out yet, kid?
Malc. |
|
| Back to top |
|
|
J. Clarke
Guest
|
| Posted:
Tue Dec 07, 2004 10:23 pm Post subject:
Re: What am I doing wrong ??? Or is Adaptec 21610SA just a c |
|
|
Malcolm Weir wrote:
| Quote: | On Tue, 07 Dec 2004 06:27:59 GMT, flux <support@fluxsoft.com> wrote:
In article <mqfar0193spg3c2n1l54ljtdm2iogjpcg0@4ax.com>,
Malcolm Weir <malc@gelt.org> wrote:
Nope. Gigabit is not relatively recent.
That's the point.
But it *is* recent.
To you, maybe. But we've pretty much established you don't know what
you're talking about.
Actually, where's the evidence that is happening at
all.
Well, *somebody* thinks it worthwhile putting it in commodity
chipsets, which means it *is* happening, despite your ignorance.
Nope. It supports the assertion that gigabit is common, which you
claimed to untrue.
To be fair, that is so, but only indirectly. It is still possible for
not to be.
Are you not familiar with the word "support"? Unlike you, I'm not
claiming absolutes.
It says nothing about how long gigabit has been available, which is
what you seem to thing is terribly significant.
The time they introduced might.
Once again, 1999.
(Actually, it predated 1999. But that was when the standard was
done).
(And I'd note you probably still haven't grasped that a tiny query can
trigger huge amounts of IO!).
So?
So you don't *need* anything more than a 2400 baud phone line to
create a workload that involves more IO than you can get out of a SATA
drive. Which was what you seemed to think required gigabit ethernet!
You want to claim that gigabit hasn't been around since 1999, and Dell
doesn't stick it in laptops and entry-level desktops, be my guest.
I didn't claim that, only that it was recent (if that).
But 1999, kid, isn't recent. It's a (drive) lifetime ago.
I'll then provide the cites, you'll look silly (well, sillier than you
Let's see them.
OK. Go to www.dell.com. Look. Be enlightened.
Go to www.google.com. Enter "1000BaseT 1999". Look. Be enlightened.
Troll.
Me, I architect multi-terabyte HSMs.
I didn't know they were so complex that to require an "architect".
I'm not surprised you didn't know that.
They are.
So you say.
Indeed. However, unlike you, I do this for a living.
What do you do, again?
Oh, wait, you didn't answer last time, did you?
School out yet, kid?
|
He slipped a bit back--he watches videos on his one ancient computer and
thinks that that's an adequate substitute for using a TV for a display.
Think about the implications of that. Odds are that he's a college kid
living in a dorm room and it hasn't occurred to him that people who have
lives generally live in more than one room and actually have friends who
don't want to all cluster around a computer screen to watch TV.
--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net) |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in