CASTalk.com

Ernest Siu · Guest

Any know know much about the solutions from Panasas? The PanFS looks
similar to clustered filesystem like CxFS, GFS, StorNext,
Lustre...etc. except it is object-based. Their solution seems to
include the backend storage, control blades, and some NFS/CIFS heads
for interop. What is the benefit of using them versus getting Lustre
and a regular SAN? Any idea?

Regards,
Ernest

Faeandar · Guest

On 1 Dec 2004 10:20:06 -0800, ernestsiu@yahoo.com (Ernest Siu) wrote:

Wolf · Guest

"Faeandar" <mr_castalot@yahoo.com> wrote in message
news:l26sq0dbkfraja25cikhoc7qmara8mma6j@4ax.com...

Keith Michaels · Guest

In article <Z0Erd.35310$bP2.18751@newssvr12.news.prodigy.com>,
"Wolf" <wolfdotcom@hotmail.com> writes:
|> "Faeandar" <mr_castalot@yahoo.com> wrote in message
|> news:l26sq0dbkfraja25cikhoc7qmara8mma6j@4ax.com...
|> > On 1 Dec 2004 10:20:06 -0800, ernestsiu@yahoo.com (Ernest Siu) wrote:
|> >
|> > >Any know know much about the solutions from Panasas? The PanFS looks
|> > >similar to clustered filesystem like CxFS, GFS, StorNext,
|> > >Lustre...etc. except it is object-based. Their solution seems to
|> > >include the backend storage, control blades, and some NFS/CIFS heads
|> > >for interop. What is the benefit of using them versus getting Lustre
|> > >and a regular SAN? Any idea?
|> > >
|> > >Regards,
|> > >Ernest
|>
|> Hey Guys. I run a large cluster with a lot of storage so that is my
|> perspective.
|>
|> > Well first thing is don't think about Lustre, it's a beast still and
|> > not ready for primetime.
|>
|> Lustre is a lot of work-there is no doubt about it. There are no
|> real tools (GUIs etc) to help. But they make their money in
|> customization I figure. What is really a good thing about Lustre
|> is that it is very scaleable. Most of the GFSes I have seen like
|> SANs and most of our boxes are NAS boxes. Lustre will let
|> us manage all our boxes (OSTs).
|>
|> > The others you mentioned are viable but the
|> > leaders today seem to be Polyserve, Ibrix, GPFS, and GFS. Personally
|> > I like Polyserve personally but Ibrix and GPFS both look good as well.
....

There is something I don't understand about these distributed
filesystems: how is the namespace mapped to the devices in a fully
distributed environment? If there is complete independence between
the name layer and the storage layer then there is an n-squared
locking problem, if devices are tied to the namespace then there
is a load-leveling/utilization problem.

Faeandar · Guest

On Thu, 2 Dec 2004 19:46:08 GMT, krm@sdc.cs.boeing.com (Keith
Michaels) wrote:

Faeandar · Guest

On Thu, 02 Dec 2004 12:38:17 GMT, "Wolf" <wolfdotcom@hotmail.com>
wrote:

Keith Michaels · Guest

In article <h8avq0dhipa8l2p7moaf3fbga9bs6h3j77@4ax.com>,
Faeandar <mr_castalot@yahoo.com> writes:
....

|> >There is something I don't understand about these distributed
|> >filesystems: how is the namespace mapped to the devices in a fully
|> >distributed environment? If there is complete independence between
|> >the name layer and the storage layer then there is an n-squared
|> >locking problem, if devices are tied to the namespace then there
|> >is a load-leveling/utilization problem.
|>
|> Yes, and yes (though maybe not squared). In most cases the products
|> use a VIP that clients mount. In cases like Polyserve and GFS the
|> client requests are round-robin'd among the node servers. Not exactly
|> load balancing but over time and enough node servers the balance finds
|> itself.
|>
|> Products like Acopia and Panasas have internal algorithms that they
|> user for latency testing and then write to the node server that
|> returns the fastest. Good to some extent but in the case of Acopia,
|> not being a clustered file system, it means a potential for one system
|> to house alot more data than the others. Maybe a problem, maybe not.
|>
|> The only time you have a potential locking issue is with a clustered
|> file system and a dedicated metadata server. This can become a
|> serious bottleneck. Some products use distributed metadata which,
|> while not alleviating the problem completely, greatly reduces it.
|>
|> ~F

If all nodes are equal (distributed metadata case) then there needs
to be a mapping from filesystem semantics to the underlying
consistency model implemented by the storage system e.g., if
filesystems guarantee strict write ordering (reads always return
the latest write) then the storage system must implement equally
strong cache coherency between all nodes, which doesn't scale.
So there's a tradeoff between performance and utilization. I was
just wondering how modern DFSs deal with this fundamental
limitation. SMPs have hardware support for cache management;
do we need similar functionality in the SAN?

Bill Todd · Guest

"Keith Michaels" <krm@sdc.cs.boeing.com> wrote in message
news:I85orB.A8u@news.boeing.com...

Faeandar · Guest

On Fri, 3 Dec 2004 14:29:21 -0500, "Bill Todd"
<billtodd@metrocast.net> wrote:

Keith Michaels · Guest

In article <18ydndgNdq8EIy3cRVn-oQ@metrocastcablevision.com>,
"Bill Todd" <billtodd@metrocast.net> writes:
|>
|> "Keith Michaels" <krm@sdc.cs.boeing.com> wrote in message
|> news:I85orB.A8u@news.boeing.com...
|> > In article <h8avq0dhipa8l2p7moaf3fbga9bs6h3j77@4ax.com>,
|> > Faeandar <mr_castalot@yahoo.com> writes:
|> > ...
|> >
|> > |> >There is something I don't understand about these distributed
|> > |> >filesystems: how is the namespace mapped to the devices in a fully
|> > |> >distributed environment? If there is complete independence between
|> > |> >the name layer and the storage layer then there is an n-squared
|> > |> >locking problem, if devices are tied to the namespace then there
|> > |> >is a load-leveling/utilization problem.
|> > |>
|> > |> Yes, and yes (though maybe not squared). In most cases the products
|> > |> use a VIP that clients mount. In cases like Polyserve and GFS the
|> > |> client requests are round-robin'd among the node servers. Not exactly
|> > |> load balancing but over time and enough node servers the balance finds
|> > |> itself.
|> > |>
|> > |> Products like Acopia and Panasas have internal algorithms that they
|> > |> user for latency testing and then write to the node server that
|> > |> returns the fastest. Good to some extent but in the case of Acopia,
|> > |> not being a clustered file system, it means a potential for one system
|> > |> to house alot more data than the others. Maybe a problem, maybe not.
|> > |>
|> > |> The only time you have a potential locking issue is with a clustered
|> > |> file system and a dedicated metadata server. This can become a
|> > |> serious bottleneck. Some products use distributed metadata which,
|> > |> while not alleviating the problem completely, greatly reduces it.
|> > |>
|> > |> ~F
|> >
|> > If all nodes are equal (distributed metadata case) then there needs
|> > to be a mapping from filesystem semantics to the underlying
|> > consistency model implemented by the storage system e.g., if
|> > filesystems guarantee strict write ordering (reads always return
|> > the latest write) then the storage system must implement equally
|> > strong cache coherency between all nodes, which doesn't scale.
|> > So there's a tradeoff between performance and utilization. I was
|> > just wondering how modern DFSs deal with this fundamental
|> > limitation. SMPs have hardware support for cache management;
|> > do we need similar functionality in the SAN?
|>
|> 'Distributed metadata' means different things in different contexts.
|>
|> On VMS clusters, it refers to distributed management of metadata stored on
|> shared devices by cooperating hosts. But even there only a single host
|> manages a given metadatum at any given time (using the distributed locking
|> facility). This approach scales to hundreds of host nodes (the nominal
|> supported limit on cluster size is 96, but sizes close to 200 have been used
|> in practice), though contention varies with the degree of access locality to
|> a given datum (the worst case obviously being if all hosts access it
|> equally, with intense update activity).
|>
|> In implementations like Lustre, it refers to low-level local metadata stored
|> on all storage units, plus file-level metadata stored in a coordinating
|> central metadata server (or metadata server cluster). The low-level local
|> metadata is completely partitioned (i.e., of only local significance) and
|> hence has no scaling problem - but by offloading that portion of metadata
|> management from the central metadata server helps it scale too.
|>
|> Another alternative is to partition higher-level metadata as well (e.g.,
|> localizing the management of metadata for any given file - or even
|> directory - to a single server (plus perhaps a mirror partner), even if the
|> file data may be spread more widely). This certainly qualifies as
|> distributed metadata, even though management of any single metadatum is not
|> distributed, and scales well at the file system level as long as the
|> partitioning granularity manages to spread out metadata loads reasonably
|> (e.g., you'd have a problem if all system activity concentrated on one
|> humongous database file - though with some ingenuity subdividing at least
|> some of the metadata management even within a single file is possible).
|>
|> Until SAN bandwidths and latencies approach those of local RAM much more
|> closely than they do today, there will likely be little reason to use
|> hardware management of distributed file caches - especially since the most
|> effective location for such caching is very often (save in cases of intense
|> contention) at the client itself, where special hardware is least likely to
|> be found.
|>
|> - bill

Thanks Bill, that's a very interesting assessment. It seems to me
that the promise of grid-based/object storage is scalability; if
we lose that we're back where we started. Lustre cleverly exploits
the difference between local and global attributes but in the end
all of these DFSs lose scalability when trying to emulate strictly
ordered filesystem semantics and centralized resource management
without making assumptions about sharing and access patterns.
Maybe there's a perfect technology coming that will do it all; in
the meantime what is the business case for DFS deployment? Maybe
a better question is how do we move beyond the filesystem model
so that these technologies fit our applications better?

Guest

In article <6hg1r0h8hide29uohvkj5kbdaaa3i19ifp@4ax.com>,
Faeandar <mr_castalot@yahoo.com> wrote:

Faeandar · Guest

On Fri, 03 Dec 2004 21:37:48 -0000,
_firstname_@lr_dot_los-gatos_dot_ca.us wrote:

Faeandar · Guest

On Fri, 3 Dec 2004 20:45:16 GMT, krm@sdc.cs.boeing.com (Keith
Michaels) wrote:

Bill Todd · Guest

"Faeandar" <mr_castalot@yahoo.com> wrote in message
news:6hg1r0h8hide29uohvkj5kbdaaa3i19ifp@4ax.com...

Keith Michaels · Guest

In article <jrp1r0hj9qjckuiq0vr7i6guj8adeqamrf@4ax.com>,
Faeandar <mr_castalot@yahoo.com> writes:
|> On Fri, 3 Dec 2004 20:45:16 GMT, krm@sdc.cs.boeing.com (Keith
|> Michaels) wrote:
|>
|> >In article <18ydndgNdq8EIy3cRVn-oQ@metrocastcablevision.com>,
|> > "Bill Todd" <billtodd@metrocast.net> writes:
|> >|>
|> >|> "Keith Michaels" <krm@sdc.cs.boeing.com> wrote in message
|> >|> news:I85orB.A8u@news.boeing.com...
|> >|> > In article <h8avq0dhipa8l2p7moaf3fbga9bs6h3j77@4ax.com>,
|> >|> > Faeandar <mr_castalot@yahoo.com> writes:
|> >|> > ...
|> >|> >
|> >|> > |> >There is something I don't understand about these distributed
|> >|> > |> >filesystems: how is the namespace mapped to the devices in a fully
|> >|> > |> >distributed environment? If there is complete independence between
|> >|> > |> >the name layer and the storage layer then there is an n-squared
|> >|> > |> >locking problem, if devices are tied to the namespace then there
|> >|> > |> >is a load-leveling/utilization problem.
|> >|> > |>
|> >|> > |> Yes, and yes (though maybe not squared). In most cases the products
|> >|> > |> use a VIP that clients mount. In cases like Polyserve and GFS the
|> >|> > |> client requests are round-robin'd among the node servers. Not exactly
|> >|> > |> load balancing but over time and enough node servers the balance finds
|> >|> > |> itself.
|> >|> > |>
|> >|> > |> Products like Acopia and Panasas have internal algorithms that they
|> >|> > |> user for latency testing and then write to the node server that
|> >|> > |> returns the fastest. Good to some extent but in the case of Acopia,
|> >|> > |> not being a clustered file system, it means a potential for one system
|> >|> > |> to house alot more data than the others. Maybe a problem, maybe not.
|> >|> > |>
|> >|> > |> The only time you have a potential locking issue is with a clustered
|> >|> > |> file system and a dedicated metadata server. This can become a
|> >|> > |> serious bottleneck. Some products use distributed metadata which,
|> >|> > |> while not alleviating the problem completely, greatly reduces it.
|> >|> > |>
|> >|> > |> ~F
|> >|> >
|> >|> > If all nodes are equal (distributed metadata case) then there needs
|> >|> > to be a mapping from filesystem semantics to the underlying
|> >|> > consistency model implemented by the storage system e.g., if
|> >|> > filesystems guarantee strict write ordering (reads always return
|> >|> > the latest write) then the storage system must implement equally
|> >|> > strong cache coherency between all nodes, which doesn't scale.
|> >|> > So there's a tradeoff between performance and utilization. I was
|> >|> > just wondering how modern DFSs deal with this fundamental
|> >|> > limitation. SMPs have hardware support for cache management;
|> >|> > do we need similar functionality in the SAN?
|> >|>
|> >|> 'Distributed metadata' means different things in different contexts.
|> >|>
|> >|> On VMS clusters, it refers to distributed management of metadata stored on
|> >|> shared devices by cooperating hosts. But even there only a single host
|> >|> manages a given metadatum at any given time (using the distributed locking
|> >|> facility). This approach scales to hundreds of host nodes (the nominal
|> >|> supported limit on cluster size is 96, but sizes close to 200 have been used
|> >|> in practice), though contention varies with the degree of access locality to
|> >|> a given datum (the worst case obviously being if all hosts access it
|> >|> equally, with intense update activity).
|> >|>
|> >|> In implementations like Lustre, it refers to low-level local metadata stored
|> >|> on all storage units, plus file-level metadata stored in a coordinating
|> >|> central metadata server (or metadata server cluster). The low-level local
|> >|> metadata is completely partitioned (i.e., of only local significance) and
|> >|> hence has no scaling problem - but by offloading that portion of metadata
|> >|> management from the central metadata server helps it scale too.
|> >|>
|> >|> Another alternative is to partition higher-level metadata as well (e.g.,
|> >|> localizing the management of metadata for any given file - or even
|> >|> directory - to a single server (plus perhaps a mirror partner), even if the
|> >|> file data may be spread more widely). This certainly qualifies as
|> >|> distributed metadata, even though management of any single metadatum is not
|> >|> distributed, and scales well at the file system level as long as the
|> >|> partitioning granularity manages to spread out metadata loads reasonably
|> >|> (e.g., you'd have a problem if all system activity concentrated on one
|> >|> humongous database file - though with some ingenuity subdividing at least
|> >|> some of the metadata management even within a single file is possible).
|> >|>
|> >|> Until SAN bandwidths and latencies approach those of local RAM much more
|> >|> closely than they do today, there will likely be little reason to use
|> >|> hardware management of distributed file caches - especially since the most
|> >|> effective location for such caching is very often (save in cases of intense
|> >|> contention) at the client itself, where special hardware is least likely to
|> >|> be found.
|> >|>
|> >|> - bill
|> >
|> >Thanks Bill, that's a very interesting assessment. It seems to me
|> >that the promise of grid-based/object storage is scalability; if
|> >we lose that we're back where we started. Lustre cleverly exploits
|> >the difference between local and global attributes but in the end
|> >all of these DFSs lose scalability when trying to emulate strictly
|> >ordered filesystem semantics and centralized resource management
|> >without making assumptions about sharing and access patterns.
|> >Maybe there's a perfect technology coming that will do it all; in
|> >the meantime what is the business case for DFS deployment? Maybe
|> >a better question is how do we move beyond the filesystem model
|> >so that these technologies fit our applications better?
|>
|> I don't think that's strictly true, that we lose scalability. A
|> single file system by today's norm is over a single host, so right out
|> of the gate the CFS's beat that, DFS's too for that matter. The
|> purpose of these file systems came out of high performance computing
|> needs, basically striping writes/reads across multiple hosts to gain
|> (in theory) linear performance increases per node addition. Obviously
|> that's not the case but it definitely improves performance
|> dramatically. So scalability is already there.
|>
|> Can it scale to 10,000 nodes? Well not currently (read-only not
|> withstanding) but 10 years ago scaling this type of file system to
|> anything beyond 1 server was voodoo in the open systems world.
|>
|> ~F

The way to preserve scalability is to stripe, which improves data
bandwidth to a host, and to partition the metadata, which controls
the cost of allocation and other metadata updates. Both of these
strategies reduce utilization; striping across a SAN gobbles
up switch bandwidth geometrically, partitioning the metadata
either suffers latency (metaserver far away), or uneven allocation
(nearby storage is full). We need global load-leveling
to achieve true grid-based storage and managing the namespace
outside the hosts is part of that; so we put the namespace and
global attributes in LDAP, let it handle metadata locality and let
OSDs handle the rest. Just my $.02

	CASTalk.com Forum Index -> Storage System	All times are GMT Goto page 1, 2 Next
Page 1 of 2

PanFS? Goto page 1, 2 Next