Combining Volume Manager and QFS on JBOD
I have combined Volume Manager and QFS, SVM to mirror the disks and QFS for a file system.
I have a testbed host with two controllers connected to two 10 slot arrays;
Node A
c3t0d0 to c3t9d0
c4t0d0 to c4t9d0
I have mirrored target 0 on controller 3 to target 0 on controller 4 ( and so on through to target 9 ) eg;
# metainit d21 1 1 c3t0d0s7
# metainit d22 1 1 c4t0d0s7
# metainit d20 -m d21
# metattach d20 d22
Then I stripe the mirror pairs with Sun QFS;
# cat >> /etc/opt/SUNWsamfs/mcf
# Sun StorEdge QFS file system configurations
#
# EquipmentEquipmentEquipmentFamilyDeviceAdditional
# IdentifierOrdinalType SetStateParameters
# ---
qfs1100 maqfs1on
/dev/md/dsk/d20101 g1qfs1
/dev/md/dsk/d30102 g2qfs1
I can create a filesystem, mount and write data to the volume, the set up works very nicely, the performance is a little slow, but I am currently using 100MB/s fibre channel and an under powered system ( B1k 900Mhz/2GB RAM, but planned system will be SMP 8 to 16 CPU modules and 16GB to 64GB RAM ).
My questions are;
Would this be a viable configuration, compared with solutions that were based on Ex500, A5x00 and Veritas?
Will it be possible achieve better performance by implementing a solution on JBOD enclosures with larger SMP systems as opposed to hardware RAID and smaller 1 to 4 way systems? Can the kernel be tweaked enough to out perform the current hardware RAID controller features given the capacity of memory available with SMP servers?
I need to scale the system to a very large number of disks and a very large capacity. FC JBOD and SATA JBOD is a cheaper route to take, and the management of the system should be easier to maintain than more complicated and more expensive RAID array firmware. I would be most grateful to hear from any users who have been working with large volume JBOD based storage systems ( A5x00 ) and large SMP servers ( suggestions/opinions/criticim of this type of solution most welcome!! ).
<b>EDIT</b> I have edited this thread in an attempt to attract some feedback, come on, where are you guys? TIA.
[2238 byte] By [
m-lennon] at [2007-11-25 22:59:51]
# 1
I have combined Volume Manager and QFS, SVM to mirror the disks and QFS for a file system.
TB> my oh my Martin, what beast are you trying to create here :-)
I can create a filesystem, mount and write data to the volume, the set up works very nicely, the performance is a little slow
TB> slow compared with what; what are you expecting; are your expectations too high?
, but I am currently using 100MB/s fibre channel and an under powered system ( B1k 900Mhz/2GB RAM, but planned system will be SMP 8 to 16 CPU modules and 16GB to 64GB RAM ).
TB> Although there will be a *difference* in performance with the different systems, I would be looking at how you are measuring your 'performance' and if the above comes in to play. If your testing a storage system out, you want to be using some kind of I/O bound test, right.
My questions are;
Would this be a viable configuration, compared with solutions that were based on Ex500, A5x00 and Veritas?
TB> Impossible to say; what arrays are you using for this, you just mention 2 x 10slots :-)
Will it be possible achieve better performance by implementing a solution on JBOD enclosures with larger SMP systems as opposed to hardware RAID and smaller 1 to 4 way systems? Can the kernel be tweaked enough to out perform the current hardware RAID controller features given the capacity of memory available with SMP servers?
TB> It is technically possible to make a software based installation out perform a RAID based system, there is no dought about that one, but you have to be looking at setting all levels of the software components, from filesystem level downwards; through the buffers and VM layers etc.
I need to scale the system to a very large number of disks and a very large capacity. FC JBOD and SATA JBOD is a cheaper route to take, and the management of the system should be easier to maintain than more complicated and more expensive RAID array firmware.
TB> I would tend to disagree slightly with the management side of a RAID based subsystem to be honest. Take the SE3000 range for example, comes with nice interface and is easy to set up alert emails SNMP/SMTP blah blah....
I would be most grateful to hear from any users who have been working with large volume JBOD based storage systems ( A5x00 ) and large SMP servers ( suggestions/opinions/criticim of this type of solution most welcome!! ).
TB> I implemented (a little while back now though) a 'really good application response-time' storage env on A5x00s with hostbased VM (Vrts 3.something) and it flew.
If your after the same type of performance out of a software based VM/JBOS than you'll get out of a RAID based system, then your going to have to do a lot of benchmarking/tuning exersises to get near. Even a badly configured RAID based system can be hard to beat with S/W. Can be done though! :-)
EDIT I have edited this thread in an attempt to attract some feedback, come on, where are you guys? TIA.
[Updated on: Thu, 18 August 2005 18:04]
# 2
TB> my oh my Martin, what beast are you trying to create here :-)
As I mentioned I am looking a a way to reduce costs, but still maintain mirrored data and a fast filesystem.
TB> slow compared with what; what are you expecting; are your expectations too high?
I have configured a similar setup using A5200, SVM and UFS, when I configured VM and QFS combination I was a little disapointed, performance was more or less the same, but this is with standard configuration ( no kernel tuning ).
TB> Although there will be a *difference* in performance with the different systems, I would be looking at how you are measuring your 'performance' and if the above comes in to play. If your testing a storage system out, you want to be using some kind of I/O bound test, right.
I will run SWAT on the system over the weekend.
TB> Impossible to say; what arrays are you using for this, you just mention 2 x 10slots :-)
The arrays are EMC FC4700 DAE, but are just intended to emulate 3510 JBOD, have a low starting bugdet!!
TB> I would tend to disagree slightly with the management side of a RAID based subsystem to be honest. Take the SE3000 range for example, comes with nice interface and is easy to set up alert emails SNMP/SMTP blah blah....
I agree, the SE3000 range is a good system and it was wrong of me to claim that the software is complicated, but nothing is as easy as SVM!
The price difference between a 12 * 73GB JBOD 3500 and a 12 * 73GB RAID 3500 is almost 18k Euro, almost the cost of a second JBOD!
# 3
As I mentioned I am looking a a way to reduce costs, but still maintain mirrored data and a fast filesystem.
TB> I was joking with you, I know what your trying to do ;)
I have configured a similar setup using A5200, SVM and UFS, when I configured VM and QFS combination I was a little disapointed, performance was more or less the same, but this is with standard configuration ( no kernel tuning ).
I will run SWAT on the system over the weekend.
TB> Good start. see what you throw up.
# 4
Martin,
I am interested in your SWAT results? I also am in the testing phase of a hardware raid system striped with QFS as well as a JBOD striped with SVM raid 5.
Like your story management is pushing for a low price solution with +++TB of space, but they don't want to hear about the performance hit. SUN Support is in process of getting me SWAT and I can post my results.
error at 2007-7-5 17:48:57 >
# 5
Error, I missed you response to this thread and only noticed it today when I was searching for some info, I am sorry. Did you run SWAT on your setup?
# 6
Have not had a chance to run SWAT yet, but will cutting the system over to a more production like enviroment on the 19th for more testing.Will keep you posted.Did your project ever an out?
error at 2007-7-5 17:48:57 >
# 7
This project is still pending, we were talking about buying various smp systems ( Ex500 or SF4800 ) a couple of months back, but I decided to wait for the Niagara systems to be released. The entry Niagara system has 6 cores and is in an affordable price range, so I am much happier to work with new systems as opposed to the EOL systems I have been working with to date. At the time I started this thread I hadn't a good grasp of performance tuning for Solaris and I have been working on some methods to improve storage performance, but I don't have very good statistics to date ( I haven't configured a production db and tested it properly yet ). I will be getting more into it now over the next few weeks because I have a couple of customers that are expressing an interest in our storage solutions.
