Network and Storage Protocols
- Home
- :
- ONTAP, AFF, and FAS
- :
- Network and Storage Protocols
- :
- Need some explaining on statit/perfstat
Network and Storage Protocols
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi
I need a few things explained and confirmed.
* In the RAID Statistics (per second) i see 22.56 partial stripes, 33.48 full stripes. Whats causing this? The aggr1_sas is 61% full and aggr1_sata is 24% full so it can't be related to full aggregate?
* blocks per stripe size is good on both aggregates?
* cpreads-chain-usecs only happens during writes and is low on this system. But i have filers where writes--chain-usecs almost matches cpreads-chain-usecs. What would cause this?
* The more disks in a Raid Group , the more load i get on the parity disks when the aggregate fills up?
Perfstat:
* How do i know the pw.over_limit is causing problems for VMware? Any number seen? 100k + ? Is there a way to calculate the precentage of impact? Same question regards pw.partial_writes.
NetApp Release 8.0.2P3 7-Mode: Thu Oct 13 15:56:25 PDT 2011
<1O>
Start time: Tue Mar 20 10:48:17 CET 2012
CPU Statistics
90.788229 time (seconds) 100 %
86.090012 system time 95 %
3.971232 rupt time 4 % (2035042 rupts x 2 usec/rupt)
82.118780 non-rupt system time 90 %
277.062900 idle time 305 %
6.979443 time in CP 8 % 100 %
0.252834 rupt time in CP 4 % (139558 rupts x 2 usec/rupt)
Multiprocessor Statistics (per second)
cpu0 cpu1 cpu2 cpu3 total
sk switches 49170.86 50303.76 58284.55 60764.53 218523.69
hard switches 16705.28 19006.45 30735.93 10775.63 77223.29
domain switches 946.14 1074.95 414.37 631.29 3066.75
CP rupts 1075.93 153.74 153.75 153.75 1537.18
nonCP rupts 15339.44 1846.22 1846.21 1846.21 20878.08
IPI rupts 0.00 0.00 0.00 0.00 0.00
grab kahuna 0.07 0.41 0.00 0.00 0.47
grab w_xcleaner 0.00 0.00 7.45 21.78 29.22
grab kahuna usec 4.59 26.15 0.00 0.00 30.74
grab w_xcleaner usec 0.00 0.00 2154.24 2229.41 4383.65
CP rupt usec 2495.91 168.28 68.58 52.09 2784.88
nonCP rupt usec 37568.92 1965.19 809.76 612.93 40956.82
idle 748292.01 740625.08 720309.80 842522.27 3051749.16
kahuna 22.38 20.05 8838.35 30476.23 39357.02
storage 45211.72 2226.46 5027.29 12387.69 64853.17
exempt 42521.46 48149.60 39653.00 2.60 130326.67
raid 13961.42 79354.67 2181.36 0.00 95497.47
target 374.44 557.08 546.86 0.00 1478.39
netcache 0.00 0.00 0.06 0.00 0.06
netcache2 0.00 0.00 0.00 0.00 0.00
cifs 489.90 522.20 440.95 0.00 1453.06
wafl_exempt 83054.39 82581.84 66435.37 113946.16 346017.79
wafl_xcleaner 1341.08 1083.16 935.20 0.00 3359.44
sm_exempt 21.02 27.64 23.48 0.00 72.15
cluster 0.00 0.00 0.00 0.00 0.00
protocol 31.13 31.49 25.86 0.00 88.49
nwk_exclusive 476.83 815.81 1183.52 0.00 2476.19
nwk_exempt 2.76 3.58 3.30 0.00 9.65
nwk_legacy 4092.84 6089.29 4399.15 0.00 14581.30
nwk_ctx1 3909.87 4752.50 4030.60 0.00 12692.98
nwk_ctx2 11949.62 26916.02 140547.07 0.00 179412.73
nwk_ctx3 0.00 0.00 0.00 0.00 0.00
nwk_ctx4 0.00 0.00 0.00 0.00 0.00
hostOS 4182.19 4109.98 4540.31 0.00 12832.51
44.913882 seconds with one or more CPUs active ( 49%)
25.689593 seconds with 2 or more CPUs active ( 28%)
9.661719 seconds with 3 or more CPUs active ( 11%)
19.224288 seconds with one CPU active ( 21%)
16.027874 seconds with 2 CPUs active ( 18%)
7.611255 seconds with 3 CPUs active ( 8%)
2.050463 seconds with all CPUs active ( 2%)
Domain Utilization of Shared Domains (per second)
0.00 idle 257418.84 kahuna
0.00 storage 0.00 exempt
0.00 raid 0.00 target
0.00 netcache 0.00 netcache2
0.00 cifs 0.00 wafl_exempt
0.00 wafl_xcleaner 0.00 sm_exempt
0.00 cluster 0.00 protocol
199955.91 nwk_exclusive 0.00 nwk_exempt
0.00 nwk_legacy 0.00 nwk_ctx1
0.00 nwk_ctx2 0.00 nwk_ctx3
0.00 nwk_ctx4 0.00 hostOS
Miscellaneous Statistics (per second)
77223.29 hard context switches 3415.84 NFS operations
35.75 CIFS operations 0.00 HTTP operations
5863.91 network KB received 428606.63 network KB transmitted
210026.79 disk KB read 2930.29 disk KB written
1566.90 NVRAM KB written 0.00 nolog KB written
51728.51 WAFL bufs given to clients 0.00 checksum cache hits ( 0%)
51541.18 no checksum - partial buffer 20.89 FCP operations
1.11 iSCSI operations
WAFL Statistics (per second)
39.07 name cache hits ( 86%) 6.21 name cache misses ( 14%)
161744.49 buf hash hits ( 58%) 115974.15 buf hash misses ( 42%)
5983.18 inode cache hits ( 100%) 4.55 inode cache misses ( 0%)
57736.59 buf cache hits ( 97%) 1939.41 buf cache misses ( 3%)
21.43 blocks read 51606.40 blocks read-ahead
1852.89 chains read-ahead 646.93 dummy reads
50999.43 blocks speculative read-ahead 616.88 blocks written
1.96 stripes written 0.00 blocks over-written
0.10 wafl_timer generated CP 0.00 snapshot generated CP
0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt generated CP
0.00 full NV-log generated CP 0.00 back-to-back CP
0.00 flush generated CP 0.00 sync generated CP
0.00 deferred back-to-back CP 0.00 container-indirect-pin CP
0.00 low mbufs generated CP 0.00 low datavecs generated CP
11457.49 non-restart messages 18.55 IOWAIT suspends
0.00 next nvlog nearly full msecs 0.00 dirty buffer susp msecs
0.00 nvlog full susp msecs 700572 buffers
RAID Statistics (per second)
78.60 xors 0.00 long dispatches [0]
0.00 long consumed [0] 0.00 long consumed hipri [0]
0.00 long low priority [0] 0.00 long high priority [0]
0.00 long monitor tics [0] 0.00 long monitor clears [0]
0.00 long dispatches [1] 0.00 long consumed [1]
0.00 long consumed hipri [1] 0.00 long low priority [1]
0.00 long high priority [1] 0.00 long monitor tics [1]
0.00 long monitor clears [1] 18 max batch
1.75 blocked mode xor 53.50 timed mode xor
0.06 fast adjustments 0.06 slow adjustments
0 avg batch start 0 avg stripe/msec
1879.85 checksum dispatches 115082.85 checksum consumed
2.36 tetrises written 0.00 master tetrises
0.00 slave tetrises 56.04 stripes written
22.56 partial stripes 33.48 full stripes
617.27 blocks written 73.02 blocks read
2.05 1 blocks per stripe size 12 0.62 2 blocks per stripe size 12
0.51 3 blocks per stripe size 12 1.10 4 blocks per stripe size 12
1.31 5 blocks per stripe size 12 0.29 6 blocks per stripe size 12
1.49 7 blocks per stripe size 12 1.43 8 blocks per stripe size 12
0.94 9 blocks per stripe size 12 1.48 10 blocks per stripe size 12
3.47 11 blocks per stripe size 12 13.78 12 blocks per stripe size 12
0.64 1 blocks per stripe size 14 0.52 2 blocks per stripe size 14
0.23 3 blocks per stripe size 14 0.21 4 blocks per stripe size 14
0.17 5 blocks per stripe size 14 0.18 6 blocks per stripe size 14
0.21 7 blocks per stripe size 14 0.21 8 blocks per stripe size 14
0.29 9 blocks per stripe size 14 0.37 10 blocks per stripe size 14
0.58 11 blocks per stripe size 14 1.32 12 blocks per stripe size 14
2.96 13 blocks per stripe size 14 19.71 14 blocks per stripe size 14
Network Interface Statistics (per second)
_CUT__
Disk Statistics (per second)
ut% is the percent of time the disk was busy.
xfers is the number of data-transfer commands issued per second.
xfers = ureads + writes + cpreads + greads + gwrites
chain is the average number of 4K blocks per command.
usecs is the average disk round-trip time per 4K block.
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/aggr1_sata/plex0/rg0:
0a.01.1 2 2.69 0.54 1.00 5531 1.81 4.87 798 0.34 8.55 1060 0.00 .... . 0.00 .... .
0a.01.2 3 2.85 0.53 1.00 6750 2.00 4.58 872 0.32 5.69 1939 0.00 .... . 0.00 .... .
0a.01.3 1 1.31 0.55 3.74 2390 0.45 11.78 605 0.31 9.86 993 0.00 .... . 0.00 .... .
4c.02.1 1 0.61 0.09 24.38 518 0.24 21.14 830 0.28 13.04 494 0.00 .... . 0.00 .... .
4c.03.1 0 0.58 0.08 9.57 746 0.24 21.50 702 0.26 12.33 557 0.00 .... . 0.00 .... .
0a.01.4 0 0.55 0.11 19.20 641 0.23 21.90 661 0.21 10.74 495 0.00 .... . 0.00 .... .
4c.02.2 1 0.70 0.14 16.38 484 0.28 18.60 798 0.29 8.69 584 0.00 .... . 0.00 .... .
4c.03.2 0 0.67 0.06 18.60 699 0.30 17.22 755 0.32 10.79 511 0.00 .... . 0.00 .... .
0a.01.5 1 0.69 0.18 22.00 446 0.25 20.22 555 0.26 10.21 469 0.00 .... . 0.00 .... .
4c.02.3 1 0.77 0.14 14.69 764 0.30 17.22 1120 0.33 9.37 520 0.00 .... . 0.00 .... .
4c.03.3 1 0.82 0.17 22.07 417 0.29 17.88 735 0.36 10.76 668 0.00 .... . 0.00 .... .
0a.01.6 1 0.89 0.20 12.61 436 0.32 16.03 789 0.37 8.15 773 0.00 .... . 0.00 .... .
4c.02.4 1 0.77 0.23 10.10 519 0.24 21.14 753 0.30 10.56 467 0.00 .... . 0.00 .... .
4c.03.4 1 0.87 0.29 15.31 606 0.24 21.14 566 0.34 10.06 567 0.00 .... . 0.00 .... .
/aggr1_sata/plex0/rg1:
4c.02.5 1 0.73 0.00 .... . 0.34 21.13 608 0.39 10.37 570 0.00 .... . 0.00 .... .
4c.03.5 1 0.74 0.00 .... . 0.36 19.97 669 0.37 8.79 719 0.00 .... . 0.00 .... .
0a.01.7 1 0.89 0.31 11.71 460 0.29 17.88 602 0.30 7.48 931 0.00 .... . 0.00 .... .
4c.02.6 0 0.72 0.21 9.42 955 0.24 21.09 532 0.26 8.96 763 0.00 .... . 0.00 .... .
4c.03.6 0 0.59 0.10 22.11 467 0.23 22.10 519 0.26 10.13 650 0.00 .... . 0.00 .... .
0a.01.8 1 0.90 0.34 14.32 300 0.30 17.19 756 0.26 9.21 833 0.00 .... . 0.00 .... .
4c.02.7 0 0.52 0.10 15.22 591 0.21 24.42 390 0.21 10.74 588 0.00 .... . 0.00 .... .
4c.03.7 0 0.57 0.19 20.71 332 0.20 25.78 476 0.19 13.59 532 0.00 .... . 0.00 .... .
0a.01.9 1 0.80 0.33 13.80 413 0.22 23.20 474 0.25 15.96 425 0.00 .... . 0.00 .... .
4c.02.8 1 0.80 0.35 9.94 830 0.23 22.19 573 0.22 10.00 860 0.00 .... . 0.00 .... .
4c.03.8 0 0.61 0.15 27.29 230 0.23 22.10 571 0.22 11.20 625 0.00 .... . 0.00 .... .
0a.01.10 0 0.54 0.14 15.08 709 0.19 27.29 325 0.21 10.79 624 0.00 .... . 0.00 .... .
4c.02.9 1 0.74 0.21 18.16 368 0.23 22.14 540 0.30 7.85 835 0.00 .... . 0.00 .... .
4c.03.9 0 0.66 0.09 24.13 523 0.23 22.14 497 0.34 7.61 792 0.00 .... . 0.00 .... .
/aggr1_sata/plex0/rg2:
4c.03.10 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.10 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.01.11 0 0.17 0.17 18.20 333 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.03.11 0 0.18 0.18 16.19 344 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.11 0 0.22 0.22 20.15 303 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.01.12 0 0.15 0.15 17.43 406 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.03.12 0 0.10 0.10 21.89 604 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.12 0 0.13 0.13 26.67 322 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.01.13 0 0.15 0.15 14.14 955 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.03.13 0 0.07 0.07 23.50 248 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.13 0 0.12 0.12 30.45 179 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.01.14 0 0.25 0.25 11.17 747 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.03.14 0 0.11 0.11 7.80 1949 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.14 0 0.03 0.03 43.00 240 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
/aggr1_sata/plex0/rg3:
0a.01.15 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.15 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.03.15 70 0.06 0.06 26.00 431 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.01.16 0 0.26 0.26 15.67 739 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.16 0 0.14 0.14 13.92 884 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.03.16 0 0.22 0.22 17.50 414 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.01.17 0 0.13 0.13 24.67 307 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.17 0 0.09 0.09 16.63 970 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.03.17 0 0.09 0.09 18.75 387 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.01.18 0 0.08 0.08 31.00 235 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.18 0 0.12 0.12 21.45 576 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.03.18 0 0.11 0.11 20.10 647 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.01.19 0 0.10 0.10 17.67 384 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
4c.02.19 0 0.10 0.10 20.89 628 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
/aggr1_sas/plex0/rg0:
4a.10.0 0 1.23 0.20 1.00 5056 0.65 21.71 172 0.39 13.91 160 0.00 .... . 0.00 .... .
0a.00.0 1 1.33 0.20 1.00 1556 0.85 17.10 227 0.29 3.38 807 0.00 .... . 0.00 .... .
4a.10.1 11 51.18 50.13 26.29 118 0.69 18.57 381 0.35 6.78 576 0.00 .... . 0.00 .... .
0a.00.1 11 49.48 48.71 26.85 110 0.44 28.40 295 0.33 10.57 315 0.00 .... . 0.00 .... .
4a.10.2 10 48.15 47.41 27.78 108 0.43 29.59 288 0.31 10.00 336 0.00 .... . 0.00 .... .
0a.00.2 11 48.20 47.46 28.03 108 0.42 30.45 310 0.32 11.34 304 0.00 .... . 0.00 .... .
4a.10.3 10 48.62 47.74 27.39 109 0.51 24.80 323 0.37 8.15 542 0.00 .... . 0.00 .... .
0a.00.3 11 49.39 48.49 27.42 112 0.52 24.40 359 0.39 8.00 543 0.00 .... . 0.00 .... .
4a.10.4 11 48.73 47.99 27.70 110 0.46 27.60 289 0.28 10.52 395 0.00 .... . 0.00 .... .
0a.00.4 10 48.61 47.75 27.68 109 0.47 26.77 275 0.39 9.63 398 0.00 .... . 0.00 .... .
4a.10.5 11 48.17 47.33 27.96 109 0.44 28.83 324 0.40 8.33 390 0.00 .... . 0.00 .... .
0a.00.5 11 50.68 49.94 26.47 111 0.47 27.09 327 0.26 10.17 422 0.00 .... . 0.00 .... .
4a.10.6 11 48.71 47.85 27.16 114 0.46 27.57 264 0.40 10.56 284 0.00 .... . 0.00 .... .
0a.00.6 10 48.97 48.23 27.62 106 0.42 30.50 287 0.32 11.52 296 0.00 .... . 0.00 .... .
4a.10.7 10 48.50 47.72 27.73 111 0.46 27.52 270 0.32 9.72 539 0.00 .... . 0.00 .... .
0a.00.7 10 48.98 48.14 27.63 103 0.52 24.32 332 0.33 7.53 487 0.00 .... . 0.00 .... .
/aggr1_sas/plex0/rg1:
4a.10.8 0 0.90 0.00 .... . 0.51 26.61 218 0.40 11.25 212 0.00 .... . 0.00 .... .
0a.00.8 0 0.90 0.00 .... . 0.51 26.61 212 0.40 11.25 227 0.00 .... . 0.00 .... .
4a.10.9 10 48.03 47.12 27.39 106 0.53 23.40 321 0.37 9.47 491 0.00 .... . 0.00 .... .
0a.00.9 10 47.17 46.26 28.60 96 0.59 21.06 368 0.31 7.39 565 0.00 .... . 0.00 .... .
4a.10.10 10 48.44 47.47 27.24 99 0.58 21.60 419 0.39 6.71 621 0.00 .... . 0.00 .... .
0a.00.10 10 47.96 46.96 27.62 104 0.56 22.16 360 0.44 6.95 514 0.00 .... . 0.00 .... .
4a.10.11 10 45.72 44.81 28.99 94 0.59 21.22 360 0.32 7.41 572 0.00 .... . 0.00 .... .
0a.00.11 10 47.33 46.16 28.08 96 0.66 18.63 530 0.51 5.85 721 0.00 .... . 0.00 .... .
4a.10.12 10 47.38 46.30 28.25 98 0.63 19.12 413 0.45 6.80 538 0.00 .... . 0.00 .... .
0a.00.12 10 47.19 46.21 27.95 102 0.56 22.20 360 0.42 4.66 757 0.00 .... . 0.00 .... .
4a.10.13 10 46.78 45.84 28.48 93 0.54 23.04 352 0.40 6.08 680 0.00 .... . 0.00 .... .
0a.00.13 9 46.03 45.19 28.40 96 0.50 25.60 299 0.34 8.32 512 0.00 .... . 0.00 .... .
4a.10.14 10 48.55 47.55 27.38 99 0.63 19.54 466 0.37 8.00 540 0.00 .... . 0.00 .... .
0a.00.14 10 46.98 45.89 28.45 96 0.63 19.14 387 0.46 6.02 549 0.00 .... . 0.00 .... .
4a.10.15 9 47.55 46.42 27.81 94 0.67 18.25 490 0.46 5.86 622 0.00 .... . 0.00 .... .
0a.00.15 9 46.74 45.68 28.42 89 0.59 20.70 409 0.46 7.05 463 0.00 .... . 0.00 .... .
/aggr1_sas/plex0/rg2:
0a.00.16 0 1.01 0.00 .... . 0.53 25.79 283 0.48 10.05 328 0.00 .... . 0.00 .... .
4a.10.16 0 1.02 0.00 .... . 0.53 25.79 287 0.50 10.13 296 0.00 .... . 0.00 .... .
0a.00.17 10 45.83 44.65 28.77 105 0.69 17.40 533 0.48 6.18 805 0.00 .... . 0.00 .... .
4a.10.17 10 47.75 46.57 27.77 96 0.74 16.40 554 0.44 6.63 630 0.00 .... . 0.00 .... .
0a.00.18 10 46.64 45.39 28.20 95 0.72 16.89 576 0.53 6.15 786 0.00 .... . 0.00 .... .
4a.10.18 10 46.23 45.06 28.66 98 0.69 17.52 447 0.47 7.00 522 0.00 .... . 0.00 .... .
0a.00.19 10 46.87 45.63 28.38 100 0.72 16.49 523 0.52 7.00 763 0.00 .... . 0.00 .... .
4a.10.19 10 48.52 47.41 27.40 98 0.68 17.68 619 0.43 9.54 454 0.00 .... . 0.00 .... .
0a.00.20 10 46.89 45.76 28.32 98 0.70 17.19 492 0.43 7.13 568 0.00 .... . 0.00 .... .
4a.10.20 10 46.21 45.05 28.36 99 0.66 17.73 533 0.50 7.40 502 0.00 .... . 0.00 .... .
0a.00.21 10 46.93 45.54 27.97 106 0.79 14.28 600 0.61 5.93 656 0.00 .... . 0.00 .... .
4a.10.21 10 47.96 46.78 28.05 95 0.68 17.23 560 0.50 6.56 515 0.00 .... . 0.00 .... .
0a.00.22 10 47.20 45.98 27.95 108 0.70 16.64 508 0.52 7.40 557 0.00 .... . 0.00 .... .
4a.10.22 10 47.39 46.19 27.80 105 0.68 17.58 518 0.52 7.28 547 0.00 .... . 0.00 .... .
Aggregate statistics:
Minimum 0 0.00 0.00 0.00 0.00 0.00 0.00
Mean 5 19.08 18.45 0.36 0.25 0.00 0.00
Maximum 70 51.18 50.13 2.00 0.61 0.00 0.00
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Looks like you were not able to help on this issue here. We have external and internal subject matter experts in the NetApp Support Community answering questions about Data ONTAP. If you have a NOW login, this link enables you to engage them about this. Mike
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Tommy,
Was this aggregate expanded after initial configuration? If so, was a reallocate done to re-strip the volumes? I've seen similar issues that have been caused when an aggr was created with a single raid group and then a 2nd rad group was added later.
Might be something to check to see if that's possible.
