Subscribe

Node disk fragmentation

Hello,

We have several clusters with spinning disks (with and without flash pool). On most of them we observe events for "node disk fragmentation"  (high node utilization caused by high percentage of delayed disk write) despite that free space realloc is enabled everywhere. I have even tried to schedule from node shell the old-style full aggregate reallocation which should help for free space fragmentation but that's also not helping much (or for a long time). 

How can I verify that the free space realloc is working and doing it's job ? 

Re: Node disk fragmentation

Hi,

 

Refer KB https://kb.netapp.com/support/s/article/ka11A0000001Z2wQAE/oncommand-performance-manager-alert-node-disk-fragmentation (needs login)

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

Re: Node disk fragmentation

I don't think this is it. We do really have fragmentation and we do see it on most of the clusters regardless if flash pool is used or not and we are running OCPM 7.1 at the moment. 

 

In statis output I see bad RAID statistics and 2:1 cpreads:writes

 

                     RAID Statistics (per second)
   6735.19 xors                              0.00 long dispatches [0]
      0.00 long consumed [0]                 0.00 long consumed hipri [0]
      0.00 long low priority [0]             0.00 long high priority [0]
     99.89 long monitor tics [0]             0.02 long monitor clears [0]
      0.00 long dispatches [1]               0.00 long consumed [1]
      0.00 long consumed hipri [1]           0.00 long low priority [1]
     99.89 long high priority [1]           99.89 long monitor tics [1]
      0.02 long monitor clears [1]             18 max batch
      4.87 blocked mode xor               1142.43 timed mode xor
      6.54 fast adjustments                  7.73 slow adjustments
         0 avg batch start                      0 avg stripe/msec
      0.00 checksum dispatches               0.00 checksum consumed
     56.29 tetrises written                  0.00 master tetrises
      0.00 slave tetrises                 3384.92 stripes written
   3347.99 partial stripes                  36.93 full stripes
  28699.47 blocks written                26734.05 blocks read
     20.66 1 blocks per stripe size 1      152.46 1 blocks per stripe size 20
    179.10 2 blocks per stripe size 20     201.02 3 blocks per stripe size 20
    218.03 4 blocks per stripe size 20     240.61 5 blocks per stripe size 20
    251.40 6 blocks per stripe size 20     260.87 7 blocks per stripe size 20
    261.86 8 blocks per stripe size 20     248.86 9 blocks per stripe size 20
    229.94 10 blocks per stripe size 20    211.66 11 blocks per stripe size 20
    193.01 12 blocks per stripe size 20    168.48 13 blocks per stripe size 20
    144.02 14 blocks per stripe size 20    123.53 15 blocks per stripe size 20
     99.40 16 blocks per stripe size 20     76.79 17 blocks per stripe size 20
     55.66 18 blocks per stripe size 20     31.30 19 blocks per stripe size 20
     16.27 20 blocks per stripe size 20


disk             ut%  xfers  ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/AGGR_ROOT_03/plex0/rg0:
0a.30.0            2   2.70    0.35   1.51   639   2.24   9.81   779   0.11   5.91  1001   0.00   ....     .   0.00   ....     .
0a.31.0            2   3.09    0.35   1.51   687   2.73   8.40   919   0.01   1.00  5732   0.00   ....     .   0.00   ....     .
4b.32.0            2   4.46    2.93   1.08  2835   1.04  19.88   377   0.48   1.00  5714   0.00   ....     .   0.00   ....     .
/AGGR_DATA_03/plex0/rg0:
4b.32.1           22  40.03    0.00   ....     .  18.82  59.68   151  21.22  53.81   110   0.00   ....     .   0.00   ....     .
4b.31.1           23  40.86    0.00   ....     .  19.66  57.20   163  21.20  53.76   118   0.00   ....     .   0.00   ....     .
0a.30.1           78 203.91  152.31   3.84  3550  16.00  29.72   615  35.59  10.52  1027   0.00   ....     .   0.00   ....     .
4b.32.2           75 205.12  154.42   3.83  2950  15.52  31.39   554  35.18  11.00   933   0.00   ....     .   0.00   ....     .
0a.31.2           75 206.01  155.64   3.78  3100  15.55  30.66   557  34.82  11.26   848   0.00   ....     .   0.00   ....     .
0a.30.2           79 206.50  155.46   3.83  3552  15.42  31.79   603  35.62  10.92  1046   0.00   ....     .   0.00   ....     .
4b.32.3           75 206.53  155.80   3.80  3080  15.90  31.29   571  34.83  11.67   840   0.00   ....     .   0.00   ....     .
4b.31.3           74 203.55  152.00   3.92  2812  15.46  31.63   564  36.09  10.82   934   0.00   ....     .   0.00   ....     .
0a.30.3           78 205.15  154.34   3.72  3615  15.52  30.74   590  35.29  11.00   951   0.00   ....     .   0.00   ....     .
4b.32.4           75 205.29  153.46   3.89  2993  15.53  30.75   589  36.31  10.81   942   0.00   ....     .   0.00   ....     .
0a.31.4           75 203.92  153.00   3.84  3040  15.36  30.40   572  35.55  10.70   955   0.00   ....     .   0.00   ....     .
0a.30.4           79 203.39  152.56   3.91  3519  15.37  29.95   631  35.45  11.01   991   0.00   ....     .   0.00   ....     .
4b.32.5           72 198.69  147.49   3.90  2796  15.52  31.59   536  35.68  11.03   866   0.00   ....     .   0.00   ....     .
4b.31.5           75 200.52  150.44   3.94  2906  15.40  30.92   554  34.68  10.96   919   0.00   ....     .   0.00   ....     .
0a.30.5           77 200.43  148.21   3.93  3440  15.76  30.49   598  36.46  10.91   942   0.00   ....     .   0.00   ....     .
4b.32.6           76 204.50  153.70   3.89  3019  15.56  31.83   612  35.24  11.04   939   0.00   ....     .   0.00   ....     .
0a.31.6           76 211.24  160.19   3.70  3059  15.48  31.64   559  35.57  11.06   910   0.00   ....     .   0.00   ....     .
0a.30.6           78 206.84  155.03   3.79  3545  15.66  30.25   601  36.15  10.67  1036   0.00   ....     .   0.00   ....     .
4b.32.7           75 207.19  156.46   3.81  2985  15.43  31.30   557  35.30  11.15   859   0.00   ....     .   0.00   ....     .
4b.31.7           75 207.70  157.63   3.88  2885  15.35  30.69   563  34.72  11.14   908   0.00   ....     .   0.00   ....     .
0a.30.7           78 201.06  149.99   3.91  3501  15.61  31.04   588  35.46  11.24   963   0.00   ....     .   0.00   ....     .
4b.32.8           74 204.21  153.27   3.76  3029  15.33  31.27   579  35.61  10.80   889   0.00   ....     .   0.00   ....     .
/AGGR_DATA_03/plex0/rg1:
0a.31.8           21  38.83    0.00   ....     .  18.05  61.33   140  20.78  54.30   111   0.00   ....     .   0.00   ....     .
0a.30.8           22  39.07    0.00   ....     .  18.31  60.49   147  20.76  54.29   113   0.00   ....     .   0.00   ....     .
4b.32.9           76 209.23  157.00   3.80  3037  15.64  30.20   580  36.59  10.98   894   0.00   ....     .   0.00   ....     .
4b.31.9           74 201.50  149.04   4.03  2753  15.64  30.40   567  36.82  10.92   927   0.00   ....     .   0.00   ....     .
0a.30.9           80 213.00  160.05   3.71  3690  15.58  30.78   615  37.37  10.65  1030   0.00   ....     .   0.00   ....     .
4b.32.10          74 210.20  158.12   3.87  2858  15.58  30.31   578  36.50  10.90   902   0.00   ....     .   0.00   ....     .
0a.31.10          75 211.73  160.03   3.87  2896  15.54  29.41   571  36.17  11.02   916   0.00   ....     .   0.00   ....     .
0a.30.10          79 207.73  155.23   3.94  3323  15.55  31.23   576  36.94  10.83  1031   0.00   ....     .   0.00   ....     .
4b.32.11          74 199.78  146.79   3.89  2940  15.46  30.21   560  37.53  10.88   918   0.00   ....     .   0.00   ....     .
4b.31.11          75 215.56  162.85   3.77  2936  15.42  30.67   557  37.29  10.78   967   0.00   ....     .   0.00   ....     .
0a.30.11          80 209.90  158.23   3.80  3627  15.53  30.01   628  36.14  11.02  1011   0.00   ....     .   0.00   ....     .
4b.32.12          74 204.92  152.36   3.73  3067  15.51  30.20   575  37.05  10.85   902   0.00   ....     .   0.00   ....     .
0a.31.12          76 210.26  157.36   3.98  2930  15.61  30.77   581  37.28  10.95   916   0.00   ....     .   0.00   ....     .
0a.30.12          79 208.94  156.94   3.77  3383  15.32  30.44   570  36.68  10.86  1061   0.00   ....     .   0.00   ....     .
4b.32.13          73 200.18  148.01   3.86  2998  15.37  30.66   586  36.80  10.73   920   0.00   ....     .   0.00   ....     .
4b.31.13          75 206.07  153.50   3.93  3050  15.52  30.68   591  37.05  10.68   952   0.00   ....     .   0.00   ....     .
0a.30.13          79 210.68  159.01   3.91  3587  15.35  30.28   628  36.32  11.17  1010   0.00   ....     .   0.00   ....     .
4b.32.14          75 215.53  162.97   3.70  3033  15.47  30.04   596  37.09  10.42   955   0.00   ....     .   0.00   ....     .
0a.31.14          75 208.71  156.31   3.91  2933  15.58  30.40   556  36.82  10.67   968   0.00   ....     .   0.00   ....     .
0a.30.14          78 208.60  156.47   3.87  3331  15.38  31.13   592  36.75  10.86   975   0.00   ....     .   0.00   ....     .
4b.32.15          76 213.41  160.78   3.86  2947  15.53  29.74   590  37.10  10.62   967   0.00   ....     .   0.00   ....     .
4b.31.15          76 213.52  161.41   3.66  3131  15.46  30.87   592  36.65  10.94   933   0.00   ....     .   0.00   ....     .
/AGGR_DATA_03/plex0/rg2:
4b.32.16          22  39.13    0.00   ....     .  18.42  61.67   147  20.71  55.77   112   0.00   ....     .   0.00   ....     .
0a.30.15          22  39.38    0.00   ....     .  18.70  60.80   145  20.69  55.71   109   0.00   ....     .   0.00   ....     .
0a.31.16          74 206.72  155.79   3.81  2759  15.51  31.00   508  35.42  11.25   861   0.00   ....     .   0.00   ....     .
4b.32.17          75 203.32  153.05   3.80  2983  15.56  30.48   539  34.70  11.57   817   0.00   ....     .   0.00   ....     .
0a.30.22          79 207.19  157.11   3.77  3437  15.58  31.52   551  34.50  11.26   951   0.00   ....     .   0.00   ....     .
4b.31.17          75 206.84  156.15   3.78  3003  15.60  31.31   533  35.09  11.21   885   0.00   ....     .   0.00   ....     .
4b.32.18          74 206.46  156.73   3.88  2796  15.35  31.23   541  34.38  11.36   860   0.00   ....     .   0.00   ....     .
0a.30.17          79 203.93  153.23   3.78  3518  15.52  30.45   588  35.17  11.00   991   0.00   ....     .   0.00   ....     .
0a.31.18          75 209.19  158.20   3.84  2883  15.51  31.05   546  35.48  11.20   904   0.00   ....     .   0.00   ....     .
4b.32.19          75 206.44  156.06   3.64  3185  15.38  31.45   535  35.00  11.43   880   0.00   ....     .   0.00   ....     .
0a.30.18          78 212.69  161.99   3.68  3373  15.37  31.37   558  35.33  11.15  1001   0.00   ....     .   0.00   ....     .
4b.31.19          75 205.36  153.75   3.85  2967  15.92  31.47   541  35.69  11.41   907   0.00   ....     .   0.00   ....     .
4b.32.20          74 211.59  161.04   3.71  2857  15.64  30.92   537  34.91  11.34   873   0.00   ....     .   0.00   ....     .
0a.30.19          78 206.73  155.74   3.69  3505  15.71  30.55   569  35.28  11.64   926   0.00   ....     .   0.00   ....     .
0a.31.20          75 210.89  160.87   3.76  2957  15.26  30.84   556  34.76  11.25   897   0.00   ....     .   0.00   ....     .
4b.32.21          74 202.38  151.96   3.88  2891  15.68  30.79   526  34.74  11.45   840   0.00   ....     .   0.00   ....     .
0a.30.20          79 203.98  153.04   3.93  3474  15.76  30.84   575  35.17  11.58   932   0.00   ....     .   0.00   ....     .
4b.31.21          75 211.86  161.03   3.81  2907  15.57  31.63   565  35.26  11.12   909   0.00   ....     .   0.00   ....     .
4b.32.22          75 205.53  154.98   3.81  3016  15.56  30.40   560  34.98  11.16   909   0.00   ....     .   0.00   ....     .
0a.30.21          78 204.44  153.64   3.89  3430  15.70  31.15   557  35.10  11.65   936   0.00   ....     .   0.00   ....     .
0a.31.22          75 210.42  159.52   3.78  2894  15.70  30.81   544  35.19  11.51   849   0.00   ....     .   0.00   ....     .
4b.32.23          74 202.89  153.08   3.66  3028  15.18  29.69   589  34.63  11.35   857   0.00   ....     .   0.00   ....     .

 

 

Re: Node disk fragmentation

Hi There,

Having a similar situation, did you solve the issue or some explaination about it?

 

Cheers,

RomSmiley Wink