Openshift之NetApp SolidFire集成及测试

Sunglyeol

NetApp公司是一家存储和数据管理公司,主要提供软件，系统和服务来管理和存储数据，包括其专有的Data ONTAP操作系统。NetApp公司主要向全球数据密集型企业提供统一存储解决方案其 Data ONTAP是全球首屈一指的存储操作系统，公司存储解决方案涵盖了专业化的硬件、软件和服务，为开放网络环境提供了无缝的存储管理。——(来自搜狗百科)
SolidFire成立于2010年，是一家全闪存阵列的存储厂商，其存储控制器基于标准的x86服务器，最大可扩展到100个节点，2015年12月，SolidFire被NetApp收购；2017年6月，NetApp基于SolidFire推出超融合一体机。
SolidFire提供分布式块存储，类似于ceph rbd，非常灵活，支持动态扩缩容，具有良好的性能。同时具有很多企业特性：如快照，组快照，丰富的API，非常灵活的QOS配置等。

作者：潘晓华Michael
链接：https://www.jianshu.com/p/8a393c68f2c9
來源：简书
简书著作权归作者所有，任何形式的转载都请联系作者获得授权并注明出处。

什么是Trident?

NetApp是CNCF的金牌会员，它开发的Trident是一款开源存储配置程序和流程编排程序。
在没有Trident的环境下，K8s/Openshift环境要使用NetApp存储，就需要，手动在NetApp控制台上创建volume，并设置创建PV，再创建PVC。这些过程需要在两个平台切换操作，而且很麻烦。
部署了Trident后，配置好相应的storageclass，K8s/Openshift平台就可以直接通过storageclass动态自动创建PVC。K8s/Openshift平台通过Trident控制器调用NetApp设备的API从而达到控制NetApp设备目的，如创建volume，并自动创建PV，及PVC，进而让Pod能够使用，此过程是自动的，对平台使用者是无感知的。
Trident本身也是以Pod的形式在K8s/Openshift平台上运行的。

Openshift上部署与使用Trident

准备工作

1. 用system:admin登录集群

$ oc login -u <span class="hljs-symbol">system:</span>admin

2. 集群能够访问SolidFire机器的MVIP(管理VIP)及SVIP(存储SVIP)

$ telnet $MVIP <span class="hljs-number">443</span>
$ telnet $SVIP <span class="hljs-number">3260</span>

3. 安装基本包

$ ansible all -m package -a <span class="hljs-string">'name=lsscsi,iscsi-initiator-utils,sg3_utils,device-mapper-multipath state=present'</span>
$ ansible all -m shell -a <span class="hljs-string">'mpathconf --enable --with_multipathd y'</span>
$ ansible all -m service -a <span class="hljs-string">'name=iscsid enabled=true state=started'</span>
$ ansible all -m service -a <span class="hljs-string">'name=multipathd enabled=true state=started'</span>
$ ansible all -m service -a <span class="hljs-string">'name=iscsi enabled=true state=started'</span>

部署

4. 下载安装文件，并解压

$ wget <span class="hljs-symbol">https:</span>/<span class="hljs-regexp">/github.com/</span>NetApp/trident/releases/download/v18.<span class="hljs-number">10.0</span>/trident-installer-<span class="hljs-number">18.10</span>.<span class="hljs-number">0</span>.tar.gz
$ tar -xf trident-installer-<span class="hljs-number">18.10</span>.<span class="hljs-number">0</span>.tar.gz
$ cd trident-installer

5. 配置安装backend.json文件

$ cp sample-input/backend-solidfire.json setup/backend.json
<span class="hljs-comment"># &#20462;&#25913;&#37324;&#38754;&#30340;&#37197;&#32622;</span>
$ cat setup/backend.json
{
    <span class="hljs-string">"version"</span>: <span class="hljs-number">1</span>,
    <span class="hljs-string">"storageDriverName"</span>: <span class="hljs-string">"solidfire-san"</span>,
    <span class="hljs-string">"Endpoint"</span>: <span class="hljs-string">"https://{{&#29992;&#25143;&#21517;}}:{{&#23494;&#30721;}}@{{&#31649;&#29702;VIP}}/json-rpc/11.0"</span>,
    <span class="hljs-string">"SVIP"</span>: <span class="hljs-string">"{{&#23384;&#20648;VIP}}:3260"</span>,
    <span class="hljs-string">"TenantName"</span>: <span class="hljs-string">"trident"</span>,
    <span class="hljs-string">"UseCHAP"</span>: <span class="hljs-literal">true</span>,
    <span class="hljs-string">"InitiatorIFace"</span>: <span class="hljs-string">"default"</span>,
    <span class="hljs-string">"Types"</span>: [{<span class="hljs-string">"Type"</span>: <span class="hljs-string">"Bronze"</span>, <span class="hljs-string">"Qos"</span>: {<span class="hljs-string">"minIOPS"</span>: <span class="hljs-number">1000</span>, <span class="hljs-string">"maxIOPS"</span>: <span class="hljs-number">2000</span>, <span class="hljs-string">"burstIOPS"</span>: <span class="hljs-number">4000</span>}},
              {<span class="hljs-string">"Type"</span>: <span class="hljs-string">"Silver"</span>, <span class="hljs-string">"Qos"</span>: {<span class="hljs-string">"minIOPS"</span>: <span class="hljs-number">4000</span>, <span class="hljs-string">"maxIOPS"</span>: <span class="hljs-number">6000</span>, <span class="hljs-string">"burstIOPS"</span>: <span class="hljs-number">8000</span>}},
              {<span class="hljs-string">"Type"</span>: <span class="hljs-string">"Gold"</span>, <span class="hljs-string">"Qos"</span>: {<span class="hljs-string">"minIOPS"</span>: <span class="hljs-number">6000</span>, <span class="hljs-string">"maxIOPS"</span>: <span class="hljs-number">8000</span>, <span class="hljs-string">"burstIOPS"</span>: <span class="hljs-number">10000</span>}}]
}

6. 创建trident project

$ oc <span class="hljs-keyword">new</span>-project trident

7. 安装检查

$ ./tridentctl install --dry-run -n trident

这个步骤会模拟安装过程进行执行一遍，并会删除所有资源。通过模拟对整个环境进行全面的检测。以下是执行的日志

[root@master02 trident-installer]# ./tridentctl install --dry-run -n trident -d
DEBU Initialized logging.                          logLevel=debug
DEBU Running outside a pod, creating CLI-based client. 
DEBU Initialized Kubernetes CLI client.            cli=oc flavor=openshift <span class="hljs-keyword">namespace</span>=trident version=<span class="hljs-number">1.11</span><span class="hljs-number">.0</span>+d4cacc0
DEBU Validated installation environment.           installationNamespace=trident kubernetesVersion=
DEBU Deleted Kubernetes configmap.                 label=<span class="hljs-string">"app=trident-installer.netapp.io"</span> <span class="hljs-keyword">namespace</span>=trident
DEBU Namespace exists.                             <span class="hljs-keyword">namespace</span>=trident
DEBU Deleted Kubernetes object by YAML.           
DEBU Deleted installer cluster role binding.      
DEBU Deleted Kubernetes object by YAML.           
DEBU Deleted installer cluster role.              
DEBU Deleted Kubernetes object by YAML.           
DEBU Deleted installer service account.           
DEBU Removed security context constraint user.     scc=privileged user=trident-installer
DEBU Created Kubernetes object by YAML.           
INFO Created installer service account.            serviceaccount=trident-installer
DEBU Created Kubernetes object by YAML.           
INFO Created installer cluster role.               clusterrole=trident-installer
DEBU Created Kubernetes object by YAML.           
INFO Created installer cluster role binding.       clusterrolebinding=trident-installer
INFO Added security context constraint user.       scc=privileged user=trident-installer
DEBU Created Kubernetes configmap from directory.  label=<span class="hljs-string">"app=trident-installer.netapp.io"</span> name=trident-installer <span class="hljs-keyword">namespace</span>=trident path=/root/trident-installer/setup
INFO Created installer configmap.                  configmap=trident-installer
DEBU Created Kubernetes object by YAML.           
INFO Created installer pod.                        pod=trident-installer
INFO Waiting <span class="hljs-keyword">for</span> Trident installer pod to start.  
DEBU Trident installer pod <span class="hljs-keyword">not</span> yet started, waiting.  increment=<span class="hljs-number">280.357322</span>ms message=<span class="hljs-string">"pod not yet started (Pending)"</span>
DEBU Trident installer pod <span class="hljs-keyword">not</span> yet started, waiting.  increment=<span class="hljs-number">523.702816</span>ms message=<span class="hljs-string">"pod not yet started (Pending)"</span>
DEBU Trident installer pod <span class="hljs-keyword">not</span> yet started, waiting.  increment=<span class="hljs-number">914.246751</span>ms message=<span class="hljs-string">"pod not yet started (Pending)"</span>
DEBU Trident installer pod <span class="hljs-keyword">not</span> yet started, waiting.  increment=<span class="hljs-number">1.111778662</span>s message=<span class="hljs-string">"pod not yet started (Pending)"</span>
DEBU Pod started.                                  phase=Succeeded
INFO Trident installer pod started.                <span class="hljs-keyword">namespace</span>=trident pod=trident-installer
DEBU Getting logs.                                 cmd=<span class="hljs-string">"oc --namespace=trident logs trident-installer -f"</span>
DEBU Initialized logging.                          logLevel=debug
DEBU Running in a pod, creating API-based client.  <span class="hljs-keyword">namespace</span>=trident
DEBU Initialized Kubernetes API client.            cli=oc flavor=openshift <span class="hljs-keyword">namespace</span>=trident version=v1<span class="hljs-number">.11</span><span class="hljs-number">.0</span>+d4cacc0
DEBU Validated installation environment.           installationNamespace=trident kubernetesVersion=v1<span class="hljs-number">.11</span><span class="hljs-number">.0</span>+d4cacc0
DEBU Parsed requested volume size.                 quantity=<span class="hljs-number">2</span>Gi
DEBU Dumping RBAC fields.                          ucpBearerToken= ucpHost= useKubernetesRBAC=<span class="hljs-literal">true</span>
DEBU Namespace exists.                             <span class="hljs-keyword">namespace</span>=trident
DEBU PVC does <span class="hljs-keyword">not</span> exist.                           pvc=trident
DEBU PV does <span class="hljs-keyword">not</span> exist.                            pv=trident
INFO Starting storage driver.                      backend=/setup/backend.json
DEBU config: {<span class="hljs-string">"Endpoint"</span>:<span class="hljs-string">"https://admin:root1234@99.248.106.82/json-rpc/11.0"</span>,<span class="hljs-string">"InitiatorIFace"</span>:<span class="hljs-string">"default"</span>,<span class="hljs-string">"SVIP"</span>:<span class="hljs-string">"99.248.82.55:3260"</span>,<span class="hljs-string">"TenantName"</span>:<span class="hljs-string">"trident"</span>,<span class="hljs-string">"Types"</span>:[{<span class="hljs-string">"Qos"</span>:{<span class="hljs-string">"burstIOPS"</span>:<span class="hljs-number">4000</span>,<span class="hljs-string">"maxIOPS"</span>:<span class="hljs-number">2000</span>,<span class="hljs-string">"minIOPS"</span>:<span class="hljs-number">1000</span>},<span class="hljs-string">"Type"</span>:<span class="hljs-string">"Bronze"</span>},{<span class="hljs-string">"Qos"</span>:{<span class="hljs-string">"burstIOPS"</span>:<span class="hljs-number">8000</span>,<span class="hljs-string">"maxIOPS"</span>:<span class="hljs-number">6000</span>,<span class="hljs-string">"minIOPS"</span>:<span class="hljs-number">4000</span>},<span class="hljs-string">"Type"</span>:<span class="hljs-string">"Silver"</span>},{<span class="hljs-string">"Qos"</span>:{<span class="hljs-string">"burstIOPS"</span>:<span class="hljs-number">10000</span>,<span class="hljs-string">"maxIOPS"</span>:<span class="hljs-number">8000</span>,<span class="hljs-string">"minIOPS"</span>:<span class="hljs-number">6000</span>},<span class="hljs-string">"Type"</span>:<span class="hljs-string">"Gold"</span>}],<span class="hljs-string">"UseCHAP"</span>:<span class="hljs-literal">true</span>,<span class="hljs-string">"storageDriverName"</span>:<span class="hljs-string">"solidfire-san"</span>,<span class="hljs-string">"version"</span>:<span class="hljs-number">1</span>} 
DEBU Storage prefix is absent, will use <span class="hljs-keyword">default</span> prefix. 
DEBU Parsed commonConfig: {Version:<span class="hljs-number">1</span> StorageDriverName:solidfire-san BackendName: Debug:<span class="hljs-literal">false</span> DebugTraceFlags:<span class="hljs-built_in">map</span>[] DisableDelete:<span class="hljs-literal">false</span> StoragePrefixRaw:[] StoragePrefix:<nil> SerialNumbers:[] DriverContext: LimitVolumeSize:} 
DEBU Initializing storage driver.                  driver=solidfire-san
DEBU Configuration defaults                        Size=<span class="hljs-number">1</span>G StoragePrefix= UseCHAP=<span class="hljs-literal">true</span>
DEBU Parsed into solidfireConfig                   DisableDelete=<span class="hljs-literal">false</span> StorageDriverName=solidfire-san Version=<span class="hljs-number">1</span>
DEBU Decoded to &{CommonStorageDriverConfig:<span class="hljs-number">0xc42064e0a0</span> TenantName:trident EndPoint:https:<span class="hljs-comment">//admin:root1234@99.248.106.82/json-rpc/11.0 SVIP:99.248.82.55:3260 InitiatorIFace:default Types:0xc4206d26e0 LegacyNamePrefix: AccessGroups:[] UseCHAP:true DefaultBlockSize:0 SolidfireStorageDriverConfigDefaults:{CommonStorageDriverConfigDefaults:{Size:1G}}} </span>
DEBU Set <span class="hljs-keyword">default</span> block size.                       defaultBlockSize=<span class="hljs-number">512</span>
DEBU Using SF API version from config file.        version=<span class="hljs-number">11.0</span>
DEBU Initializing SolidFire API client.            cfg=<span class="hljs-string">"{trident https://admin:root1234@99.248.106.82/json-rpc/11.0  99.248.82.55:3260 default 0xc4206d26e0  [] 512 map[]}"</span> endpoint=<span class="hljs-string">"https://admin:root1234@99.248.106.82/json-rpc/11.0"</span> svip=<span class="hljs-string">"99.248.82.55:3260"</span>
ERRO Error detected in API response.               ID=<span class="hljs-number">637</span> code=<span class="hljs-number">500</span> message=xUnknownAccount name=xUnknownAccount
DEBU Account <span class="hljs-keyword">not</span> found, creating.                  error=<span class="hljs-string">"device API error: xUnknownAccount"</span> tenantName=trident
DEBU Created account.                              accountID=<span class="hljs-number">0</span> tenantName=trident
DEBU SolidFire driver initialized.                 AccountID=<span class="hljs-number">2</span> InitiatorIFace=<span class="hljs-keyword">default</span>
DEBU Using CHAP, skipped Volume Access Group logic.  AccessGroups=<span class="hljs-string">"[]"</span> SVIP=<span class="hljs-string">"99.248.82.55:3260"</span> UseCHAP=<span class="hljs-literal">true</span> driver=solidfire-san
DEBU Added pool <span class="hljs-keyword">for</span> SolidFire backend.             attributes=<span class="hljs-string">"map[media:{Offers: ssd} IOPS:{Min: 1000, Max: 2000} snapshots:{Offer:  true} clones:{Offer:  true} encryption:{Offer:  false} provisioningType:{Offers: thin} backendType:{Offers: solidfire-san}]"</span> backend=solidfire_99<span class="hljs-number">.248</span><span class="hljs-number">.82</span><span class="hljs-number">.55</span> pool=Bronze
DEBU Added pool <span class="hljs-keyword">for</span> SolidFire backend.             attributes=<span class="hljs-string">"map[clones:{Offer:  true} encryption:{Offer:  false} provisioningType:{Offers: thin} backendType:{Offers: solidfire-san} media:{Offers: ssd} IOPS:{Min: 4000, Max: 6000} snapshots:{Offer:  true}]"</span> backend=solidfire_99<span class="hljs-number">.248</span><span class="hljs-number">.82</span><span class="hljs-number">.55</span> pool=Silver
DEBU Added pool <span class="hljs-keyword">for</span> SolidFire backend.             attributes=<span class="hljs-string">"map[snapshots:{Offer:  true} clones:{Offer:  true} encryption:{Offer:  false} provisioningType:{Offers: thin} backendType:{Offers: solidfire-san} media:{Offers: ssd} IOPS:{Min: 6000, Max: 8000}]"</span> backend=solidfire_99<span class="hljs-number">.248</span><span class="hljs-number">.82</span><span class="hljs-number">.55</span> pool=Gold
DEBU Storage driver initialized.                   driver=solidfire-san
INFO Storage driver loaded.                        driver=solidfire-san
INFO Dry run completed, no problems found.        
DEBU Received EOF from pod logs.                   container= pod=trident-installer
INFO Waiting <span class="hljs-keyword">for</span> Trident installer pod to finish. 
DEBU Pod finished.                                 phase=Succeeded
INFO Trident installer pod finished.               <span class="hljs-keyword">namespace</span>=trident pod=trident-installer
DEBU Deleted Kubernetes pod.                       label=<span class="hljs-string">"app=trident-installer.netapp.io"</span> <span class="hljs-keyword">namespace</span>=trident
INFO Deleted installer pod.                        pod=trident-installer
DEBU Deleted Kubernetes configmap.                 label=<span class="hljs-string">"app=trident-installer.netapp.io"</span> <span class="hljs-keyword">namespace</span>=trident
INFO Deleted installer configmap.                  configmap=trident-installer
INFO In-cluster installation completed.           
DEBU Deleted Kubernetes object by YAML.           
INFO Deleted installer cluster role binding.      
DEBU Deleted Kubernetes object by YAML.           
INFO Deleted installer cluster role.              
DEBU Deleted Kubernetes object by YAML.           
INFO Deleted installer service account.           
INFO Removed security context constraint user.     scc=privileged user=trident-installer

8. 正式安装

$ ./tridentctl install -n trident

该步骤是真正的执行。会创建serviceaccount, clusterrolebinding,configmap配置，trident-install pod（该pod在部署完trident deployment后会删除）等， 并会创建一个pv与trident pvc进行初始化操作，最终会创建trident deployment，完成trident的安装。

• trident的安装支持自定义一些配置。
• --etcd-image可指定etcd的镜像（默认是quay.io/coreos/etcd，下载会比较慢）
• --trident-image指定trident的镜像
• --volume-size指定trident持久存储的大小（默认为2GiB）
• --volume-name指定volume名字（默认是etcd-vol）
• --pv指定pv名字（默认是trident）
• --pvc指定pvc名字（默认是trident）
• --generate-custom-yaml将使用的所有配置进行导出到一个setup文件夹，不会对集群做任何操作
• --use-custom-yaml安装setup下的所有yaml文件进行部署trident

以下是执行的日志

[root@master02 trident-installer]# ./tridentctl install -n trident 
INFO Created installer service account.            serviceaccount=trident-installer
INFO Created installer cluster role.               clusterrole=trident-installer
INFO Created installer cluster role binding.       clusterrolebinding=trident-installer
INFO Added security context constraint user.       scc=privileged user=trident-installer
INFO Created installer configmap.                  configmap=trident-installer
INFO Created installer pod.                        pod=trident-installer
INFO Waiting <span class="hljs-keyword">for</span> Trident installer pod to start.  
INFO Trident installer pod started.                <span class="hljs-keyword">namespace</span>=trident pod=trident-installer
INFO Starting storage driver.                      backend=/setup/backend.json
INFO Storage driver loaded.                        driver=solidfire-san
INFO Starting Trident installation.                <span class="hljs-keyword">namespace</span>=trident
INFO Created service account.                     
INFO Created cluster role.                        
INFO Created cluster role binding.                
INFO Added security context constraint user.       scc=anyuid user=trident
INFO Created PVC.                                 
INFO Controller serial numbers.                    serialNumbers=<span class="hljs-string">"4BZXJB2,85Q8JB2,4BXXJB2,4BXTJB2"</span>
INFO Created iSCSI CHAP secret.                    secret=trident-chap-solidfire<span class="hljs-number">-99</span><span class="hljs-number">-248</span><span class="hljs-number">-82</span><span class="hljs-number">-55</span>-trident
INFO Created PV.                                   pv=trident
INFO Waiting <span class="hljs-keyword">for</span> PVC to be bound.                  pvc=trident
INFO Created Trident deployment.                  
INFO Waiting <span class="hljs-keyword">for</span> Trident pod to start.            
INFO Trident pod started.                          <span class="hljs-keyword">namespace</span>=trident pod=trident<span class="hljs-number">-57</span>ccdff48f-gtflx
INFO Waiting <span class="hljs-keyword">for</span> Trident REST interface.          
INFO Trident REST interface is up.                 version=<span class="hljs-number">18.10</span><span class="hljs-number">.0</span>
INFO Trident installation succeeded.              
INFO Waiting <span class="hljs-keyword">for</span> Trident installer pod to finish. 
INFO Trident installer pod finished.               <span class="hljs-keyword">namespace</span>=trident pod=trident-installer
INFO Deleted installer pod.                        pod=trident-installer
INFO Deleted installer configmap.                  configmap=trident-installer
INFO In-cluster installation completed.           
INFO Deleted installer cluster role binding.      
INFO Deleted installer cluster role.              
INFO Deleted installer service account.           
INFO Removed security context constraint user.     scc=privileged user=trident-installer

9. 执行完install后，trident并不会安装之前配置的backend，需要另外再单独添加。（个人觉得netapp这点考虑得有点多余，因为dry-run的时候已经对backend.json作了检查了，直接install将它添加上岂不是更方便）

$ ./tridentctl -n trident create backend -f setup/backend.json
$ ./tridentctl -n trident get backend
+------------------------+----------------+--------+---------+
<span class="hljs-params">|          NAME          |</span> STORAGE DRIVER <span class="hljs-params">| ONLINE |</span> VOLUMES <span class="hljs-params">|
+------------------------+----------------+--------+---------+
|</span> solidfire_99.<span class="hljs-number">248.82</span>.<span class="hljs-number">55</span> <span class="hljs-params">| solidfire-san  |</span> <span class="hljs-literal">true</span>   <span class="hljs-params">|       0 |</span>
+------------------------+----------------+--------+---------+

10. 添加基本的storageclass
    将sample-input/storage-class-basic.yaml.templ中的BACKEND_TYPE用指定的backend中的STORAGE DRIVER值替换（此例中为solidfire-san）

$ cat sample-input/storage-<span class="hljs-class"><span class="hljs-keyword">class</span>-<span class="hljs-title">basic</span>.<span class="hljs-title">yaml</span>.<span class="hljs-title">templ</span></span>
<span class="hljs-symbol">apiVersion:</span> storage.k8s.io/v1
<span class="hljs-symbol">kind:</span> StorageClass
<span class="hljs-symbol">metadata:</span>
  <span class="hljs-symbol">name:</span> basic
<span class="hljs-symbol">provisioner:</span> netapp.io/trident
<span class="hljs-symbol">parameters:</span>
  <span class="hljs-symbol">backendType:</span> <span class="hljs-string">"__BACKEND_TYPE__"</span>
$ sed <span class="hljs-string">"s/__BACKEND_TYPE__/solidfire-san/"</span> sample-input/storage-<span class="hljs-class"><span class="hljs-keyword">class</span>-<span class="hljs-title">basic</span>.<span class="hljs-title">yaml</span>.<span class="hljs-title">templ</span> | <span class="hljs-title">oc</span> <span class="hljs-title">create</span> -<span class="hljs-title">f</span> -</span>

11. 根据backend中的Type创建对应的storageclass

$ cat storage-<span class="hljs-class"><span class="hljs-keyword">class</span>-<span class="hljs-title">gold</span>.<span class="hljs-title">yaml</span></span>
<span class="hljs-symbol">apiVersion:</span> storage.k8s.io/v1
<span class="hljs-symbol">kind:</span> StorageClass
<span class="hljs-symbol">metadata:</span>
  <span class="hljs-symbol">name:</span> gold
  <span class="hljs-symbol">annotations:</span>
    storageclass.kubernetes.io/is-default-<span class="hljs-class"><span class="hljs-keyword">class</span>: "<span class="hljs-title">true</span>"</span>
<span class="hljs-symbol">provisioner:</span> netapp.io/trident
<span class="hljs-symbol">parameters:</span>
  <span class="hljs-symbol">storagePools:</span> <span class="hljs-string">"solidfire_99.248.82.55:Gold"</span> <span class="hljs-comment"># solidfire_99.248.82.55&#20026;backend name;Gold&#20026;&#25351;&#23450;&#30340;Type</span>
$ oc create -f storage-<span class="hljs-class"><span class="hljs-keyword">class</span>-<span class="hljs-title">gold</span>.<span class="hljs-title">yaml</span></span>

查看当前的storageclass

$ oc get sc
NAME             PROVISIONER         AGE
basic            netapp.io/trident   <span class="hljs-number">2</span><span class="hljs-function">h
<span class="hljs-title">gold</span> <span class="hljs-params">(<span class="hljs-keyword">default</span>)</span>   netapp.io/trident   1h
</span>

 

使用：创建PVC

12. 创建第一个PVC

$ cat test-pvc.yaml
<span class="hljs-symbol">apiVersion:</span> v1
<span class="hljs-symbol">kind:</span> PersistentVolumeClaim
<span class="hljs-symbol">metadata:</span>
  <span class="hljs-symbol">annotations:</span>
    volume.beta.kubernetes.io/storage-<span class="hljs-class"><span class="hljs-keyword">class</span>: <span class="hljs-title">gold</span></span>
    volume.beta.kubernetes.io/storage-<span class="hljs-symbol">provisioner:</span> netapp.io/trident
    trident.netapp.io/<span class="hljs-symbol">reclaimPolicy:</span> <span class="hljs-string">"Retain"</span>
  <span class="hljs-symbol">name:</span> testpvc
  <span class="hljs-symbol">namespace:</span> test
<span class="hljs-symbol">spec:</span>
  <span class="hljs-symbol">accessModes:</span>
  - ReadWriteOnce
  <span class="hljs-symbol">resources:</span>
    <span class="hljs-symbol">requests:</span>
      <span class="hljs-symbol">storage:</span> <span class="hljs-number">1</span>Gi
$ oc create -f test-pvc.yaml

PVC创建的说明:

• volume.beta.kubernetes.io/storage-class为10，11步创建的storageclass
• volume.beta.kubernetes.io/storage-provisioner指定为netapp的trident
• trident.netapp.io/reclaimPolicy指定创建PV的reclaimPolicy,默认为"Delete",支持"Delete"和"Retain"，不支持"Recycle"
• accessModes因SolidFire是块存储，只支持ReadWriteOnce

SolidFire功能测试

快照恢复数据

创建快照

创建快照

基于已有快照恢复pvc数据

基于快照创建新的PVC

指定快照，创建新的存储

基于已有快照恢复volume

查看新建的volume的IQN

查看新建的volume的IQN

基于新的volume创建PV

$ cat test-clone-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: netapp.io/trident
    volume.beta.kubernetes.io/storage-<span class="hljs-class"><span class="hljs-keyword">class</span>:</span> gold
  name: test-dd-testxx-volume
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: <span class="hljs-number">100</span>Gi
  iscsi:
    chapAuthDiscovery: <span class="hljs-literal">true</span>
    chapAuthSession: <span class="hljs-literal">true</span>
    fsType: ext4
    iqn: iqn<span class="hljs-number">.2010</span><span class="hljs-number">-01.</span>com.solidfire:fs69.test-dd-testxx-volume<span class="hljs-number">.169</span>
    iscsiInterface: <span class="hljs-keyword">default</span>
    lun: <span class="hljs-number">0</span>
    secretRef:
      name: trident-chap-solidfire<span class="hljs-number">-99</span><span class="hljs-number">-248</span><span class="hljs-number">-82</span><span class="hljs-number">-55</span>-trident
      <span class="hljs-keyword">namespace</span>: trident
    targetPortal: <span class="hljs-number">99.248</span><span class="hljs-number">.82</span><span class="hljs-number">.55</span>:<span class="hljs-number">3260</span>
  persistentVolumeReclaimPolicy: Delete
  storageClassName: gold
$ oc create -f test-clone-pv.yaml

创建pvc使用手动创建的pv

$ cat <span class="hljs-built_in">test</span>-clone-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test111x
  namespace: <span class="hljs-built_in">test</span>-dd
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

组快照

组快照与快照类似，不同之处，它把多个存储卷在同一时间的数据做快照，从而避免数据不一致的情况。同时在恢复的时候，也同时将备份时刻的数据进行恢复。

组快照创建.png

克隆已有的pvc数据

添加annotations配置trident.netapp.io/cloneFromPVC: test-pvc,创建新的pvc基于已有的PVC test-pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    trident.netapp.io/cloneFromPVC: <span class="hljs-built_in">test</span>-pvc
  name: <span class="hljs-built_in">test</span>-clone-pvc
  namespace: <span class="hljs-built_in">test</span>-dd
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: gold

SolidFire性能测试

测试环境说明

• openshift 3.11物理机部署：3 Masters 4 Nodes
• SolidFire 4台Node：型号SF9605 每台Node上为10块SSD盘，每个Node的IOPS为5w，集群最高IOPS 20w
• 每块PV存储设置为gold类型storageclass：{"Type": "Gold", "Qos": {"minIOPS": 6000, "maxIOPS": 8000, "burstIOPS": 10000}}

dd测试

<span class="hljs-comment"># &#27979;&#35797;&#21629;&#20196;</span>
$ dd <span class="hljs-keyword">if</span>=<span class="hljs-regexp">/dev/zero</span> of=<span class="hljs-regexp">/data/dd</span>.test bs=<span class="hljs-number">4</span>k count=<span class="hljs-number">200000</span> oflag=direct

1. 单个pod，单个pv作dd命令测试
   创建deployment进行测试

$ cat test<span class="hljs-number">0</span>-pvc.yaml
<span class="hljs-symbol">kind:</span> PersistentVolumeClaim
<span class="hljs-symbol">metadata:</span>
  <span class="hljs-symbol">annotations:</span>
    volume.beta.kubernetes.io/storage-<span class="hljs-class"><span class="hljs-keyword">class</span>: <span class="hljs-title">gold</span></span>
    volume.beta.kubernetes.io/storage-<span class="hljs-symbol">provisioner:</span> netapp.io/trident
  <span class="hljs-symbol">name:</span> test<span class="hljs-number">0</span>
  <span class="hljs-symbol">namespace:</span> test-dd
<span class="hljs-symbol">spec:</span>
  <span class="hljs-symbol">accessModes:</span>
  - ReadWriteOnce
  <span class="hljs-symbol">resources:</span>
    <span class="hljs-symbol">requests:</span>
      <span class="hljs-symbol">storage:</span> <span class="hljs-number">10</span>Gi
$ oc create -f test<span class="hljs-number">0</span>-pvc.yaml <span class="hljs-comment">## &#21019;&#24314;&#27979;&#35797;&#30340;&#23384;&#20648;</span>
$ cat dd.yaml
<span class="hljs-symbol">apiVersion:</span> apps.openshift.io/v1
<span class="hljs-symbol">kind:</span> DeploymentConfig
<span class="hljs-symbol">metadata:</span>
  <span class="hljs-symbol">labels:</span>
    <span class="hljs-symbol">run:</span> ddtest
  <span class="hljs-symbol">name:</span> ddtest
<span class="hljs-symbol">spec:</span>
  <span class="hljs-symbol">replicas:</span> <span class="hljs-number">1</span>
  <span class="hljs-symbol">selector:</span>
    <span class="hljs-symbol">run:</span> ddtest
  <span class="hljs-symbol">strategy:</span>
    <span class="hljs-symbol">type:</span> Recreate
  <span class="hljs-symbol">template:</span>
    <span class="hljs-symbol">metadata:</span>
      <span class="hljs-symbol">labels:</span>
        <span class="hljs-symbol">run:</span> ddtest
    <span class="hljs-symbol">spec:</span>
      <span class="hljs-symbol">containers:</span>
      - <span class="hljs-symbol">command:</span>
        - <span class="hljs-regexp">/bin/bash</span>
        - <span class="hljs-string">'-c'</span>
        - <span class="hljs-params">|
          #/bin/bash
          dd <span class="hljs-keyword">if</span>=/dev/zero of=/data/out.test1 bs=4k count=200000 oflag=direct
        image: tools/iqperf:latest
        imagePullPolicy: Always
        name: ddtest
        volumeMounts:
        - mountPath: /data
          name: volume-spq10
      volumes:
      - name: volume-spq10
        persistentVolumeClaim:
          claimName: test0
  triggers:
  - type: ConfigChange
$ oc create -f dd.yaml
</span>

在webconsole上查看日志如下

<span class="hljs-number">200000</span>+<span class="hljs-number">0</span> records <span class="hljs-keyword">in</span>
<span class="hljs-number">200000</span>+<span class="hljs-number">0</span> records <span class="hljs-keyword">out</span>
<span class="hljs-number">819200000</span> bytes (<span class="hljs-number">819</span> MB) copied, <span class="hljs-number">68.8519</span> s, <span class="hljs-number">11.9</span> MB/s

NetApp的管理平台上查看集群IO状态，如图（只需要看11:32时间以后部分）

单个pod，单个pv作dd命令测试.png

IOPS为2908

2. 1个pod，1个pv，8个dd进程
   将1中的deploymentconfig中的command内容更新为：

...
- <span class="hljs-built_in">command</span>:
  - <span class="hljs-string">'/bin/bash'</span>
  - <span class="hljs-string">'-c'</span>
  - |
    <span class="hljs-comment">#/bin/bash</span>
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> {1..8}
    <span class="hljs-keyword">do</span> 
      dd <span class="hljs-keyword">if</span>=/dev/zero of=/data/dd.test<span class="hljs-variable">$i</span> bs=4k count=200000 oflag=direct &
    <span class="hljs-keyword">done</span>    
    sleep 1000000
...

 1个pod，1个pv，8个dd进程.png

IOPS为10000

3. 8个pod，8个pv同时使用dd命令测试
   创建statefulset，设置volumeClaimTemplates批量创建存储

$ cat dd-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: testdd
  namespace: <span class="hljs-built_in">test</span>-dd
spec:
  serviceName: testdd
  replicas: 8
  selector:
    matchLabels:
      app: testdd
  template:
    metadata:
      labels:
        app: testdd
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: testdd
          containers:
          - <span class="hljs-built_in">command</span>:
            - /bin/bash
            - <span class="hljs-string">'-c'</span>
            - |
              <span class="hljs-comment">#!/bin/bash</span>
              dd <span class="hljs-keyword">if</span>=/dev/zero of=/data/out.test1 bs=4k count=2000000 oflag=direct
          image: <span class="hljs-string">'harbor.apps.it.mbcloud.com/tools/iqperf:latest'</span>
          imagePullPolicy: Always
          name: testdd
          image: <span class="hljs-string">'tools/dd:latest'</span>
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: gold
        resources:
          requests:
            storage: 100Gi

8个pod，8个pv同时使用dd命令测试.png

IOPS为33883

4. 8个pod，8个pv同时每个pod启用8个dd进程，共64个dd进程测试
   更改3中statefulset的command命令如下：

...
- <span class="hljs-built_in">command</span>:
  - <span class="hljs-string">'/bin/bash'</span>
  - <span class="hljs-string">'-c'</span>
  - |
    <span class="hljs-comment">#/bin/bash</span>
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> {1..8}
    <span class="hljs-keyword">do</span> 
      dd <span class="hljs-keyword">if</span>=/dev/zero of=/data/dd.test<span class="hljs-variable">$i</span> bs=4k count=200000 oflag=direct &
    <span class="hljs-keyword">done</span>    
    sleep 1000000
...

8个pod，8个pv同时每个pod启用8个dd进程，共64个dd进程测试.png

IOPS为76832达到了gold Type下设置的IOPS上限

5. 50个pod，50个pv同时每个pod启用3个dd进程，共150个dd进程测试

图片.png

此时单个PV存储的详情

图片.png

IOPS为205545达到了gold Type下设置的IOPS上限
综合结果如下：

pod数	pv数	dd进程数	IOPS
1	1	1	2908
1	1	8	10000
8	8	8	33883
8	8	64	76832
50	50	150	205545

 

数据库测试

测试工具mydbtest
测试配置

$ mysql -uapp -h172.<span class="hljs-number">30.213</span>.<span class="hljs-number">17</span> -papp app -e <span class="hljs-string">"create table t_mytest(col1 int);"</span>
$ cat test.conf
option

name app

loop <span class="hljs-number">20000</span>

user app/app@172.<span class="hljs-number">30.213</span>.<span class="hljs-number">17</span><span class="hljs-symbol">:</span><span class="hljs-number">3306</span><span class="hljs-symbol">:app</span>

declare

a int <span class="hljs-number">10</span> <span class="hljs-number">30000</span>

<span class="hljs-keyword">begin</span>

<span class="hljs-comment">#select * from t_mytest where col1 = :a; # &#26597;&#35810;</span>
insert into t_mytest set col1 = <span class="hljs-symbol">:a</span>;  <span class="hljs-comment"># &#25554;&#20837;</span>

<span class="hljs-keyword">end</span>

执行测试过程

./mydbtest_64.bin query=test.conf  degree=40

执行结果

<span class="hljs-comment"># &#25554;&#20837;&#25968;&#25454;</span>
2019-01-23 17:59:35 Total  tran=20000=312/s, qtps=40000=624/s, ela=64046 ms, avg=3202 us
Summary: SQL01 <span class="hljs-built_in">exec</span>=800000, rows=0=0/e, avg=65 us
Summary: SQL02 <span class="hljs-built_in">exec</span>=800000, rows=800000=100/e, avg=3135 us
Summary: <span class="hljs-built_in">exec</span>=12307/s, qtps=24615/s

<span class="hljs-comment"># &#21019;&#24314;&#23436;&#32034;&#24341;&#21518;&#65292;&#35835;&#25968;&#25454;&#65288;&#21442;&#32771;&#24847;&#20041;&#19981;&#22823;&#65289;</span>
2019-01-23 17:56:31 Total  tran=20000=3835/s, qtps=40000=7670/s, ela=5203 ms, avg=260 us
Summary: SQL01 <span class="hljs-built_in">exec</span>=800000, rows=22668078=2833/e, avg=174 us
Summary: SQL02 <span class="hljs-built_in">exec</span>=800000, rows=0=0/e, avg=69 us
Summary: <span class="hljs-built_in">exec</span>=133333/s, qtps=266666/s

插入的qtps为24615/s,性能不错。

小礼物走一走，来简书关注我

Jeff_Yao

谢谢分享