Solved: OMF and netapp plugin

gasparuben · ‎2013-10-30

Hi,

It turns out we are using OMF for both PDB$SEED and our PDB's. While trying to use the Netapp plug-in I get:

--using different paths

sys@DODCDB2:SQL> create pluggable database RUBEN04 from RUBEN03 file_name_convert=('/ORA/dbs03/RUBEN03','/ORA/dbs03/RUBEN04') snapshot copy;

create pluggable database RUBEN04 from RUBEN03 file_name_convert=('/ORA/dbs03/RUBEN03','/ORA/dbs03/RUBEN04') snapshot copy

*

ERROR at line 1:

ORA-01276: Cannot add file /ORA/dbs03/RUBEN04/DODCDB2/E9957ADE4BDF78E3E0438C08100A182D/datafile/o1_mf_system__179231698634_.dbf. File has an Oracle Managed Files file name.

--looking for file

sys@DODCDB2:SQL> create pluggable database RUBEN04 from RUBEN03 file_name_convert=('__17923','totoclone') snapshot copy;

create pluggable database RUBEN04 from RUBEN03 file_name_convert=('__17923','totoclone') snapshot copy

*

ERROR at line 1:

ORA-01276: Cannot add file /ORA/dbs03/RUBEN03/DODCDB2/E9957ADE4BDF78E3E0438C08100A182D/datafile/o1_mf_systemtotoclone1698634_.dbf. File has an Oracle Managed Files file name.

It looks like more a restriction on the "create pluggable database" command than in the plug-in, it also fails for an standard PDB creation from seed:

alter session set db_create_file_dest='';

create pluggable database RUBEN04 ADMIN USER ruben_admin IDENTIFIED BY ChangeMe13 ROLES=(DBA)

DEFAULT TABLESPACE RUBEN DATAFILE '/ORA/dbs03/RUBEN05/datafile/myfile.dbf' SIZE 250M AUTOEXTEND ON

path_prefix='/ORA/dbs03/RUBEN05'

file_name_convert=('/ORA/dbs03/DODCDB2/','/ORA/dbs03/RUBEN05/')

TEMPFILE REUSE;

ERROR at line 1:

ORA-01276: Cannot add file /ORA/dbs03/RUBEN05/E97092FD38474BE4E0438C08100AAFFB/datafile/o1_mf_system__20748329356_.dbf. File has an Oracle Managed Files file name.

sys@DODCDB2:SQL> @pdbs_data_files.sql

PDB_ID PDB_NAME FILE_ID TABLESPACE_NAME FILE_NAME

------ --------------- ------- ------------------------------ ------------------------------------------------------------------------------------------------------------------------

2 PDB$SEED 6 USERTBS /ORA/dbs03/DODCDB2/E97092FD38474BE4E0438C08100AAFFB/datafile/o1_mf_usertbs__20802017065_.dbf

2 PDB$SEED 4 SYSAUX /ORA/dbs03/DODCDB2/E97092FD38474BE4E0438C08100AAFFB/datafile/o1_mf_sysaux__20776046082_.dbf

2 PDB$SEED 2 SYSTEM /ORA/dbs03/DODCDB2/E97092FD38474BE4E0438C08100AAFFB/datafile/o1_mf_system__20748329356_.dbf

alter session set db_create_file_dest='';

Session altered.

create pluggable database RUBEN04 ADMIN USER ruben_admin IDENTIFIED BY ChangeMe13 ROLES=(DBA)

DEFAULT TABLESPACE RUBEN DATAFILE '/ORA/dbs03/RUBEN05/datafile/myfile.dbf' SIZE 250M AUTOEXTEND ON

path_prefix='/ORA/dbs03/RUBEN05'

file_name_convert=('/ORA/dbs03/DODCDB2/E97092FD38474BE4E0438C08100AAFFB/datafile/','/ORA/dbs03/RUBEN05/datafile/')

5 TEMPFILE REUSE;

create pluggable database RUBEN04 ADMIN USER ruben_admin IDENTIFIED BY ChangeMe13 ROLES=(DBA)

*

ERROR at line 1:

ORA-01276: Cannot add file /ORA/dbs03/RUBEN05/datafile/o1_mf_system__20748329356_.dbf. File has an Oracle Managed Files file name.

Please let me know if there is something I am doing wrong.

Thanks for your time,

Ruben

gasparuben · ‎2014-02-25

Hi Karthikeyan,

Sorry for my delay to get back to you.

db_create_file_dest='/ORA/dbs03/RUBEN'

sys@DODCDB1:SQL> create pluggable database ruben03_clone from ruben03 snapshot copy path_prefix = '/ORA/dbs03/RUBEN03_CLONE';

create pluggable database ruben03_clone from ruben03 snapshot copy path_prefix = '/ORA/dbs03/RUBEN03_CLONE'

*

ERROR at line 1:

ORA-00604: error occurred at recursive SQL level

ORA-17517: Database cloning using storage snapshot failed on file :

While looking at the trace files I see:

*** 2014-02-25 17:22:46.511

*** SESSION ID:(287.18491) 2014-02-25 17:22:46.511

*** SERVICE NAME:(SYS$USERS) 2014-02-25 17:22:46.511

*** MODULE NAME:(sqlplus@XXXXX.cern.ch (TNS V1-V3)) 2014-02-25 17:22:46.511

*** ACTION NAME:() 2014-02-25 17:22:46.511

*** CONTAINER ID:(1) 2014-02-25 17:22:46.511

Storage clones not supported on /ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_iops__4498075797710_.dbf storage_type=0

ORA-17517: Database cloning using storage snapshot failed on file 1:/ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_iops__4498075797710_.dbf

The storage has a valid clone and snaprestore license.

Looking at the documentation I see that this error my be linked to: https://communities.netapp.com/servlet/JiveServlet/previewBody/30115-102-1-50847/NetApp%20Cloning%20Plug-in%201%200%20for%20Oracle%20Database%20Instal... page 25:

1) Verify that the alert_db.log contains “Found NAS server NetApp Release”.

2) If the alert_db.log does not contain “Found NAS server NetApp Release‖, perform the following

steps:

a) Install net-snmp-* rpms.

b) Run the snmpwalk command.

I certainly dont see this "Found NAS..." but the net-snmp rpms are installed. I hesitate installation of those will be reflected in the alert log. About the snmpwalk there is another reference on page 6 of the same document. But I have following remarks. I am working on C-mode cluster, please could you tell me where I should look for:

snmp.access all

snmp.enable on

options nfs.export.auto-update

My mount points look like this:

vserver-dbnasb402:/ORA/dbs03/RUBEN03 on /ORA/dbs03/RUBEN03 type nfs (rw,bg,hard,nointr,tcp,nfsvers=3,actimeo=0,timeo=600,rsize=65536,wsize=65536,addr=XXXXX)

sys@DODCDB1:SQL> select client from v$client_secrets;

CLIENT

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

vserver-dbnasb402

This is the lif I used to mount, and the credentials are defined using vsadmin user:

sys@DODCDB0:SQL> ADMINISTER KEY MANAGEMENT ADD SECRET 'xxxxx' FOR CLIENT 'vserver-dbnasb402' USING TAG 'vsadmin' IDENTIFIED BY xxx WITH BACKUP;

keystore altered.

Also the firewall of that lif is modified so snmp traffic goes through:

c02::> net int show -vserver dbvs -lif dbvs_lif3 -instance

(network interface show)

Vserver Name: dbvs

Logical Interface Name: dbvs_lif3

Role: data

Data Protocol: nfs

Home Node: dbnasb402

Home Port: a0c

Current Node: dbnasb402

Current Port: a0c

Operational Status: up

Extended Status: -

Is Home: true

Network Address: XXXXXXX

Netmask: 255.255.255.192

Bits in the Netmask: 26

IPv4 Link Local: -

Routing Group Name: XXXXXXX/26

Administrative Status: up

Failover Policy: nextavail

Firewall Policy: datandmgmt

Auto Revert: false

Fully Qualified DNS Zone Name: vserver-dbnasb402.cern.ch

DNS Query Listen Enable: true

Failover Group Name: dbvs_10g_IP26

FCP WWPN: -

Address family: ipv4

Comment: -

c02::> firewall policy show -policy datandmgmt

(system services firewall policy show)

Policy Service Action IP-List

---------------- ---------- ------ --------------------

datandmgmt

dns allow 0.0.0.0/0

http allow 0.0.0.0/0

https allow 0.0.0.0/0

ndmp allow 0.0.0.0/0

ntp allow 0.0.0.0/0

snmp allow 0.0.0.0/0

ssh allow 0.0.0.0/0

telnet deny 0.0.0.0/0

8 entries were displayed.

itrac>-RAC>-DODCDB11:/ORA/dbs01/oracle/product/rdbms/dbs$ snmpget -c public -v 2c vserver-dbnasb402 1.3.6.1.4.1.789.1.1.2.0

Timeout: No Response from vserver-dbnasb402.

Using the cluster management IP:

trac>-RAC>-DODCDB11:/ORA/dbs01/oracle/product/rdbms/dbs$ snmpget -c public -v 2c IPCLUSTERMANAGEMENT 1.3.6.1.4.1.789.1.1.2.0

SNMPv2-SMI::enterprises.789.1.1.2.0 = STRING: "NetApp Release 8.2P3 Cluster-Mode: Thu Sep 12 01:13:07 PDT 2013"

So please could you clarify how to enable snmp and check auto update exports on the vserver to which my data lifs belong to? How should I run snmpwalk?

Thank you,

Ruben

View solution in original post

nkarthik · ‎2013-10-30

Good question, Let me check and get back to you.

Regards,

Karthikeyan.N

nkarthik · ‎2013-12-11

Hello gasparuben,

Sorry for the delay, I got the answer for you. OMF with NetApp Plugin (snapshot copy ) is supported. You need to change the 'db_create_file_dest' parameter to NetApp storage mountpoint and no need to pass 'file_name_convert' with create pluggable database.

Please check the below link with an example ( Might be too detailed ) to show the OMF based PDB create and delete (Netapp flexclone PDB) using NetApp cloning Plug-In:

https://communities.netapp.com/docs/DOC-31695

Please let me know, if you need any more details.

Regards,

Karthikeyan.N

nkarthik · ‎2014-01-22

Hello gasparuben,

Any update for your query. Does it answer your query? Please check the document and let me know, if you need any changes.

Regards,

Karthikeyan.N

gasparuben · ‎2014-02-25

Hi Karthikeyan,

Sorry for my delay to get back to you.

db_create_file_dest='/ORA/dbs03/RUBEN'

sys@DODCDB1:SQL> create pluggable database ruben03_clone from ruben03 snapshot copy path_prefix = '/ORA/dbs03/RUBEN03_CLONE';

create pluggable database ruben03_clone from ruben03 snapshot copy path_prefix = '/ORA/dbs03/RUBEN03_CLONE'

*

ERROR at line 1:

ORA-00604: error occurred at recursive SQL level

ORA-17517: Database cloning using storage snapshot failed on file :

While looking at the trace files I see:

*** 2014-02-25 17:22:46.511

*** SESSION ID:(287.18491) 2014-02-25 17:22:46.511

*** SERVICE NAME:(SYS$USERS) 2014-02-25 17:22:46.511

*** MODULE NAME:(sqlplus@XXXXX.cern.ch (TNS V1-V3)) 2014-02-25 17:22:46.511

*** ACTION NAME:() 2014-02-25 17:22:46.511

*** CONTAINER ID:(1) 2014-02-25 17:22:46.511

Storage clones not supported on /ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_iops__4498075797710_.dbf storage_type=0

ORA-17517: Database cloning using storage snapshot failed on file 1:/ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_iops__4498075797710_.dbf

The storage has a valid clone and snaprestore license.

Looking at the documentation I see that this error my be linked to: https://communities.netapp.com/servlet/JiveServlet/previewBody/30115-102-1-50847/NetApp%20Cloning%20Plug-in%201%200%20for%20Oracle%20Database%20Instal... page 25:

1) Verify that the alert_db.log contains “Found NAS server NetApp Release”.

2) If the alert_db.log does not contain “Found NAS server NetApp Release‖, perform the following

steps:

a) Install net-snmp-* rpms.

b) Run the snmpwalk command.

I certainly dont see this "Found NAS..." but the net-snmp rpms are installed. I hesitate installation of those will be reflected in the alert log. About the snmpwalk there is another reference on page 6 of the same document. But I have following remarks. I am working on C-mode cluster, please could you tell me where I should look for:

snmp.access all

snmp.enable on

options nfs.export.auto-update

My mount points look like this:

vserver-dbnasb402:/ORA/dbs03/RUBEN03 on /ORA/dbs03/RUBEN03 type nfs (rw,bg,hard,nointr,tcp,nfsvers=3,actimeo=0,timeo=600,rsize=65536,wsize=65536,addr=XXXXX)

sys@DODCDB1:SQL> select client from v$client_secrets;

CLIENT

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

vserver-dbnasb402

This is the lif I used to mount, and the credentials are defined using vsadmin user:

sys@DODCDB0:SQL> ADMINISTER KEY MANAGEMENT ADD SECRET 'xxxxx' FOR CLIENT 'vserver-dbnasb402' USING TAG 'vsadmin' IDENTIFIED BY xxx WITH BACKUP;

keystore altered.

Also the firewall of that lif is modified so snmp traffic goes through:

c02::> net int show -vserver dbvs -lif dbvs_lif3 -instance

(network interface show)

Vserver Name: dbvs

Logical Interface Name: dbvs_lif3

Role: data

Data Protocol: nfs

Home Node: dbnasb402

Home Port: a0c

Current Node: dbnasb402

Current Port: a0c

Operational Status: up

Extended Status: -

Is Home: true

Network Address: XXXXXXX

Netmask: 255.255.255.192

Bits in the Netmask: 26

IPv4 Link Local: -

Routing Group Name: XXXXXXX/26

Administrative Status: up

Failover Policy: nextavail

Firewall Policy: datandmgmt

Auto Revert: false

Fully Qualified DNS Zone Name: vserver-dbnasb402.cern.ch

DNS Query Listen Enable: true

Failover Group Name: dbvs_10g_IP26

FCP WWPN: -

Address family: ipv4

Comment: -

c02::> firewall policy show -policy datandmgmt

(system services firewall policy show)

Policy Service Action IP-List

---------------- ---------- ------ --------------------

datandmgmt

dns allow 0.0.0.0/0

http allow 0.0.0.0/0

https allow 0.0.0.0/0

ndmp allow 0.0.0.0/0

ntp allow 0.0.0.0/0

snmp allow 0.0.0.0/0

ssh allow 0.0.0.0/0

telnet deny 0.0.0.0/0

8 entries were displayed.

itrac>-RAC>-DODCDB11:/ORA/dbs01/oracle/product/rdbms/dbs$ snmpget -c public -v 2c vserver-dbnasb402 1.3.6.1.4.1.789.1.1.2.0

Timeout: No Response from vserver-dbnasb402.

Using the cluster management IP:

trac>-RAC>-DODCDB11:/ORA/dbs01/oracle/product/rdbms/dbs$ snmpget -c public -v 2c IPCLUSTERMANAGEMENT 1.3.6.1.4.1.789.1.1.2.0

SNMPv2-SMI::enterprises.789.1.1.2.0 = STRING: "NetApp Release 8.2P3 Cluster-Mode: Thu Sep 12 01:13:07 PDT 2013"

So please could you clarify how to enable snmp and check auto update exports on the vserver to which my data lifs belong to? How should I run snmpwalk?

Thank you,

Ruben

nkarthik · ‎2014-02-25

Hello Ruben,

In your case, Please do the following:

1. create a LIF in vserver for public access using below

cluster1::> network interface create –vserver vs1_dnfs_rac –lif vs1_public_access -role data -home-node TESO-03 -home-port e2a-1164 –address 10.63.164.19 -netmask 255.255.255.0 -failover-policy nextavail -firewall-policy mgmt -auto-revert false

2. check the Lif properties

TESO::> n i show -vserver vs1_dnfs_rac -lif vs1_public_access

(network interface show)

Vserver Name: vs1_dnfs_rac

Logical Interface Name: vs1_public_access

Role: data

Data Protocol: nfs, cifs, fcache

Home Node: TESO-03

Home Port: e2a-1164

Current Node: TESO-03

Current Port: e2a-1164

Operational Status: up

Extended Status: -

Is Home: true

Network Address: 10.63.164.19

Netmask: 255.255.255.0

Bits in the Netmask: 24

IPv4 Link Local: -

Routing Group Name: d10.63.164.0/24

Administrative Status: up

Failover Policy: nextavail

Firewall Policy: mgmt

Auto Revert: false

Fully Qualified DNS Zone Name: none

DNS Query Listen Enable: false

Failover Group Name: system-defined

FCP WWPN: -

Address family: ipv4

Comment:

3. enable snmp for vs1_dnfs_rac

TESO::> security login create -username public -application snmp -authmethod community -role vsadmin-readonly -vserver vs1_dnfs_rac

4. check Snmp access for vserver new lif(10.63.164.19)

[oracle@RHEL64 ~]$ snmpget -c public -v 2c 10.63.164.19 1.3.6.1.4.1.789.1.1.2.0

SNMPv2-SMI::enterprises.789.1.1.2.0 = STRING: "NetApp Release 8.2.1RC1 Cluster-Mode: Wed Nov 20 21:01:05 PST 2013"

[oracle@RHEL64 ~]$

Please let me know, if you need any help.

Regards,

Karthikeyan.N

gasparuben · ‎2014-02-28

Hi Karthikeyan,

That helped. Thanks.

I am getting now this error:

sys@DODCDB1:SQL> alter session set db_create_file_dest='/ORA/dbs03/RUBEN03';

Session altered.

sys@DODCDB1:SQL> create pluggable database ruben03_clone from ruben03 snapshot copy;

create pluggable database ruben03_clone from ruben03 snapshot copy

*

ERROR at line 1:

ORA-65028: Unable to open plugin data file at path /ORA/dbs03/RUBEN03/DODCDB1/F37C479FCE7FCF05E043A506100A84E2/datafile/o1_mf_system_9k1myvhs_.dbf

ORA-17503: ksfdopn:6 Failed to open file /ORA/dbs03/RUBEN03/DODCDB1/F37C479FCE7FCF05E043A506100A84E2/datafile/o1_mf_system_9k1myvhs_.dbf

ORA-17500: ODM err:KGNFS READ FAIL:Invalid argument

I have the database open in read only mode on just one node, on the other node of the Oracle cluster it's closed. I have tried with different db_create_file_dest but I get always the same error.

Our directory structure is like:

--CDB mounts

dbnast1332-priv:/CRS/dbs03/DODCDB1 on /CRS/dbs03/DODCDB1 type nfs (rw,bg,hard,nointr,tcp,vers=3,actimeo=0,timeo=600,rsize=32768,wsize=32768,addr=10.30.16.34)

dbnast1332-priv:/CRS/dbs00/DODCDB1 on /CRS/dbs00/DODCDB1 type nfs (rw,bg,hard,nointr,tcp,vers=3,actimeo=0,timeo=600,rsize=32768,wsize=32768,addr=10.30.16.34)

dbnast1332-priv:/ORA/dbs02/DODCDB1 on /ORA/dbs02/DODCDB1 type nfs (rw,bg,hard,nointr,tcp,vers=3,actimeo=0,timeo=600,rsize=32768,wsize=32768,addr=10.30.16.34)

dbnast1332-priv:/ORA/dbs0a/DODCDB1 on /ORA/dbs0a/DODCDB1 type nfs (rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,addr=10.30.16.34)

dbnast1332-priv:/ORA/dbs03/DODCDB1 on /ORA/dbs03/DODCDB1 type nfs (rw,bg,hard,nointr,tcp,vers=3,actimeo=0,timeo=600,rsize=32768,wsize=32768,addr=10.30.16.34)

dbnast1332-priv:/ORA/dbs04/DODCDB1 on /ORA/dbs04/DODCDB1 type nfs (rw,bg,hard,nointr,tcp,vers=3,actimeo=0,timeo=600,rsize=32768,wsize=32768,addr=10.30.16.34)

dbnast1332-priv:/CRS/dbs02/DODCDB1 on /CRS/dbs02/DODCDB1 type nfs (rw,bg,hard,nointr,tcp,vers=3,actimeo=0,timeo=600,rsize=32768,wsize=32768,addr=10.30.16.34)

dbnast1332-priv:/ORA/dbs00/DODCDB1 on /ORA/dbs00/DODCDB1 type nfs (rw,bg,hard,nointr,tcp,vers=3,actimeo=0,timeo=600,rsize=32768,wsize=32768,addr=10.30.16.34)

--PDB mounts

db-dbnasb402:/ORA/dbs03/RUBEN03 on /ORA/dbs03/RUBEN03 type nfs (rw,bg,hard,nointr,tcp,nfsvers=3,actimeo=0,timeo=600,rsize=65536,wsize=65536,addr=10.16.128.201)

db-dbnasb402:/ORA/dbs03/RUBEN01 on /ORA/dbs03/RUBEN01 type nfs (rw,bg,hard,nointr,tcp,nfsvers=3,actimeo=0,timeo=600,rsize=65536,wsize=65536,addr=10.16.128.201)

db-dbnasb402:/ORA/dbs03/RUBEN02 on /ORA/dbs03/RUBEN02 type nfs (rw,bg,hard,nointr,tcp,nfsvers=3,actimeo=0,timeo=600,rsize=65536,wsize=65536,addr=10.16.128.138)

Here all the availble data files:

sys@DODCDB1:SQL> l

1 SELECT p.PDB_ID, p.PDB_NAME, d.FILE_ID, d.TABLESPACE_NAME, d.FILE_NAME

2 FROM DBA_PDBS p, CDB_DATA_FILES d

3 WHERE p.PDB_ID = d.CON_ID

4* ORDER BY p.PDB_ID

PDB_ID PDB_NAME FILE_ID TABLESPACE_NAME FILE_NAME

------ --------------- ------- ------------------------------ ------------------------------------------------------------------------------------------------------------------------

2 PDB$SEED 4 SYSAUX /ORA/dbs03/DODCDB1/EAA8BEE1715CE8AAE043A506100AA1BF/datafile/o1_mf_sysaux__1448002357603_.dbf

2 PDB$SEED 2 SYSTEM /ORA/dbs03/DODCDB1/EAA8BEE1715CE8AAE043A506100AA1BF/datafile/o1_mf_system__1447994720172_.dbf

2 PDB$SEED 6 USERTBS /ORA/dbs03/DODCDB1/EAA8BEE1715CE8AAE043A506100AA1BF/datafile/o1_mf_usertbs__1448018775315_.dbf

3 ANDREI_PDB 11 USERTBS /ORA/dbs03/DODCDB1/EAEA588C9581F362E043A506100A36F8/datafile/o1_mf_usertbs__1729742098491_.dbf

3 ANDREI_PDB 10 SYSAUX /ORA/dbs03/DODCDB1/EAEA588C9581F362E043A506100A36F8/datafile/o1_mf_sysaux__1729524029687_.dbf

3 ANDREI_PDB 9 SYSTEM /ORA/dbs03/DODCDB1/EAEA588C9581F362E043A506100A36F8/datafile/o1_mf_system__1729742100341_.dbf

4 ANDREI_PDB2 14 USERTBS /ORA/dbs03/DODCDB1/EAEA588C9582F362E043A506100A36F8/datafile/o1_mf_usertbs__1729883100555_.dbf

4 ANDREI_PDB2 12 SYSTEM /ORA/dbs03/DODCDB1/EAEA588C9582F362E043A506100A36F8/datafile/o1_mf_system__1729883118573_.dbf

4 ANDREI_PDB2 13 SYSAUX /ORA/dbs03/DODCDB1/EAEA588C9582F362E043A506100A36F8/datafile/o1_mf_sysaux__1729883101209_.dbf

5 RUBEN02 22 SYSAUX /ORA/dbs03/RUBEN02/DODCDB1/EB11E8A3070EC232E043A506100A5F02/datafile/o1_mf_sysaux__1899663749988_.dbf

5 RUBEN02 37 TOOLS /ORA/dbs03/DODCDB1/EB11E8A3070EC232E043A506100A5F02/datafile/o1_mf_tools__4226018417178_.dbf

5 RUBEN02 36 DBOD /ORA/dbs03/DODCDB1/EB11E8A3070EC232E043A506100A5F02/datafile/o1_mf_dbod__4226017809732_.dbf

5 RUBEN02 24 RUBEN /ORA/dbs03/RUBEN02/DODCDB1/EB11E8A3070EC232E043A506100A5F02/datafile/o1_mf_ruben__1899670039558_.dbf

5 RUBEN02 23 USERTBS /ORA/dbs03/RUBEN02/DODCDB1/EB11E8A3070EC232E043A506100A5F02/datafile/o1_mf_usertbs__1899663740574_.dbf

5 RUBEN02 21 SYSTEM /ORA/dbs03/RUBEN02/DODCDB1/EB11E8A3070EC232E043A506100A5F02/datafile/o1_mf_system__1899663720533_.dbf

5 RUBEN02 39 IOPS /ORA/dbs03/RUBEN02/DODCDB1/EB11E8A3070EC232E043A506100A5F02/datafile/o1_mf_iops__4495986687838_.dbf

6 RUBEN01 38 IOPS /ORA/dbs03/RUBEN01/DODCDB1/EB11E8A3070FC232E043A506100A5F02/datafile/o1_mf_iops__4487547997417_.dbf

6 RUBEN01 33 TPCCTAB /ORA/dbs03/RUBEN01/DODCDB1/EB11E8A3070FC232E043A506100A5F02/datafile/o1_mf_tpcctab__3888591021276_.dbf

6 RUBEN01 28 RUBEN /ORA/dbs03/RUBEN01/DODCDB1/EB11E8A3070FC232E043A506100A5F02/datafile/o1_mf_ruben__1899690893913_.dbf

6 RUBEN01 27 USERTBS /ORA/dbs03/RUBEN01/DODCDB1/EB11E8A3070FC232E043A506100A5F02/datafile/o1_mf_usertbs__1899686875908_.dbf

6 RUBEN01 25 SYSTEM /ORA/dbs03/RUBEN01/DODCDB1/EB11E8A3070FC232E043A506100A5F02/datafile/o1_mf_system__1899686875586_.dbf

6 RUBEN01 26 SYSAUX /ORA/dbs03/RUBEN01/DODCDB1/EB11E8A3070FC232E043A506100A5F02/datafile/o1_mf_sysaux__1899686880864_.dbf

7 RUBEN03 29 SYSTEM /ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_system__2683129838730_.dbf

7 RUBEN03 30 SYSAUX /ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_sysaux__2683129851875_.dbf

7 RUBEN03 31 USERTBS /ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_usertbs__2683129862767_.dbf

7 RUBEN03 32 RUBEN /ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_ruben__2683144897824_.dbf

7 RUBEN03 34 DBOD /ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_dbod__4152914319944_.dbf

7 RUBEN03 35 TOOLS /ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_tools__4152915012393_.dbf

7 RUBEN03 40 IOPS /ORA/dbs03/RUBEN03/DODCDB1/EBC852DD72A67D43E043A506100AA921/datafile/o1_mf_iops__4498075797710_.dbf

This is what I see in the alert log for this instance:

create pluggable database ruben03_clone from ruben03 snapshot copy

Fri Feb 28 19:08:20 2014

Direct NFS: Read FAIL status 22

Fri Feb 28 19:08:20 2014

Direct NFS: NFSERR 305 error encountered. Server db-dbnasb402 path db-dbnasb402

ORA-65028 signalled during: create pluggable database ruben03_clone from ruben03 snapshot copy...

Dont find anything in metalink (MOS) about this dnfs error.

The access to all mount points is via dnfs:

sys@DODCDB1:SQL> select * from v$dnfs_servers;

ID

--------------

SVRNAME

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

DIRNAME

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

MNTPORT NFSPORT NFSVERSI WTMAX RTMAX CON_ID

-------------- -------------- -------- -------------- -------------- --------------

1

dbnast1332-priv

/ORA/dbs00/DODCDB1