Re: Problem with deleting archive log files

ANDREAS_JANKOWIAK · ‎2012-09-24

Dear Community,

I have a problem with our snapcreator ver 3.5 implementatation.

We take snapshots for some db2 (SAP) databases serially, so we have db2 plugin configurations which quiesces the dbs, after that we take a snapshot of the relevant volumes and unquiesece the dbs. After the unquiesce we run:

POST_APP_UNQUIESCE_CMD01=snapcreator.exe -profile Q-Landscape -config <SID1> -action arch

POST_APP_UNQUIESCE_CMD02=snapcreator.exe -profile Q-Landscape -config <SID2> -action arch

POST_APP_UNQUIESCE_CMD03=snapcreator.exe -profile Q-Landscape -config <SID3> -action arch

POST_APP_UNQUIESCE_CMD04=snapcreator.exe -profile Q-Landscape -config <SID4> -action arch

for each db in the main configuration.

The db specific configuration looks like this:

ARCHIVE_LOG_ENABLE=Y

ARCHIVE_LOG_RETENTION=9

ARCHIVE_LOG_DIR=/db2/<SID>/archdir/db2<sid>/<SID>/NODE0000/

ARCHIVE_LOG_EXT=LOG

ARCHIVE_LOG_RECURSIVE_SEARCH=Y

Sometimes the automatic erease of archive logs is working but often we get the following error, ie:

[Sun Sep 23 23:00:26 2012] ERROR: Could not delete Archive Log .../NODE0000/C0000008/S0000003.LOG: Inappropriate ioctl for device

[Sun Sep 23 23:00:26 2012] ERROR: [scf-00112] Archive Log Deletion failed, Exiting!

[Sun Sep 23 23:00:26 2012] DEBUG: Exiting with error code - 1

Sometimes the log file creation data is very strangly recongnized:

INFO: Performing recursive search on dir ...NODE0000

DEBUG: Found following Archive Log ...NODE0000/C0000008/S0000000.LOG, was created on Jan 01 1970 01:00:00

DEBUG: Found following Archive Log ...NODE0000/C0000008/S0000001.LOG, was created on Jan 01 1970 01:00:00

DEBUG: Found following Archive Log ....NODE0000/C0000008/S0000002.LOG, was created on Jan 01 1970 01:00:00

DEBUG: Found following Archive Log ..../NODE0000/C0000008/S0000004.LOG, was created on Sep 14 2012 00:04:02

DEBUG: Found following Archive Log ...PHA/NODE0000/C0000008/S0000005.LOG, was created on Sep 14 2012 00:05:14

For some files the deletion is working

[Sun Sep 23 23:00:25 2012] INFO: Archive Log ....NODE0000/C0000008/S0000111.LOG is not older than 9 days, Skipping!

[Sun Sep 23 23:00:25 2012] INFO: Archive Log ...NODE0000/C0000008/S0000110.LOG is not older than 9 days, Skipping!

[Sun Sep 23 23:00:25 2012] INFO: Archive Log .....NODE0000/C0000008/S0000011.LOG is older than 9 days, Deleting!

[Sun Sep 23 23:00:25 2012] INFO: Archive Log ....NODE0000/C0000008/S0000011.LOG has been successfully deleted

After restarting the snapcreator agent it works again! But after 1 or 2 days the next error occurs.

Does anyone has the same problem?

Thanks a lot for your help.

Best Regards,

Andreas

ktenzer · ‎2012-09-24

Strange it sounds like agent is holding file handles open or something. The error is an OS error but I agree it is something strange with SC agent. I know I have seen this issue before and I think we fixed it in 3.6 but am not 100% so dont hold me to it.

The only thing I can think of trying is using SC 3.6 we did make a lot of fixes and improvements so maybe the issue is resolved?

There isnt really a workaround I can think of for 3.5 as it appears to be a bug.

You need to upgrade SC server to 3.6 to use 3.6 agent

Keith

ANDREAS_JANKOWIAK · ‎2012-09-24

Hi Keith,

thanks again for your quick response.

The bad thing is that we use a N-Series storage... and my information is that SnapCreator 3.6 is not officially approved by IBM yet.

Do you have information on that?

Thanks!

BR,

Andreas

ktenzer · ‎2012-09-24

We released a version for IBM for 3.6 at same time 3.6 released on NOW site which was end of July. Not sure how long it takes IBM to make it available but you could at least download NetApp version and test it and if it works then you know you have a fix for it

Regards,

Keith

ANDREAS_JANKOWIAK · ‎2012-09-25

Hi Keith,

testing the version 3.6 in the customer landscape will be difficult for me.

If this is a known problem, will there be no fix for version 3.5 ?

I check with IBM, the version is still not approved ;o(

BR,

Andreas

ktenzer · ‎2012-09-25

I dont know if this is a known problem, I know we saw something similar in QA in 3.6 and fixed it but it could have been a different issue.

Regardless the recommendation is to go to SC 3.6

This would not be fixed in SC 3.5. We dont fix old releases, we fix code in current releases. There is a SC 3.6P1 coming out soon for example with some fixes (not related to this).

If this issue is not addressed in SC 3.6 then you need to open case, engineering will either provide customer patch or fix in a future release depending on what support and engineering decide.

For now just dont use this feature. You could write a simple script which deletes logs based on retention? If so you can run your script from SC and then you have same solution

Hope this helps

Keith

brauntvr2swiss · ‎2012-10-17

Hi Andreas

We have the same Issue on one Server (Oracle on Linux with sc). We have sc agent 3.5.0.1 installed. Do you have fixed this Problem in your environment? When yes How? (fe.Upgrade to 3.6)

TIA

Thomas

ANDREAS_JANKOWIAK · ‎2012-10-17

Hi Thomas,

I implemented SC 3.6 this week in our environment and it looks better.

I've monitored now for 2 days, seems to work fine.

So upgrade to version 3.6 should fix your problem.

BR,

Andreas

brauntvr2swiss · ‎2012-10-17

Hi Andreas

Thanks for your quick response.. In the moment we cant upgrade to SC 3.6, because something does not working with the Protection Manager Integration in our environment (with 3.6) :-(..

I have to wait for 3.6P1 with the hope our Protection Manager Problem is solved in the P1 Version......

regards

Thomas

brauntvr2swiss · ‎2012-10-18

FYI....

We have solved the Problem without upgrade to 3.6 :-)... The only thing we have changed: scAgent started multithreaded and the Problem was away...

I dont know the deep why, but it work now...

Thomas

ktenzer · ‎2012-10-18

Hi Thomas,

Cool!

As mentioned it is definitely a bug we fixed in 3.6 but if multithreaded agent fixes issue feel free to use that

Just be aware multithreaded feature of agent does not exist on windows, it is a unix only thing...in SC 4.1 agent will be multithreaded by default on both windows and unix.

Keith