How to get totally confused in one easy step! Had some one-to-one Netapp training today, and came up with an issue that had both of us scratching our heads. We still don't fully know the answers!
It started with us running our performance autosupport file through the Netapp partner tool. This indicated misalignment on one of our luns.
Knowing that the Netapp block size is 4k (4096 bytes), so we checked the starting offset for the disk by running the msinfo32.exe (in the c:\Windows\servicepackfiles\I386), this showed the offset as 32256, dividing this by 4096 gave a figure of 7.875, ie not directly divisible, thus confirming the blocks as bring out of alignment.
We therefore did some more digging and found this article:
http://msdn.microsoft.com/en-us/library/dd758814(v=sql.100).aspx
Working through this it appears that the 32256 is the default and if you use the Microsoft disk management tool there is no way to change this, the solution is to use the diskpart program.
We therefore destroyed the partition and set about recreating it with the diskpart commands:
diskpart
list disk
select disk <number>
create partition primary align=32
assign letter=<drive letter>
We then went back into the disk management tool and formatted it as ntfs.
Upon checking in the msinfo32 tool again, the offset is now 32768, and therefore directly divisible by 4096.
So far so good.
Let's check the alignment, to do this we set off a large copy of data to the Lun and in another window set off a perfstat to the filer with a 1 minute duration. Perfstat can be downloaded from the software area of the Netapp website.
We then checked the perfstat output file, searching for the string "perfstat_lun" which is at the start of the section we are interested in.
Here we found the the writes were being performed to bucket 1, thereby being 512 bytes out of alignment (all reads and writes should go to bucket 0 when in alignment).
Strange! The offset is divisible by 4096, but is out of alignment by 512. When the offset was 512 less, we were still showing misalignment!
So we thought, let's recreate the Lun again with the original offset (32256), by using the disk management tool again.
Repeating the test by writing data and running a perfstat job at the same time. Thus time the report showed all the writes into bucket 0!
Even stranger, the lun is now in alignment with the same as the original settings!
So why is it now in alignment when it originally wasn't?
How come it is in alignment when the starting offset is not divisible by 4096?
All, we seen to have learnt on this is how to check for maligned luns, but not how to select a suitable offset should maligned luns be found. However it appears that it is always worth recreation luns with the same settings should misalignment be found!
It started with us running our performance autosupport file through the Netapp partner tool. This indicated misalignment on one of our luns.
Knowing that the Netapp block size is 4k (4096 bytes), so we checked the starting offset for the disk by running the msinfo32.exe (in the c:\Windows\servicepackfiles\I386), this showed the offset as 32256, dividing this by 4096 gave a figure of 7.875, ie not directly divisible, thus confirming the blocks as bring out of alignment.
We therefore did some more digging and found this article:
http://msdn.microsoft.com/en-us/library/dd758814(v=sql.100).aspx
Working through this it appears that the 32256 is the default and if you use the Microsoft disk management tool there is no way to change this, the solution is to use the diskpart program.
We therefore destroyed the partition and set about recreating it with the diskpart commands:
diskpart
list disk
select disk <number>
create partition primary align=32
assign letter=<drive letter>
We then went back into the disk management tool and formatted it as ntfs.
Upon checking in the msinfo32 tool again, the offset is now 32768, and therefore directly divisible by 4096.
So far so good.
Let's check the alignment, to do this we set off a large copy of data to the Lun and in another window set off a perfstat to the filer with a 1 minute duration. Perfstat can be downloaded from the software area of the Netapp website.
We then checked the perfstat output file, searching for the string "perfstat_lun" which is at the start of the section we are interested in.
Here we found the the writes were being performed to bucket 1, thereby being 512 bytes out of alignment (all reads and writes should go to bucket 0 when in alignment).
Strange! The offset is divisible by 4096, but is out of alignment by 512. When the offset was 512 less, we were still showing misalignment!
So we thought, let's recreate the Lun again with the original offset (32256), by using the disk management tool again.
Repeating the test by writing data and running a perfstat job at the same time. Thus time the report showed all the writes into bucket 0!
Even stranger, the lun is now in alignment with the same as the original settings!
So why is it now in alignment when it originally wasn't?
How come it is in alignment when the starting offset is not divisible by 4096?
All, we seen to have learnt on this is how to check for maligned luns, but not how to select a suitable offset should maligned luns be found. However it appears that it is always worth recreation luns with the same settings should misalignment be found!
If I understand correctly, you recreated the partition once using the same LUN and then created a new LUN on the storage and used the old settings. I suspect you might have had different versions of Data ONTAP during the original LUN creation and the new LUN creation. We have improved how ONTAP creates LUNs to properly align better. Have you seen this KB?: https://kb.netapp.com/support/index?page=content&id=1010803
ReplyDeleteThanks for that.
ReplyDeleteThe data was originally on a Filer running Data ONTAP 7.3.3, and was moved to a Filer running 7.3.4
We never did any checking of alignment of Luns on the old 7.3.3 filer, so it could well have been misaligned on there.
Copying it to the 7.3.4 filer (through copy/paste in Windows 2003) showed the misalignment on the new Lun.
We then created a separate volume and Lun and moved the data to this which was in alignment. We then re-created the first lun on 7.3.4 and copied the data back - this was then shown as being aligned.
We also played around with the offset on the Luns on the 7.3.4 filer and could not get them to align when the start block was directly divisible by 4096. This is what confused us!
I wonder if it was misaligned on 7.3.3, then the copy to 7.3.4 carried this over. Copying 7.3.4 to 7.3.4 then brought it into alignment?
It was strange as we have 5 Luns on our filer and this was the only 1 to show this behaviour. The other 4 were aligned first time when we checked!
"Copying it to the 7.3.4 filer (through copy/paste in Windows 2003) showed the misalignment on the new Lun."
ReplyDeleteDo I understand you to be saying you're mounting the NetApp volume that contains the lun and copying the actual lun via CIFS? While this works, we definitely recommend doing something like ndmpcopy or snapmirror instead. The CIFS protocol can cause some unexpected issues here and could be a reason we wound up offset during one of the copies.
Oh it's worth confirming also that if you copied the raw lun to a new system by any of these means, it would have carried over the same filesystem (Ex: NTFS -> WAFL) offset and been unaligned.
ReplyDeleteGerald,
ReplyDeleteMany thanks for all your help on this.
Not sure how we would use NDMP Copy to do this as we are talking about separate filers and we do not have the Snap Mirror licence!
Anyway, we have since moved all the data back to the correct final luns and confirmed using perfstat that the luns are all now in alignment.
This information will be here should we need to do this in future!
Thanks for your help again.
The reason the partition with the offset as 32256 is showing up as aligned is because it is a raw disk/lun. When you create the lun and select the correct lun type (in this case windows), the filer does all the alignment stuff on the filer side. And so there is no need to do this manually in diskpart, in fact if you do, you will end up with unaligned disks.
ReplyDeleteAtleast this is my understanding of it.
You should only pre partiton your disks if they are virtual, and then set the offset to 32k or 64k. Your layer below should already be aligned if you selected the correct lun type at creation (windows for ntfs/hyper v and VMware for esx/vmware).
RAW device mappings dont require diskpart or anythign similar, selecting the right LUN type at creation will sort out all your alignment things for you.