software raid rebuilding time under linux

Submitted by mator
on July 25, 2006 - 6:47am

using an old celeron 466MHz machine with 2 new Maxtor 6Y080L0 disks in raid1 software array. DMA and 32bit-access is enabled (hdparm) for both disks.

# uptime
 17:43:19 up  2:09,  3 users,  load average: 1.29, 1.06, 1.06
# mdadm -D /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Mon Jul 24 12:35:51 2006
     Raid Level : raid1
     Array Size : 72685952 (69.32 GiB 74.43 GB)
    Device Size : 72685952 (69.32 GiB 74.43 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Jul 25 15:34:25 2006
          State : dirty, recovering
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0


 Rebuild Status : 43% complete

    Number   Major   Minor   RaidDevice State
       0       3        6        0      active sync   /dev/hda6
       1      22        6        1      active sync   /dev/hdc6
           UUID : 5937930e:dd70ac10:9bfff6fe:1421d511
         Events : 0.5046
# cat /proc/mdstat 
Personalities : [raid1]
md0 : active raid1 hdc6[1] hda6[0]
      72685952 blocks [2/2] [UU]
      [========>............]  resync = 44.4% (32314240/72685952) finish=128.7min speed=5223K/sec

as you can see for 2 hours only 40% is done. I wander is it any faster to rebuild 63Gb array on faster processor?

add/edit 23/01/2007 :
looking at md.c from 2.4.34 kernel sources:
/*
* Current RAID-1,4,5 parallel reconstruction 'guaranteed speed limit'
* is 100 KB/sec, so the extra system load does not show up that much.
* Increase it if you want to have more _guaranteed_ speed. Note that
* the RAID driver will use the maximum available bandwith if the IO
* subsystem is idle. There is also an 'absolute maximum' reconstruction
* speed limit - in case reconstruction slows down your system despite
* idle IO detection.
*
* you can change it via /proc/sys/dev/raid/speed_limit_min and _max.
*/

So, increasing this 2 variables can dramatically reduce rebuild time, for example, rebuilding 1Tb raid5 with 3 sata (each 500gb) disks with default settings 100/10000 gives me aprox.rebuild end in 800 minutes. Increasing speed_limit_max to 50000 cut rebuild time to 300 minutes.

# mdadm -D /dev/md5
/dev/md5:
        Version : 00.90.00
  Creation Time : Fri Jan 19 15:06:16 2007
     Raid Level : raid5
     Array Size : 976767872 (931.52 GiB 1000.21 GB)
    Device Size : 488383936 (465.76 GiB 500.11 GB)
   Raid Devices : 3
  Total Devices : 4
Preferred Minor : 5
    Persistence : Superblock is persistent

    Update Time : Tue Jan 23 11:40:34 2007
          State : active, degraded, recovering
 Active Devices : 2
Working Devices : 2
 Failed Devices : 2
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 32% complete

           UUID : d43db220:e08760aa:fc62ddd8:61d2e6ab
         Events : 0.12

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8        1        1      active sync   /dev/sda1
       2       0        0        2      removed

       3       8       33        3      spare   /dev/sdc1
# cat /proc/mdstat 
md5 : active raid5 sdb1[0] sdc1[3] sda1[1]
      976767872 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
      [======>..............]  recovery = 30.2% (147512916/488383936) finish=162.8min speed=34882K/sec

Hope this helps.

PS: thanks to linux kernel developers and Neil Brown for the excellent mdadm tool.

Linux software raid rebuild

Tony240ZT (not verified)
on
August 18, 2006 - 2:18pm

My PIII 600 with two 20 gig drives UDMA 33 are rebuilding at about 7,000K/sec which estimates about a 45minute raid 1 build. Yours is only going at 5,223K/sec, which doesn't seem all that great. If you can get your DMA speed up to 66, 100, or 133 you'll obviously have a much quicker disk IO speed. If your onboard IDE controler doesn't support higher speeds, but your drives do you could always through in a PCI controler card like those offered by Promise, with added support through your kernel. The drives will be relabeled to /dev/sdX though.

Since software raid does use a lot of IO and cpu while rebuilding, yes a faster processor would help. While I'm rebuilding an array my load is at ~1.10

re: rebuild time

mator
on
August 30, 2006 - 7:59am

and yet another example:
/dev/md2 is 2Gb raid1(mirror) over 2 partitions (sda6,sdb6), which is 2 sata disks in sun fire x2100 server (Dual Core AMD Opteron Processor 175).


Aug 30 18:28:09 doltan kernel: md: syncing RAID array md2
Aug 30 18:28:09 doltan kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
Aug 30 18:28:09 doltan kernel: md: using maximum available idle IO bandwith (but not more than 200000 KB/sec) for reconstruction.
Aug 30 18:28:09 doltan kernel: md: using 128k window, over a total of 1959808 blocks.
Aug 30 18:28:46 doltan kernel: md: md2: sync done.

as you can see 2Gb raid sync was done in 37 seconds, really nice! :-)

PS: i'm not used nvidia chipset bios raid functionality on this server MB, since i learn about dmraid utility only today.

hdparm

Anonymous (not verified)
on
January 18, 2007 - 6:59pm

If you have this problem, turning on DMA for your disks will increase the speed greatly.

test speed:
hdparm -tT /dev/hda

enable dma for all array members:
hdparm -d1 /dev/hda
hdparm -d1 /dev/hdb
hdparm -d1 /dev/hdc
hdparm -d1 /dev/hdd

test speed:
hdparm -tT /dev/hda

see how fast it is?

hdparm

mator
on
November 21, 2008 - 4:48pm

thanks for the comment, i have noted in the first post about this hdparm setting.

i'm always trying to enable DMA transfer mode for the hard disks. But only recent sata disks visible as a scsi devices to kernel (/dev/sd?) and hdparm gives error trying to operate on it.
Talking about hdparm, it's also usefull to enable 32 bit transfers with hdparm -c 1 /dev/hda.
Check your redhatish (fedora) /etc/sysconfig/harddisks file.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.