An array loses a drive. A replacement goes in and the rebuild starts. Those few hours are the most fragile the storage system will be all year.
Most people picture a rebuild as the system quietly filling data back in. What it actually does is more aggressive: it reads every remaining member drive from end to end. Normal operation only touches hot data, and cold regions may not have been read in years. During a rebuild, they will be.
1. A second drive drops mid-rebuild
Symptom. RAID 5 loses a drive, a replacement goes in, and at sixty or seventy percent through the rebuild another member throws an error. The set is gone.
The mechanism. This is an unrecoverable read error (URE). Every mechanical drive datasheet specifies a URE rate, commonly one per 10^14 or 10^15 bits read. You never encounter it in normal operation because normal operation never reads that much consecutively. A rebuild does.
The larger the drive, the more data a full pass reads. The older the fleet and the closer the batch, the more likely several members are degrading together. Those two factors compound, and that is why RAID 5 on high-capacity drives carries real risk.
What to do:
- RAID 6 (dual parity) tolerates a second failure during the rebuild. On large drives this is a baseline requirement, not an upgrade.
- Replace drives before they degrade, not after they fail. See the SMART trend section in When a failed drive is not the problem.
- Stage the replacement before the rebuild. Sourcing a part after the failure means the waiting period is spent with no redundancy at all.
One practical reality. Drives in out-of-warranty systems usually went in as one batch on one day. When one reaches end of life, the others are close behind. An entire set entering its high-risk window together is something the plan has to account for.
2. Throughput halves while every drive reports healthy
Symptom. All array members show normal status and there are no alerts, but throughput and latency are clearly worse.
This is the hardest class of problem on aging equipment, because nothing is broken. Three possible causes:
A drive has entered internal retry or media scan. Individual responses stretch to hundreds of milliseconds without crossing the controller’s error threshold. Per-drive latency distribution finds it: if one member’s P99 is markedly worse than its peers, that is the one.
The controller downgraded its cache policy. The usual reason is a failed RAID cache battery, which drops the controller from write-back to write-through and costs an order of magnitude on writes. That case is covered in Nothing failed, the machine just got slower.
A background task is running. Consistency checks, patrol reads and rebuilds all consume bandwidth. Many platforms schedule a consistency check weekly or monthly by default, and if nobody has adjusted the window it may be landing squarely on your peak hours.
Order of work. Check controller state and the background task list first — it is the fastest step — then per-drive latency distribution, and only then suspect hardware. Reversing that order wastes a great deal of time.
3. The controller was replaced and the array is unrecognizable
Replacing a storage controller is the highest-risk routine action in storage maintenance.
Symptom. After the swap, the existing array configuration is lost, misread, or comes back with the disk order scrambled.
Three common causes:
- Firmware incompatibility. The new controller’s firmware cannot correctly read the configuration metadata already written on the disks.
- No exported configuration, or disks not returned in order. Most controllers can reconstruct configuration from on-disk metadata, but only if the ordering information is intact.
- On dual-controller systems, the second controller was touched before synchronization finished. This forks the configuration between the two sides.
The procedure — this is a procedure, not a suggestion:
- Export and save the configuration before starting
- Record disk order bay by bay, with photographs
- Follow the vendor’s replacement process step by step; do not skip steps from experience
- On dual-controller systems, change one at a time and wait for synchronization
There is no shortcut here. The cost of improvising on site is the whole data set.
4. Dual-controller storage running on one controller for six months
Symptom. One controller failed long ago, the other has carried the load ever since, and nothing was replaced.
This is a common state on out-of-warranty storage.
Three layers of risk:
- There is no redundancy. A problem on the survivor is a full stop, not a degradation.
- Performance is halved, while the entire load runs through one controller.
- The survivor’s own failure probability is rising, because sustained full load is accelerated ageing.
Why it stays that way. Usually not by choice: the post-warranty quote was too high, or the OEM no longer offers the part at all.
What is actually true. Controllers of this class are generally available through channel stock and decommissioned assets. The question is not whether one can be bought — it is whether its firmware will read the existing configuration, which brings you back to the previous section.
The longer a system runs degraded, the more the final failure costs. It is a liability that does not appear on any balance sheet and settles in full on the day of the outage.
A few rules for array design
- On drives of 8TB and above, use RAID 6 or better. Not RAID 5.
- More members means a longer rebuild, which means a wider risk window.
- A hot spare shortens the gap between failure and the start of the rebuild. It is worth the slot.
- Check when consistency checks are scheduled and move them to off-peak hours.
- Stagger replacement of same-batch drives to spread out their failure curves.
On spare parts
Half the answer to rebuild risk is having the part before you need it.
Seven Small Services catalogues 11,894 part numbers, 4,786 physically stocked, with regional depots in six cities in China and local warehouses in 35+ countries and regions. For drives and controllers — the parts a rebuild depends on — it is worth confirming stock location against your equipment list in advance. Search Parts Lookup or send us the list.
This describes common failure modes for this class of equipment, not a specific incident at a named site.