ZFS and Proxmox performance on consumer SSDs
Background - I have a pair of Samsung QVO 8TB consumer grade drives, and some Seagate Ironwolf SSD 4TB enterprise drives. The 8TB drives were configured as a mirror and were performing terribly - like 3 fsyncs a second terrible. The enterprise drives work great. The following is my conclusions: After several days and nights of testing I think I now understand my problem. Its overcomplicated, and the rule "just use enterprise SSDs for ZFS" holds but given I am sitting on 48 TB of consumer grade SSDs I wanted to understand what what happening. The root issue was the consumer grade performance and the design of consumer SSDs necessitating trimming. Once enough bytes have been written (say by moving a VM), their performance falls off a cliff. Proxmox has a handy tool called pveperf that tells you Fsyncs a second. Running them on one of my old drives in situ was giving 10(!) a second. The enterprise was giving 1100! Surely I can make this better. So I emptied the drive recreat