Helsinki is clearly (and logically) having higher latency to the German nodes. This adds up enormously in time.
We are using 6 virtual nodes at Hetzner, all the cheapest variant: 2 GB RAM, 20GB NVMe SSD disk, 1 Skylake IBS CPU core. All are on gbit. GFS 1+2+3 are all in the same location (Nuremberg). The rest speak for themselves.
Copies. If the amount of copies equals the amount of servers, all servers have a local copy of the data. Then no read latency occurs, as everything can be served locally. Additionally, all other nodes in the gluster can go down and data can still be read. However, if there are less copies than servers, read latency occurs as data needs to be fetched over the network with every read, and all servers must be alive for it to function.
This test was simple: Extract WordPress:
cd ~ wget https://wordpress.org/latest.tar.gz tar -zxf latest.tar.gz -C /path/to/volume/mount
This extracted/created 193 directories, filled with a total of 1835 files. A total of 2028 objects. 42695316 bytes total, 21052.92 bytes per file on average. Or 20.6 KiB/object. Negligible sizes (and big overhead).
|Vol||Set-up||Highest Lat.||1 Replica||Full replicas||Difference||Per object (1)||Per object (all)|
|gv0||Nuremberg||0.060 ms||5s||5s||0s||2.47 ms||2.47 ms|
|gv1||Nuremberg + Falkenstein||3.635 ms||39s||97s||+58s||19.23 ms||47.83 ms|
|gv2||Nuremberg + Helsinki||25.441 ms||227s||595s||+368s||111.93 ms||293.39 ms|
|gv3||Nuremberg + Falkenstein + Helsinki||26.262 ms||179s||597s||+418s||88.26 ms||294.38 ms|
|gv4||GFS 1||0.159 ms||5s||5s||0s||2.47 ms||2.47 ms|
|gv5||GFS 1 + GFS 2||0.995 ms||14s||34s||+20s||6.90 ms||16.77 ms|
|gv6||GFS 1 + GFS 3||1.128 ms||12s||31s||+19s||5.92 ms||15.29 ms|
|gv6||GFS 1 + GFS 2 + GFS 3||1.128 ms||15s||34s||+19s||7.40 ms||16.77 ms|
Extracting WordPress locally on a
tmpfs-disk takes 0.6s. Let's say 1 second. Rounding decimals to full seconds.