GlusterFS
Helsinki is clearly (and logically) having higher latency to the German nodes. This adds up enormously in time.
We are using 6 virtual nodes at Hetzner, all the cheapest variant: 2 GB RAM, 20GB NVMe SSD disk, 1 Skylake IBS CPU core. All are on gbit. GFS 1+2+3 are all in the same location (Nuremberg). The rest speak for themselves.
Copies. If the amount of copies equals the amount of servers, all servers have a local copy of the data. Then no read latency occurs, as everything can be served locally. Additionally, all other nodes in the gluster can go down and data can still be read. However, if there are less copies than servers, read latency occurs as data needs to be fetched over the network with every read, and all servers must be alive for it to function.
This test was simple: Extract WordPress:
cd ~
wget https://wordpress.org/latest.tar.gz
tar -zxf latest.tar.gz -C /path/to/volume/mountThis extracted/created 193 directories, filled with a total of 1835 files. A total of 2028 objects. 42695316 bytes total, 21052.92 bytes per file on average. Or 20.6 KiB/object. Negligible sizes (and big overhead).
| Vol | Set-up | Highest Lat. | 1 Replica | Full replicas | Difference | Per object (1) | Per object (all) |
|---|---|---|---|---|---|---|---|
| gv0 | Nuremberg | 0.060 ms | 5s | 5s | 0s | 2.47 ms | 2.47 ms |
| gv1 | Nuremberg + Falkenstein | 3.635 ms | 39s | 97s | +58s | 19.23 ms | 47.83 ms |
| gv2 | Nuremberg + Helsinki | 25.441 ms | 227s | 595s | +368s | 111.93 ms | 293.39 ms |
| gv3 | Nuremberg + Falkenstein + Helsinki | 26.262 ms | 179s | 597s | +418s | 88.26 ms | 294.38 ms |
| gv4 | GFS 1 | 0.159 ms | 5s | 5s | 0s | 2.47 ms | 2.47 ms |
| gv5 | GFS 1 + GFS 2 | 0.995 ms | 14s | 34s | +20s | 6.90 ms | 16.77 ms |
| gv6 | GFS 1 + GFS 3 | 1.128 ms | 12s | 31s | +19s | 5.92 ms | 15.29 ms |
| gv6 | GFS 1 + GFS 2 + GFS 3 | 1.128 ms | 15s | 34s | +19s | 7.40 ms | 16.77 ms |
Extracting WordPress locally on a tmpfs-disk takes 0.6s. Let's say 1 second. Rounding decimals to full seconds.
- Doing it on a 3-node cluster on the same network with 1 replica takes a maximum of 15 seconds.
- But doing it on a 3-node cluster with a geographically different network: 179 seconds. Almost 12 times as long.
- But doing it on a 3-node cluster with a geographically different network: 179 seconds. Almost 12 times as long.
- Doing it on a 3-node cluster on the same network with a replica per server takes a maximum 34 seconds.
- But doing it on a 3-node cluster with a geographically different network: 597 seconds. Almost 18 times as long.