====== Ceph ====== We use a Ceph Storage Cluster for our storage system. [[http://docs.ceph.com/docs/master/]]\\ Ceph has an underlying system called RADOS which stores the data. On top of that three different mechanisms are provided to access and use the underlying RADOS layer. - RadosGW: Bucket based REST API for integration with S3/Openstack, need for a rados gateway service/server - RBD: Rados Block Devices, creates a full block device in the storage cluster, can be mounted and used (create filesystem) as any other block device - CEPH FS: A POSIX compliant file system, can be mounted with FUSE driver, need for a metadata service/server {{ :infrastructure:ceph-stack.png?700 |}} The cluster itself consists of basically two components: ''Monitors'' and ''OSD Daemons'' Some important concepts to understand are Pools, the Crush Map and the Placement groups.\\ **Pools:** A pool is a logical partition for storing objects. It is basically a part of the overall storage cluster. Each pool has its own number of Placement groups and its own Crush Map. (Different number of replicas, failure domains, etc.)\\ ---- **Placement groups:** One pool has a certain number of placement groups, when an object is added it is hashed and put in one placement group, each placement group (based on the number of replicas) puts its object on certain OSDs. \\ Here is a Placement group calculator for choosing the number of PGs for a pool. [[https://ceph.com/pgcalc/]]\\ More on Placement Groups: [[http://docs.ceph.com/docs/master/rados/operations/placement-groups/]]\\ Placement Groups can be in different states:\\ * Inactive: Cannot provess reads or write because waiting for an OSD with most up-to-date data to come up and in * Unclean: Contains data that is not replicated the desired number of times, should be recovering * Stale: Unkown state - all the osds that host them have not reported in a while ---- **Crush Map:** The Crush Map finally maps the objects in a placement group to one or several (number of replicas) OSDs. The crush algorithm can be tweaked to take different failure domains in to account. ===== Overview Infrastructure ===== ^Server^Purpose^N-Disks^Raw Space^N-Journal Disks^Journal Space^ |mon01-cm|Monitor + (CephFS Metadata)| - | - |1 x 800GB|800GB| |sto01-cm|OSD Daemon + Monitor|15 x 9,9TB|148,5TB|2 x 120GB|240GB| |sto02-cm|OSD Daemon + Monitor|15 x 9,9TB|148,5TB|2 x 120GB|240GB| ^Pool Name^Purpose^Replica Size^Note^ |rbd|Default Pool|repl: 2|Used for Rados Block devices| |one|Opennebula pool|repl: 2|VM disks for Opennebula| |fs|Ceph FS Data Pool|repl: 2|-| |fs_meta|Ceph FS Meta Data Pool| repl: 3|-| |ec|Erasure Coded Pool|erasure: k=8 m=4|-| ===== CephFS Export through NFS ===== To export CephFS namespaces nfs-ganesha needs to be used. * Install packages: sudo apt install libcephfs2 nfs-ganesha nfs-ganesha-ceph ceph-fuse * Copy client keyrings in /etc/ceph and add "ceph." at the beginning of the keyring files * client.I11.fs.sto.student.keyring -> ceph.client.I11.fs.sto.student.keyring * Make sure you can mount CephFS directories with ceph-fuse, it will be used from nfs-ganesha * sudo ceph-fuse -n client.I11.fs.sto.student -r /I11/sto/student /mnt/public * Edit nfs-ganesha configuration file: sudo vim /etc/ganesha/ganesha.conf EXPORT { # Export Id (mandatory, each EXPORT must have a unique Export_Id) Export_Id = 2; # Exported path (mandatory) Path = /I11/sto/student; # Pseudo Path (required for NFS v4) Pseudo = /mnt/public; # Exporting FSAL FSAL { Name = CEPH; User_Id = "I11.fs.sto.student"; } # Export to clients CLIENT { Clients = 131.159.24.0/23, 172.24.24.0/23; Squash = None; Access_Type = RW; } } * Restart ganesha service: sudo service nfs-ganesha restart * Have a look into /var/log/ganesha/ganesha.log to check for problems * Mount directory on a remote host: sudo mount -t nfs -o nfsvers=4.1,proto=tcp cephex:/mnt/public /data/ceph/public ===== Usage ===== === Get PGs Number === * Get placement groups in a pool ceph osd pool get pg_num === Set PGs Number === * First calculate number of PGs according to environment: [[https://ceph.com/pgcalc/|Ceph PG Calc]] * Number of Placement Groups can not be decreased! ceph osd pool set pg_num ceph osd pool set pgp_num #first command splits data, second command makes the number available to crush algorithm # values should be equal * [[https://docs.ceph.com/docs/master/releases/|Ceph Version]] ceph tell mon.* version * Show config of ceph daemons, log in on the host the daemon runs on! ceph daemon mds.sto01 config show ceph daemon osd.0 config show * Short Status ceph health * Storage Cluster status ceph status * Cluster usage stats: Show free space in the pools ceph df * List users and their capabilities ceph auth list * Very quick benchmark of a cluster pool, simple throughput benchmark with 1GB in 4-MB increments ceph tell osd. bench * Monitor Map: Contains cluster fsid, position, name address and port of each monitor, indicates current epoch, when map was created, when map was last changed ceph mon dump * OSD Map: Contains cluster fsid, when map was created and last modified, list of pools, replica sizes, PG numbers, list of OSDs and their status (up,in) ceph osd dump * PG Map: Contains PG version, time stamp, last ODS map epoch, full ratios, details on each Placement group such as PG ID, UP Set, Acting Set, state of PG and data usage for each pool ceph pg dump ceph pg map * Crush Map: Contains a list of storage devices, failure domain hierarchy when storing data ceph osd getcrushmap -o #decompile map crushtool -d -o #view with vim or cat * MDS Map: metadata server map, when map was created and last time changed, conatins pool for storing metadata, list of metadata servers and which are up (up,in) ceph fs dump * MDS Clients: show all cephfs clients connected to this metadata server - using the filesystem ceph tell mds.0 session ls ==== Add OSDs ==== * Prepare the OSD ceph-deploy osd prepare : * Activate the OSD ceph-deploy osd activate : ==== CephFS commands ==== * List all connected fs client / mounted directories ceph tell mds.* session ls ==== Remove OSDs ==== * Change crush weight for rebalancing ceph osd crush reweight osd. 0.0 * Weight for rebalance to complete! * Completely remove the OSD #take the osd out of the cluster ceph osd out #stop the osd daemon for that drive on the host it is running sudo systemctl stop ceph-osd@ #remove osd from crush map ceph osd crush remove osd. #remove authentication key ceph auth del osd. #remove the osd ceph osd rm (optional) #delete partition table on node sudo umount /dev/ sudo wipefs -a /dev/ * Disable rebalancing ceph osd set nodown ceph osd set noout * Also stop CephFS and MDs Services sudo umount -lf /mnt/cephfs sudo service ceph stop mds * Disable scrubing and deep-scrubing ceph osd set noscrub ceph osd set nodeep-srub * Reenable all the servcices sudo service ceph start osd. sudo service ceph start mds sudo mount /mnt/cephfs ceph osd unset noscrub ceph osd unset nodeep-scrub ceph osd unset noout ceph osd unset nodown * Get Crush Map ceph osd getcrushmap -o crush-com * Decompile Crush Map crushtool -d crush-com -o crush-dec * Edit Map with text editing tool vim crush-dec * Compile new Crush Map crushtool -c crush-dec -o crush-com * Set new Map for usage in cluster ceph osd setcrushmap -i crush-com ---- The actual Map consists of four components:\\ - ''Devices'': One device for each OSD daemon/disk\\ - ''Bucket Type'': Define the buckets used in Crush hierarchy\\ - ''Bucket Instance'': Define the buckets to each other, sets the devices in failure domains\\ - ''Rules'': Rules determine data placement for pools\\ More Information on the Crush Map can be found [[http://docs.ceph.com/docs/master/rados/operations/crush-map/#crush-map-parameters|here]] ===CLI Crush Map Configuration=== * Add/Move an OSD inside Crush Map ceph osd crush set root= ... ceph osd crush set 0 9.01598 root=default host=sto01-1 * Remove OSD from Crush Map ceph osd crush remove osd. * Add bucket ceph osd crush add-bucket ceph osd crush add-bucket sto01-1 host * Move bucket ceph osd crush move ceph osd crush move sto01-1 root=ssd room=room1 * Delete bucket ceph osd crush remove * Set the replication size of the pool ceph osd pool set size * Get the replication size ceph osd dump | grep 'replicated size' * Create a new pool ceph osd pool create replicated * List all Pools ceph osd pool ls * Delete a pool ceph osd pool delete --yes-i-really-really-mean-it * Rename a pool ceph osd pool rename * List all ceph services/units on a node sudo systemctl status ceph\*.service ceph\*.target * Restart all osd services sudo systemctl stop ceph.target sudo systemctl start ceph.target * Controlling daemons by type sudo systemctl stop ceph-mon\*.service ceph-mon.target sudo systemctl stop ceph-osd\*.service ceph-osd.target sudo systemctl stop ceph-mds\*.service ceph-mds.target #start services sudo systemctl start ceph-osd.target sudo systemctl start ceph-mon.target sudo systemctl start ceph-mds.target * Start specific daemons sudo systemctl start ceph-osd@{id} sudo systemctl start ceph-mon@{hostname} sudo systemctl start ceph-mds@{hostname} * Create new Block Device rbd create --size 4096 / sudo rbd feature disable / exclusive-lock object-map fast-diff deep-flatten sudo rbd map / --name client.admin sudo mkfs.ext4 -m0 /dev/rbd1 sudo mount /dev/rbd1 /mnt * List all block devices rbd ls #list in specific pool rbd ls * Retrieve image information rbd info / * Resize Block Device rbd resize --size 2048 / # increase rbd resize --size 2048 / --allow-shrink #decrease * Remove RBD sudo umount /mnt sudo rbd unmap /dev/rbd1 rbd rm / #if still watchers look where rbd showmapped sudo service rbdmap stop rbd rm / #or rbd info / rados -p rbd listwatchers rbd_header. * Get all users and cluster rights / authorization ceph auth list * Log in on a ceph admin node ceph auth add client.fs_user mon 'allow r' osd 'allow rwx pool=fs, allow rwx pool=fs_meta' mds 'allow r' * Get user key ceph auth get-key client.fs_user | tee client.fs_user.key * Get user keyring ceph auth get client.fs_user -o ceph.client.fs_user.keyring * Copy the key and keyring to the node * Install ceph on node ceph-deploy install 10.0.60.4 ceph-deploy config push 10.0.60.4 * Now log in to the node. Make sure that the md servers and all storage servers are reachable by ip and hostname! (Add maas-10.0.10.1 as DNS server) * Move secret and key to ceph directory sudo mv ~/ceph.client.fs_user.keyring /etc/ceph/ sudo mv ~/client.fs_user.key /etc/ceph/ * Check Authentication ceph --id=fs_user health * Install Ceph Filesystem package sudo apt install ceph-fs-common * Mount Filesystem sudo mount -t ceph 10.0.10.1:6789:/ /mnt/cephfs -o name=fs_user,secretfile=/etc/ceph/client.fs_user.key ''Change user permission'' * Change permissions ceph auth caps client.fs_user mds ' allow rw' mon 'allow r' osd 'allow rwx pool=fs' ''Path restriction'' * Path restriction works only when mounted with ceph-fuse * MDS: allow read + write on path mds 'allow r' -> allow read only for whole fs mds 'allow rw' -> allow read and write for whole fs mds 'allow r, allo rw path=/data_tonetto' -> allow read for whole fs and only write for path data_tonetto mds 'allow rw path=/datasets' -> allow read and write only to the path datasets * MON: only read necessary mon ' allow r' -> get object map, access to cluster * OSD: allow read and write to fs pool osd 'allow rwx pool=fs' -> allow read and write to fs pool * All command for path restriction ceph auth caps client.fs_user mds 'allow rw path=/datasets, allow rw path=/data_tonetto' mon 'allow r' osd 'allow rw pool=fs' #on client #install fuse sudo apt install ceph-fuse #mount directories sudo ceph-fuse -n client.fs_user --keyring=/etc/ceph/ceph.client.fs_user.keyring -r /data_tonetto /data sudo ceph-fuse -n client.fs_user --keyring=/etc/ceph/ceph.client.fs_user.keyring -r /datasets /datasets ===== Installation ===== The installation will be made from one server with ceph-deploy.\\ Following this guide: [[http://docs.ceph.com/docs/master/rados/deployment/]] * Initial installation of ceph-admin and nodes ssh mon01-cm mkdir sto_cluster cd sto_cluster wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add - echo deb https://download.ceph.com/debian-kraken/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list sudo apt-get update && sudo apt-get install ceph-deploy ceph-deploy new mon01-cm vim ceph.conf ------------------------------------------------------------ [global] fsid = b2fe6c5c-10d5-4eb3-af02-121d6493d6bf mon_initial_members = mon01 mon_host = 10.0.10.1 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 10.0.0.0/16 osd_journal_size = 14336 #reasonable number of replicas and placement groups osd_pool_default_size = 3 #Write an object 3 times osd_pool_default_min_size = 1 #Allow writing 1 copy in degraded state osd_pool_default_pg_num = 256 osd_pool_default_pgp_num = 256 ------------------------------------------------------------ #install python on nodes sto01, sto02 sudo apt install python python-apt # enable nat on gateway ceph-deploy install mon01-cm sto01 sto02 ceph-deploy disk list sto01 ceph-deploy mon create-initial #sometimes two monitors are running mon01-cm + mon01 #stop mon01: sudo systemctl stop ceph-mon@mon01 #run create-initial command again sudo chmod +r /etc/ceph/ceph.client.admin.keyring ceph-deploy admin sto01 sto02 #run command on both nodes: sudo chmod +r /etc/ceph/ceph.client.admin.keyring ceph status * Add OSDs ceph-deploy osd prepare sto01:/dev/sdb sto01:/dev/sdc sto01:/dev/sdd sto01:/dev/sde sto01:/dev/sdf sto01:/dev/sdg sto01:/dev/sdh sto01:/dev/sdi sto01:/dev/sdj sto01:/dev/sdk sto01:/dev/sdl sto01:/dev/sdo sto01:/dev/sdp sto01:/dev/sdq sto01:/dev/sdr sto02:/dev/sdb sto02:/dev/sdc sto02:/dev/sdd sto02:/dev/sde sto02:/dev/sdf sto02:/dev/sdg sto02:/dev/sdh sto02:/dev/sdi sto02:/dev/sdj sto02:/dev/sdk sto02:/dev/sdl sto02:/dev/sdo sto02:/dev/sdp sto02:/dev/sdq sto02:/dev/sdr ceph-deploy osd activate sto01:/dev/sdb1 sto01:/dev/sdc1 sto01:/dev/sdd1 sto01:/dev/sde1 sto01:/dev/sdf1 sto01:/dev/sdg1 sto01:/dev/sdh1 sto01:/dev/sdi1 sto01:/dev/sdj1 sto01:/dev/sdk1 sto01:/dev/sdl1 sto01:/dev/sdo1 sto01:/dev/sdp1 sto01:/dev/sdq1 sto01:/dev/sdr1 sto02:/dev/sdb1 sto02:/dev/sdc1 sto02:/dev/sdd1 sto02:/dev/sde1 sto02:/dev/sdf1 sto02:/dev/sdg1 sto02:/dev/sdh1 sto02:/dev/sdi1 sto02:/dev/sdj1 sto02:/dev/sdk1 sto02:/dev/sdl1 sto02:/dev/sdo1 sto02:/dev/sdp1 sto02:/dev/sdq1 sto02:/dev/sdr1 #cluster to reach health ok ceph osd pool rbd set size 2 * Create Erasure Coded pool A profile must be set when creating a new pool. The profile can not be changed later! A new pool has to be created and all the data moved from the first pool to the second. #create custom profile ceph osd erasure-code-profile get default ceph osd erasure-code-profile set fs1 k=8 m=4 ruleset-failure-domain=osd #optional take another pool ceph osd erasure-code-profile set ruleset-root=ssd #create pool ceph osd pool create ec 12 12 erasure fs1 * Create and Mount CephFS #create FS pools - recommended to use higher replication level for metadata pool #any data loss can render whole filesystem inaccessible ceph osd pool create fs 256 256 replicated ceph osd pool create fs_meta 256 256 replicated ceph osd pool set fs size 2 ceph osd pool set fs_meta size 3 #create filesystem ceph fs new ceph_fs fs_meta fs #mount filesystem on client ceph-deploy install emu11 #on client: sudo apt install ceph-fs-common ceph-fuse cat /etc/ceph/ceph.client.admin.keyring #copy only key into new file /etc/ceph/admin.secret sudo mount -t ceph 10.0.10.1:6789:/ /mnt/cephfs -o name=admin,secretfile=/etc/ceph/admin.secret #unmount ceph-fuse / ceph kernel client sudo fusermount -u /mnt/cephfs sudo umount /mnt/cephfs * Benchmark cluster #network benchmark sudo apt install iperf sto01: iperf -s emu12: iperf -c sto01 #disk benchmark sudo hdparm -tT /dev/sdc sudo hdparm -tT --direct /dev/sdc sudo mount /dev/sdc /mnt/tmp cd /mnt/tmp sudo dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc sudo dd if=/dev/zero of=tempfile2 bs=1G count=5 conv=fdatasync,notrunc sudo dd if=tempfile of=/dev/null bs=1M count=1024 sudo dd if=tempfile2 of=/dev/null bs=1G count=5 #bench rados cluster #normal cluster rados bench -p rbd 60 write --no-cleanup #read random for 60 seconds rados bench -p rbd 60 rand #Cleanup Mess on every pool rados -p cleanup #network benchmark sudo apt install iperf sto01: iperf -s emu12: iperf -c sto01 9.40 Gbits/sec - sto01 -> sto02 9.41 Gbits/sec - sto01 -> sto02 9.39 Gbits/sec - emu12 -> sto02 9.35 Gbits/sec - emu12 -> sto01 #disk benchmark sudo hdparm -tT /dev/sdc Timing cached reads: 20318 MB in 2.00 seconds = 10168.52 MB/sec Timing buffered disk reads: 700 MB in 3.01 seconds = 232.81 MB/sec sudo hdparm -tT --direct /dev/sdc Timing O_DIRECT cached reads: 886 MB in 2.00 seconds = 442.57 MB/sec Timing O_DIRECT disk reads: 358 MB in 3.00 seconds = 119.31 MB/sec sudo mkdir /mnt/tmp sudo mount /dev/sdc /mnt/tmp cd /mnt/tmp sudo dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.10234 s, 210 MB/s sudo dd if=/dev/zero of=tempfile2 bs=1G count=5 conv=fdatasync,notrunc 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 25.7669 s, 208 MB/s sudo dd if=tempfile of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.204041 s, 5.9 GB/s sudo dd if=tempfile2 of=/dev/null bs=1G count=5 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.280824 s, 4.8 GB/s #disk benchmark - ssd sudo hdparm -tT /dev/sdn4 Timing cached reads: 20768 MB in 2.00 seconds = 10394.23 MB/sec Timing buffered disk reads: 1458 MB in 3.00 seconds = 485.62 MB/sec sudo hdparm -tT --direct /dev/sdn4 Timing O_DIRECT cached reads: 646 MB in 2.00 seconds = 322.43 MB/sec Timing O_DIRECT disk reads: 1494 MB in 3.00 seconds = 497.91 MB/sec sudo mkfs.xfs -f -i size=2048 /dev/sdn4 sudo dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.14751 s, 135 MB/s sudo dd if=/dev/zero of=tempfile2 bs=1G count=5 conv=fdatasync,notrunc 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 40.4716 s, 133 MB/s sudo dd if=tempfile of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.204041 s, 5.8 GB/s sudo dd if=tempfile2 of=/dev/null bs=1G count=5 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.280824 s, 5.2 GB/s #bench rados cluster #normal cluster rados bench -p rbd 60 write --no-cleanup 2017-03-30 16:58:01.468876 min lat: 0.0414839 max lat: 1.96491 avg lat: 0.378728 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 2529 2513 167.516 96 0.265697 0.378728 Total time run: 60.665624 Total writes made: 2530 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 166.816 Stddev Bandwidth: 57.3734 Max bandwidth (MB/sec): 336 Min bandwidth (MB/sec): 84 Average IOPS: 41 Stddev IOPS: 14 Max IOPS: 84 Min IOPS: 21 Average Latency(s): 0.383575 Stddev Latency(s): 0.259392 Max latency(s): 1.96491 Min latency(s): 0.0414839 # two concurrent on both nodes Total time run: 30.422575 Total writes made: 744 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 97.8221 Stddev Bandwidth: 73.2887 Max bandwidth (MB/sec): 352 Min bandwidth (MB/sec): 16 Average IOPS: 24 Stddev IOPS: 18 Max IOPS: 88 Min IOPS: 4 Average Latency(s): 0.653725 Stddev Latency(s): 0.609964 Max latency(s): 3.22897 Min latency(s): 0.0601762 #read random for 60 seconds rados bench -p rbd 60 rand Total time run: 60.054755 Total reads made: 39269 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 2615.55 Average IOPS: 653 Stddev IOPS: 26 Max IOPS: 725 Min IOPS: 604 Average Latency(s): 0.0237641 Max latency(s): 0.179591 Min latency(s): 0.00335397 #ssd journal cluster rados bench -p data-ssd 60 write --no-cleanup Total time run: 60.250558 Total writes made: 2516 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 167.036 Stddev Bandwidth: 11.7987 Max bandwidth (MB/sec): 196 Min bandwidth (MB/sec): 144 Average IOPS: 41 Stddev IOPS: 3 Max IOPS: 49 Min IOPS: 36 Average Latency(s): 0.383063 Stddev Latency(s): 0.126893 Max latency(s): 1.07908 Min latency(s): 0.0446201 # two concurrent on both nodes Total time run: 30.744405 Total writes made: 682 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 88.7316 Stddev Bandwidth: 14.6719 Max bandwidth (MB/sec): 140 Min bandwidth (MB/sec): 60 Average IOPS: 22 Stddev IOPS: 3 Max IOPS: 35 Min IOPS: 15 Average Latency(s): 0.713941 Stddev Latency(s): 0.306251 Max latency(s): 1.82924 Min latency(s): 0.0950345 #reads Total time run: 60.049589 Total reads made: 37142 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 2474.09 Average IOPS: 618 Stddev IOPS: 24 Max IOPS: 669 Min IOPS: 561 Average Latency(s): 0.0251675 Max latency(s): 0.208302 Min latency(s): 0.00336694 #--------------------------------------- #rados Block Device sudo rbd create image02 --size 4096 --pool data-ssd sudo rbd feature disable image02 exclusive-lock object-map fast-diff deep-flatten --pool data-ssd sudo rbd map image02 --pool data-ssd --name client.admin sudo mkfs.ext4 -m0 /dev/rbd1 sudo mount /dev/rbd1 /mnt/device-bl2 rbd bench-write image02 --pool data-ssd rbd - pool elapsed: 12 ops: 262144 ops/sec: 21747.94 bytes/sec: 89079547.35 data-ssd - pool elapsed: 11 ops: 262144 ops/sec: 23236.06 bytes/sec: 95174905.63 * RBD Normal Benchmarks sudo hdparm -tT /dev/rbd0 # normal Timing cached reads: 20254 MB in 2.00 seconds = 10139.08 MB/sec Timing buffered disk reads: 2722 MB in 3.00 seconds = 907.20 MB/sec sudo hdparm --direct -tT /dev/rbd0 Timing O_DIRECT cached reads: 3720 MB in 2.00 seconds = 1860.49 MB/sec Timing O_DIRECT disk reads: 4096 MB in 2.96 seconds = 1384.69 MB/sec sudo hdparm -tT /dev/rbd1 #ssd Timing cached reads: 20072 MB in 2.00 seconds = 10047.58 MB/sec Timing buffered disk reads: 2766 MB in 3.00 seconds = 921.83 MB/sec sudo hdparm --direct -tT /dev/rbd1 Timing O_DIRECT cached reads: 3872 MB in 2.00 seconds = 1936.37 MB/sec Timing O_DIRECT disk reads: 4096 MB in 1.77 seconds = 2311.77 MB/sec #normal sudo dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.1852 s, 257 MB/s sudo dd if=tempfile of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.184213 s, 5.8 GB/s #ssd sudo dd if=/dev/zero of=tempfile2 bs=1M count=1024 conv=fdatasync,notrunc 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.91692 s, 155 MB/s sudo dd if=tempfile of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.184213 s, 5.8 GB/s Raw Network Speed iperf -s iperf -c
9.40 Gbits/sec - sto01 -> sto02 9.41 Gbits/sec - sto01 -> sto02 9.39 Gbits/sec - emu12 -> sto02 9.35 Gbits/sec - emu12 -> sto01 -> ca. 1170 MB/sec Raw Speed sudo hdparm -tT /dev/ ssd Timing cached reads: 10220.44 MB/sec Timing cached reads: 10014.83 MB/sec Timing cached reads: 10406.89 MB/sec Timing cached reads: 10233.53 MB/sec Timing buffered reads: 321.97 MB/sec Timing buffered reads: 372.85 MB/sec Timing buffered reads: 324.69 MB/sec Timing buffered reads: 371.13 MB/sec normal Timing cached reads: 9862.59 MB/sec 10251.11 MB/sec 10005.25 MB/sec 10581.66 MB/sec Timing buffered reads: 237.98 MB/sec 240.19 MB/sec 233.71 MB/sec 208.83 MB/sec ssd-DIRECT Timing cached reads: 345.82 MB/sec 372.97 MB/sec 377.45 MB/sec 370.21 MB/sec Timing buffered reads: 404.91 MB/sec 436.78 MB/sec 436.65 MB/sec 411.83 MB/sec normal-DIRECT Timing cached reads: 879.61 MB/sec 853.61 MB/sec 437.60 MB/sec 867.01 MB/sec Timing buffered reads: 119.51 MB/sec 121.01 MB/sec 175.96 MB/sec 112.51 MB/sec Rados Bench rados bench -p 10/20/30/60 write --no-cleanup ssd-primary Bandwidth (MB/sec): 246.749 Bandwidth (MB/sec): 216.592 Bandwidth (MB/sec): 232.526 Bandwidth (MB/sec): 186.368 ssd-only Bandwidth (MB/sec): 118.422 Bandwidth (MB/sec): 121.656 Bandwidth (MB/sec): 116.717 Bandwidth (MB/sec): 108.17 rbd-normal Bandwidth (MB/sec): 260.393 Bandwidth (MB/sec): 282.246 Bandwidth (MB/sec): 268.01 Bandwidth (MB/sec): 271.671 erasure-coded Bandwidth (MB/sec): 244.602 Bandwidth (MB/sec): 231.217 Bandwidth (MB/sec): 242.125 Bandwidth (MB/sec): 229.342 Rados Read Seq rados bench -p 10/20/30 seq ssd-primary Bandwidth (MB/sec): 975.046 Bandwidth (MB/sec): 1025.61 Bandwidth (MB/sec): 1030.69 ssd-only Bandwidth (MB/sec): 1071.45 Bandwidth (MB/sec): 1065.46 Bandwidth (MB/sec): 1045.66 rbd-normal Bandwidth (MB/sec): 1009.87 Bandwidth (MB/sec): 1022.06 Bandwidth (MB/sec): 1054.07 erasure-coded Bandwidth (MB/sec): 962.095 Bandwidth (MB/sec): 956.349 Bandwidth (MB/sec): 974.866 Rados Read Rand rados bench -p 10/20/30 rand ssd-primary Bandwidth (MB/sec): 993.921 Bandwidth (MB/sec): 1034.7 Bandwidth (MB/sec): 1062.02 ssd-only Bandwidth (MB/sec): 1055.95 Bandwidth (MB/sec): 1039.78 Bandwidth (MB/sec): 1063.78 rbd-normal Bandwidth (MB/sec): 1037.47 Bandwidth (MB/sec): 1057.92 Bandwidth (MB/sec): 1049.02 erasure-coded Bandwidth (MB/sec): 964.135 Bandwidth (MB/sec): 952.147 Bandwidth (MB/sec): 950.543 --> Network bottleneck: 1170 MB/sec --> Single Hard Drive Read: 230 MB/sec normal 130 MB/sec direct --> Single SSD Read: 330 MB/sec normal 430 MB/sec direct --> All Cluster Read Operations are capped by network: 1050 MB/sec --> Erasure Coded Pool slightly less 955 MB/sec --> Single Hard Drive Write: 210 MB/sec --> Single SSD Write: 130 MB/sec --> Cluster Write Operations: 265 MB/sec normal 120 MB/sec ssd 235 MB/sec erasure-coded