Simplified distributed block storage with strong consistency, like in Ceph (repository mirror)

Overview

Vitastor

Читать на русском

The Idea

Make Software-Defined Block Storage Great Again.

Vitastor is a small, simple and fast clustered block storage (storage for VM drives), architecturally similar to Ceph which means strong consistency, primary-replication, symmetric clustering and automatic data distribution over any number of drives of any size with configurable redundancy (replication or erasure codes/XOR).

Features

Vitastor is currently a pre-release, a lot of features are missing and you can still expect breaking changes in the future. However, the following is implemented:

  • Basic part: highly-available block storage with symmetric clustering and no SPOF
  • Performance ;-D
  • Multiple redundancy schemes: Replication, XOR n+1, Reed-Solomon erasure codes based on jerasure library with any number of data and parity drives in a group
  • Configuration via simple JSON data structures in etcd
  • Automatic data distribution over OSDs, with support for:
    • Mathematical optimization for better uniformity and less data movement
    • Multiple pools
    • Placement tree, OSD selection by tags (device classes) and placement root
    • Configurable failure domains
  • Recovery of degraded blocks
  • Rebalancing (data movement between OSDs)
  • Lazy fsync support
  • I/O statistics reporting to etcd
  • Generic user-space client library
  • QEMU driver (built out-of-tree)
  • Loadable fio engine for benchmarks (also built out-of-tree)
  • NBD proxy for kernel mounts
  • Inode removal tool (vitastor-rm)
  • Packaging for Debian and CentOS
  • Per-inode I/O and space usage statistics
  • Inode metadata storage in etcd
  • Snapshots and copy-on-write image clones

Roadmap

  • Better OSD creation and auto-start tools
  • Other administrative tools
  • Plugins for OpenStack, Kubernetes, OpenNebula, Proxmox and other cloud systems
  • iSCSI proxy
  • Operation timeouts and better failure detection
  • Scrubbing without checksums (verification of replicas)
  • Checksums
  • SSD+HDD optimizations, possibly including tiered storage and soft journal flushes
  • RDMA and NVDIMM support
  • Web GUI
  • Compression (possibly)
  • Read caching using system page cache (possibly)

Architecture

Similarities:

  • Just like Ceph, Vitastor has Pools, PGs, OSDs, Monitors, Failure Domains, Placement Tree.
  • Just like Ceph, Vitastor is transactional (even though there's a "lazy fsync mode" which doesn't implicitly flush every operation to disks).
  • OSDs also have journal and metadata and they can also be put on separate drives.
  • Just like in Ceph, client library attempts to recover from any cluster failure so you can basically reboot the whole cluster and only pause, but not crash, your clients (I consider this a bug if the client crashes in that case).

Some basic terms for people not familiar with Ceph:

  • OSD (Object Storage Daemon) is a process that stores data and serves read/write requests.
  • PG (Placement Group) is a container for data that (normally) shares the same replicas.
  • Pool is a container for data that has the same redundancy scheme and placement rules.
  • Monitor is a separate daemon that watches cluster state and handles failures.
  • Failure Domain is a group of OSDs that you allow to fail. It's "host" by default.
  • Placement Tree groups OSDs in a hierarchy to later split them into Failure Domains.

Architectural differences from Ceph:

  • Vitastor's primary focus is on SSDs. Proper SSD+HDD optimizations may be added in the future, though.
  • Vitastor OSD is (and will always be) single-threaded. If you want to dedicate more than 1 core per drive you should run multiple OSDs each on a different partition of the drive. Vitastor isn't CPU-hungry though (as opposed to Ceph), so 1 core is sufficient in a lot of cases.
  • Metadata and journal are always kept in memory. Metadata size depends linearly on drive capacity and data store block size which is 128 KB by default. With 128 KB blocks metadata should occupy around 512 MB per 1 TB (which is still less than Ceph wants). Journal doesn't have to be big, the example test below was conducted with only 16 MB journal. A big journal is probably even harmful as dirty write metadata also take some memory.
  • Vitastor storage layer doesn't have internal copy-on-write or redirect-write. I know that maybe it's possible to create a good copy-on-write storage, but it's much harder and makes performance less deterministic, so CoW isn't used in Vitastor.
  • The basic layer of Vitastor is block storage with fixed-size blocks, not object storage with rich semantics like in Ceph (RADOS).
  • There's a "lazy fsync" mode which allows to batch writes before flushing them to the disk. This allows to use Vitastor with desktop SSDs, but still lowers performance due to additional network roundtrips, so use server SSDs with capacitor-based power loss protection ("Advanced Power Loss Protection") for best performance.
  • PGs are ephemeral. This means that they aren't stored on data disks and only exist in memory while OSDs are running.
  • Recovery process is per-object (per-block), not per-PG. Also there are no PGLOGs.
  • Monitors don't store data. Cluster configuration and state is stored in etcd in simple human-readable JSON structures. Monitors only watch cluster state and handle data movement. Thus Vitastor's Monitor isn't a critical component of the system and is more similar to Ceph's Manager. Vitastor's Monitor is implemented in node.js.
  • PG distribution isn't based on consistent hashes. All PG mappings are stored in etcd. Rebalancing PGs between OSDs is done by mathematical optimization - data distribution problem is reduced to a linear programming problem and solved by lp_solve. This allows for almost perfect (96-99% uniformity compared to Ceph's 80-90%) data distribution in most cases, ability to map PGs by hand without breaking rebalancing logic, reduced OSD peer-to-peer communication (on average, OSDs have fewer peers) and less data movement. It also probably has a drawback - this method may fail in very large clusters, but up to several hundreds of OSDs it's perfectly fine. It's also easy to add consistent hashes in the future if something proves their necessity.
  • There's no separate CRUSH layer. You select pool redundancy scheme, placement root, failure domain and so on directly in pool configuration.

Understanding Storage Performance

The most important thing for fast storage is latency, not parallel iops.

The best possible latency is achieved with one thread and queue depth of 1 which basically means "client load as low as possible". In this case IOPS = 1/latency, and this number doesn't scale with number of servers, drives, server processes or threads and so on. Single-threaded IOPS and latency numbers only depend on how fast a single daemon is.

Why is it important? It's important because some of the applications can't use queue depth greater than 1 because their task isn't parallelizable. A notable example is any ACID DBMS because all of them write their WALs sequentially with fsync()s.

fsync, by the way, is another important thing often missing in benchmarks. The point is that drives have cache buffers and don't guarantee that your data is actually persisted until you call fsync() which is translated to a FLUSH CACHE command by the OS.

Desktop SSDs are very fast without fsync - NVMes, for example, can process ~80000 write operations per second with queue depth of 1 without fsync - but they're really slow with fsync because they have to actually write data to flash chips when you call fsync. Typical number is around 1000-2000 iops with fsync.

Server SSDs often have supercapacitors that act as a built-in UPS and allow the drive to flush its DRAM cache to the persistent flash storage when a power loss occurs. This makes them perform equally well with and without fsync. This feature is called "Advanced Power Loss Protection" by Intel; other vendors either call it similarly or directly as "Full Capacitor-Based Power Loss Protection".

All software-defined storages that I currently know are slow in terms of latency. Notable examples are Ceph and internal SDSes used by cloud providers like Amazon, Google, Yandex and so on. They're all slow and can only reach ~0.3ms read and ~0.6ms 4 KB write latency with best-in-slot hardware.

And that's in the SSD era when you can buy an SSD that has ~0.04ms latency for 100 $.

I use the following 6 commands with small variations to benchmark any storage:

  • Linear write: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/sdX
  • Linear read: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=read -runtime=60 -filename=/dev/sdX
  • Random write latency (T1Q1, this hurts storages the most): fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/sdX
  • Random read latency (T1Q1): fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -rw=randread -runtime=60 -filename=/dev/sdX
  • Parallel write iops (use numjobs if a single CPU core is insufficient to saturate the load): fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=128 [-numjobs=4 -group_reporting] -rw=randwrite -runtime=60 -filename=/dev/sdX
  • Parallel read iops (use numjobs if a single CPU core is insufficient to saturate the load): fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=128 [-numjobs=4 -group_reporting] -rw=randread -runtime=60 -filename=/dev/sdX

Vitastor's Theoretical Maximum Random Access Performance

Replicated setups:

  • Single-threaded (T1Q1) read latency: 1 network roundtrip + 1 disk read.
  • Single-threaded write+fsync latency:
    • With immediate commit: 2 network roundtrips + 1 disk write.
    • With lazy commit: 4 network roundtrips + 1 disk write + 1 disk flush.
  • Saturated parallel read iops: min(network bandwidth, sum(disk read iops)).
  • Saturated parallel write iops: min(network bandwidth, sum(disk write iops / number of replicas / write amplification)).

EC/XOR setups:

  • Single-threaded (T1Q1) read latency: 1.5 network roundtrips + 1 disk read.
  • Single-threaded write+fsync latency:
    • With immediate commit: 3.5 network roundtrips + 1 disk read + 2 disk writes.
    • With lazy commit: 5.5 network roundtrips + 1 disk read + 2 disk writes + 2 disk fsyncs.
    • 0.5 in actually (k-1)/k which means that an additional roundtrip doesn't happen when the read sub-operation can be served locally.
  • Saturated parallel read iops: min(network bandwidth, sum(disk read iops)).
  • Saturated parallel write iops: min(network bandwidth, sum(disk write iops * number of data drives / (number of data + parity drives) / write amplification)). In fact, you should put disk write iops under the condition of ~10% reads / ~90% writes in this formula.

Write amplification for 4 KB blocks is usually 3-5 in Vitastor:

  1. Journal block write
  2. Journal data write
  3. Metadata block write
  4. Another journal block write for EC/XOR setups
  5. Data block write

If you manage to get an SSD which handles 512 byte blocks well (Optane?) you may lower 1, 3 and 4 to 512 bytes (1/8 of data size) and get WA as low as 2.375.

Lazy fsync also reduces WA for parallel workloads because journal blocks are only written when they fill up or fsync is requested.

Example Comparison with Ceph

Hardware configuration: 4 nodes, each with:

  • 6x SATA SSD Intel D3-4510 3.84 TB
  • 2x Xeon Gold 6242 (16 cores @ 2.8 GHz)
  • 384 GB RAM
  • 1x 25 GbE network interface (Mellanox ConnectX-4 LX), connected to a Juniper QFX5200 switch

CPU powersaving was disabled. Both Vitastor and Ceph were configured with 2 OSDs per 1 SSD.

All of the results below apply to 4 KB blocks and random access (unless indicated otherwise).

Raw drive performance:

  • T1Q1 write ~27000 iops (~0.037ms latency)
  • T1Q1 read ~9800 iops (~0.101ms latency)
  • T1Q32 write ~60000 iops
  • T1Q32 read ~81700 iops

Ceph 15.2.4 (Bluestore):

  • T1Q1 write ~1000 iops (~1ms latency)
  • T1Q1 read ~1750 iops (~0.57ms latency)
  • T8Q64 write ~100000 iops, total CPU usage by OSDs about 40 virtual cores on each node
  • T8Q64 read ~480000 iops, total CPU usage by OSDs about 40 virtual cores on each node

T8Q64 tests were conducted over 8 400GB RBD images from all hosts (every host was running 2 instances of fio). This is because Ceph has performance penalties related to running multiple clients over a single RBD image.

cephx_sign_messages was set to false during tests, RocksDB and Bluestore settings were left at defaults.

In fact, not that bad for Ceph. These servers are an example of well-balanced Ceph nodes. However, CPU usage and I/O latency were through the roof, as usual.

Vitastor:

  • T1Q1 write: 7087 iops (0.14ms latency)
  • T1Q1 read: 6838 iops (0.145ms latency)
  • T2Q64 write: 162000 iops, total CPU usage by OSDs about 3 virtual cores on each node
  • T8Q64 read: 895000 iops, total CPU usage by OSDs about 4 virtual cores on each node
  • Linear write (4M T1Q32): 2800 MB/s
  • Linear read (4M T1Q32): 1500 MB/s

T8Q64 read test was conducted over 1 larger inode (3.2T) from all hosts (every host was running 2 instances of fio). Vitastor has no performance penalties related to running multiple clients over a single inode. If conducted from one node with all primary OSDs moved to other nodes the result was slightly lower (689000 iops), this is because all operations resulted in network roundtrips between the client and the primary OSD. When fio was colocated with OSDs (like in Ceph benchmarks above), 1/4 of the read workload actually used the loopback network.

Vitastor was configured with: --disable_data_fsync true --immediate_commit all --flusher_count 8 --disk_alignment 4096 --journal_block_size 4096 --meta_block_size 4096 --journal_no_same_sector_overwrites true --journal_sector_buffer_count 1024 --journal_size 16777216.

EC/XOR 2+1

Vitastor:

  • T1Q1 write: 2808 iops (~0.355ms latency)
  • T1Q1 read: 6190 iops (~0.16ms latency)
  • T2Q64 write: 85500 iops, total CPU usage by OSDs about 3.4 virtual cores on each node
  • T8Q64 read: 812000 iops, total CPU usage by OSDs about 4.7 virtual cores on each node
  • Linear write (4M T1Q32): 3200 MB/s
  • Linear read (4M T1Q32): 1800 MB/s

Ceph:

  • T1Q1 write: 730 iops (~1.37ms latency)
  • T1Q1 read: 1500 iops with cold cache (~0.66ms latency), 2300 iops after 2 minute metadata cache warmup (~0.435ms latency)
  • T4Q128 write (4 RBD images): 45300 iops, total CPU usage by OSDs about 30 virtual cores on each node
  • T8Q64 read (4 RBD images): 278600 iops, total CPU usage by OSDs about 40 virtual cores on each node
  • Linear write (4M T1Q32): 1950 MB/s before preallocation, 2500 MB/s after preallocation
  • Linear read (4M T1Q32): 2400 MB/s

NBD

NBD is currently required to mount Vitastor via kernel, but it imposes additional overhead due to additional copying between the kernel and userspace. This mostly hurts linear bandwidth, not iops.

Vitastor with single-thread NBD on the same hardware:

  • T1Q1 write: 6000 iops (0.166ms latency)
  • T1Q1 read: 5518 iops (0.18ms latency)
  • T1Q128 write: 94400 iops
  • T1Q128 read: 103000 iops
  • Linear write (4M T1Q128): 1266 MB/s (compared to 2800 MB/s via fio)
  • Linear read (4M T1Q128): 975 MB/s (compared to 1500 MB/s via fio)

Installation

Debian

  • Trust Vitastor package signing key: wget -q -O - https://vitastor.io/debian/pubkey | sudo apt-key add -
  • Add Vitastor package repository to your /etc/apt/sources.list:
    • Debian 11 (Bullseye/Sid): deb https://vitastor.io/debian bullseye main
    • Debian 10 (Buster): deb https://vitastor.io/debian buster main
  • For Debian 10 (Buster) also enable backports repository: deb http://deb.debian.org/debian buster-backports main
  • Install packages: apt update; apt install vitastor lp-solve etcd linux-image-amd64 qemu

CentOS

  • Add Vitastor package repository:
    • CentOS 7: yum install https://vitastor.io/rpms/centos/7/vitastor-release-1.0-1.el7.noarch.rpm
    • CentOS 8: dnf install https://vitastor.io/rpms/centos/8/vitastor-release-1.0-1.el8.noarch.rpm
  • Enable EPEL: yum/dnf install epel-release
  • Enable additional CentOS repositories:
    • CentOS 7: yum install centos-release-scl
    • CentOS 8: dnf install centos-release-advanced-virtualization
  • Enable elrepo-kernel:
    • CentOS 7: yum install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
    • CentOS 8: dnf install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
  • Install packages: yum/dnf install vitastor lpsolve etcd kernel-ml qemu-kvm

Building from Source

  • Install Linux kernel 5.4 or newer, for io_uring support. 5.8 or later is highly recommended because there is at least one known io_uring hang with 5.4 and an HP SmartArray controller.
  • Install liburing 0.4 or newer and its headers.
  • Install lp_solve.
  • Install etcd, at least version 3.4.15. Earlier versions won't work because of various bugs, for example #12402. You can also take 3.4.13 with this specific fix from here: https://github.com/vitalif/etcd/, branch release-3.4.
  • Install node.js 10 or newer.
  • Install gcc and g++ 8.x or newer.
  • Clone https://yourcmc.ru/git/vitalif/vitastor/ with submodules.
  • Install QEMU 3.0+, get its source, begin to build it, stop the build and copy headers:
    • <qemu>/include<vitastor>/qemu/include
    • Debian:
      • Use qemu packages from the main repository
      • <qemu>/b/qemu/config-host.h<vitastor>/qemu/b/qemu/config-host.h
      • <qemu>/b/qemu/qapi<vitastor>/qemu/b/qemu/qapi
    • CentOS 8:
      • Use qemu packages from the Advanced-Virtualization repository. To enable it, run yum install centos-release-advanced-virtualization.noarch and then yum install qemu
      • <qemu>/config-host.h<vitastor>/qemu/b/qemu/config-host.h
      • For QEMU 3.0+: <qemu>/qapi<vitastor>/qemu/b/qemu/qapi
      • For QEMU 2.0+: <qemu>/qapi-types.h<vitastor>/qemu/b/qemu/qapi-types.h
    • config-host.h and qapi are required because they contain generated headers
  • You can also rebuild QEMU with a patch that makes LD_PRELOAD unnecessary to load vitastor driver. See qemu-*.*-vitastor.patch.
  • Install fio 3.7 or later, get its source and symlink it into <vitastor>/fio.
  • Build & install Vitastor with mkdir build && cd build && cmake .. && make -j8 && make install. Pay attention to the QEMU_PLUGINDIR cmake option - it must be set to qemu-kvm on RHEL.

Running

Please note that startup procedure isn't currently simple - you specify configuration and calculate disk offsets almost by hand. This will be fixed in near future.

  • Get some SATA or NVMe SSDs with capacitors (server-grade drives). You can use desktop SSDs with lazy fsync, but prepare for inferior single-thread latency.
  • Get a fast network (at least 10 Gbit/s).
  • Disable CPU powersaving: cpupower idle-set -D 0 && cpupower frequency-set -g performance.
  • Check /usr/lib/vitastor/mon/make-units.sh and /usr/lib/vitastor/mon/make-osd.sh and put desired values into the variables at the top of these files.
  • Create systemd units for the monitor and etcd: /usr/lib/vitastor/mon/make-units.sh
  • Create systemd units for your OSDs: /usr/lib/vitastor/mon/make-osd.sh /dev/disk/by-partuuid/XXX [/dev/disk/by-partuuid/YYY ...]
  • You can edit the units and change OSD configuration. Notable configuration variables:
    • disable_data_fsync 1 - only safe with server-grade drives with capacitors.
    • immediate_commit all - use this if all your drives are server-grade.
    • disable_device_lock 1 - only required if you run multiple OSDs on one block device.
    • flusher_count 256 - flusher is a micro-thread that removes old data from the journal. You don't have to worry about this parameter anymore, 256 is enough.
    • disk_alignment, journal_block_size, meta_block_size should be set to the internal block size of your SSDs which is 4096 on most drives.
    • journal_no_same_sector_overwrites true prevents multiple overwrites of the same journal sector. Most (99%) SSDs don't need this option. But Intel D3-4510 does because it doesn't like when you overwrite the same sector twice in a short period of time. The setting forces Vitastor to never overwrite the same journal sector twice in a row which makes D3-4510 almost happy. Not totally happy, because overwrites of the same block can still happen in the metadata area... When this setting is set, it is also required to raise journal_sector_buffer_count setting, which is the number of dirty journal sectors that may be written to at the same time.
  • systemctl start vitastor.target everywhere.
  • Create global configuration in etcd: etcdctl --endpoints=... put /vitastor/config/global '{"immediate_commit":"all"}' (if all your drives have capacitors).
  • Create pool configuration in etcd: etcdctl --endpoints=... put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":1,"pg_count":256,"failure_domain":"host"}}'. For jerasure pools the configuration should look like the following: 2:{"name":"ecpool","scheme":"jerasure","pg_size":4,"parity_chunks":2,"pg_minsize":2,"pg_count":256,"failure_domain":"host"}.
  • At this point, one of the monitors will configure PGs and OSDs will start them.
  • You can check PG states with etcdctl --endpoints=... get --prefix /vitastor/pg/state. All PGs should become 'active'.
  • Run tests with (for example): fio -thread -ioengine=libfio_vitastor.so -name=test -bs=4M -direct=1 -iodepth=16 -rw=write -etcd=10.115.0.10:2379/v3 -pool=1 -inode=1 -size=400G.
  • Upload VM disk image with qemu-img (for example):
    qemu-img convert -f qcow2 debian10.qcow2 -p -O raw 'vitastor:etcd_host=10.115.0.10\:2379/v3:pool=1:inode=1:size=2147483648'
    
    Note that the command requires to be run with LD_PRELOAD=/usr/lib/x86_64-linux-gnu/qemu/block-vitastor.so qemu-img ... if you use unmodified QEMU.
  • Run QEMU with (for example):
    qemu-system-x86_64 -enable-kvm -m 1024
      -drive 'file=vitastor:etcd_host=10.115.0.10\:2379/v3:pool=1:inode=1:size=2147483648',format=raw,if=none,id=drive-virtio-disk0,cache=none
      -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=off,physical_block_size=4096,logical_block_size=512
      -vnc 0.0.0.0:0
    
  • Remove inode with (for example):
    vitastor-rm --etcd_address 10.115.0.10:2379/v3 --pool 1 --inode 1 --parallel_osds 16 --iodepth 32
    

Known Problems

  • Object deletion requests may currently lead to 'incomplete' objects in EC pools if your OSDs crash during deletion because proper handling of object cleanup in a cluster should be "three-phase" and it's currently not implemented. Just repeat the removal request again in this case.

Implementation Principles

  • I like architecturally simple solutions. Vitastor is and will always be designed exactly like that.
  • I also like reinventing the wheel to some extent, like writing my own HTTP client for etcd interaction instead of using prebuilt libraries, because in this case I'm confident about what my code does and what it doesn't do.
  • I don't care about C++ "best practices" like RAII or proper inheritance or usage of smart pointers or whatever and I don't intend to change my mind, so if you're here looking for ideal reference C++ code, this probably isn't the right place.
  • I like node.js better than any other dynamically-typed language interpreter because it's faster than any other interpreter in the world, has neutral C-like syntax and built-in event loop. That's why Monitor is implemented in node.js.

Author and License

Copyright (c) Vitaliy Filippov (vitalif [at] yourcmc.ru), 2019+

Join Vitastor Telegram Chat: https://t.me/vitastor

All server-side code (OSD, Monitor and so on) is licensed under the terms of Vitastor Network Public License 1.1 (VNPL 1.1), a copyleft license based on GNU GPLv3.0 with the additional "Network Interaction" clause which requires opensourcing all programs directly or indirectly interacting with Vitastor through a computer network and expressly designed to be used in conjunction with it ("Proxy Programs"). Proxy Programs may be made public not only under the terms of the same license, but also under the terms of any GPL-Compatible Free Software License, as listed by the Free Software Foundation. This is a stricter copyleft license than the Affero GPL.

Please note that VNPL doesn't require you to open the code of proprietary software running inside a VM if it's not specially designed to be used with Vitastor.

Basically, you can't use the software in a proprietary environment to provide its functionality to users without opensourcing all intermediary components standing between the user and Vitastor or purchasing a commercial license from the author 😀 .

Client libraries (cluster_client and so on) are dual-licensed under the same VNPL 1.1 and also GNU GPL 2.0 or later to allow for compatibility with GPLed software like QEMU and fio.

You can find the full text of VNPL-1.1 in the file VNPL-1.1.txt. GPL 2.0 is also included in this repository as GPL-2.0.txt.

Issues
  • [vitastor] vitastor often appears io hang

    [vitastor] vitastor often appears io hang

    When I installed the ubuntu18.04 system by mounting the optical drive through virtualization, the virtual machine stopped moving in the install kernel, and I saw the following log in syslog.

    And has been stopped, it seems to be hanging, unable to continue.

    Jul 12 16:59:57 vitastor-1 vitastor-osd[248596]: [OSD 2] avg latency for subop 15 (ping): 185 us
    Jul 12 16:59:57 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_RX: client=0x555c3200e5a0 len=28 prog=536903814 vers=1 proc=6 type=0 status=0 serial=4689
    Jul 12 16:59:57 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x555c3200e5a0 len=188 prog=536903814 vers=1 proc=6 type=1 status=0 serial=4689
    Jul 12 16:59:57 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_RX: client=0x555c3200e5a0 len=40 prog=536903814 vers=1 proc=344 type=0 status=0 serial=4690
    Jul 12 16:59:57 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x555c3200e5a0 len=3924 prog=536903814 vers=1 proc=344 type=1 status=0 serial=4690
    Jul 12 16:59:59 vitastor-1 vitastor-osd[248623]: [OSD 1] avg latency for subop 15 (ping): 236 us
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34062 inode=1000000000004 offset=1280640000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34064 inode=1000000000004 offset=1280680000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34065 inode=1000000000004 offset=12806a0000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34066 inode=1000000000004 offset=12806c0000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34068 inode=1000000000004 offset=1280700000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34070 inode=1000000000004 offset=1280740000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34071 inode=1000000000004 offset=1280760000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34072 inode=1000000000004 offset=1280780000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34074 inode=1000000000004 offset=12807c0000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34076 inode=1000000000004 offset=1280800000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34077 inode=1000000000004 offset=28800000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34079 inode=1000000000004 offset=28840000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34080 inode=1000000000004 offset=28860000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34081 inode=1000000000004 offset=28880000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34083 inode=1000000000004 offset=288c0000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34085 inode=1000000000004 offset=28900000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34086 inode=1000000000004 offset=28920000 len=14000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34087 inode=1000000000004 offset=808c3d000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34088 inode=1000000000004 offset=808c3e000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34104 inode=1000000000004 offset=88a010000 len=8000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34121 inode=1000000000004 offset=808c3f000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34122 inode=1000000000004 offset=808c40000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34123 inode=1000000000004 offset=28620000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34124 inode=1000000000004 offset=808c41000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34125 inode=1000000000004 offset=808c42000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34126 inode=1000000000004 offset=808c43000 len=2000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34127 inode=1000000000004 offset=808c45000 len=6000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34128 inode=1000000000004 offset=808c4b000 len=2000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34129 inode=1000000000004 offset=808c4d000 len=2000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34130 inode=1000000000004 offset=808c4f000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34131 inode=1000000000004 offset=808c50000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34132 inode=1000000000004 offset=808c51000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34133 inode=1000000000004 offset=808c52000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34134 inode=1000000000004 offset=808c53000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34135 inode=1000000000004 offset=28640000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34137 inode=1000000000004 offset=808c54000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34138 inode=1000000000004 offset=808c55000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34139 inode=1000000000004 offset=808c56000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34140 inode=1000000000004 offset=808c57000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34141 inode=1000000000004 offset=808c58000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34142 inode=1000000000004 offset=808c59000 len=5000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34143 inode=1000000000004 offset=808c5e000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34144 inode=1000000000004 offset=808c5f000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34151 inode=1000000000004 offset=28940000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34160 inode=1000000000004 offset=28700000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34162 inode=1000000000004 offset=28740000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34164 inode=1000000000004 offset=28780000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34165 inode=1000000000004 offset=287a0000 len=14000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34171 inode=1000000000004 offset=808c80000 len=5000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248623]: [OSD 1] Slow op from client 11: primary_write id=34172 inode=1000000000004 offset=808c85000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34061 inode=1000000000004 offset=1280620000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34063 inode=1000000000004 offset=1280660000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34067 inode=1000000000004 offset=12806e0000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34069 inode=1000000000004 offset=1280720000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34073 inode=1000000000004 offset=12807a0000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34075 inode=1000000000004 offset=12807e0000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34078 inode=1000000000004 offset=28820000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34082 inode=1000000000004 offset=288a0000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34084 inode=1000000000004 offset=288e0000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34136 inode=1000000000004 offset=28660000 len=12000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34145 inode=1000000000004 offset=808c60000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34146 inode=1000000000004 offset=808c61000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34147 inode=1000000000004 offset=808c62000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34148 inode=1000000000004 offset=808c63000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34149 inode=1000000000004 offset=808c64000 len=3000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34150 inode=1000000000004 offset=808c67000 len=3000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34152 inode=1000000000004 offset=28960000 len=a000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34153 inode=1000000000004 offset=808c6a000 len=2000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34154 inode=1000000000004 offset=808c6c000 len=2000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34155 inode=1000000000004 offset=808c6e000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34156 inode=1000000000004 offset=808c6f000 len=4000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34157 inode=1000000000004 offset=808c73000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34158 inode=1000000000004 offset=808c74000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34159 inode=1000000000004 offset=808c75000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34161 inode=1000000000004 offset=28720000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34163 inode=1000000000004 offset=28760000 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34166 inode=1000000000004 offset=808c76000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34167 inode=1000000000004 offset=808c77000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34168 inode=1000000000004 offset=808c78000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34169 inode=1000000000004 offset=808c79000 len=5000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 8: primary_write id=34170 inode=1000000000004 offset=808c7e000 len=2000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158793 1000000000004:1280640000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158795 1000000000004:1280680000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158797 1000000000004:12806a0000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158799 1000000000004:12806c0000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158801 1000000000004:1280700000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158803 1000000000004:1280740000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158805 1000000000004:1280760000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158807 1000000000004:1280780000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158809 1000000000004:12807c0000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158811 1000000000004:1280800000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158813 1000000000004:28800000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158815 1000000000004:28840000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158817 1000000000004:28860000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158819 1000000000004:28880000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158821 1000000000004:288c0000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158823 1000000000004:28900000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158825 1000000000004:28920000 v1 offset=0 len=14000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158827 1000000000004:808c20000 v10 offset=1d000 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158831 1000000000004:88a000000 v2 offset=10000 len=8000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158833 1000000000004:808c40000 v1 offset=0 len=1000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158835 1000000000004:28620000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158837 1000000000004:28640000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158839 1000000000004:28940000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158841 1000000000004:28700000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158843 1000000000004:28740000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158845 1000000000004:28780000 v1 offset=0 len=20000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158847 1000000000004:287a0000 v1 offset=0 len=14000
    Jul 12 17:00:00 vitastor-1 vitastor-osd[248596]: [OSD 2] Slow op from client 10: write_stable id=158849 1000000000004:808c80000 v1 offset=0 len=5000
    Jul 12 17:00:00 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_RX: client=0x555c3200e5a0 len=28 prog=536903814 vers=1 proc=6 type=0 status=0 serial=4691
    Jul 12 17:00:00 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x555c3200e5a0 len=188 prog=536903814 vers=1 proc=6 type=1 status=0 serial=4691
    Jul 12 17:00:00 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_RX: client=0x555c3200e5a0 len=40 prog=536903814 vers=1 proc=344 type=0 status=0 serial=4692
    Jul 12 17:00:00 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x555c3200e5a0 len=3924 prog=536903814 vers=1 proc=344 type=1 status=0 serial=4692
    Jul 12 17:00:02 vitastor-1 vitastor-osd[248623]: [OSD 1] avg latency for subop 15 (ping): 202 us
    Jul 12 17:00:03 vitastor-1 vitastor-osd[248596]: [OSD 2] avg latency for subop 15 (ping): 232 us
    Jul 12 17:00:03 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_RX: client=0x555c3200e5a0 len=28 prog=536903814 vers=1 proc=6 type=0 status=0 serial=4693
    Jul 12 17:00:03 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x555c3200e5a0 len=188 prog=536903814 vers=1 proc=6 type=1 status=0 serial=4693
    Jul 12 17:00:03 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_RX: client=0x555c3200e5a0 len=40 prog=536903814 vers=1 proc=344 type=0 status=0 serial=4694
    Jul 12 17:00:03 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x555c3200e5a0 len=3924 prog=536903814 vers=1 proc=344 type=1 status=0 serial=4694
    Jul 12 17:00:06 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_RX: client=0x555c3200e5a0 len=28 prog=536903814 vers=1 proc=6 type=0 status=0 serial=4695
    Jul 12 17:00:06 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x555c3200e5a0 len=188 prog=536903814 vers=1 proc=6 type=1 status=0 serial=4695
    Jul 12 17:00:06 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_RX: client=0x555c3200e5a0 len=40 prog=536903814 vers=1 proc=344 type=0 status=0 serial=4696
    Jul 12 17:00:06 vitastor-1 libvirtd[406644]: RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x555c3200e5a0 len=3924 prog=536903814 vers=1 proc=344 type=1 status=0 serial=4696
    Jul 12 17:00:08 vitastor-1 vitastor-osd[248623]: [OSD 1] avg latency for subop 15 (ping): 228 us
    
    opened by lnsyyj 27
  • [qemu] qemu data error

    [qemu] qemu data error

    hi, @vitalif i use windows 10 as guest os and found that when it start, some of the qemu iov array item use the same address at a IO request。for example in a iovcnt = 5 request,iov[0] and iov[3] use the same address。when client slice the reqeust (in two part_op), part_op[0] use iov[0]-iov[2] and send to one osd , part_op[1] use iov[3]-iov[4] and send to another osd. if part_op[1] return earlier than part_op[0], the address of iov[0] and iov[3] will use the data of iov[0], this result is not same as use qcow2 file as vm disk. so, when the request opcode is read, i use another malloc part_op iov buf to save the data read from different osd, and memcpy data to qemu iov after all part_op done. but when i calc crc32 at part_op iov and qemu iov, some time it is different。i mprotect the qemu iov PROT_READ before all part_op done. then mprotect it PROT_READ|PROT_WRITE. but this way will cause qemu error: kvm run failed Bad address.

    opened by mirrorll 8
  • Is there a libvirt patch for vitastor

    Is there a libvirt patch for vitastor

    Hi Vitaliy, I want to use the virsh command to manage virtual machines through libvirt, but libvirt does not support vitastor, is there a corresponding patch?

    [email protected]:~# virsh  define vm_win10
    error: Failed to define domain from vm_win10
    error: unsupported configuration: unknown protocol type 'vitastor'
    
    opened by lnsyyj 8
  • [vitastor-cli rm] Layer to remove argument is missing

    [vitastor-cli rm] Layer to remove argument is missing

    In release 0.6.6, I got this error when I'm trying to remove node (Release 0.6.5, it remove normally): This error also makes cinder-vitastor driver can't remove volume in openstack.

    vitastor-cli rm --etcd_address 127.0.0.1:2379/v3 --pool 1 --inode 1
    Layer to remove argument is missing
    
    vitastor-cli rm --etcd_address 127.0.0.1:2379/v3 --pool 1 --inode 1 --parallel_osds 16 --iodepth 32
    Layer to remove argument is missing
    

    Node Data in etcd: (I run Create global configuration, Create pool configuration and Name an image command exactly like in Readme)

    etcdctl --endpoints=http://127.0.0.1:2379 get /vitastor/config/inode/1/1
    
    /vitastor/config/inode/1/1
    {"name":"testimg","size":2147483648}
    
    opened by moly7x 7
  • [QEMU and vitastor-mon] qemu-img: Unknown protocol 'vitastor' and Type Error

    [QEMU and vitastor-mon] qemu-img: Unknown protocol 'vitastor' and Type Error

    Hi @vitalif, I hope you can help me solve this Issues soon 😢😢 . I really don’t know what should I do now. Host:

    lsb_release -a
    Distributor ID: Ubuntu
    Description:    Ubuntu 21.04
    Release:        21.04
    Codename:       hirsute
    
    dpkg -l | grep vitastor
    ii  qemu                                  1:5.2+dfsg-10+vitastor1                                              amd64        fast processor emulator, dummy package
    ii  vitastor                              0.6.5-1bullseye                                                      amd64        Vitastor, a fast software-defined clustered block storage
    

    Cinder_volume (Inside container):

    lsb_release -a
    No LSB modules are available.
    Distributor ID: Ubuntu
    Description:    Ubuntu 20.04.3 LTS
    Release:        20.04
    Codename:       focal
    
    dpkg -l | grep qemu
    ii  qemu                           1:5.2+dfsg-10+vitastor1           amd64        fast processor emulator, dummy package
    ii  qemu-block-extra:amd64         1:4.2-3ubuntu6.17                 amd64        extra block backend modules for qemu-system and qemu-utils
    ii  qemu-utils                     1:4.2-3ubuntu6.17                 amd64        QEMU utilities
    
    dpkg -l | grep vitastor
    ii  qemu                           1:5.2+dfsg-10+vitastor1           amd64        fast processor emulator, dummy package
    ii  vitastor                       0.6.5-1bullseye                   amd64        Vitastor, a fast software-defined clustered block storage
    

    Problem 1 (QEMU):

    Before rebuilding QEMU: I already installed vitastor-cli inside Cinder_volume but when I try using qemu-img (Inside Cinder_volume container). It always has a problem (Both 2 method)

    qemu-img convert -f raw cirros-0.4.0-x86_64-disk.img -p -O raw 'vitastor:etcd_host=127.0.0.1\:2379/v3:image=testimg'
    qemu-img: Unknown protocol 'vitastor'
    
    LD_PRELOAD=/usr/lib/x86_64-linux-gnu/qemu/block-vitastor.so qemu-img convert -f raw cirros-0.4.0-x86_64-disk.img -p -O raw 'vitastor:etcd_host=127.0.0.1\:2379/v3:image=testimg'
    qemu-img: Unknown protocol 'vitastor'
    

    After building QEMU: I rebuilded QEMU (1:4.2-3ubuntu6.17) and applying a Patch in vitastor/patch/ qemu-4.2-vitastor.patch But it still got same problem.

    qemu-img convert -f raw cirros-0.4.0-x86_64-disk.img -p -O raw 'vitastor:etcd_host=127.0.0.1\:2379/v3:image=testimg'
    Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-vitastor.so
    Note: only modules from the same build can be loaded.
    Failed to open module: /var/run/qemu/Debian_1_4.2-3ubuntu6.17/block-vitastor.so: failed to map segment from shared object
    qemu-img: Unknown protocol 'vitastor'
    
    
    LD_PRELOAD=/usr/lib/x86_64-linux-gnu/qemu/block-vitastor.so qemu-img convert -f raw cirros-0.4.0-x86_64-disk.img -p -O raw 'vitastor:etcd_host=127.0.0.1\:2379/v3:image=testimg'
    qemu-img: /home/moly7x/qemu3/qemu-4.2/util/module.c:125: module_load_file: Assertion `QTAILQ_EMPTY(&dso_init_list)' failed.
    Aborted (core dumped)
    

    This is how I rebuild QEMU (I build in another VM, Ubuntu 20.04)

    sudo apt install dpkg-dev
    
    #Uncomment Deb source
    sudo nano /etc/apt/sources.list
    
    sudo apt-update
    sudo apt-get source qemu=1:4.2-3ubuntu6.17
    cd ./qemu-4.2/
    sudo apt build-dep qemu=1:4.2-3ubuntu6.17
    
    export QUILT_PATCHES=debian/patches
    export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
    sudo quilt import ../qemu-4.2-vitastor.patch
    sudo quilt push -a
    sudo quilt refresh
    
    dpkg-buildpackage -rfakeroot -b -uc -us
    

    Problem 2 (BigInt):

    Everytime I start my VM (host), I always need to start etcd either. (Not autostart)

    systemctl status etcd
    ● etcd.service - etcd for vitastor
         Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
         Active: active (running) since Wed 2021-10-20 04:12:51 UTC; 3h 31min ago
        Process: 12802 ExecStartPre=chown -R etcd /var/lib/etcd0.etcd (code=exited, status=0/SUCCESS)
       Main PID: 12805 (etcd)
          Tasks: 11
         Memory: 113.1M
    

    After I installed vitastor, I started vitastor-mon and vitastor.target services. And run command like example in README.

    etcdctl --endpoints=http://127.0.0.1:2379 put /vitastor/config/global '{"immediate_commit":"all"}'
    
    etcdctl --endpoints=http://127.0.0.1:2379 put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":1,"pg_count":256,"failure_domain":"host"}}'
    
    etcdctl --endpoints=http://127.0.0.1:2379 put /vitastor/config/inode/1/1 '{"name":"testimg","size":2147483648}'
    

    And run fio testing

    fio -thread -ioengine=libfio_vitastor.so -name=test -bs=4M -direct=1 -iodepth=16 -rw=write -etcd=127.0.0.1:2379/v3 -image=testimg
    
    test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=vitastor_cluster, iodepth=16
    fio-3.25
    Starting 1 thread
    No RDMA devices found
    [OSD 0] Couldnt initialize RDMA, proceeding with TCP only
    Jobs: 1 (f=1): [W(1)][97.3%][w=23.9MiB/s][w=5 IOPS][eta 00m:01s]
    test: (groupid=0, jobs=1): err= 0: pid=28987: Thu Sep 23 03:28:17 2021
      write: IOPS=14, BW=57.7MiB/s (60.5MB/s)(2048MiB/35521msec); 0 zone resets
        slat (usec): min=84, max=1037, avg=188.40, stdev=81.25
        clat (msec): min=380, max=2349, avg=1107.34, stdev=345.66
         lat (msec): min=381, max=2349, avg=1107.53, stdev=345.65
        clat percentiles (msec):
         |  1.00th=[  443],  5.00th=[  600], 10.00th=[  701], 20.00th=[  785],
         | 30.00th=[  894], 40.00th=[  995], 50.00th=[ 1062], 60.00th=[ 1150],
         | 70.00th=[ 1250], 80.00th=[ 1435], 90.00th=[ 1603], 95.00th=[ 1754],
         | 99.00th=[ 1989], 99.50th=[ 2022], 99.90th=[ 2366], 99.95th=[ 2366],
         | 99.99th=[ 2366]
       bw (  KiB/s): min= 8159, max=130549, per=100.00%, avg=62443.09, stdev=28879.73, samples=65
       iops        : min=    1, max=   31, avg=14.60, stdev= 7.06, samples=65
      lat (msec)   : 500=1.17%, 750=13.09%, 1000=27.73%, 2000=57.23%, >=2000=0.78%
      cpu          : usr=0.28%, sys=2.74%, ctx=1105, majf=0, minf=290
      IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=58.2%, 16=40.4%, 32=0.0%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=95.4%, 8=4.4%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
         issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=16
    
    Run status group 0 (all jobs):
      WRITE: bw=57.7MiB/s (60.5MB/s), 57.7MiB/s-57.7MiB/s (60.5MB/s-60.5MB/s), io=2048MiB (2147MB), run=35521-35521msec
    

    But after I restart my VM, and check vitastor-mon status. It got a problem.

    systemctl status vitastor-mon
    ● vitastor-mon.service - Vitastor monitor
         Loaded: loaded (/etc/systemd/system/vitastor-mon.service; enabled; vendor preset: enabled)
         Active: active (running) since Wed 2021-10-20 07:46:46 UTC; 47s ago
       Main PID: 318383 (node)
          Tasks: 7
         Memory: 15.3M
         CGroup: /system.slice/vitastor-mon.service
                 └─318383 node /usr/lib/vitastor/mon/mon-main.js --etcd_url http://127.0.0.1:2379 --etcd_prefix /vitastor --etcd_start_timeou>
    
    Oct 20 07:46:46 controller systemd[1]: Started Vitastor monitor.
    Oct 20 07:46:46 controller node[318383]: Waiting to become master
    Oct 20 07:46:51 controller node[318383]: Waiting to become master
    Oct 20 07:46:56 controller node[318383]: Waiting to become master
    Oct 20 07:47:01 controller node[318383]: Waiting to become master
    Oct 20 07:47:06 controller node[318383]: Waiting to become master
    Oct 20 07:47:11 controller node[318383]: Waiting to become master
    Oct 20 07:47:16 controller node[318383]: Waiting to become master
    Oct 20 07:47:21 controller node[318383]: Became master
    Oct 20 07:47:21 controller node[318383]: Bad key in etcd: /vitastor/pool/stats/1 = {"used_raw_tb":0.001953125,"total_raw_tb":0.01949596405029297,"raw_to_usable":0.5,"space_efficiency":1}
    
    
    
    Oct 20 07:50:40 controller node[318383]: TypeError: Do not know how to serialize a BigInt
    Oct 20 07:50:40 controller node[318383]:     at JSON.stringify (<anonymous>)
    Oct 20 07:50:40 controller node[318383]:     at Mon.update_total_stats (/usr/lib/vitastor/mon/mon.js:1355:33)
    Oct 20 07:50:40 controller node[318383]:     at Timeout._onTimeout (/usr/lib/vitastor/mon/mon.js:1380:18)
    Oct 20 07:50:40 controller node[318383]:     at listOnTimeout (internal/timers.js:554:17)
    Oct 20 07:50:40 controller node[318383]:     at processTimers (internal/timers.js:497:7)
    
    opened by moly7x 7
  • [vitastor-mon] Mon's responsibilities

    [vitastor-mon] Mon's responsibilities

    Hi @vitalif ,

    I have a question that I didn’t think about clearly. Why is a POST request sent in load_config? Is the original intention of this code to read value from etcd? Or create a directory to etcd?

    https://github.com/vitalif/vitastor/blob/712576ca75acf3a61812bafdf68518b3bf19f043/mon/mon.js#L1459

    opened by lnsyyj 7
  • [libvirt] error: internal error: argument key 'etcd_prefix' must not have null value

    [libvirt] error: internal error: argument key 'etcd_prefix' must not have null value

    Hi @vitalif ,

    I used vitastor's libvirt patch, and I want to try to use virsh to start the defined virtual machine template through libvirt. The xml is as follows. When I start the virtual machine, the following output appears.

    Can you help me take a look?

      <devices>
        <emulator>/usr/bin/qemu-system-x86_64</emulator>
          <disk type='network' device='disk'>
            <target dev='vda' bus='virtio' />
            <driver name='qemu' type='raw' />
            <!-- name is Vitastor image name -->
            <!-- config (optional) is the path to Vitastor's configuration file -->
            <!-- query (optional) is Vitastor's etcd_prefix -->
            <source protocol='vitastor' name='vitastor-inode-1' query='/vitastor' config='/etc/vitastor/vitastor.conf'>
              <!-- hosts = etcd addresses -->
              <host name='192.168.3.100' port='2379' />
            </source>
            <!-- required because Vitastor only supports 4k physical sectors -->
            <blockio physical_block_size="4096" logical_block_size="512" />
          </disk>
    
    [email protected]:~/qemuxml# virsh define win10_3
    Domain 'vdi_win10_3' defined from win10_3
    
    [email protected]:~/qemuxml# virsh start win10_3
    error: Failed to start domain 'win10_3'
    error: internal error: argument key 'etcd_prefix' must not have null value
    
    opened by lnsyyj 6
  • [nbd] XFS (nbd0): metadata I/O error

    [nbd] XFS (nbd0): metadata I/O error

    When I use nbd to mount the vitastor image and read and write, the following error occurs

    [Mon Jul 19 10:37:34 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:34 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:34 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:34 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:34 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:34 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:34 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:34 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:34 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:34 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:34 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:34 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:34 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:34 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:34 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:34 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:34 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:34 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:34 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:34 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:34 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:34 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:34 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:34 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:34 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:34 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:34 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:34 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:34 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:34 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:34 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:34 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:34 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:34 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:35 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:35 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:35 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:35 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:35 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:35 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:35 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:35 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:35 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:35 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:35 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:35 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:35 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:35 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:35 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:35 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:35 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:35 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:35 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:35 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:35 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:35 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:35 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:35 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:35 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:35 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:35 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:35 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:35 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:35 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:35 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:35 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:35 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:35 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:35 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:35 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:35 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:35 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:35 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:35 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:35 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:35 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:35 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:35 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:35 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:35 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:35 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:35 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:35 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:35 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:35 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:35 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:35 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:35 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:35 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:35 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Metadata corruption detected at xfs_dinode_verify.part.0+0x172/0x640 [xfs], inode 0x14506b6d dinode
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:37:35 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:37:35 2021] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Mon Jul 19 10:37:35 2021] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:37:35 2021] 00000020: 60 f4 e3 3f 07 da c4 4e 60 f4 e3 3f 07 da c4 4e  `..?...N`..?...N
    [Mon Jul 19 10:37:35 2021] 00000030: 60 f4 e3 3f 07 da c4 4e 00 00 00 00 00 00 1b f4  `..?...N........
    [Mon Jul 19 10:37:35 2021] 00000040: 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 01  ................
    [Mon Jul 19 10:37:35 2021] 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 ca c6 e6 ca  ................
    [Mon Jul 19 10:37:35 2021] 00000060: ff ff ff ff dd d6 04 ae 00 00 00 00 00 00 00 05  ................
    [Mon Jul 19 10:37:35 2021] 00000070: 00 00 00 02 00 01 da a8 00 00 00 00 00 00 00 00  ................
    [Mon Jul 19 10:38:42 2021] XFS (nbd0): Corruption warning: Metadata has LSN (7:27) ahead of current LSN (3:28176). Please unmount and run xfs_repair (>= v4.3) to resolve.
    [Mon Jul 19 10:38:42 2021] XFS (nbd0): Metadata CRC error detected at xfs_inobt_read_verify+0x12/0x90 [xfs], xfs_inobt block 0x12e423e8 
    [Mon Jul 19 10:38:42 2021] XFS (nbd0): Unmount and run xfs_repair
    [Mon Jul 19 10:38:42 2021] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Mon Jul 19 10:38:42 2021] 00000000: 54 5a 69 66 32 00 00 00 00 00 00 00 00 00 00 00  TZif2...........
    [Mon Jul 19 10:38:42 2021] 00000010: 00 00 00 00 00 00 00 07 00 00 00 07 00 00 00 1b  ................
    [Mon Jul 19 10:38:42 2021] 00000020: 00 00 00 3b 00 00 00 07 00 00 00 1a 80 00 00 00  ...;............
    [Mon Jul 19 10:38:42 2021] 00000030: 91 05 fc 00 da 62 04 38 4c 9f 27 c8 4d 97 2b f8  .....b.8L.'.M.+.
    [Mon Jul 19 10:38:42 2021] 00000040: 4e 7d e2 78 4e fd 8b b8 4f 77 0d f8 50 66 fe f9  N}.xN...Ow..Pf..
    [Mon Jul 19 10:38:42 2021] 00000050: 51 60 2a 79 52 46 e0 f9 53 40 0c 79 54 26 c2 f9  Q`*[email protected]&..
    [Mon Jul 19 10:38:42 2021] 00000060: 55 1f ee 79 56 06 a4 fa 56 ff d0 7a 57 e6 86 fa  U..yV...V..zW...
    [Mon Jul 19 10:38:42 2021] 00000070: 58 df b2 7b 59 c6 68 fb 5a bf 94 7b 5b af 85 7b  X..{Y.h.Z..{[..{
    [Mon Jul 19 10:38:42 2021] XFS: metadata IO error: 1716 callbacks suppressed
    [Mon Jul 19 10:38:42 2021] XFS (nbd0): metadata I/O error in "xfs_btree_read_buf_block.constprop.0+0x95/0xd0 [xfs]" at daddr 0x12e423e8 len 8 error 74
    [Mon Jul 19 10:38:42 2021] XFS (nbd0): xfs_do_force_shutdown(0x1) called from line 296 of file fs/xfs/xfs_trans_buf.c. Return address = 00000000f86604ca
    [Mon Jul 19 10:38:42 2021] XFS (nbd0): I/O Error Detected. Shutting down filesystem
    [Mon Jul 19 10:38:42 2021] XFS (nbd0): Please unmount the filesystem and rectify the problem(s)
    [Mon Jul 19 10:38:42 2021] XFS (nbd0): xfs_difree_inobt: xfs_inobt_lookup() returned error -117.
    
    
    opened by lnsyyj 5
  • [vitastor] When vdbench rewrites 8000000 small files, it reports an error

    [vitastor] When vdbench rewrites 8000000 small files, it reports an error

    Hi @vitalif ,

    When I format the xfs file system with vitastor nbd and use the test tool vdbench to read and write, the following problems occur occasionally. The performance of vitastor is very good now, and I think it will be perfect if I increase the stability.

    vdbench log

    Mar 15, 2022 ..Interval.. .ReqstdOps... ...cpu%...  read ....read..... ....write.... ..mb/sec... mb/sec .xfer.. ...mkdir.... ...rmdir.... ...create... ....open.... ...close.... ...delete... ..getattr... ..setattr... ...a[502/1854]
                                rate   resp total  sys   pct   rate   resp   rate   resp  read write  total    size  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  
    14:57:08.012            1    0.0  0.000  10.9 5.12   0.0    0.0  0.000    0.0  0.000  0.00  0.00   0.00       0   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000  
    14:57:08.021      avg_2-1 NaN  0.000 NaN NaN   0.0 NaN  0.000 NaN  0.000 NaN NaN NaN       0 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000                                       
    14:57:08.021      std_2-1                                                                                                                                                                                                             
    14:57:08.021      max_2-1                                                                                                                                                                                                             
    14:57:08.130                                                                                                                                                                                                                          
    14:57:08.130 Miscellaneous statistics:                                                                                                                                                                                                
    14:57:08.130 (These statistics do not include activity between the last reported interval and shutdown.)                                                                                                                              
    14:57:08.130 DIR_EXISTS          Directory may not exist (yet):                   14          4/sec                                                                                                                                   
    14:57:08.130 FILE_MAY_NOT_EXIST  File may not exist (yet):                        16          5/sec                                                                                                                                   
    14:57:08.130                                                                                                                                                                                                                          
    14:57:38.173 Waiting for slave synchronization: hd1-0. Building and validating file structure(s).                                                                                                                                     
    14:58:08.063 hd1-0: Completing the creation of internal file structure for anchor=/root/performance/fg-mp1/verilog_1: 4,096,000 files.                                                                                                
    14:58:08.202 Waiting for slave synchronization: hd1-0. Building and validating file structure(s).                                                                                                                                     
    14:58:38.232 Waiting for slave synchronization: hd1-0. Building and validating file structure(s).                                                                                                                                     
    14:59:08.262 Waiting for slave synchronization: hd1-0. Building and validating file structure(s).                                                                                                                                     
    14:59:08.411 hd1-0: Completing the creation of internal file structure for anchor=/root/performance/fg-mp2/verilog_2: 4,096,000 files.                                                                                                
    14:59:09.000 Starting RD=rd2; elapsed=300; fwdrate=max. For loops: xfersize=64k threads=128                                                                                                                                           
    14:59:09.461 hd1-0: 14:59:09.460 op: write  lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0064.file lba:            0 0x00000000 xfer:     1024 err
    no: EINVAL: 'Invalid argument'           
    14:59:09.475 hd1-0: 14:59:09.470 Write error using file /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0064.file                                         
    Error:         EINVAL: 'Invalid argument'                                                                          
    lba:           0                                                                                                                                                                                                                      
    xfersize:      1024                                                                                                
    blocks_done:   0                                                                                                                                                                                                                      
    bytes_done:    0                                                                                                   
    open_for_read: false                                                                                                                                                                                                                  
    fhandle:       17                                                                                                  
    14:59:09.506 hd1-0: 14:59:09.472 op: read   lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0008.file lba:            0 0x00000000 xfer:     2048 err
    no: EINVAL: 'Invalid argument'                                                                                     
    14:59:09.508 hd1-0: 14:59:09.507 Read error using file /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0008.file                                          
    Error:         EINVAL: 'Invalid argument'
    lba:           0                                                                                                                                                                                                                      
    xfersize:      2048                                                                                                
    blocks_done:   0                                                                                                                                                                                                                      
    bytes_done:    0
    open_for_read: true
    fhandle:       13
    14:59:09.633 hd1-0: 14:59:09.633 op: read   lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0474.file lba:            0 0x00000000 xfer:     1024 err
    no: EINVAL: 'Invalid argument' 
    14:59:09.635 hd1-0: 14:59:09.634 op: read   lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0473.file lba:            0 0x00000000 xfer:     1024 err
    no: EINVAL: 'Invalid argument' 
    14:59:09.635 hd1-0: 14:59:09.635 Read error using file /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0474.file
    Error:         EINVAL: 'Invalid argument'
    lba:           0
    xfersize:      1024
    blocks_done:   0
    bytes_done:    0
    open_for_read: true
    fhandle:       97
    14:59:09.636 hd1-0: 14:59:09.636 op: read   lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0472.file lba:            0 0x00000000 xfer:     1024 err
    no: EINVAL: 'Invalid argument' 
    

    file size

    [email protected]:~/output# ls -l /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0064.file
    -rw-r--r-- 1 root root 1024 Mar 15 14:36 /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0064.file
    [email protected]:~/output# ls -l /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0008.file
    -rw-r--r-- 1 root root 2048 Mar 15 14:49 /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0008.file
    [email protected]:~/output# ls -l  /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0474.file
    -rw-r--r-- 1 root root 1024 Mar 15 15:18 /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0474.file
    
    opened by lnsyyj 4
  • qemu upload disk image error: Co-routine re-entered recursively

    qemu upload disk image error: Co-routine re-entered recursively

    I installed vitastor 0.5.11 and qemu from yum using CentOS 7.9, and an error was reported when I executed the following command:

    [[email protected] ~]# qemu-img convert -f qcow2 -p /o/disk/f33.qcow2 -O raw 'vitastor:etcd_host=192.168.15.11\:2379/v3:pool=1:inode=2:size=13098811392' Co-routine re-entered recursively Aborted [[email protected] ~]#

    fio can run normally.

    opened by donglisi 4
  • [vitastor-osd] read subop failed

    [vitastor-osd] read subop failed

    Hi @vitalif ,

    When I used fio to do a read test in the ubuntu18.04 virtual machine, read subop failed: retval = -11 (expected 131072) appeared.

    I created two 200G disks from vitastor. One is used as a system disk, and the other is used as a test fio. When using fio to test the test disk, randread 4K, 8K, 16K, 256K are all normal, and an error occurs when randread 512k, 1m.

    I use fio to test the command:

    fio -ioengine=libaio -numjobs=16 -direct=1 -size=150g -iodepth=512 -runtime=600 -rw=randread -ba=512k -bs=512k -filename=/dev/vdb -name=iscsi_libaio_512_randread_512k -group_reporting
    
    fio -ioengine=libaio -numjobs=16 -direct=1 -size=150g -iodepth=512 -runtime=600 -rw=randread -ba=1m -bs=1m -filename=/dev/vdb -name=iscsi_libaio_512_randread_1m -group_reporting
    

    I uploaded the log to: https://github.com/lnsyyj/vitastor-log/tree/main/read-subop-failed

    opened by lnsyyj 3
  • compile error.

    compile error.

    while compile the source code, i got the following error

    vitastor-0.7.1/src/cluster_client.cpp:969:13: sorry, unimplemented: non-trivial designated initializers not supported
    
    vitastor-0.7.1/src/cluster_client.cpp:1021:13: sorry, unimplemented: non-trivial designated initializers not supported
    
    
    opened by meagon 1
  • [vitastor] Structure needs cleaning

    [vitastor] Structure needs cleaning

    Hi @vitalif ,

    When I use nbd to do xfs, mount to a local directory, and write files to the mount point.

    [email protected]:~/yujiang/vdbench# ls -l /root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1219.file
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1219.file': Structure needs cleaning
    [email protected]:~/yujiang/vdbench# ls -l /root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1219.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1220.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1221.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1222.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1224.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1223.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1225.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1227.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1226.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1229.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1230.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1234.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1232.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1233.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1231.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1228.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1235.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1236.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1237.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1238.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1239.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1242.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1240.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1243.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1241.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1244.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1245.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1247.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1246.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1248.file': Structure needs cleaning
    ls: cannot access '/root/performance/fg-mp2/verilog_2/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_2.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f1249.file': Structure needs cleaning
    total 4368
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1000.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1001.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1002.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1003.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:32 vdb_f1004.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1005.file
    -rw-r--r-- 1 root root  131072 Mar 16  2022 vdb_f1006.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1007.file
    -rw-r--r-- 1 root root    4096 Mar 15 15:33 vdb_f1008.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:32 vdb_f1009.file
    -rw-r--r-- 1 root root   65536 Mar 15 15:31 vdb_f1010.file
    -rw-r--r-- 1 root root   16384 Mar 16  2022 vdb_f1011.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1012.file
    -rw-r--r-- 1 root root   65536 Mar 15 15:31 vdb_f1013.file
    -rw-r--r-- 1 root root   16384 Mar 16  2022 vdb_f1014.file
    -rw-r--r-- 1 root root    4096 Mar 15 15:20 vdb_f1015.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1016.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1017.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1018.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:32 vdb_f1019.file
    -rw-r--r-- 1 root root    4096 Mar 15 15:22 vdb_f1020.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1021.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:32 vdb_f1022.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1023.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1024.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1025.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:22 vdb_f1026.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:20 vdb_f1027.file
    -rw-r--r-- 1 root root    4096 Mar 15 15:32 vdb_f1028.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1029.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1030.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1031.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1032.file
    -rw-r--r-- 1 root root   65536 Mar 15  2022 vdb_f1033.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1034.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1035.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:31 vdb_f1036.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1037.file
    -rw-r--r-- 1 root root    8192 Mar 15  2022 vdb_f1038.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1039.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:32 vdb_f1040.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1041.file
    -rw-r--r-- 1 root root    4096 Mar 15  2022 vdb_f1042.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1043.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:23 vdb_f1044.file
    -rw-r--r-- 1 root root   32768 Mar 15 15:23 vdb_f1045.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:22 vdb_f1046.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1047.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1048.file
    -rw-r--r-- 1 root root    8192 Mar 15  2022 vdb_f1049.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:31 vdb_f1050.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:32 vdb_f1051.file
    -rw-r--r-- 1 root root    4096 Mar 16  2022 vdb_f1052.file
    -rw-r--r-- 1 root root    8192 Mar 15 14:50 vdb_f1053.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:32 vdb_f1054.file
    -rw-r--r-- 1 root root   16384 Mar 15  2022 vdb_f1055.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1056.file
    -rw-r--r-- 1 root root    8192 Mar 15  2022 vdb_f1057.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:22 vdb_f1058.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:22 vdb_f1059.file
    -rw-r--r-- 1 root root 1048576 Mar 15 15:31 vdb_f1060.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1061.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:31 vdb_f1062.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1063.file
    -rw-r--r-- 1 root root   16384 Mar 16  2022 vdb_f1064.file
    -rw-r--r-- 1 root root   16384 Mar 15 15:31 vdb_f1065.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1066.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1067.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1068.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1069.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1070.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1071.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:20 vdb_f1072.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:31 vdb_f1073.file
    -rw-r--r-- 1 root root    4096 Mar 16  2022 vdb_f1074.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1075.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:18 vdb_f1076.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1077.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1078.file
    -rw-r--r-- 1 root root    4096 Mar 15  2022 vdb_f1079.file
    -rw-r--r-- 1 root root   65536 Mar 16  2022 vdb_f1080.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1081.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1082.file
    -rw-r--r-- 1 root root   65536 Mar 16  2022 vdb_f1083.file
    -rw-r--r-- 1 root root    8192 Mar 15  2022 vdb_f1084.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1085.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1086.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1087.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1088.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1089.file
    -rw-r--r-- 1 root root   65536 Mar 15 15:20 vdb_f1090.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1091.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1092.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1093.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1094.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1095.file
    -rw-r--r-- 1 root root    8192 Mar 15  2022 vdb_f1096.file
    -rw-r--r-- 1 root root    4096 Mar 15 15:19 vdb_f1097.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1098.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1099.file
    -rw-r--r-- 1 root root  131072 Mar 15 15:33 vdb_f1100.file
    -rw-r--r-- 1 root root   65536 Mar 15 15:19 vdb_f1101.file
    -rw-r--r-- 1 root root  131072 Mar 16  2022 vdb_f1102.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1103.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1104.file
    -rw-r--r-- 1 root root    8192 Mar 15  2022 vdb_f1105.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1106.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1107.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1108.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1109.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1110.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1111.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1112.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1113.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1114.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1115.file
    -rw-r--r-- 1 root root    4096 Mar 16  2022 vdb_f1116.file
    -rw-r--r-- 1 root root   65536 Mar 16  2022 vdb_f1117.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1118.file
    -rw-r--r-- 1 root root    8192 Mar 15 14:26 vdb_f1119.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1120.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1121.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1122.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1123.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1124.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1125.file
    -rw-r--r-- 1 root root   16384 Mar 16  2022 vdb_f1126.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1127.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1128.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1129.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1130.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1131.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1132.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1133.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:20 vdb_f1134.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1135.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1136.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1137.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:35 vdb_f1138.file
    -rw-r--r-- 1 root root    8192 Mar 15  2022 vdb_f1139.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1140.file
    -rw-r--r-- 1 root root  131072 Mar 15 15:19 vdb_f1141.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1142.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1143.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:23 vdb_f1144.file
    -rw-r--r-- 1 root root   32768 Mar 16  2022 vdb_f1145.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1146.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1147.file
    -rw-r--r-- 1 root root   65536 Mar 16  2022 vdb_f1148.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1149.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1150.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1151.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1152.file
    -rw-r--r-- 1 root root  131072 Mar 15 15:35 vdb_f1153.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1154.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1155.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1156.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1157.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1158.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1159.file
    -rw-r--r-- 1 root root    8192 Mar 15  2022 vdb_f1160.file
    -rw-r--r-- 1 root root   16384 Mar 15 15:35 vdb_f1161.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1162.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1163.file
    -rw-r--r-- 1 root root   65536 Mar 16  2022 vdb_f1164.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:32 vdb_f1165.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1166.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:22 vdb_f1167.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:35 vdb_f1168.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1169.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1170.file
    -rw-r--r-- 1 root root  131072 Mar 16  2022 vdb_f1171.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1172.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1173.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1174.file
    -rw-r--r-- 1 root root   65536 Mar 15 15:33 vdb_f1175.file
    -rw-r--r-- 1 root root   16384 Mar 16  2022 vdb_f1176.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1177.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1178.file
    -rw-r--r-- 1 root root    4096 Mar 16  2022 vdb_f1179.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1180.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1181.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1182.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1183.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1184.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1185.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1186.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1187.file
    -rw-r--r-- 1 root root    4096 Mar 16  2022 vdb_f1188.file
    -rw-r--r-- 1 root root   65536 Mar 15  2022 vdb_f1189.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1190.file
    -rw-r--r-- 1 root root    8192 Mar 15 15:33 vdb_f1191.file
    -rw-r--r-- 1 root root    4096 Mar 15 15:23 vdb_f1192.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1193.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1194.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1195.file
    -rw-r--r-- 1 root root    4096 Mar 15 15:19 vdb_f1196.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1197.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1198.file
    -rw-r--r-- 1 root root   65536 Mar 16  2022 vdb_f1199.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1200.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1201.file
    -rw-r--r-- 1 root root   65536 Mar 15 15:32 vdb_f1202.file
    -rw-r--r-- 1 root root   16384 Mar 16  2022 vdb_f1203.file
    -rw-r--r-- 1 root root    4096 Mar 16  2022 vdb_f1204.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1205.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1206.file
    -rw-r--r-- 1 root root    4096 Mar 16  2022 vdb_f1207.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1208.file
    -rw-r--r-- 1 root root   16384 Mar 15 15:33 vdb_f1209.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1210.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1211.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1212.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1213.file
    -rw-r--r-- 1 root root    4096 Mar 15 17:49 vdb_f1214.file
    -rw-r--r-- 1 root root   16384 Mar 16  2022 vdb_f1215.file
    -rw-r--r-- 1 root root    4096 Mar 16  2022 vdb_f1216.file
    -rw-r--r-- 1 root root    8192 Mar 15  2022 vdb_f1217.file
    -rw-r--r-- 1 root root    8192 Mar 16  2022 vdb_f1218.file
    -????????? ? ?    ?          ?            ? vdb_f1219.file
    -????????? ? ?    ?          ?            ? vdb_f1220.file
    -????????? ? ?    ?          ?            ? vdb_f1221.file
    -????????? ? ?    ?          ?            ? vdb_f1222.file
    -????????? ? ?    ?          ?            ? vdb_f1223.file
    -????????? ? ?    ?          ?            ? vdb_f1224.file
    -????????? ? ?    ?          ?            ? vdb_f1225.file
    -????????? ? ?    ?          ?            ? vdb_f1226.file
    -????????? ? ?    ?          ?            ? vdb_f1227.file
    -????????? ? ?    ?          ?            ? vdb_f1228.file
    -????????? ? ?    ?          ?            ? vdb_f1229.file
    -????????? ? ?    ?          ?            ? vdb_f1230.file
    -????????? ? ?    ?          ?            ? vdb_f1231.file
    -????????? ? ?    ?          ?            ? vdb_f1232.file
    -????????? ? ?    ?          ?            ? vdb_f1233.file
    -????????? ? ?    ?          ?            ? vdb_f1234.file
    -????????? ? ?    ?          ?            ? vdb_f1235.file
    -????????? ? ?    ?          ?            ? vdb_f1236.file
    -????????? ? ?    ?          ?            ? vdb_f1237.file
    -????????? ? ?    ?          ?            ? vdb_f1238.file
    -????????? ? ?    ?          ?            ? vdb_f1239.file
    -????????? ? ?    ?          ?            ? vdb_f1240.file
    -????????? ? ?    ?          ?            ? vdb_f1241.file
    -????????? ? ?    ?          ?            ? vdb_f1242.file
    -????????? ? ?    ?          ?            ? vdb_f1243.file
    -????????? ? ?    ?          ?            ? vdb_f1244.file
    -????????? ? ?    ?          ?            ? vdb_f1245.file
    -????????? ? ?    ?          ?            ? vdb_f1246.file
    -????????? ? ?    ?          ?            ? vdb_f1247.file
    -????????? ? ?    ?          ?            ? vdb_f1248.file
    -????????? ? ?    ?          ?            ? vdb_f1249.file
    

    dmesg log

    [Tue Mar 15 17:49:36 2022] 00000040: 00 00 00 00 00 00 00 00 62 30 41 a4 0f 78 b3 5b  ........b0A..x.[
    [Tue Mar 15 17:49:36 2022] 00000050: 62 30 bc 01 39 e0 48 c0 62 30 40 d1 39 9f 6c 5a  [email protected]
    [Tue Mar 15 17:49:36 2022] 00000060: 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00 01  ................
    [Tue Mar 15 17:49:36 2022] 00000070: 00 00 00 00 00 00 00 01 00 00 00 02 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Metadata corruption detected at xfs_buf_ioend+0x162/0x570 [xfs], xfs_inode block 0x381ee15c8 xfs_inode_buf_verify
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Unmount and run xfs_repair
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Tue Mar 15 17:49:36 2022] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Tue Mar 15 17:49:36 2022] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] 00000020: 62 30 41 a4 19 b9 45 c2 49 4e 81 a4 03 02 00 00  b0A...E.IN......
    [Tue Mar 15 17:49:36 2022] 00000030: 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] 00000040: 00 00 00 00 00 00 00 00 62 30 41 a4 0f 78 b3 5b  ........b0A..x.[
    [Tue Mar 15 17:49:36 2022] 00000050: 62 30 bc 01 39 e0 48 c0 62 30 40 d1 39 9f 6c 5a  [email protected]
    [Tue Mar 15 17:49:36 2022] 00000060: 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00 01  ................
    [Tue Mar 15 17:49:36 2022] 00000070: 00 00 00 00 00 00 00 01 00 00 00 02 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Metadata corruption detected at xfs_buf_ioend+0x162/0x570 [xfs], xfs_inode block 0x381ee15c8 xfs_inode_buf_verify
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Unmount and run xfs_repair
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Tue Mar 15 17:49:36 2022] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Tue Mar 15 17:49:36 2022] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] 00000020: 62 30 41 a4 19 b9 45 c2 49 4e 81 a4 03 02 00 00  b0A...E.IN......
    [Tue Mar 15 17:49:36 2022] 00000030: 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] 00000040: 00 00 00 00 00 00 00 00 62 30 41 a4 0f 78 b3 5b  ........b0A..x.[
    [Tue Mar 15 17:49:36 2022] 00000050: 62 30 bc 01 39 e0 48 c0 62 30 40 d1 39 9f 6c 5a  [email protected]
    [Tue Mar 15 17:49:36 2022] 00000060: 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00 01  ................
    [Tue Mar 15 17:49:36 2022] 00000070: 00 00 00 00 00 00 00 01 00 00 00 02 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Metadata corruption detected at xfs_buf_ioend+0x162/0x570 [xfs], xfs_inode block 0x381ee15c8 xfs_inode_buf_verify
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Unmount and run xfs_repair
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Tue Mar 15 17:49:36 2022] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Tue Mar 15 17:49:36 2022] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] 00000020: 62 30 41 a4 19 b9 45 c2 49 4e 81 a4 03 02 00 00  b0A...E.IN......
    [Tue Mar 15 17:49:36 2022] 00000030: 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] 00000040: 00 00 00 00 00 00 00 00 62 30 41 a4 0f 78 b3 5b  ........b0A..x.[
    [Tue Mar 15 17:49:36 2022] 00000050: 62 30 bc 01 39 e0 48 c0 62 30 40 d1 39 9f 6c 5a  [email protected]
    [Tue Mar 15 17:49:36 2022] 00000060: 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00 01  ................
    [Tue Mar 15 17:49:36 2022] 00000070: 00 00 00 00 00 00 00 01 00 00 00 02 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Metadata corruption detected at xfs_buf_ioend+0x162/0x570 [xfs], xfs_inode block 0x381ee15c8 xfs_inode_buf_verify
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Unmount and run xfs_repair
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): First 128 bytes of corrupted metadata buffer:
    [Tue Mar 15 17:49:36 2022] 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
    [Tue Mar 15 17:49:36 2022] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] 00000020: 62 30 41 a4 19 b9 45 c2 49 4e 81 a4 03 02 00 00  b0A...E.IN......
    [Tue Mar 15 17:49:36 2022] 00000030: 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] 00000040: 00 00 00 00 00 00 00 00 62 30 41 a4 0f 78 b3 5b  ........b0A..x.[
    [Tue Mar 15 17:49:36 2022] 00000050: 62 30 bc 01 39 e0 48 c0 62 30 40 d1 39 9f 6c 5a  [email protected]
    [Tue Mar 15 17:49:36 2022] 00000060: 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00 01  ................
    [Tue Mar 15 17:49:36 2022] 00000070: 00 00 00 00 00 00 00 01 00 00 00 02 00 00 00 00  ................
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Metadata corruption detected at xfs_buf_ioend+0x162/0x570 [xfs], xfs_inode block 0x381ee15c8 xfs_inode_buf_verify
    [Tue Mar 15 17:49:36 2022] XFS (nbd0): Unmount and run xfs_repair
    
    opened by lnsyyj 2
  • [vitastor] probably corrupted filesystem

    [vitastor] probably corrupted filesystem

    Hi @vitalif ,

    I have a vistator cluster, I created 11TB of images in vistator, used vitastor-nbd map, formatted XFS to mount it locally, and wrote files to it.

    [email protected]:~# etcdctl --endpoints=192.168.1.215:2379 put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":3,"pg_minsize":1,"pg_count":256,"failure_domain":"host"}}'
    
    [email protected]:~# etcdctl --endpoints=http://192.168.1.215:2379/v3 put /vitastor/config/inode/1/1 '{"name":"testimg","size":12094627905536}'
    
    [email protected]:~# vitastor-nbd map --etcd_address 192.168.1.215:2379/v3 --image testimg
    
    [email protected]:~# mkfs.xfs /dev/nbd0
    
    [email protected]:~# mount /dev/nbd0 /home/vita-xfs/
    
    During data writing the vistator-nbd process is gone, when I mount it again,
    
    [email protected]:~# mount /dev/nbd0 /home/vita-xfs/
    mount: /home/vita-xfs: cannot mount; probably corrupted filesystem on /dev/nbd0.
    
    
    opened by lnsyyj 1
  • [question] Can vistator run on HDD without considering performance?

    [question] Can vistator run on HDD without considering performance?

    Hi @vitalif ,

    We have a requirement that we want to use vitastor on HDD, because now most customers consider the large capacity of HDD.

    1. Can vistator run on HDD without considering performance?
    2. Can it be deployed to distribute wal on SSD and data on HDD?
    opened by lnsyyj 2
  • [tcmu-runner] vitastor supports iscsi feature

    [tcmu-runner] vitastor supports iscsi feature

    Hi @vitalif ,

    Are you interested in supporting the iscsi protocol? I think this feature is very useful for iscsi users, and can directly connect to linux, windows, vmware and other software in the future.

    ceph wrote a plugin in tcmu-runner: https://github.com/open-iscsi/tcmu-runner

    opened by lnsyyj 0
  • can not remove inode when cluster state error

    can not remove inode when cluster state error

    with a 3 node and 6 osd cluster.(node1: [osd.1, osd.2]. node2:[osd.3, osd.4], node3:[osd.5, osd.6] ).

    1. if node3 down, then some pg become degrade, in this time can not remove inode.
    2. after osd_out_time pg become active, can remove, and remove it.
    3. then in some times node3 up, join to cluster again. but osd.5 and osd.6 has some object of the already remove inode, with peering they write to other osd.
    opened by mirrorll 0
Strong Othello AI based on Egaroucid4, which got 1st place in the world

Egaroucid5 Strong Othello AI based on Egaroucid4, which got 1st place in the world Abstract Egaroucid5 is an Othello AI. You can play light version of

Takuto Yamana 24 Aug 1, 2022
Beringei is a high performance, in-memory storage engine for time series data.

** THIS REPO HAS BEEN ARCHIVED AND IS NO LONGER BEING ACTIVELY MAINTAINED ** Beringei A high performance, in memory time series storage engine In the

Meta Archive 3.1k Aug 4, 2022
Mirror of compiler project code. Not for SVC purpose.

Compiler-proj Project progress is updated here. Progress 2021/11/28: Started! Set up Makefile and finished basic scanner. 2021/10/24: Repo created. Ac

Yuheng 0 Dec 23, 2021
Zenotech 7 Nov 13, 2020
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

The Apache Software Foundation 20k Aug 5, 2022
Distributed machine learning platform

Veles Distributed platform for rapid Deep learning application development Consists of: Platform - https://github.com/Samsung/veles Znicz Plugin - Neu

Samsung 898 Jul 25, 2022
A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

Light Gradient Boosting Machine LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed a

Microsoft 14k Jul 31, 2022
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eXtreme Gradient Boosting Community | Documentation | Resources | Contributors | Release Notes XGBoost is an optimized distributed gradient boosting l

Distributed (Deep) Machine Learning Community 23k Aug 6, 2022
OpenEmbedding is an open source framework for Tensorflow distributed training acceleration.

OpenEmbedding English version | 中文版 About OpenEmbedding is an open-source framework for TensorFlow distributed training acceleration. Nowadays, many m

4Paradigm 19 Jul 25, 2022
Distributed Pose Graph Optimization

Distributed Pose Graph Optimization

MIT Aerospace Controls Laboratory 85 Aug 1, 2022
LineFS: Efficient SmartNIC Offload of a Distributed File System with Pipeline Parallelism

LineFS repository for the Artifact Evaluation of SOSP 2021 1. System requirements (Tested environment) 1.1. Hardware requirements 1.1.1. Host machine

null 48 Jul 8, 2022
Fairring (FAIR + Herring) is a plug-in for PyTorch that provides a process group for distributed training that outperforms NCCL at large scales

Fairring (FAIR + Herring): a faster all-reduce TL;DR: Using a variation on Amazon’s "Herring" technique, which leverages reduction servers, we can per

Meta Research 45 Jul 13, 2022
A program developed using MPI for distributed computation of Histogram for large data and their performance anaysis on multi-core systems

mpi-histo A program developed using MPI for distributed computation of Histogram for large data and their performance anaysis on multi-core systems. T

Raj Shrestha 2 Dec 21, 2021
A library for distributed ML training with PyTorch

moolib moolib - a communications library for distributed ML training moolib offers general purpose RPC with automatic transport selection (shared memo

Meta Research 332 Jul 18, 2022
PaRSEC: the Parallel Runtime Scheduler and Execution Controller for micro-tasks on distributed heterogeneous systems.

PaRSEC is a generic framework for architecture aware scheduling and management of micro-tasks on distributed, GPU accelerated, many-core heterogeneous architectures. PaRSEC assigns computation threads to the cores, GPU accelerators, overlaps communications and computations and uses a dynamic, fully-distributed scheduler based on architectural features such as NUMA nodes and algorithmic features such as data reuse.

null 9 Jun 19, 2022
functorch is a prototype of JAX-like composable function transforms for PyTorch.

functorch Why functorch? | Install guide | Transformations | Future Plans functorch is a prototype of JAX-like composable FUNCtion transforms for pyTO

Richard Zou 1.1k Aug 6, 2022
functorch is a prototype of JAX-like composable function transforms for PyTorch.

functorch Why functorch? | Install guide | Transformations | Future Plans functorch is a prototype of JAX-like composable FUNCtion transforms for pyTO

Facebook Research 1.1k Jul 28, 2022
AlphaZero like implementation for Oware Abapa game

CGZero AlphaZero like implementation for Oware abapa game, in Codingame (https://www.codingame.com/multiplayer/bot-programming/oware-abapa) See https:

null 21 Jul 2, 2022
Handwrite the Korean alphabet, Hangul, like a native.

BTS Magic Wand lets you improve your Hangul handwriting! Using an Arduino, IMU sensor, and TensorFlow Lite for Microcontrollers, we trained a tiny machine learning model to recognize the Korean alphabet, Hangul, you are writing.

BTS-MicroNet (Team Butler) 15 Nov 18, 2021