双路超线程物理服务器的QEMU CPU affinity调整

首先运行 virsh capabilities 查看物理机物理核心及其线程与逻辑处理器（线程）的关系：

...
    <topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>24671680</memory>
          <pages unit='KiB' size='4'>6167920</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='16'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,8'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,9'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,10'/>
            <cpu id='3' socket_id='0' core_id='3' siblings='3,11'/>
            <cpu id='4' socket_id='1' core_id='0' siblings='4,12'/>
            <cpu id='5' socket_id='1' core_id='1' siblings='5,13'/>
            <cpu id='6' socket_id='1' core_id='2' siblings='6,14'/>
            <cpu id='7' socket_id='1' core_id='3' siblings='7,15'/>
            <cpu id='8' socket_id='0' core_id='0' siblings='0,8'/>
            <cpu id='9' socket_id='0' core_id='1' siblings='1,9'/>
            <cpu id='10' socket_id='0' core_id='2' siblings='2,10'/>
            <cpu id='11' socket_id='0' core_id='3' siblings='3,11'/>
            <cpu id='12' socket_id='1' core_id='0' siblings='4,12'/>
            <cpu id='13' socket_id='1' core_id='1' siblings='5,13'/>
            <cpu id='14' socket_id='1' core_id='2' siblings='6,14'/>
            <cpu id='15' socket_id='1' core_id='3' siblings='7,15'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <cache>
      <bank id='0' level='3' type='both' size='8' unit='MiB' cpus='0-3,8-11'/>
      <bank id='1' level='3' type='both' size='8' unit='MiB' cpus='4-7,12-15'/>
    </cache>
...

...

</distances>

</cpus>

</cell>

</cells>

</topology>

<cache>

</cache>

...

其中的socket_id代表CPU的槽位，core_id代表CPU物理核心，siblings表示哪些逻辑处理器（线程）是属于同一个物理核心的。

例如id为0以及8的逻辑处理器，都属于同一个CPU槽位且属于同一个物理核心。

在这台物理机上建立的虚拟机，想得到最佳的性能，首先不要跨CPU插槽，其次，不要让多个虚拟机CPU共享同一个物理核心。

根据上面的信息，4核心8线程的虚拟机CPU亲和度配置如下：

<domain>
...
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='9'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='10'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='7' cpuset='11'/>
  </cputune>
...
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
...
</domain>

...

</cputune>

...

</cpu>

...

</domain>

其中vcpu 0和1在虚拟机中对应一个物理核心的两个线程，将其亲和度设定到逻辑处理器0和8，后面vcpu 2-7，同理。

启动虚拟机，通过 virsh vcpupin 进行验证：

VCPU: CPU Affinity
----------------------------------
   0: 0
   1: 8
   2: 1
   3: 9
   4: 2
   5: 10
   6: 3
   7: 11

VCPU: CPU Affinity

----------------------------------

0: 0

1: 8

2: 1

3: 9

4: 2

5: 10

6: 3

7: 11

到虚拟机中查看/proc/cpuinfo：

# cat /proc/cpuinfo | grep "core id"
core id		: 0
core id		: 0
core id		: 1
core id		: 1
core id		: 2
core id		: 2
core id		: 3
core id		: 3

# cat /proc/cpuinfo | grep "core id"

core id : 0

core id : 1

core id : 2

core id : 3

上面输出的信息可以确定设定无误，因为第一、二个逻辑核心对应了第一个物理核心，后面第三到第八个也同理。如果输出的core id与cputune处设定时所期望的顺序不对，调整cputune中的参数即可。

Tags: affinity, CPU, hyperthreading, libvirt, libvirtd, qemu, 亲和度, 超线程

You might also like:

Unix Like

qemu with pool/volume storage: Could not open ‘xxxxxxx’: Permission denied

volume信息：

# virsh vol-list test
 Name                 Path                                    
------------------------------------------------------------------------------
 test.qcow2           /virt/test.qcow2

# virsh vol-list test

Name Path

------------------------------------------------------------------------------

test.qcow2 /virt/test.qcow2

虚拟机disk配置：

# virsh dumpxml Test
...
    <disk type='volume' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source pool='test' volume='test.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
...

# virsh dumpxml Test

...

</disk>

...

启动虚拟机：

# virsh start Test
error: Failed to start domain Test
error: internal error: process exited while connecting to monitor: 2019-02-06T12:54:47.722297Z qemu-system-x86_64: -drive file=/virt/test.qcow2,format=qcow2,if=none,id=drive-virtio-disk0: Could not open '/virt/test.qcow2': Permission denied

# virsh start Test

error: Failed to start domain Test

error: internal error: process exited while connecting to monitor: 2019-02-06T12:54:47.722297Z qemu-system-x86_64: -drive file=/virt/test.qcow2,format=qcow2,if=none,id=drive-virtio-disk0: Could not open '/virt/test.qcow2': Permission denied

查看syslog：

# cat /var/log/syslog
...
kernel: [ 6551.331932] audit: type=1400 audit(1549457961.800:209): apparmor="DENIED" operation="open" profile="libvirt-5831a051-78ee-43b4-a15d-6e520b1b3ab7" name="/virt/test.qcow2" pid=27204 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
...

# cat /var/log/syslog

...

kernel: [ 6551.331932] audit: type=1400 audit(1549457961.800:209): apparmor="DENIED" operation="open" profile="libvirt-5831a051-78ee-43b4-a15d-6e520b1b3ab7" name="/virt/test.qcow2" pid=27204 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

...

似乎没找比较好的方法解决此问题。一个选择是弃用type=’volume’，改成type=’file’。另一种选择是关闭apparmor：

# vim /etc/libvirt/qemu.conf
...
security_driver = "none"
...
# systemctl restart libvirtd

# vim /etc/libvirt/qemu.conf

...

security_driver = "none"

...

# systemctl restart libvirtd

&n […]

Unix Like

CLion调试Linux Kernel

1. 编译Kernel 先按照https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html说明的config设定编译内核并且安装到guest上。 menuconfig中按“/”是可以搜索的，例如要找“CONFIG_GDB_SCRIPTS”，按“/”输入“GDB_SCRIPTS”。 qemu启用了virtio的话，记得选上VIRTIO_BLK和VIRTIO_NET。 2. QEMU 2.1 关闭KVM 关闭KVM，关闭KVM，关闭KVM，重要的事情说三次！不然step by step的时候，会无限跳到arch […]

Unix Like

libvirtd: virNetTLSContextValidCertificate:994 : Unable to verify TLS peer: No certificate was found.

Jan 29 06:20:12 debian libvirtd[3363]: 2019-01-29 11:20:12.116+0000: 3363: error : virNetTLSContextValidCertificate:994 : Unable to verify TLS peer: No certificate was found.
Jan 29 06:20:12 debian libvirtd[3363]: 2019-01-29 11:20:12.116+0000: 3363: warning : virNetTLSContextCheckCertificate:1125 : Certificate check failed Unable to verify TLS peer: No certificate was found.
Jan 29 06:20:12 debian libvirtd[3363]: 2019-01-29 11:20:12.116+0000: 3363: error : virNetTLSContextCheckCertificate:1128 : authentication failed: Failed to verify peer's certificate

Jan 29 06:20:12 debian libvirtd[3363]: 2019-01-29 11:20:12.116+0000: 3363: error : virNetTLSContextValidCertificate:994 : Unable to verify TLS peer: No certificate was found.

Jan 29 06:20:12 debian libvirtd[3363]: 2019-01-29 11:20:12.116+0000: 3363: warning : virNetTLSContextCheckCertificate:1125 : Certificate check failed Unable to verify TLS peer: No certificate was found.

Jan 29 06:20:12 debian libvirtd[3363]: 2019-01-29 11:20:12.116+0000: 3363: error : virNetTLSContextCheckCertificate:1128 : authentication failed: Failed to verify peer's certificate

使用TLS连接libvirtd时，默认需要验证客户端是否拥有由CA签名的证书，详见libvirtd配置文件（/etc/libvirt/libvirtd.conf）：

# Flag to disable verification of client certificates
#
# Client certificate verification is the primary authentication mechanism.
# Any client which does not present a certificate signed by the CA
# will be rejected.
#
# Default is to always verify. Uncommenting this will disable
# verification - make sure an IP whitelist is set
#tls_no_verify_certificate = 1

# Flag to disable verification of client certificates

# Client certificate verification is the primary authentication mechanism.

# Any client which does not present a certificate signed by the CA

# will be rejected.

# Default is to always verify. Uncommenting this will disable

# verification - make sure an IP whitelist is set

#tls_no_verify_certificate = 1

取消此行的注释即可关闭此验证（当然你得启用其它身份认证途径）：

#tls_no_verify_certificate = 1

1	#tls_no_verify_certificate = 1

也可选择以让CA给客户端签发证书：

# 自行修改CA私钥与证书的路径
CA_KEY_FILE_PATH="/etc/pki/libvirt/private/cakey.pem"
CA_CERTIFICATE_FILE_PATH="/etc/pki/CA/cacert.pem"

# 客户端私钥，CSR，证书保存路径
PKI_PREFIX="/tmp/client_pki/"
CLIENT_PRIVATE_KEY_FILE_PATH="${PKI_PREFIX}/clientkey.pem"
CLIENT_CSR_FILE_PATH="${PKI_PREFIX}/clientcsr.pem"
CLIENT_CERTIFICATE_FILE_PATH="${PKI_PREFIX}/clientcert.pem"

oldUmask=$(umask)
umask 0077
mkdir -p ${PKI_PREFIX}

# 生成客户端私钥
openssl genrsa -out ${CLIENT_PRIVATE_KEY_FILE_PATH} 2048
# 生成客户端CSR，根据自己需要自行修改subjects
openssl req -new -key ${CLIENT_PRIVATE_KEY_FILE_PATH} -subj "/C=CN/ST=China/L=China/O=China/OU=China/CN=ClientCommonName" -out ${CLIENT_CSR_FILE_PATH}
# CA通过CSR签发证书，根据需要自行修改有效期
openssl x509 -req -in ${CLIENT_CSR_FILE_PATH} -CA ${CA_CERTIFICATE_FILE_PATH} -CAkey ${CA_KEY_FILE_PATH} -CAcreateserial -out ${CLIENT_CERTIFICATE_FILE_PATH} -days 3650
# 把CA证书放进去
cat ${CA_CERTIFICATE_FILE_PATH} > ${PKI_PREFIX}/cacert.pem

umask ${oldUmask}

# 把${PKI_PREFIX}传送到客户端
# scp -r ${PKI_PREFIX} username@host:~/libvirt-server-pki

# 自行修改CA私钥与证书的路径

CA_KEY_FILE_PATH="/etc/pki/libvirt/private/cakey.pem"

CA_CERTIFICATE_FILE_PATH="/etc/pki/CA/cacert.pem"

# 客户端私钥，CSR，证书保存路径

PKI_PREFIX="/tmp/client_pki/"

CLIENT_PRIVATE_KEY_FILE_PATH="${PKI_PREFIX}/clientkey.pem"

CLIENT_CSR_FILE_PATH="${PKI_PREFIX}/clientcsr.pem"

CLIENT_CERTIFICATE_FILE_PATH="${PKI_PREFIX}/clientcert.pem"

oldUmask=$(umask)

umask 0077

mkdir -p ${PKI_PREFIX}

# 生成客户端私钥

openssl genrsa -out ${CLIENT_PRIVATE_KEY_FILE_PATH} 2048

# 生成客户端CSR，根据自己需要自行修改subjects

openssl req -new -key ${CLIENT_PRIVATE_KEY_FILE_PATH} -subj "/C=CN/ST=China/L=China/O=China/OU=China/CN=ClientCommonName" -out ${CLIENT_CSR_FILE_PATH}

# CA通过CSR签发证书，根据需要自行修改有效期

openssl x509 -req -in ${CLIENT_CSR_FILE_PATH} -CA ${CA_CERTIFICATE_FILE_PATH} -CAkey ${CA_KEY_FILE_PATH} -CAcreateserial -out ${CLIENT_CERTIFICATE_FILE_PATH} -days 3650

# 把CA证书放进去

cat ${CA_CERTIFICATE_FILE_PATH} > ${PKI_PREFIX}/cacert.pem

umask ${oldUmask}

# 把${PKI_PREFIX}传送到客户端

# scp -r ${PKI_PREFIX} username@host:~/libvirt-server-pki

完成上面的操作后，把整个${PKI_PRE […]

Unix Like

[Updated] libvirt & qemu change VNC password without restart

文章【libvirt & qemu无需重启（在线）更改VNC密码】已于2019-02-17更新。有两个方法，一个是通过libvirt的virDomainUpdateDeviceFlags接口，另一个是通过qemu-monitor。以下把“DOMAIN_NAME”替换为虚拟机的名称，“YOU_NEW_VNC_PASSWORD”替换为你的新密码。通过virDomainUpdateDeviceFlags接口使用libvirt管理虚拟机的情况下，这个方法是首选，libvirt官方是不推荐使用了libvirt的情况下操纵qemu-monitor的。首先编写VNC graphic的XML […]