linux在线增加删除存储lun测试报告

linux-dash

A beautiful web dashboard for Linux

项目地址：https://gitcode.com/gh_mirrors/li/linux-dash

免费下载资源

bigclouder

7748人浏览 · 2015-02-05 14:39:36

bigclouder · 2015-02-05 14:39:36 发布

l 随着业务数据的快速增长，我们经常需要对数据库空间进行扩容，以满足空间快速增长的需要。目前数据库使用存储做为存储介质，数据库数据文件、控制文件、在线重做日志均存储在存储上。

AIX操作系统增加存储能够做到在线认盘。Linux操作系统目前的操作是增加存储lun后，重启服务器来认盘。

l 为了进一步提高业务可用性，减少停机时间。探索在linux平台上也实现增加存储后能在线认盘。

本次测试基于redhat 5.4、redhat 6.2两个操作系统版本，测试在有压力的情况下增加存储lun。

二、多路径软件安装

操作系统为redhat6.2时，多路径软件版本：hdlm7.2

操作系统为redhat5.4时，多路径软件版本：hdlm6.3

两个版本的安装步骤一样，以hdlm6.3为例：

1）解压缩hdlm6.3_linux.zip

unziphdlm6.3_linux.zip

2）把所有文件的权限改777

chmod -R777 *

3）把license拷贝到/var/tmp

cphdlm_license /var/tmp

4）安装

./installhdlm

5）重启服务器

三、在线增加存储lun

Redhat 6.2 和redhat 5.4操作系统版本下，步骤稍有不同。Redhat 6.2环境下可自动认盘，redhat 5.4环境下需要手动认盘。下面分别描述具体步骤：

1. redhat 6.2操作系统下，在线增加存储lun

操作系统：Red HatEnterprise Linux Server release 6.2 (Santiago) x86_64

存储： Hitachi storage HUS150

在测试过程中，运行IO测试工具orion：

#./orion_linux_x86-64 -run oltp -testname test-num_disks 4

1.1先用管理软件把新增的lun分配到相应的host group.

测试环境hitachi管理软件：http://192.168.6.38:23015/StorageNavigatorModular/jsp/Title.jsp

选择存储array name: HUS150_93012124

Lun: 0600 当前链路0

操作host group: TESTZHF_0A、TESTZHF_1A、TESTZHF_0B、TESTZHF_1B

Edit host group—available volumes, 把lun 0600 分别加到4个host group, 因为有四条链路。

现在每个host group有5个lun: 0104，0200，0300，0404，0600

1.2.linux上，动态探测新设备的SCSI/FC bus

操作系统版本redhat 6 , QLogic HBA驱动

# echo 1 > /sys/class/fc_host/host1/issue_lip

# echo 1 > /sys/class/fc_host/host2/issue_lip

# echo "- - -" >/sys/class/scsi_host/host1/scan

# echo "- - -" >/sys/class/scsi_host/host2/scan

注：/sys/class/scsi_host/host1/scan

Host后的数值n表示: host port number

1.3．检查操作系统是否认到盘

#cat/proc/scsi/scsi

Attached devices:

Host: scsi0 Channel: 00 Id: 32 Lun: 00

Vendor:DP Model: BACKPLANE Rev: 1.09

Type: Enclosure ANSI SCSI revision: 05

Host: scsi0 Channel: 02 Id: 00 Lun: 00

Vendor:DELL Model: PERC H700 Rev: 2.10

Type: Direct-Access ANSI SCSI revision: 05

Host: scsi1 Channel: 00 Id: 00 Lun: 00

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi1 Channel: 00 Id: 00 Lun: 01

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi1 Channel: 00 Id: 00 Lun: 02

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi1 Channel: 00 Id: 00 Lun: 03

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi1 Channel: 00 Id: 01 Lun: 00

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi1 Channel: 00 Id: 01 Lun: 01

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi1 Channel: 00 Id: 01 Lun: 02

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi1 Channel: 00 Id: 01 Lun: 03

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2 Channel: 00 Id: 00 Lun: 00

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2 Channel: 00 Id: 00 Lun: 01

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2 Channel: 00 Id: 00 Lun: 02

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2 Channel: 00 Id: 00 Lun: 03

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2 Channel: 00 Id: 01 Lun: 00

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2 Channel: 00 Id: 01 Lun: 01

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2 Channel: 00 Id: 01 Lun: 02

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2 Channel: 00 Id: 01 Lun: 03

Vendor:HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi1Channel: 00 Id: 00 Lun: 04

Vendor: HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi1Channel: 00 Id: 01 Lun: 04

Vendor: HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2Channel: 00 Id: 00 Lun: 04

Vendor: HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

Host: scsi2Channel: 00 Id: 01 Lun: 04

Vendor: HITACHI Model: DF600F Rev: 0000

Type: Direct-Access ANSI SCSI revision: 04

lun编码为04的即为新加的盘，已认到

1.4.更新HDLM配置信息

#/sbin/dlmcfgmgr -r

KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y

KAPL10341-I The HDLM device configurations havebeen changed.

KAPL10302-I /sbin/dlmcfgmgr completed normally.

1.5.确认结果

#/sbin/dlmcfgmgr -v

HDevName Management Device Host Channel Target Lun

/dev/sddlmaa configured /dev/sdb 1 0 0 0

/dev/sdf 1 0 1 0

/dev/sdj 2 0 0 0

/dev/sdn 2 0 1 0

/dev/sddlmab configured /dev/sdc 1 0 0 1

/dev/sdg 1 0 1 1

/dev/sdk 2 0 0 1

/dev/sdo 2 0 1 1

/dev/sddlmac configured /dev/sdd 1 0 0 2

/dev/sdh 1 0 1 2

/dev/sdl 2 0 0 2

/dev/sdp 2 0 1 2

/dev/sddlmad configured /dev/sde 1 0 0 3

/dev/sdi 1 0 1 3

/dev/sdm 2 0 0 3

/dev/sdq 2 0 1 3

/dev/sddlmae configured /dev/sdr 1 0 0 4

/dev/sds 1 0 1 4

/dev/sdt 2 0 0 4

/dev/sdu 2 0 1 4

KAPL10302-I /sbin/dlmcfgmgr completed normally.

--/dev/sddlmae为新加的盘.

如果/dev/sddlmae 状态为unconfigured，表示未被HDLM多路径软件管理。

执行以下命令让HDLM管理/dev/sddlmae

#/sbin/dlmcfgmgr -i /dev/sddlmae

再次确认：

#/sbin/dlmcfgmgr -v

查看多路径信息，已能识别设备： /dev/sddlmae，并且4条链路都是online.

#/opt/DynamicLinkManager/bin/dlnkmgr view -lu

Product : DF600F

SerialNumber : 93012124

LUs : 5

iLU HDevName Device PathIDStatus

0104 sddlmaa /dev/sdb 000000 Online

/dev/sdf 000004 Online

/dev/sdj 000008 Online

/dev/sdn 000009 Online

0200 sddlmab /dev/sdc 000001 Online

/dev/sdg 000005 Online

/dev/sdk 000010 Online

/dev/sdo 000011 Online

0300 sddlmac /dev/sdd 000002 Online

/dev/sdh 000006 Online

/dev/sdl 000012 Online

/dev/sdp 000013 Online

0404 sddlmad /dev/sde 000003 Online

/dev/sdi 000007 Online

/dev/sdm 000014 Online

/dev/sdq 000015 Online

0600 sddlmae /dev/sdr 000016 Online

/dev/sds 000017 Online

/dev/sdt 000018 Online

/dev/sdu 000019 Online

KAPL01001-I The HDLM command completed normally.Operation name = view, completion time = 2014/01/10 15:34:02

Orion测试结果显示，现有存储lun无明显异常。测试结果见：《redhat 5.4在线认盘_orion测试结果》

2． Redhat 5.4操作系统下，在线增加存储lun

此处以生产历史库的操作为例。

操作系统版本：Red HatEnterprise Linux Server release 5.4 (Tikanga) x86_64

存储：Hitachi storage AMS2300

2.1先用管理软件把新增的lun分配到相应的host group.

新增lun id: 2000

2.2 手动添加SCSI设备

# cd/proc/scsi/qla2xxx/

# ls

0 1

使用以下命令：

# echo"scsi-qlascan" > /proc/scsi/qla2xxx/0

# echo"scsi-qlascan" > /proc/scsi/qla2xxx/1

查看：Id:Lun部分

# cat/proc/scsi/qla2xxx/0

( 0: 7):Total reqs 0, Pending reqs 0, flags 0x0*, Dflags 0x0, 0:0:81 00

( 1: 7):Total reqs 0, Pending reqs 0, flags 0x0*, Dflags 0x0, 0:0:82 00

# cat/proc/scsi/qla2xxx/1

( 0: 7):Total reqs 0, Pending reqs 0, flags 0x0*, Dflags 0x0, 1:0:81 00

( 1: 7):Total reqs 0, Pending reqs 0, flags 0x0*, Dflags 0x0, 1:0:82 00

手动加入:

# echo"scsi add-single-device 0 0 0 7" > /proc/scsi/scsi

# echo"scsi add-single-device 0 0 1 7" > /proc/scsi/scsi

# echo"scsi add-single-device 1 0 0 7" > /proc/scsi/scsi

# echo"scsi add-single-device 1 0 1 7" > /proc/scsi/scsi

验证:

cat/proc/scsi/scsi

2.3更新HDLM配置信息

#/sbin/dlmcfgmgr -r

KAPL10339-IThis operation will change the configuration of HDLM devices. Do you want tocontinue? [y/n]: y

KAPL10341-IThe HDLM device configurations have been changed.

KAPL10302-I/sbin/dlmcfgmgr completed normally.

2.4 确认结果

#/sbin/dlmcfgmgr -v

/dev/sddlmah configured /dev/sdad 0 0 0 7

/dev/sdae 0 0 1 7

/dev/sdaf 1 0 0 7

/dev/sdag 1 0 1 7

验证发现已认到盘

四、在线认到lun后，重启服务器看路径是否变化

此处以生产历史库的操作为例。新增加的lun id: 2000 设备名称:/dev/sddlmah

测试过程概述：先在新增的lun上创建文件系统并挂载，验证是否能正常读写。

重启服务器，看存储链路是否不变，能否正常mount设备，并正常读写。

1．重启前路径状态

[10:12:13 root(grid)@histdb ~]# /sbin/dlmcfgmgr -v

HDevName Management Device Host Channel Target Lun

/dev/sddlmaa configured /dev/sdc 0 0 0 2

/dev/sdj 0 0 1 2

/dev/sdq 1 0 0 2

/dev/sdx 1 0 1 2

/dev/sddlmab configured /dev/sdo 1 0 0 0

/dev/sdv 1 0 1 0

/dev/sdh 0 0 1 0

/dev/sda 0 0 0 0

/dev/sddlmac configured /dev/sdp 1 0 0 1

/dev/sdw 1 0 1 1

/dev/sdi 0 0 1 1

/dev/sdb 0 0 0 1

/dev/sddlmad configured /dev/sdd 0 0 0 3

/dev/sdk 0 0 1 3

/dev/sdr 1 0 0 3

/dev/sdy 1 0 1 3

/dev/sddlmae configured /dev/sde 0 0 0 4

/dev/sdl 0 0 1 4

/dev/sds 1 0 0 4

/dev/sdz 1 0 1 4

/dev/sddlmaf configured /dev/sdf 0 0 0 5

/dev/sdm 0 0 1 5

/dev/sdt 1 0 0 5

/dev/sdaa 1 0 1 5

/dev/sddlmag configured /dev/sdg 0 0 0 6

/dev/sdn 0 0 1 6

/dev/sdu 1 0 0 6

/dev/sdab 1 0 1 6

/dev/sddlmah configured /dev/sdad 0 0 0 7

/dev/sdae 0 0 1 7

/dev/sdaf 1 0 0 7

/dev/sdag 1 0 1 7

[09:38:15 root(grid)@histdb bin]# ./dlnkmgr view-lu

Product : AMS

SerialNumber : 85003997

LUs : 8

iLU HDevName Device PathIDStatus

0021 sddlmab /dev/sda 000000 Online

/dev/sdh 000006 Online

/dev/sdo 000012 Online

/dev/sdv 000013 Online

0022 sddlmac /dev/sdb 000001 Online

/dev/sdi 000007 Online

/dev/sdp 000014 Online

/dev/sdw 000015 Online

0023 sddlmaa /dev/sdc 000002 Online

/dev/sdj 000008 Online

/dev/sdq 000016 Online

/dev/sdx 000017 Online

0024 sddlmad /dev/sdd 000003 Online

/dev/sdk 000009 Online

/dev/sdr 000018 Online

/dev/sdy 000019 Online

0025 sddlmae /dev/sde 000004 Online

/dev/sdl 000010 Online

/dev/sds 000020 Online

/dev/sdz 000021 Online

0026 sddlmaf /dev/sdf 000005 Online

/dev/sdm 000011 Online

/dev/sdt 000022 Online

/dev/sdaa 000023 Online

0027 sddlmag /dev/sdg 000024 Online

/dev/sdn 000025 Online

/dev/sdu 000026 Online

/dev/sdab 000027 Online

2000 sddlmah /dev/sdad 000028 Online

/dev/sdae 000029 Online

/dev/sdaf 000030 Online

/dev/sdag 000031Online

2．/dev/sddlmah划分区,整个lun划一个分区

# fdisk /dev/sddlmah

Device contains neither a valid DOS partitiontable, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel. Changes will remainin memory only,

until you decide to write them. After that, ofcourse, the previous

content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4will be corrected by w(rite)

Command (m for help): p

Disk /dev/sddlmah: 1073 MB, 1073741824 bytes

34 heads, 61 sectors/track, 1011 cylinders

Units = cylinders of 2074 * 512 = 1061888 bytes

Device Boot Start End Blocks Id System

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1011, default 1): 1

Last cylinder or +size or +sizeM or +sizeK(1-1011, default 1011):

Using default value 1011

Command (m for help): p

Disk /dev/sddlmah: 1073 MB, 1073741824 bytes

34 heads, 61 sectors/track, 1011 cylinders

Units = cylinders of 2074 * 512 = 1061888 bytes

Device Boot Start End Blocks Id System

/dev/sddlmah1 1 1011 1048376+ 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

3．创建文件系统并挂载

mkfs.ext3 /dev/sddlmah1

创建目录： mkdir /data1

在/etc/fstab中加入:

/dev/sddlmah1 /data1 ext3 defaults 1 0

挂载目录:

mount /data1 //此处可以确认/etc/fstab中是否正确

4.测试能否正常读写

cd /data1

vi test.txt

hello~

online test for read write on /dev/sddlmah.

cat test.txt

5.重启

1）关闭数据库

$jdb

$sqlplus /”as sysdba”

alter system archive log current;

alter system checkpoint;

shutdown immediate;

2）关闭ASM

$jgrid

$sqlplus /”as sysdba”

shutdown immediate;

3）关闭listener

$jdb

$lsnrctl status

$lsnrctl stop

4) check

$jdb

$lsnrctl status

$ps -ef|grep ora_

$ps -ef|grep asm

5)重启服务器

#sync;sync;sync;

#reboot

6.服务器重启后启动数据库

查看crs资源是否启动：./crsctl stat res –t

启动listener:

$jdb

$lsnrctl start

启动asm:

$srvctl start asm

启动数据库：

$sqlplus /”as sysdba”

Startup;

7.检查新加lun的盘符

# /sbin/dlmcfgmgr -v

#/opt/DynamicLinkManager/bin/dlnkmgr view -lu

盘的 Host ID (host port number)、 Channel number (bus number)、Target ID、 Lun (host LUnumber)

不变

8.验证重启后能否正常读写

cd /data1

vi testaf.txt

hello~

read write test on /dev/sddlmah after reboot.

9测试完成,umount /data1

#umount /data1

#vi /etc/fstab

删除这行:/dev/sddlmah1 /data1 ext3 defaults 1 0

五、在线删除lun

经测试redhat 5.4与redhat 6.2环境下的操作步骤相似，这里以redhat 6.2环境下为例展示具体操作步骤。删除lun过程中，数据库运行压力测试。收集每秒IOSTAT信息

1) 停止访问需要删除的LUN

2) 用管理软件从相应的host group中去除lun,即删除链路

3) 删除scsi设备

从hdlm管理中去除/dev/sddlmae

#dlmcfgmgr -o /dev/sddlmae

KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y

KAPL10341-I The HDLM device configurations havebeen changed.

KAPL10302-I /sbin/dlmcfgmgr completed normally.

去除多路径：

#echo "scsi remove-single-device 1 0 04" > /proc/scsi/scsi

#echo "scsi remove-single-device 2 0 04" > /proc/scsi/scsi

#echo "scsi remove-single-device 2 0 14" > /proc/scsi/scsi

#echo "scsi remove-single-device 1 0 14" > /proc/scsi/scsi

说明：以上的数值a,b,c,d 分别代表

a: Host ID (host port number)

b: Channel number (bus number)

c: Target ID

d: Lun (host LU number)

以上值可以在/sbin/dlmcfgmgr -v命令看到

#dlmcfgmgr -i /dev/sddlmae

KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y

KAPL10341-I The HDLM device configurations havebeen changed.

KAPL10302-I /sbin/dlmcfgmgr completed normally.

4）执行HDLM配置定义，使变更生效

#/sbin/dlmcfgmgr -r

KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y

KAPL10302-I /sbin/dlmcfgmgr completed normally.

5）确认操作,被删除的设备前有“-”

[13:38:26 root(grid)@testzhf zhf] #/sbin/dlmcfgmgr-v

HDevName Management Device Host Channel Target Lun

/dev/sddlmaa configured /dev/sdb 1 0 0 0

/dev/sdq 1 0 1 0

/dev/sdg 2 0 0 0

/dev/sdl 2 0 1 0

/dev/sddlmab configured /dev/sdc 1 0 0 1

/dev/sdr 1 0 1 1

/dev/sdh 2 0 0 1

/dev/sdm 2 0 1 1

/dev/sddlmac configured /dev/sdd 1 0 0 2

/dev/sds 1 0 1 2

/dev/sdi 2 0 0 2

/dev/sdn 2 0 1 2

/dev/sddlmad configured /dev/sde 1 0 0 3

/dev/sdt 1 0 1 3

/dev/sdj 2 0 0 3

/dev/sdo 2 0 1 3

/dev/sddlmae configured - 1 0 0 4

- 1 0 1 4

- 2 0 0 4

- 2 0 1 4

KAPL10302-I /sbin/dlmcfgmgr completed normally.

6) 从HDLM中删除路径

针对被删除lun执行以下命令，从HDLM中删除路径

#/sbin/dlmcfgmgr -u /dev/sddlmae

KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y

KAPL10341-I The HDLM device configurations havebeen changed.

KAPL10302-I /sbin/dlmcfgmgr completed normally.

7) 确认lun已被删除

#/sbin/dlmcfgmgr -v

HDevName Management Device Host Channel Target Lun

/dev/sddlmaa configured /dev/sdb 1 0 0 0

/dev/sdq 1 0 1 0

/dev/sdg 2 0 0 0

/dev/sdl 2 0 1 0

/dev/sddlmab configured /dev/sdc 1 0 0 1

/dev/sdr 1 0 1 1

/dev/sdh 2 0 0 1

/dev/sdm 2 0 1 1

/dev/sddlmac configured /dev/sdd 1 0 0 2

/dev/sds 1 0 1 2

/dev/sdi 2 0 0 2

/dev/sdn 2 0 1 2

/dev/sddlmad configured /dev/sde 1 0 0 3

/dev/sdt 1 0 1 3

/dev/sdj 2 0 0 3

/dev/sdo 2 0 1 3

KAPL10302-I /sbin/dlmcfgmgr completed normally.

在执行以上操作时，数据库不停止，仍运行压力测试，收集iostat数据

iostat -xmnztd 1 >/home/oracle/zhf/storage_scan_test/iostat_0108.txt

iostat数据分析结果:iostat 数据连续，IO无明显异常

iostat 结果详见：iostat_0108_在线删lun_redhat6.2.txt

GitHub 加速计划 / li / linux-dash

10.39 K

1.2 K

下载

A beautiful web dashboard for Linux

最近提交(Master分支：2 个月前 )

186a802e added ecosystem file for PM2 4 年前

5def40a3 Add host customization support for the NodeJS version 4 年前

GitCode 开源社区

旨在为数千万中国开发者提供一个无缝且高效的云端环境，以支持学习、使用和贡献开源项目。

更多推荐

[转载]在Windows环境下安装GNU Radio

转自：在Windows环境下安装GNURadio_恐弱智_新浪博客GNU Radio是用Python开发的，大部分开源的工程能够在Linux环境下运行良好，而Windows下却运行的很勉强，而且安装配置都很复杂。GNU Radio算是个例外了，不光提供了Windows的二进制安装，还有比较详细的说明。我是Python小白，所以折腾了好久才弄好，特意记录下来，免得以后再装还折腾。GNU Radio的

GitCode 开源社区

centOS 8 使用dnf安装Docker

DNF是什么？CentOS 8使用YUM软件包管理器版本v4.0.4。现在，该版本使用DNF(已删除YUM)。DNF是软件包管理器。它会在Linux发行版上安装，执行更新并删除软件包。使用DNF安装Docker跳过具有损坏依赖性的程序包一个有效的解决方案是使您的CentOS 8系统使用以下--nobest命令安装最符合条件的版本：sudo dnf install docker...

GitCode 开源社区

定时同步数据库表(mysql+linux+crontab)

sync.sh里面的参数需要改变，ip/username/password/database/tablesync.sh#!/bin/sh# Please change the IP and password of the data source db.# Then change the table name.filename=/home/nington/db/$(date +%Y-%m

GitCode 开源社区

所有评论(0)

查看更多评论

bigclouder

已为社区贡献3条内容