1 前情提要
话说上次DBA小倩通过删除home lv，把空间扩给了/分区，问题暂时得到了解决。
没过几天，领导找到小倩下达任务，客户说数据库在本地磁盘空间太小了又快要满了，由于之前用的服务器本地磁盘，性能也比较差，现在不想扩展服务器本地硬盘了。正好这台服务器有HBA卡，客户从别的高端存储上划了个2TB的LUN过来，映射给这台数据库服务器，想让DBA小倩来做一下数据迁移，当然停机时间己经客户己经安排好了，就差小倩上线了。
2 准备工作
正式停机之前还有一些准备工作要作，比如先把存储空间挂载到服务器上（后做也可以，用不了多久）。
检查一下当前磁盘空间情况，根目录满上要用光了。
[root@dbserver ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 1.9G 259M 1.6G 14% /dev/shm
tmpfs tmpfs 1.9G 12M 1.9G 1% /run
tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 496G 493G 2.8G 100% /
/dev/sda1 xfs 1014M 151M 864M 15% /boot
tmpfs tmpfs 378M 0 378M 0% /run/user/0
再看一下磁盘情况，目前只是识别到了一块磁盘
[root@dbserver ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 500G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 499G 0 part ├─centos-root 253:0 0 495.1G 0 lvm /└─centos-swap 253:1 0 3.9G 0 lvm [SWAP]
sr0 11:0 1 4.4G 0 rom
下面说一下怎么识别磁盘，正常这个操作是需要客户的系统管理员来操作（非必要DBA不要操作）。我这里用虚拟机给大家演示，就直接简化操作了（实际生产环境有可能还需要配置ISCSI、多路径，比较复杂，一般有专人处理，这个放在后面单独一章简单给大家做个知识补充）
我这里用虚拟机模拟，那么操作就是这样，扫描之后发现2T磁盘
[root@dbserver scsi_host]# lsscsi
[0:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda
[2:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0
[root@dbserver scsi_host]# echo "- - -" >/sys/class/scsi_host/host0/scan
[root@dbserver scsi_host]#
[root@dbserver scsi_host]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 500G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 499G 0 part ├─centos-root 253:0 0 495.1G 0 lvm /└─centos-swap 253:1 0 3.9G 0 lvm [SWAP]
sdb 8:16 0 2T 0 disk
sr0 11:0 1 4.4G 0 rom
下面对2T磁盘进行分区，并标记为LVM格式（划分区再标记这是标准做法，这样避免以后被别人当成空盘，直接给用了。当然也可以直接pvcreate /dev/sdb，但是这样就没有标记了。）
[root@dbserver ~]# parted -s /dev/sdb mklabel gpt
[root@dbserver ~]# parted -s /dev/sdb mkpart primary 2048s 100%
[root@dbserver ~]# parted -s /dev/sdb toggle 1 lvm[root@dbserver ~]# fdisk -l /dev/sdb
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.Disk /dev/sdb: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
Disk identifier: 96DE5EB7-29A5-42AA-9651-03995AF95DEC# Start End Size Type Name1 2048 4294965247 2T Linux LVM primary下一步就是创建lvm，创建挂载点，挂载到操作系统上。
[root@dbserver ~]# pvcreate /dev/sdb1 Physical volume "/dev/sdb1" successfully created.
[root@dbserver ~]# vgcreate oradatavg /dev/sdb1 Volume group "oradatavg" successfully created
[root@dbserver ~]# lvcreate -l 100%FREE -n oralv oradatavgLogical volume "oralv" created.
[root@dbserver ~]# lvsLV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convertroot centos -wi-ao---- 495.12g swap centos -wi-ao---- <3.88g oralv oradatavg -wi-a----- <2.00t
[root@dbserver ~]# vgsVG #PV #LV #SN Attr VSize VFreecentos 1 2 0 wz--n- <499.00g 0 oradatavg 1 1 0 wz--n- <2.00t 0
[root@dbserver ~]# mkfs.xfs /dev/mapper/oradatavg-oralv
meta-data=/dev/mapper/oradatavg-oralv isize=512 agcount=4, agsize=134217472 blks= sectsz=512 attr=2, projid32bit=1= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=536869888, imaxpct=5= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=262143, version=2= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
3 开始迁移
开始停机，先让客户结束业务办理，把应用系统都关了，然后建议是先把LOCAL=NO进程全KILL，不然生产库一般很难正常shutdown immediate。操作步骤：先杀进程、关监听、关库、关EM
ps -ef |grep LOCAL=NO|grep -v grep|cut -c 9-15|xargs kill -9
[oracle@dbserver ~]$ lsnrctl stopLSNRCTL for Linux: Version 11.2.0.4.0 - Production on 11-SEP-2024 23:07:17Copyright (c) 1991, 2013, Oracle. All rights reserved.Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
The command completed successfully
[oracle@dbserver ~]$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.4.0 Production on Wed Sep 11 23:07:00 2024Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options[oracle@dbserver ~]$ emctl stop dbconsole
OC4J Configuration issue. /u01/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/OC4J_DBConsole_dbserver_oracle not found.
操作步骤是这样：
1、先重命名/u01目录为/u01.old
2、创建新的/u01目录，然后挂载/dev/mapper/oradatavg-oralv到/u01
3、给/u01目录赋予oracle:oinstall权限
4、使用rsync迁移/u01.old目录里的文件到/u01目录下（这里写成脚本用nohup运行，防止终端断开），这里注意rsync源目录后面要带/，就是比如这样/u01.old/
5、启动数据库恢复业务
[root@dbserver ~]#cd /
[root@dbserver /]#mv /u01 /u01.old
[root@dbserver /]# mkdir -p /u01
[root@dbserver /]# echo "/dev/mapper/oradatavg-oralv /u01 xfs defaults 0 0" >> /etc/fstab
[root@dbserver /]# mount -a
[root@dbserver /]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 12M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root 496G 493G 2.8G 100% /
/dev/sda1 1014M 151M 864M 15% /boot
tmpfs 378M 0 378M 0% /run/user/0
/dev/mapper/oradatavg-oralv 2.0T 33M 2.0T 1% /u01
[root@dbserver /]# chown oracle:oinstall /u01
[root@dbserver /]# echo "/usr/bin/rsync -av /u01.old/ /u01" >rsync.sh
[root@dbserver /]# chmod +x rsync.sh
[root@dbserver /]# nohup ./rsync.sh &
#如果需要查看同步情况,最后显示完成
[root@dbserver /]# tail -f nohup.out
app/oracle/product/11.2.0/dbhome_1/xdk/mesg/
app/oracle/product/11.2.0/dbhome_1/xdk/mesg/lpxus.msb
app/oracle/product/11.2.0/dbhome_1/xdk/mesg/lpxus.msg
app/oracle/product/11.2.0/dbhome_1/xdk/mesg/lpxzhs.msb
app/oracle/product/11.2.0/dbhome_1/xdk/mesg/lsxus.msb
app/oracle/product/11.2.0/dbhome_1/xdk/mesg/lsxus.msg
app/oracle/product/11.2.0/dbhome_1/xdk/mesg/lsxzhs.msbsent 6,049,894,594 bytes received 709,411 bytes 145,797,686.87 bytes/sec
total size is 6,046,010,967 speedup is 1.00最后启动数据库，成功运行后，确认无错误，就可以把/u01.old目录删除释放空间了。
[oracle@dbserver ~]$ lsnrctl startLSNRCTL for Linux: Version 11.2.0.4.0 - Production on 11-SEP-2024 23:29:57Copyright (c) 1991, 2013, Oracle. All rights reserved.Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr: please wait...TNSLSNR for Linux: Version 11.2.0.4.0 - Production
System parameter file is /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/dbserver/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dbserver)(PORT=1521)))Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date 11-SEP-2024 23:29:57
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/dbserver/listener/alert/log.xml
Listening Endpoints Summary...(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dbserver)(PORT=1521)))
The listener supports no services
The command completed successfully
[oracle@dbserver ~]$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.4.0 Production on Wed Sep 11 23:30:03 2024Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to an idle instance.SQL> startup;
ORACLE instance started.Total System Global Area 1586708480 bytes
Fixed Size 2253624 bytes
Variable Size 989859016 bytes
Database Buffers 587202560 bytes
Redo Buffers 7393280 bytes
Database mounted.
Database opened.
SQL> exit
4 扩展知识
1、如果生产服务器用的是hba卡的话一般都2块卡，需要重新扫描一下hba卡就可以发现新硬盘了，我给大家看一下我这有一个生产环境样例是这样：主机有2块HBA卡为host1、host2。host0是本地的硬盘控制器。
[root@rac1]# cd /sys/class/fc_host/
[root@rac1 fc_host]# ls -l
total 0
lrwxrwxrwx 1 root root 0 Aug 6 03:02 host1 -> ../../devices/pci0000:00/0000:00:03.0/0000:04:00.0/host1/fc_host/host1
lrwxrwxrwx 1 root root 0 Aug 6 03:02 host2 -> ../../devices/pci0000:40/0000:40:02.2/0000:47:00.0/host2/fc_host/host2
[root@rac1 fc_host]# cd /sys/class/scsi_host/
[root@rac1 scsi_host]# ls -l
total 0
lrwxrwxrwx 1 root root 0 Aug 6 03:02 host0 -> ../../devices/pci0000:00/0000:00:02.0/0000:02:00.0/host0/scsi_host/host0
lrwxrwxrwx 1 root root 0 Aug 6 03:02 host1 -> ../../devices/pci0000:00/0000:00:03.0/0000:04:00.0/host1/scsi_host/host1
lrwxrwxrwx 1 root root 0 Aug 6 03:02 host2 -> ../../devices/pci0000:40/0000:40:02.2/0000:47:00.0/host2/scsi_host/host2
[root@rac1 scsi_host]# lsscsi
[0:0:0:0] disk HP LOGICAL VOLUME 4.04 /dev/sda
[0:3:0:0] storage HP P830i 4.04 -
[1:0:0:0] disk 3PARdata VV 3315 /dev/sdb
[1:0:0:1] disk 3PARdata VV 3315 /dev/sdc
[1:0:0:2] disk 3PARdata VV 3315 /dev/sdd
[1:0:0:3] disk 3PARdata VV 3315 /dev/sde
[1:0:0:4] disk 3PARdata VV 3315 /dev/sdf
[1:0:0:254] enclosu 3PARdata SES 3315 -
[1:0:1:0] disk 3PARdata VV 3315 /dev/sdg
[1:0:1:1] disk 3PARdata VV 3315 /dev/sdh
[1:0:1:3] disk 3PARdata VV 3315 /dev/sdi
[1:0:1:4] disk 3PARdata VV 3315 /dev/sdj
[1:0:1:5] disk 3PARdata VV 3315 /dev/sdk
[1:0:1:254] enclosu 3PARdata SES 3315 -
[2:0:0:0] disk 3PARdata VV 3315 /dev/sdl
[2:0:0:1] disk 3PARdata VV 3315 /dev/sdm
[2:0:0:3] disk 3PARdata VV 3315 /dev/sdn
[2:0:0:4] disk 3PARdata VV 3315 /dev/sdo
[2:0:0:5] disk 3PARdata VV 3315 /dev/sdp
[2:0:0:254] enclosu 3PARdata SES 3315 -
[root@rac1 scsi_host]#
那么需要进行的操作就是扫描两块HBA卡。
echo "- - -" >/sys/class/scsi_host/host1/scan
echo "- - -" >/sys/class/scsi_host/host2/scan
2、如果是用万兆ISCSI连接的存储，那么就需要配置ISCSI，主机需要安装iscsi-initiator。生产一般会接2个网卡做为冗余，假设主机这边对接ISCSI存储的IP是192.168.1.2，存储侧是192.168.1.101、192.168.1.102
iscsiadm -m discovery -t st -p 192.168.1.101
iscsiadm -m node -p 192.168.1.101 -l
iscsiadm -m discovery -t st -p 192.168.1.102
iscsiadm -m node -p 192.168.1.102 -l
systemctl enable iscsi.service
iscsiadm -m node -o update -n node.startup -v automatic
3、上面无论是FC-HBA或是ISCSI连接完存储后，剩下需要做的就是配置多路径multipath或者直接安装存储厂商的多路径软件（如果客户现场没有多个存储要同时用的话，推荐使用存储厂商的多路径软件，这样几乎不用啥配置），此处略长，大家要有兴趣我回头再写一篇。
5 未完待续
解决了这个问题，又来了一个棘手的活， 这回客户也是数据库装在根目录，然后根目录总满了，目前是一直手动清日志保证业务能用。但不一样的是，这次客户用的是虚拟机系统，而且虚拟机没用LVM装的系统，客户系统又不能停机，看看下期DBA小倩如何处理。
