使用Rclone进行文件备份

环境概述

本地环境为由Beijing、shanghai、guangzhou三台Linux主机构成的数据备份源；
远端备份目的为本地自建S3存储和紫光云存储；
使用rclone将本地多主机文件备份到远端存储；

具体工具有：rclone、minio和sftp。

主机地址

主机名	角色	IP地址
Rclone	同步工具	192.168.174.150
Minio	开源S3服务器	192.168.174.160
UNicloud	公网S3服务器	*
Beijing	备份站点	192.168.174.141
Shanghai	备份站点	192.168.174.142
Guangzhou	备份站点	192.168.174.143

Minio的建立

Mnino是一个轻巧、快速的开源S3服务器，其官方主页为：minio https://min.io/

# 下载二进制执行文件
wget https://dl.minio.io/server/minio/release/linux-amd64/minio
# 添加可执行权限
chmod +x minio
# 复制到指定路径
cp minio /usr/local/bin/
# 建立程序运行账户
useradd -r minio -s /sbin/nologin
# 对程序属主进行配置
chown minio:minio /usr/local/bin/minio
# 配置程序启动参数
touch /etc/default/minio
cat>/etc/default/minio<<EOF
MINIO_VOLUMES="/usr/local/share/minio/"
MINIO_OPTS="-C /etc/minio --address 192.168.174.160:9000"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="miniostorage"
EOF
# 配置systemctl管理程序
touch minio.service
cat >> minio.service <<EOF
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local/

User=minio
Group=minio

EnvironmentFile=/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"

ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target
# Built for ${project.name}-${project.version} (${project.name})
EOF

# 复制配置文件到指定位置
cp minio.service /etc/systemd/system
systemctl daemon-reload
systemctl enable minio.service --now

# 配置防火墙
firewall-cmd --permanent --add-service={ssh,http,https}
firewall-cmd --permanent --add-port=9000/tcp
firewall-cmd --reload

建立桶存储并上传文件

[root@minio ~]# ls /usr/local/share/minio/
test
[root@minio ~]# ls /usr/local/share/minio/test/
This is MinIO.mp4

接下来，我们可以使用官方客户端mc来访问我们建立的桶：

wget http://dl.minio.org.cn/client/mc/release/linux-amd64/mc
chmod +x mc
./mc config host add minio http://192.168.174.160:9000 minio miniostroage
[root@minio ~]# ./mc ls minio
[2020-06-01 23:50:56 CST]      0B test/
[root@minio ~]# ./mc ls minio/test/
[2020-06-01 23:51:01 CST]  8.7MiB This is MinIO.mp4

Rclone部署

rclone的部署很简单，从官方网站下载安装rclone包即可。

wget https://downloads.rclone.org/v1.52.0/rclone-v1.52.0-linux-amd64.rpm
yum localinstall -y rclone-v1.52.0-linux-amd64.rpm

然后，使用“rclone config”进行配置即可。配置完成的配置文件如下：

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = minio
secret_access_key = miniostorage
region = us-east-1
endpoint = http://192.168.174.160:9000

[unicloud]
type = s3
provider = Other
env_auth = false
access_key_id = __
secret_access_key = ______
endpoint = ______.unicloudsrv.com
acl = private

接下来，我们进行测试：

[root@rclone ~]# rclone lsd minio:
          -1 2020-06-01 23:50:56        -1 test
[root@rclone ~]# rclone ls minio:test
  9117336 This is MinIO.mp4
[root@rclone ~]# rclone copy rclone-v1.52.0-linux-amd64.rpm minio:test
[root@rclone ~]# rclone ls minio:test
  9117336 This is MinIO.mp4
 13825921 rclone-v1.52.0-linux-amd64.rpm

备份其他上传

各个服务器需要开启时间同步服务并配置rclone主机同各个备份主机的ssh免密登录；
使用rclone config 建立sftp类型的配置文件；

测试

[root@beijing opt]# touch beijing/192.168.174.141
[root@beijing opt]# hostname >> ./beijing/192.168.174.141 

[root@rclone ~]# rclone copy beijing:/opt/beijing minio:/test
[root@rclone ~]# rclone ls minio:/test/
        9 192.168.174.141
  9117336 This is MinIO.mp4
 13825921 rclone-v1.52.0-linux-amd64.rpm
# 在公有云新建桶
[root@rclone ~]# rclone mkdir unicloud:sujx
[root@rclone ~]# rclone lsd unicloud:
          -1 2020-05-27 14:16:27        -1 backup-ucd-public-prod
          -1 2020-05-27 14:16:22        -1 backup-ucd-public-test
          -1 2020-05-20 09:20:30        -1 crmbak
          -1 2020-05-25 16:36:46        -1 fs-pcf
          -1 2020-05-25 16:36:35        -1 fs-pcf-temp
          -1 2020-06-01 19:28:14        -1 sujx
# 在本地S3和远端S3 同步
[root@rclone ~]# rclone sync minio:test unicloud:sujx
[root@rclone ~]# rclone ls unicloud:sujx
        9 192.168.174.141
  9117336 This is MinIO.mp4
 13825921 rclone-v1.52.0-linux-amd64.rpm