【分布式文件存储】MinIO部署及实现文件上传下载_百度小程序

您的位置：上海毫米网络优化公司 > 小程序 > 百度小程序 >

概述

MinIO | 高性能, Kubernetes 原生对象存储

【分布式文件存储】MinIO部署及实现文件上传下载,第1张

MinIO是全球领先的对象存储先锋，目前在全世界有数百万的用户。

高性能，在标准硬件上，读/写速度上高达183GB/秒和171GB/秒，拥有更高的吞吐量和更低的延迟
可扩展性，为对象存储带来了简单的缩放模型，通过添加更多集群可以扩展空间
简单，极简主义是MinIO的指导性设计原则，即可在几分钟内安装和配置
与Amazon S3兼容，亚马逊云的 S3 API（接口协议）是在全球范围内达到共识的对象存储的协议，是全世界内大家都认可的标准

数据安全，使用纠删码来保护数据免受硬件故障和无声数据损坏

MinIO集群部署

基于docker-compose部署入门案例

准备docker-compose.yml

【分布式文件存储】MinIO部署及实现文件上传下载,第2张

version: '3.7'
# 所有容器通用的设置和配置
x-minio-common: &minio-common
  image: minio/minio
  command: server --console-address ":9001" http://minio{1...4}/data
  expose:
    - "9000"
  # environment:
    # MINIO_ROOT_USER: minioadmin
    # MINIO_ROOT_PASSWORD: minioadmin
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
    interval: 30s
    timeout: 20s
    retries: 3
# 启动4个docker容器运行minio服务器实例
# 使用nginx反向代理9000端口，负载均衡, 你可以通过9001、9002、9003、9004端口访问它们的web console
services:
  minio1:
    <<: *minio-common
    hostname: minio1
    ports:
      - "9001:9001"
    volumes:
      - ./data/data1:/data
  minio2:
    <<: *minio-common
    hostname: minio2
    ports:
      - "9002:9001"
    volumes:
      - ./data/data2:/data
  minio3:
    <<: *minio-common
    hostname: minio3
    ports:
      - "9003:9001"
    volumes:
      - ./data/data3:/data
  minio4:
    <<: *minio-common
    hostname: minio4
    ports:
      - "9004:9001"
    volumes:
      - ./data/data4:/data
  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./config/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

在当前目录下，创建config，创建nginx.conf

【分布式文件存储】MinIO部署及实现文件上传下载,第3张

user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
events {
    worker_connections  4096;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;
    # include /etc/nginx/conf.d/*.conf;
    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }
    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;
        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;
            proxy_pass http://minio;
        }
    }
}

测试启动

docker-compose up -d

【分布式文件存储】MinIO部署及实现文件上传下载,第4张

访问任意节点

默认账户密码：minioadmin minioadmin

【分布式文件存储】MinIO部署及实现文件上传下载,第5张

【分布式文件存储】MinIO部署及实现文件上传下载,第6张

MinIO用户管理

需要创建一个用户，客户端用户可上传下载权限。

注意：长度最好是32位，否则后续调用API会报错

【分布式文件存储】MinIO部署及实现文件上传下载,第7张

Buckets管理

MinIO uses buckets to organize objects. A bucket is similar to a folder or directory in a filesystem, where each bucket can hold an arbitrary number of objects.

MinIO使用bucket来组织对象。bucket类似于文件系统中的文件夹或目录，其中每个bucket可以容纳任意数量的对象。

【分布式文件存储】MinIO部署及实现文件上传下载,第8张

创建Buckets

注意桶名的命名规范，否则后续调用api创建桶会报错

【分布式文件存储】MinIO部署及实现文件上传下载,第9张

可以在控制台管理文件，上传下载预览删除等

【分布式文件存储】MinIO部署及实现文件上传下载,第10张

Simple Storage Service（Amazon S3）存储桶命名要求 - AWS CloudTrail

【分布式文件存储】MinIO部署及实现文件上传下载,第11张

MinIO客户端

官方接口API文档

Java Client API Reference — MinIO Object Storage for Linux

MinIO支持多种客户端，本文以java为例

【分布式文件存储】MinIO部署及实现文件上传下载,第12张

引入依赖


    io.minio
    minio
    8.5.3

文件上传下载Demo

package pers.kw.minio.demo;
import com.alibaba.fastjson.JSON;
import io.minio.*;
import io.minio.errors.*;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.List;
public class MinIODemo {
    private static final String accessKey = "PPeCeQbHtFsEcGNxnlmSpbYasxELHoGy";
    private static final String secretKey = "GEMQDMUhQSHaIJCqmxperpESNYCJXvjJ";
    private static final String endpoint = "http://192.168.8.88:9000";
    public static void main(String[] args)
            throws ServerException, InsufficientDataException, ErrorResponseException,
            IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
        String bucket = "test01";
        String objectName = "small125750JkEMk1625374670.jpg";
        downloadFile(bucket, objectName);
    }
    public static void downloadFile(String bucket, String objectName)
            throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
        MinioClient minioClient =
                MinioClient.builder()
                        .endpoint(endpoint)
                        .credentials(accessKey, secretKey)
                        .build();
        //todo 需要关闭流
        try (InputStream inputStream = minioClient.getObject(
                GetObjectArgs.builder()
                        .bucket(bucket)
                        .object(objectName)
                        .build())) {
            // Read data from stream
            byte[] bytes = IOUtils.toByteArray(inputStream);
            String fileName = "22.jpg";
            System.out.println("下载文件：" + bytes.length);
            FileUtils.writeByteArrayToFile(new File("C:\Users\pc\Desktop\" + fileName), bytes);
            System.out.println("文件下载成功");
            IOUtils.closeQuietly(inputStream);
        }
    }
    public static void uploadFile(String bucket) throws IOException, NoSuchAlgorithmException, InvalidKeyException {
        try {
            // Create a minioClient with the MinIO server playground, its access key and secret key.
            MinioClient minioClient =
                    MinioClient.builder()
                            .endpoint(endpoint)
                            .credentials(accessKey, secretKey)
                            .build();
            // Make 'asiatrip' bucket if not exist.
            boolean found =
                    minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
            if (!found) {
                // Make a new bucket called bucket.
                minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());
            } else {
                System.out.println("Bucket " + bucket + " already exists.");
            }
            // Upload '/home/user/Photos/asiaphotos.zip' as object name 'asiaphotos-2015.zip' to bucket
            // 'asiatrip'.
            String fileName = "C:\Users\pc\Desktop\gateway-server.rar";
            String objectName = "gateway-server_2023058.rar";
            minioClient.uploadObject(
                    UploadObjectArgs.builder()
                            .bucket(bucket)
                            .object(objectName)
                            .filename(fileName)
                            .build());
            System.out.println(
                    fileName + " is successfully uploaded as"
                            + objectName + " to bucket " + bucket + ".");
        } catch (MinioException e) {
            System.out.println("Error occurred: " + e);
            System.out.println("HTTP trace: " + e.httpTrace());
        }
    }
}

调用API碰到的问题

问题一：

java.lang.NoSuchMethodError: okhttp3.RequestBody.create ( BLokhttp3/Medialype:)Lokhttp3/RequestBody;

【分布式文件存储】MinIO部署及实现文件上传下载,第13张

解决: 引入如下依赖


    com.squareup.okhttp3
    okhttp
    4.9.0

问题二：

bucket name does not follow Amazon S3 standards.

命名不符合规则换个名字，最好在代码中增加正则校验

【分布式文件存储】MinIO部署及实现文件上传下载,第14张

问题三：

RequestTimeTooSkewed

客户端时间与服务器时间相差过大导致

【分布式文件存储】MinIO部署及实现文件上传下载,第15张

设置服务器时间

 ntpdate time.windows.com

问题4：

InvalidAccessKeyId

秘钥信息太短

【分布式文件存储】MinIO部署及实现文件上传下载,第16张

上一篇：14、Nginx---缓存服务

下一篇：【注册表】windows系统注册表常用修改方案