搭建：基于nginx的上传功能

文章目录

• 搭建：基于nginx的上传功能
• • 一、准备
  • 二、安装nginx
  • • 1.1 解压nginx和nginx插件
    • 1.2 编译并安装nginx
    • 三、启动一个python后台服务，用于上传到临时路径文件，转移到正式路径
    • 四、添加nginx配置
    • • （1）配置一个静态文件服务
      • （2）配置上传文件服务
      • 五、启动nginx和python服务
      • 六、python程序测试上传

        一、准备

        开发一个基于nginx的上传功能。

        • 下载nginx：https://nginx.org/en/download.html

          nginx-1.18.0.tar.gz

        • 下载nginx-upload-module插件：https://github.com/vkholodkov/nginx-upload-module/tags

          nginx-upload-module-2.3.0.tar.gz

          二、安装nginx

          1.1 解压nginx和nginx插件

          先解压nginx-1.18.0.tar.gz, 然后将nginx-upload-module-2.3.0.tar.gz解压到nginx-1.18.0目录下。

          [root@VM-4-3-centos nginx-1.18.0]# ll
          总用量 788
          drwxr-xr-x 6 1001 1001   4096 8月  19 09:52 auto
          -rw-r--r-- 1 1001 1001 302863 4月  21 2020 CHANGES
          -rw-r--r-- 1 1001 1001 462213 4月  21 2020 CHANGES.ru
          drwxr-xr-x 2 1001 1001   4096 8月  19 09:52 conf
          -rwxr-xr-x 1 1001 1001   2502 4月  21 2020 configure
          drwxr-xr-x 4 1001 1001   4096 8月  19 09:52 contrib
          drwxr-xr-x 2 1001 1001   4096 8月  19 09:52 html
          -rw-r--r-- 1 1001 1001   1397 4月  21 2020 LICENSE
          drwxr-xr-x 2 1001 1001   4096 8月  19 09:52 man
          drwxrwxr-x 3 root root   4096 8月   2 2018 nginx-upload-module-2.3.0
          -rw-r--r-- 1 1001 1001     49 4月  21 2020 README
          drwxr-xr-x 9 1001 1001   4096 8月  19 09:52 src
          [root@VM-4-3-centos nginx-1.18.0]#
          

          1.2 编译并安装nginx

          yum -y install make zlib zlib-devel gcc-c++ libtool  openssl openssl-devel
          

          安装PCRE，让nginx支持Rewrite功能：

          • 下载pcre ：http://downloads.sourceforge.net/project/pcre/pcre/8.35/pcre-8.35.tar.gzhttp://downloads.sourceforge.net/project/pcre/pcre/8.35/pcre-8.35.tar.gz

            [root@VM-4-3-centos servers]# cd pcre-8.35/
            [root@VM-4-3-centos pcre-8.35]# ./configure
            [root@VM-4-3-centos pcre-8.35]# make && make install
            ......
            [root@VM-4-3-centos pcre-8.35]# pcre-config --version
            8.35
            [root@VM-4-3-centos pcre-8.35]#
            

            编译并安装nginx

            mkdir /usr/local/nginx

            ./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --add-module=./nginx-upload-module-2.3.0 --with-pcre=/data/dev/servers/pcre-8.35
            make && make install
            

            三、启动一个python后台服务，用于上传到临时路径文件，转移到正式路径

            • 安装python3:https://blog.csdn.net/hefrankeleyn/article/details/122635487

              #! python3
              # -*- coding: UTF-8 -*-
              ###########################
              #
              # description: nginx上传文件，将临时路径的文件拷贝到正式路径
              # author: LiFei  
              # mail: hefrankeleyn@gmail.com
              # date: 2023-08-19
              #  pip install Flask
              #
              ###########################
              import logging,os,datetime, shutil, json,socket
              from flask import Flask,request,abort,make_response
              logging.basicConfig(level=logging.INFO, format="%(asctime)s- %(name)s %(levelname)s- %(message)s")
              # 创建一个应用
              app = Flask(__name__)
              base_dir="/data/dev/nginxUploadFiles"
              # 创建目录，如果目录不存在
              def createDirIfNotExists(dirPath):
                  if not dirPath:
                      return
                  if not (os.path.exists(dirPath) and os.path.isdir(dirPath)):
                      os.makedirs(dirPath)
              # 创建子路径
              def createSubDir(params):
                  sub_file_dir = ""
                  if "busType" in params:
                      busType = str(params["busType"]).replace(' ', '')
                  if busType:
                      sub_file_dir = os.path.join(sub_file_dir, busType)
                  one_day = datetime.datetime.now().strftime('%Y%m%d')
                  sub_file_dir = os.path.join(sub_file_dir, one_day)
                  createDirIfNotExists(os.path.join(base_dir, sub_file_dir))
                  return sub_file_dir
              # 将临时文件拷贝到正式文件目录
              def exeUpload(params):
                  if not params:
                      abort(400)
                  bus_file_name = params["file.name"]
                  file_type = params["file.content_type"]
                  tmp_file_path = params["file.path"]
                  file_md5 = params["file.md5"]
                  file_size = params["file.size"]
                  real_filename = file_md5 + (bus_file_name[bus_file_name.rfind("."):] if bus_file_name.rfind(".")!=-1 else "")
                  subDirPath = createSubDir(params)
                  target_file_path = os.path.join(base_dir, subDirPath, real_filename)
                  # 将临时路径文件拷贝到正式路径
                  shutil.copyfile(tmp_file_path, target_file_path)
                  # 这个路径，可以访问正式目录下的文件
                  base_url = "http://%s:8088/" % (socket.gethostbyname(socket.gethostname()))
                  # 返回结果
                  json_data = json.dumps({
                      "file_name": bus_file_name,
                      "content_type": file_type,
                      "file_md5": file_md5,
                      "file_size": file_size,
                      "file_path": target_file_path,
                      "base_url": base_url,
                      "sub_file_path": os.path.join(subDirPath, real_filename)
                  })
                  response = make_response(json_data)
                  headers = {
                      "content-type": "application/json; charset=utf-8"
                  }
                  response.headers = headers
                  return response
              @app.route("/upload", methods=["POST", 'GET'])
              def nginxUpload():
                  if request.method == "POST":
                      try:
                          params = request.form.to_dict()
                          response = exeUpload(params)
                          return response
                      except Exception as e:
                          err_str = str(e)
                          response = make_response(err_str, 500)
                          headers = {
                              "content-type": "text/plain; charset=utf-8"
                          }
                          response.headers = headers
                          return response
                  else:
                      response = make_response("
              错误的请求类型
              ", 500)
                      headers = {
                          "content-type": "text/plain; charset=utf-8"
                      }
                      response.headers = headers
                      return response
              if __name__ =="__main__":
                  app.run(host="0.0.0.0", port=2230)
              

              启动服务：

              nohup python3 nginxUploadPassServer.py 1>nohup.out 2>&1 &

              四、添加nginx配置

              nginx.conf

              user root;
              worker_processes  1;
              events {
                  worker_connections  1024;
              }
              http {
                  include       mime.types;
                  default_type  application/octet-stream;
                  sendfile        on;
                  keepalive_timeout  65;
                  include /usr/local/nginx/conf/conf.d/*.conf;
              }
              

              conf.d/nginx_80.conf

                  server {
                      listen       80;
                      server_name  localhost;
                      location / {
                          root   html;
                          index  index.html index.htm;
                      }
                      error_page   500 502 503 504  /50x.html;
                      location = /50x.html {
                          root   html;
                      }
                  }
              

              （1）配置一个静态文件服务

              conf.d/nginx_80.conf

                  server {
                      listen       8088;
                      server_name  localhost;
                      #charset koi8-r;
                      #access_log  logs/host.access.log  main;
                      location / {
                          alias  /data/dev/nginxUploadFiles/;
                          autoindex on;
                          #root   html;
                          #index  index.html index.htm;
                      }
                 }
              

              （2）配置上传文件服务

              conf.d/nginxUplaod.conf

              server {
                 listen 5312;
                 client_max_body_size 100m;
                 
                 location /upload {
                     # 转到后端处理的URL，表示Nginx接受完上传的文件后，然后交给后端处理的地址
                     upload_pass @fileserver_backend;
                     # 上传模块接收到文件，临时存放的路径，1 表示方法，该方法需要在/tmp/nginx_upload 下创建以0 到 9位目录，上传的时候会进行一散列处理。 mkdir -p /tmp/nginx_upload/{0,1,2,3,4,5,6,7,8,9}; chmod 777 -R /tmp/nginx_upload
                     upload_store /tmp/nginx_upload 1;
                     # 上传文件的权限，rw表示读写，r只读
                     upload_store_access user:rw group:rw all:rw;
                     set $upload_field_name "file";
                     # http 报头， pass 到后台页面后能获取set到报头字段
                     upload_set_form_field $upload_field_name.name "$upload_file_name";
                     upload_set_form_field $upload_field_name.content_type "$upload_content_type";
                     upload_set_form_field $upload_field_name.path "$upload_tmp_path";
                     # upload 模块自动生成一些信息，如文件大小，文件的md值
                     upload_aggregate_form_field "$upload_field_name.md5" "$upload_file_md5";
                     upload_aggregate_form_field "$upload_field_name.size" "$upload_file_size";
                     # 允许的字段
                     upload_pass_form_field "^.*$";
                     #upload_pass_form_field "^submit$|^description$";
                     # 如果pass页面出现以下状态码，就删除本次上传的临时文件
                     upload_cleanup 400 404 499 500-505;
                     # 打开开关， 把前段脚本请求的参数传给后端的脚本语言
                     upload_pass_args on;
                 }
                 
                 location @fileserver_backend {
                    proxy_pass http://localhost:2230;
                 }
              }
              

              五、启动nginx和python服务

              启动nginx

              cd /usr/local/nginx

              ./sbin/nginx

              nginx其它命令

              ./sbin/nginx -s reload
              ./sbin/nginx -s stop
              

              启动python服务：

              nohup python3 nginxUploadPassServer.py 1>nohup.out 2>&1 &
              

              六、python程序测试上传

              #! python3
              # -*- coding: UTF-8 -*-
              ###########################
              #
              # description: 执行上传
              # author: LiFei  
              # mail: hefrankeleyn@gmail.com
              # date: 2023-08-19
              #
              ###########################
              import logging, os, requests
              logging.basicConfig(level=logging.INFO, format="%(asctime)s- %(name)s %(levelname)s- %(message)s")
              def uploadFile(upload_url, file_path, content_type):
                  try:
                      if not (os.path.exists(file_path) and os.path.isfile(file_path)):
                          raise Exception("要上传的文件不存在：" + str(file_path))
                      data = {
                          "busType": "myPro"
                      }
                      fileName = os.path.split(file_path)[-1]
                      files = {
                          "file": (fileName, open(file_path, 'rb'), content_type)
                      }
                      response = requests.post(upload_url, files=files, data=data)
                      return response
                  except Exception as e:
                      logging.error(e)
                      raise e
              if __name__=="__main__":
                  upload_url = "http://myip:5312/upload"
                  file_path = r"/Users/lifei/Documents/workspace/git_test_wp/myproject/wuyishan.jpg"
                  content_type = "image/jpeg"
                  response = uploadFile(upload_url, file_path, content_type)
                  logging.info(response.json())
              

              之后就可以访问到：

              http://myip:8088/myPro/20230819/cad0d40e01e0930cad9030f8cc32f68b.jpg