oss集成
2020-11-19 03:09:25 # oss #oss #阿里云 #华为云 #对象存储 #本地存储

oss 的血与泪

oss 这玩意也是毕业后,参加工作才知道了了,之前都是之前的同事封装的,我就直接调用就是,但是今年我单独一个人负责一个项目的时候,遇到一个需求,有一个500M以上的视频,通过普通的文件上传的时候死活传不上来,之前的接口都是通过接口把文件提交到服务器,服务器再次提交到阿里云,自己去看文档的时候,就准备分片上传,

开始测试,写一个main方法可以上传,但是在通过接口上传的时候,完了,流的问题(在上传到阿里云的时候前端传递的流没有关闭), 就算解决流的问题,如果把文件传递到后端服务器,这样的受限与服务器的带宽,也需要占用服务器的空间,
又去了解了,授权上传的方式,这样前端的的富文本也不用通过后端,由前端获得链接后直接上传到阿里云,
前端的大文件上传还是有问题,分片发实现,前端直接实现分片上传的方式,但是前端直接上传的是需要ak和sk,这样是极度危险的,所以使用sts临时授权的方式上传图片

动态

之前做公司官网的时候,老大说,让动态的实现oss的管理,包括短信通知,这些插件的动态配置

我的实现思路是这样的,这些东西都是来自系统配置,建表的话比较多,而且这些参数一般不会添加多少,于是我就枚举的方式,并用过两个表来记录数据值,在不同的插件查询参数,

设计三层

一级枚举(插件类型):

1
2
3
4
5
6
7
8
public enum ConfigTypeEnum {
	STORAGE("STORAGE","对象存储"),
	SMS_NOTIFY("SMS_NOTIFY","短信通知"),
	WX("WX","微信配置"),
	MAIL("MAIL","邮件通知"),
	//getter setter
	}

二级枚举(不同插件的名称),如对象存储的 阿里云oss

1
2
3
4
5
6
7
8

public enum HuaweiStorageConstant {
	ENDPOINT ("ENDPOINT","endPoint"),
	AK("AK","ak"),
	SK("SK","sk"),
	BUCKET_NAME("BUCKET_NAME","bucketName"),
	CDN("cdn","cdn(访问域名)")
	}

基本上每个插件都需要实现一个枚举所以我的定枚举的是时候特别长,就是区分每一个 key,
定义两张表,一张记录每一种插件的启用,禁用,另一个就是记录每一个插件的value,
在系统启动的时候加载查询数据,装配不同的oss客户端对象,由于我收到之前同事的影响,我封装的插件在也沿用了之前的方式,在项目启动的时候查询数据,进行转,我可以在使用的时候查询,就可以动态的切换不同的插件,
后来想想上边的设计好low,上边的参数都是key-value的形式,我完全可以使用一个json,一个插件的所有数据都放到一条数据中,使用一个type进行分组

多种封装的实现

只有又进行了脚手架中插件的封装, 实现本地存储,阿里oss ,华为obs
在上一次的坑之后我就进行设计的优化,常用的接口如下
实现为阿里云

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

/***
 * @author xs-shuai
 */
public interface StorageUtil {
/***
	 * 文件上传后的path (文件夹+文件名)]
	 * @Param   fileKey 文件名
	 * @return
	 */
	String getPath(String fileKey);

	/***
	 * 访问前缀
	 * @return
	 */
	String getPreUrl();

	/***
	 * 简单上传
	 * @param inputStream
	 * @param contentLength
	 * @param contentType
	 * @param keyName
	 */
	void store(InputStream inputStream, long contentLength, String contentType, String keyName);

	/***
	 * 获得授权上传链接
	 * @param keyName
	 * @param contentType
	 * @return
	 */
	OssAuthBean getAuthorizeUrl(String keyName, String contentType);
	/***
	 * 删除
	 * @param filePath 文件路径
	 * @return
	 */
	boolean delete(String filePath);

	/**
	 * 批量删除
	 * @param filePaths
	 * @return
	 */
	boolean deleteBatch(Collection<String> filePaths);

	/***
	 * 加载资源
	 * @param file
	 * @return
	 */
	Resource loadResource(String file);
}

本地实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
public class LocalStorageUtil implements StorageUtil {

   private final Log logger = LogFactory.getLog(LocalStorageUtil.class);

   private String address;
   private String storagePath;
   private Path rootLocation;


   public String getStoragePath() {
      return storagePath;
   }

   public void setStoragePath(String storagePath) {
      this.storagePath = storagePath;
      this.rootLocation = Paths.get(storagePath);
      try {
         Files.createDirectories(rootLocation);
      } catch (IOException e) {
         logger.error(e.getMessage(), e);
      }
   }

   @Override
   public void store(InputStream inputStream, long contentLength, String contentType, String keyName) {
      try {
         Files.copy(inputStream, rootLocation.resolve(keyName), StandardCopyOption.REPLACE_EXISTING);
      } catch (IOException e) {
         throw new RuntimeException("Failed to store file " + keyName, e);
      }

   }

   @Override
   public boolean delete(String filePath) {
      Path file = load(filePath);
      try {
         Files.delete(file);
      } catch (IOException e) {
         logger.error(e.getMessage(), e);
      }
      return true;
   }

   /***
    * 批量删除 循环调用 delete
    * @param filePaths
    * @return
    */
   @Override
   public boolean deleteBatch(Collection<String> filePaths) {
      if (CollectionUtils.isEmpty(filePaths)){
         return false;
      }
      filePaths.forEach(x->{
         delete(x);
      });
      return true;
   }


   public Path load(String filename) {
      return rootLocation.resolve(filename);
   }


   /***
    * 本地模式无上传
    * @param keyName
    * @param contentType
    * @return
    */
   @Override
   @Deprecated
   public OssAuthBean getAuthorizeUrl(String keyName, String contentType) {
      throw new RuntimeException("当前模式,没有此功能,请切换到oss存储");
   }

   /**
    * 加载
    * @param file
    * @return
    */
   @Override
   public Resource loadResource(String file) {
      try {
         Path path = load(file);
         Resource resource = new UrlResource(path.toUri());
         if (resource.exists() || resource.isReadable()) {
            return resource;
         } else {
            return null;
         }
      } catch (MalformedURLException e) {
         logger.error(e.getMessage(), e);
         return null;
      }

   }

   /***
    * 文件路径
    * @param fileKey
    * @return
    */
   @Override
   public String getPath(String fileKey) {
      return fileKey;
   }

   /***
    * 文件访问前缀
    * @return
    */
   @Override
   public String getPreUrl() {
      return address;
   }
}

阿里云实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
@Data
public class AliStorageUtil implements StorageUtil {
   private final Log logger = LogFactory.getLog(AliStorageUtil.class);


   private String endpoint;
   private String accessKeyId;
   private String accessKeySecret;
   private String bucketName;
   private String cdn;

   private OSS getOSS() {
      return new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret);
   }

   /**
    * 保存
    * @param inputStream
    * @param contentLength
    * @param contentType
    * @param keyName
    */
   @Override
   public void store(InputStream inputStream, long contentLength, String contentType, String keyName) {
      OSS oss = getOSS();
      try {
         // 简单文件上传, 最大支持 5 GB, 适用于小文件上传, 建议 20M以下的文件使用该接口
         ObjectMetadata objectMetadata = new ObjectMetadata();
         objectMetadata.setContentLength(contentLength);
         objectMetadata.setContentType(contentType);
         // 对象键（Key）是对象在存储桶中的唯一标识。
         PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, keyName, inputStream, objectMetadata);
         PutObjectResult putObjectResult = oss.putObject(putObjectRequest);
         logger.info(putObjectResult);
      } catch (Exception ex) {
         logger.error(ex.getMessage(), ex);
      }
      close(oss);
   }

   /**
    * 删除
    * @param filePath 文件路径
    * @return
    */
   @Override
   public boolean delete(String filePath) {
      OSS oss = getOSS();
      try {
         if (!StringUtils.isEmpty(filePath)) {
            oss.deleteObject(bucketName, filePath);
         }
      } catch (Exception e) {
         logger.error(e.getMessage(), e);
      }
      close(oss);
      return true;
   }

   /**
    * 批量删除
    * @param filePaths
    * @return
    */
   @Override
   public boolean deleteBatch(Collection<String> filePaths) {
      if (CollectionUtils.isEmpty(filePaths)){
         return false;
      }
      OSS oss = getOSS();
      ArrayList<String> filePathsKey = new ArrayList<>(filePaths);
      DeleteObjectsResult deleteObjectsResult = oss.deleteObjects(new DeleteObjectsRequest(bucketName).withKeys(filePathsKey));
      List<String> deletedObjects = deleteObjectsResult.getDeletedObjects();

      close(oss);

      return true;
   }


   /**
    * 授权上传
    * @param keyName
    * @param contentType
    * @return
    */
   @Override
   public OssAuthBean getAuthorizeUrl(String keyName, String contentType) {
      OSS oss = getOSS();

      GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(bucketName, keyName, HttpMethod.PUT);
      request.setContentType(contentType);
      Date expiration = new Date(System.currentTimeMillis() + 1000 * 60 * 60);
      request.setExpiration(expiration);
      URL url = oss.generatePresignedUrl(request);
      OssAuthBean ossBean = new OssAuthBean();
      ossBean.setPath(keyName);
      ossBean.setUploadUrl(url.toString());

      close(oss);

      return ossBean;
   }

   @Override
   public String getPath(String fileKey) {
      return fileKey;
   }

   @Override
   public String getPreUrl() {
      return cdn ;
   }

   @Override
   @Deprecated
   public Resource loadResource(String file) {
      return  null;
   }

   private void close(OSS oss){
      oss.shutdown();
   }
}

华为云实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
@Data
public class HuaWeiStorageUtil implements StorageUtil {

   private String endPoint;
   private String ak;
   private String sk;
   private String bucketName;
   private String cdn;

   private ObsClient getObsClient() {
      return new ObsClient(ak, sk,endPoint );
   }


   /***
    * 删除
    * @param filePath 文件路径
    * @return
    */
   @Override
   public boolean delete(String filePath) {
      ObsClient obsClient = getObsClient();
      DeleteObjectResult deleteObjectResult = obsClient.deleteObject(bucketName, filePath);

      close(obsClient);
      return true;
   }


   /***
    * 批量删除 根据文档无法 自定义文件名删除,文档至此 前缀,按照顺序删除 目前任使用循环删除方式
    * @param filePaths
    * @return
    */
   @Override
   public boolean deleteBatch(Collection<String> filePaths) {

      if (CollectionUtils.isEmpty(filePaths)){
         return false;
      }
      ObsClient obsClient = getObsClient();

      filePaths.forEach(x->{
         obsClient.deleteObject(bucketName,x);
      });

      close(obsClient);

      return true;
      //todo 实现自定义删除 文档地址 https://support.huaweicloud.com/sdk-java-devg-obs/obs_21_0804.html
   }

   /**
    * 文件上传
    * @param inputStream
    * @param contentLength
    * @param contentType
    * @param keyName
    */
   @Override
   public void store(InputStream inputStream, long contentLength, String contentType, String keyName) {
      ObsClient obsClient = getObsClient();
      PutObjectResult putObjectResult = obsClient.putObject(bucketName, keyName, inputStream);
      putObjectResult.getObjectUrl();

      close(obsClient);

   }

   /***
    * 返回文件名
    * @param fileKey
    * @return
    */
   @Override
   public String getPath(String fileKey) {
      return fileKey;
   }

   /**
    * 访问前缀
    * @return
    */
   @Override
   public String getPreUrl() {
      return  cdn;
   }

   /***
    * 授权上传
    * @param keyName
    * @param contentType
    * @return
    */
   @Override
   public OssAuthBean getAuthorizeUrl(String keyName, String contentType) {

      ObsClient obsClient = new ObsClient(ak, sk, endPoint);
      long expireSeconds = 3600L;

      Map<String, String> headers = new HashMap<String, String>();
      headers.put(HTTP.CONTENT_TYPE, contentType);

      TemporarySignatureRequest request = new TemporarySignatureRequest(HttpMethodEnum.PUT, expireSeconds);
      request.setBucketName(bucketName);
      request.setObjectKey(keyName);
      request.setHeaders(headers);
      TemporarySignatureResponse response = obsClient.createTemporarySignature(request);
      OssAuthBean bean = new OssAuthBean();
      bean.setPath(keyName);
      bean.setUploadUrl(response.getSignedUrl());

      close(obsClient);

      return bean;
   }



   /***
    * 加载资源,
    * @param file
    * @return
    */
   @Override
   @Deprecated
   public Resource loadResource(String file) {
      return null;
   }

   private void close(ObsClient obsClient ){
      try {
         obsClient.close();
      } catch (IOException e) {
         e.printStackTrace();
      }
   }
}

公用配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
@Data
@Configuration
@ConfigurationProperties(prefix = "xs.storage")
public class StorageProperties {


   private StorageType active;
   private Local local;
   private Ali ali;
   private HuaWei huaWei;



   @Data
   public static class Local {
      private String address;
      private String storagePath;
   }


   @Data
   public static class HuaWei {
      private String endPoint;
      private String ak;
      private String sk;
      private String bucketName;
      private String cdn;
   }

   @Data
   public static class Ali {
      private String endpoint;
      private String accessKeyId;
      private String accessKeySecret;
      private String bucketName;
      private String cdn;
   }
}

对象注入

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
@Slf4j
@Configuration
@EnableConfigurationProperties(StorageProperties.class)
public class StorageAutoConfiguration {

   @Autowired
   private StorageProperties properties;


   @Bean
   public StorageUtil getInstance() {
      StorageUtil storageUtil;
      switch (properties.getActive()) {
         case LOCAL:
            storageUtil = getLocalStorage();
            break;
         case ALI:
            storageUtil = getAliStorageUtil();
            break;
         case HUAWEI:
            storageUtil = getHuaWeiConfig();
            break;
         default:
            throw new RuntimeException("对象存储配置错误");
      }
      return storageUtil;
   }


   public LocalStorageUtil getLocalStorage() {
      LocalStorageUtil localStorageUtil = new LocalStorageUtil();
      localStorageUtil.setAddress(properties.getLocal().getAddress());
      localStorageUtil.setStoragePath(properties.getLocal().getStoragePath());
      return localStorageUtil;
   }


   public AliStorageUtil getAliStorageUtil() {
      AliStorageUtil aliStorageUtil = new AliStorageUtil();
      aliStorageUtil.setAccessKeyId(properties.getAli().getAccessKeyId());
      aliStorageUtil.setAccessKeySecret(properties.getAli().getAccessKeySecret());
      aliStorageUtil.setBucketName(properties.getAli().getBucketName());
      aliStorageUtil.setEndpoint(properties.getAli().getEndpoint());
      aliStorageUtil.setCdn(properties.getAli().getCdn());
      return aliStorageUtil;
   }


   public HuaWeiStorageUtil getHuaWeiConfig() {
      HuaWeiStorageUtil huaWeiStorageUtil = new HuaWeiStorageUtil();
      huaWeiStorageUtil.setAk(properties.getHuaWei().getAk());
      huaWeiStorageUtil.setEndPoint(properties.getHuaWei().getEndPoint());
      huaWeiStorageUtil.setSk(properties.getHuaWei().getSk());
      huaWeiStorageUtil.setBucketName(properties.getHuaWei().getBucketName());
      huaWeiStorageUtil.setCdn(properties.getHuaWei().getCdn());
      return huaWeiStorageUtil;
   }

}

MATE-IN中spring.factories

1
2
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
  com.xs.java.common.storage.config.StorageAutoConfiguration

配置中 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
xs:
  storage:
    active: LOCAL # LOCAL  ALI  HUAWEI
    local:
      storagePath: /data/file  # 存储文件位置
      # 文件显示地址，文件fetch接口对应的地址
      address: http://192.168.0.161:8061/a/file/fetch/
    ali:
      endpoint: XXXXX
      accessKeyId: XXXXX
      accessKeySecret: XXXXX
      bucketName: XXX
      cdn: XXXXX   # 建议使用cdn,如不暂时没有cdn,使用bucketName和endpoint的组合
    huaWei:
      endpoint: XXXXX
      aK: XXXXX
      sK: XXXXX
      bucketName: XXXXX
      cdn: XXXXX   # 建议配置cdn,如不使用cdn,使用bucketName和endpoint的组合

service 中注入 

1
2
@Autowired
private StorageUtil storageUtil;

就可以使用想对应的注解,

以上只是多种oss的组合,接入七牛,腾讯也是如此,其他稍微复杂的也是如此,详细相关文档,

上一页
2020-11-19 03:09:25 # oss #oss #阿里云 #华为云 #对象存储 #本地存储
下一页