需求: 应公司监管要求,需要把公司在企业微信的聊天记录下载下来保存到私有服务器.
参考博客:
企业微信官网:https://developer.work.weixin.qq.com/document/path/9136
博客: https://blog.csdn.net/qq_42851002/article/details/119460215
参考demo:https://github.com/hufeiyaya/QyChat
基本思路:
0,登录企业微信管理后台开启企微存档权限,并配置可信任IP和公钥.
1,创建项目并集成企业微信SDK;
2,初始化SDK;
3,调用SDK接口拉取数据并保存加密数据;
4,对数据进行解密并分类保存;
5,拉取媒体文件并保存
//目录结构
示例代码:
public void pullWechatMsg() {
//1,初始化sdk
//初始化时请填入自己企业的corpid与secrectkey。
log.info("开始执行初始化sdk操作");
long ret = 0;
long sdk = Finance.NewSdk();
ret = Finance.Init(sdk, corpId, secret);
if(ret != 0){
Finance.DestroySdk(sdk);
log.error("初始化sdk失败,失败消息为: ret = {}" , ret);
return;
}
log.info("初始化sdk成功,开始执行拉取消息操作");
//2,调用sdk接口拉取消息
//每次使用GetChatData拉取存档前需要调用NewSlice获取一个slice，在使用完slice中数据后，还需要调用FreeSlice释放。
long slice = Finance.NewSlice();
Long latestSeq = encryptMapper.selectLatestSeq();
if(latestSeq == null) {
latestSeq = 1L;
} else {
latestSeq ++ ;
}
proxy = "xxx.xxx.xxx.xxx:8080/qywechat-huihua";
// proxy = ""; //注意:没有代理时proxy和passwd设置为空即可
passwd = "";
ret = Finance.GetChatData(sdk, latestSeq, limit, proxy, passwd, timeout, slice);
if (ret != 0) {
log.error("调用sdk拉取消息接口失败,失败消息为 ret = {}" , ret);
Finance.FreeSlice(slice);
return;
}
String contentResult = Finance.GetContentFromSlice(slice);
log.info("调用sdk拉取消息接口:seq = {} ,拉取密文结果contentResult = {}" , latestSeq ,contentResult);
Finance.FreeSlice(slice);
ChatDataResult chatDataResult = JSON.parseObject(contentResult, ChatDataResult.class);
List<ChatData> chatDataList = chatDataResult.getChatdata();
WechatChatdataEncrypt chatdataEncrypt = null;
for(ChatData chatData : chatDataList) {
log.info("将加密数据保存到数据库中");
chatdataEncrypt = new WechatChatdataEncrypt();
chatdataEncrypt.setId(UUID.randomUUID().toString());
chatdataEncrypt.setSeq(chatData.getSeq());
chatdataEncrypt.setMsgid(chatData.getMsgid());
chatdataEncrypt.setPublickeyVer(chatData.getPublickey_ver());
chatdataEncrypt.setEncryptRandomKey(chatData.getEncrypt_random_key());
chatdataEncrypt.setEncryptChatMsg(chatData.getEncrypt_chat_msg());
encryptMapper.addEncrypeData(chatdataEncrypt);
log.info("开始执行解密操作");
//对消息进行解密处理
String encryptRandomKey = chatData.getEncrypt_random_key();
String encryptChatMsg = chatData.getEncrypt_chat_msg();
//每次使用DecryptData解密会话存档前需要调用NewSlice获取一个slice，在使用完slice中数据后，还需要调用FreeSlice释放。
String encrypt_key = null;
try {
encrypt_key = RSAUtil.getEncryptKey(encryptRandomKey);
} catch (Exception e) {
e.printStackTrace();
}
long msg = Finance.NewSlice();
ret = Finance.DecryptData(sdk, encrypt_key, encryptChatMsg, msg);
if (ret != 0) {
// System.out.println("调用sdk解密接口失败, 失败消息为: " + ret);
log.info("调用sdk解密接口失败, 失败消息ret = {}" , ret);
Finance.FreeSlice(msg);
return;
}
String encryptResult = Finance.GetContentFromSlice(msg);
log.info("解密后的消息为 encryptResult = {} " , encryptResult);
JSONObject encryptChatDataJsonObject = JSON.parseObject(encryptResult);
String msgtype = encryptChatDataJsonObject.getString("msgtype");
// 将解密后的消息存储到数据库中
WechatChatdataDecrypt chatdataDecrypt = new WechatChatdataDecrypt();
if("switch".equals(encryptChatDataJsonObject.getString("action"))) {
//切换企业日志
log.info("处理switch 消息"+encryptChatDataJsonObject);
chatdataDecrypt.setMsgid(encryptChatDataJsonObject.getString("msgid"));
chatdataDecrypt.setAction(encryptChatDataJsonObject.getString("action"));
chatdataDecrypt.setFrom(encryptChatDataJsonObject.getString("user"));
chatdataDecrypt.setMsgtime(encryptChatDataJsonObject.getLong("time"));
chatdataDecrypt.setMsgtype("switch");
} else {
chatdataDecrypt.setId(UUID.randomUUID().toString());
chatdataDecrypt.setMsgid(encryptChatDataJsonObject.getString("msgid"));
chatdataDecrypt.setAction(encryptChatDataJsonObject.getString("action"));
chatdataDecrypt.setFrom(encryptChatDataJsonObject.getString("from"));
chatdataDecrypt.setToList(encryptChatDataJsonObject.getString("tolist"));
chatdataDecrypt.setRoomid(encryptChatDataJsonObject.getString("roomid"));
chatdataDecrypt.setMsgtime(encryptChatDataJsonObject.getLong("msgtime"));
chatdataDecrypt.setMsgtype(encryptChatDataJsonObject.getString("msgtype"));
if("text".equals(msgtype) || "revoke".equals(msgtype)
|| "agree".equals(msgtype) || "disagree".equals(msgtype)
|| "card".equals(msgtype) || "location".equals(msgtype)
|| "link".equals(msgtype) || "weapp".equals(msgtype)
|| "chatrecord".equals(msgtype) || "todo".equals(msgtype)
|| "vote".equals(msgtype) ||"collect".equals(msgtype)
|| "redpacket".equals(msgtype) ||"meeting".equals(msgtype)
|| "calendar".equals(msgtype) || "sphfeed".equals(msgtype)) {
//文本,撤回消息,同意会话,名片,位置,链接,小程序,会话记录,待办消息,投票消息,填表消息,红包消息,会议邀请消息,日程消息,视频号消息
chatdataDecrypt.setContent(encryptChatDataJsonObject.getString(msgtype));
} else if("image".equals(msgtype) || "voice".equals(msgtype)
|| "video".equals(msgtype) || "emotion".equals(msgtype)
|| "file".equals(msgtype)) {
//图片,语音,视频,表情,文件
chatdataDecrypt.setContent(encryptChatDataJsonObject.getString(msgtype));
log.info("开始执行拉取媒体文件操作");
JSONObject fileJsonObject = encryptChatDataJsonObject.getJSONObject(msgtype);
// if (saveMediafile(sdk, msgtype, fileJsonObject, sdkfileid, md5sum)) return;
saveMediafile(sdk, chatData.getMsgid(), msgtype, fileJsonObject);
} else if("docmsg".equals(msgtype)){
//在线文档消息
chatdataDecrypt.setContent(encryptChatDataJsonObject.getString("doc"));
} else if("markdown".equals(msgtype) || "news".equals(msgtype)
|| "voiptext".equals(msgtype) || "qydiskfile".equals(msgtype)) {
//MarkDown格式消息,图文消息,音视频通话,微盘文件
chatdataDecrypt.setContent(encryptChatDataJsonObject.getString("info"));
} else if("mixed".equals(msgtype)) {
chatdataDecrypt.setContent(encryptChatDataJsonObject.getString(msgtype));
//混合消息
JSONObject mixedObject = encryptChatDataJsonObject.getJSONObject("mixed");
JSONArray itemJSONArray = mixedObject.getJSONArray("item");
for(int i = 0; i < itemJSONArray.size(); i ++) {
JSONObject itemJsonObject = itemJSONArray.getJSONObject(i);
String itemType = itemJsonObject.getString("type");
if("image".equals(itemType) || "voice".equals(itemType)
|| "video".equals(itemType) || "emotion".equals(itemType)
|| "file".equals(itemType)) {
//如果混合消息item属于这五种类型,则需要下载对应的媒体文件
String contentString = itemJsonObject.getString("content");
JSONObject contentJSONObject = JSONObject.parseObject(contentString);
String sdkfileid = contentJSONObject.getString("sdkfileid");
String md5sum = contentJSONObject.getString("md5sum");
// if (saveMediafile(sdk, msgtype, fileJsonObject, sdkfileid, md5sum)) return;
saveMediafile(sdk, chatData.getMsgid(),msgtype, contentJSONObject);
}
}
} else if("meeting_voice_call".equals(msgtype)) {
// 音频存档消息
chatdataDecrypt.setRoomid(encryptChatDataJsonObject.getString("voiceid"));
chatdataDecrypt.setContent(encryptChatDataJsonObject.getString(msgtype));
JSONObject meetingVoiceCallJSONObject = encryptChatDataJsonObject.getJSONObject("meeting_voice_call");
saveMediafile(sdk, chatData.getMsgid(), msgtype, meetingVoiceCallJSONObject);
} else if("voip_doc_share".equals(msgtype)) {
//音频共享文档消息
chatdataDecrypt.setRoomid(encryptChatDataJsonObject.getString("voipid"));
chatdataDecrypt.setContent(encryptChatDataJsonObject.getString(msgtype));
JSONObject voipDocShareJSONObject = encryptChatDataJsonObject.getJSONObject("voip_doc_share");
saveMediafile(sdk, chatData.getMsgid(),msgtype, voipDocShareJSONObject);
} else if ("external_redpacket".equals(msgtype)) {
//互通红包消息
chatdataDecrypt.setContent(encryptChatDataJsonObject.getString("redpacket"));
}
}
// 将明文消息保存到数据库中
decrptMapper.addDecrypeData(chatdataDecrypt);
}
}
文件存储部分代码:(我们是把文件存储到SDS对象存储桶中)
private boolean saveMediafile(long sdk,String msgId, String msgtype, JSONObject fileJsonObject) {
String sdkfileid = fileJsonObject.getString("sdkfileid");
String md5sum = "";
if("meeting_voice_call".equals(msgtype)) {
md5sum = Md5Util.getMd5(sdkfileid);
} else {
md5sum = fileJsonObject.getString("md5sum");
}
long ret;
String suffix = "";
switch(msgtype) {
case "image" :
suffix = ".jpg";
break;
case "voice" :
suffix = ".amr";
break;
case "video" :
suffix = ".mp4";
break;
case "emotion" :
Integer type = fileJsonObject.getInteger("type");
if(1 == type) {
suffix = ".gif";
} else if(2 == type) {
suffix = ".png";
}
break;
case "file" :
suffix = "." + fileJsonObject.getString("fileext");
break;
}
String saveFileName = "";
if("voip_doc_share".equals(msgtype)) {
saveFileName = fileJsonObject.getString("filename");
} else {
saveFileName = md5sum + suffix;
}
String savefile = rootPath + saveFileName;
File savePathFile = new File(savefile);
if(!savePathFile.getParentFile().exists()) {
savePathFile.getParentFile().mkdirs();
}
String indexbuf = "";
while(true){
//每次使用GetMediaData拉取存档前需要调用NewMediaData获取一个media_data，在使用完media_data中数据后，还需要调用FreeMediaData释放。
long media_data = Finance.NewMediaData();
ret = Finance.GetMediaData(sdk, indexbuf, sdkfileid, proxy, passwd, timeout, media_data);
if(ret!=0){
log.info("调用获取媒体文件接口失败,错误信息为 :" + ret);
Finance.FreeMediaData(media_data);
return true;
}
log.info("调用获取媒体文件接口返回值信息 outindex len:%d, data_len:%d, is_finis:%d\n",Finance.GetIndexLen(media_data),Finance.GetDataLen(media_data), Finance.IsMediaDataFinish(media_data));
try {
//大于512k的文件会分片拉取，此处需要使用追加写，避免后面的分片覆盖之前的数据。
FileOutputStream outputStream = new FileOutputStream(new File(savefile), true);
outputStream.write(Finance.GetData(media_data));
outputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
if(Finance.IsMediaDataFinish(media_data) == 1)
{
//已经拉取完成最后一个分片
Finance.FreeMediaData(media_data);
break;
}
else
{
//获取下次拉取需要使用的indexbuf
indexbuf = Finance.GetOutIndexBuf(media_data);
Finance.FreeMediaData(media_data);
}
}
//将文件存储到EOS,并将信息存储到数据库
long fileSize = savePathFile.length();
if(fileSize < fileSingleMaxSize) {
//小于100M的使用整体上传
//将文件保存在存储桶中
XSKYObjectStorageUtils.upladFile(savePathFile,saveFileName);
} else {
//大于100M的使用分段上传
XSKYObjectStorageUtils.multipartUpload(savePathFile,saveFileName);
}
//将本地文件删除
savePathFile.delete();
//将文件信息保存到数据库中
WechatMediadata mediadata = new WechatMediadata();
mediadata.setId(UUID.randomUUID().toString());
mediadata.setMsgid(msgId);
mediadata.setSdkfileid(sdkfileid);
mediadata.setFileName(saveFileName);
mediadata.setFileType(msgtype);
mediadata.setFileSize(fileSize);
mediadata.setAwsKey(saveFileName);
mediadataMapper.addMediaData(mediadata);
return true;
}
SDS工具类:
package cn.com.sinosure.dsp.storage.utils;
import java.io.BufferedInputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
import javax.servlet.http.HttpServletResponse;
import com.amazonaws.services.s3.model.*;
import com.amazonaws.services.s3.model.lifecycle.LifecycleFilter;
import com.amazonaws.services.s3.model.lifecycle.LifecyclePrefixPredicate;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.util.FileCopyUtils;
import org.springframework.web.multipart.MultipartFile;
import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.internal.SSEResultBase;
import com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion;
import cn.com.sinosure.dsp.file.dao.DspFile;
import cn.com.sinosure.dsp.utils.DspException;
/**
* xsky对象存储API
*/
public class XSKYObjectStroageUtils {
private static final Logger logger = LoggerFactory.getLogger(XSKYObjectStroageUtils.class);
private static AmazonS3 client;
private static String bucketName;
/**
* 获取aws实例
*
* @return
*/
public static void initAmazonS3Client(String akey, String skey, String endpoint, String bucketName) {
logger.info("获取client实例 bucketName={}", bucketName);
XSKYObjectStroageUtils.bucketName = bucketName;
// 初始化S3 configure 的实例
AWSCredentials credentials = new BasicAWSCredentials(akey, skey);
// 设置连接时的参数
ClientConfiguration clientConfig = new ClientConfiguration();
// 设置连接方式为HTTP，可选参数为HTTP和HTTPS
clientConfig.setProtocol(Protocol.HTTP);
// 设置use expect continue标志位
clientConfig.setUseExpectContinue(false);
clientConfig.setMaxConnections(500);
try {
// 设置Endpoint
AwsClientBuilder.EndpointConfiguration end_point = new AwsClientBuilder.EndpointConfiguration(
endpoint, "us-east-1");
// 创建连接,替换原AmazonS3Client接口
XSKYObjectStroageUtils.client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withClientConfiguration(clientConfig)
.withEndpointConfiguration(end_point)
.withPathStyleAccessEnabled(true).build();
} catch (AmazonServiceException e) {
e.printStackTrace();
} catch (SdkClientException e) {
e.printStackTrace();
}
}
/**
* 上传文件
*
* @param multipartFile
* 文件对象
* @param keyName
* 新文件名称
* @return
*/
public static void uploadFile(MultipartFile multipartFile, String keyName) {
// 上传一个文件作为一个对象
// Specify server-side encryption.
InputStream inputStream = null ;
try {
inputStream = multipartFile.getInputStream();
PutObjectRequest putRequest = new PutObjectRequest(bucketName, keyName,
inputStream, new ObjectMetadata());
putRequest.withCannedAcl(CannedAccessControlList.BucketOwnerFullControl);
PutObjectResult putResult = client.putObject(putRequest);
logger.info("上传文件eTag：" + putResult.getETag() + " contentMD5：" + putResult.getContentMd5());
} catch (AmazonServiceException ase) {
logger.error("Caught an AmazonServiceException when put object_acl of \"" + keyName + "\".");
logger.error("Error Message: " + ase.getMessage() + " AWS Error Code: " + ase.getErrorCode()
+ " Error Type: " + ase.getErrorType());
throw new AmazonServiceException("对象存储保存异常！");
} catch (Exception e) {
e.printStackTrace();
throw new DspException("文件上传失败！");
}finally {
if(inputStream!=null){
try {
inputStream.close();
} catch (IOException e) {
}
}
}
}
/**
* 上传文件
*
* @param file
* 文件对象
* @param keyName
* 新文件名称
* @return
*/
public static void uploadFile(File file, String keyName) {
try {
PutObjectRequest putRequest = new PutObjectRequest(bucketName, keyName, file);
putRequest.withCannedAcl(CannedAccessControlList.BucketOwnerFullControl);
PutObjectResult putResult = client.putObject(putRequest);
logger.info("上传文件eTag：" + putResult.getETag() + " contentMD5：" + putResult.getContentMd5());
} catch (AmazonServiceException ase) {
logger.error("Caught an AmazonServiceException when put object_acl of \"" + keyName + "\".");
logger.error("Error Message: " + ase.getMessage() + " AWS Error Code: " + ase.getErrorCode()
+ " Error Type: " + ase.getErrorType());
throw new AmazonServiceException("对象存储保存异常！");
} catch (Exception e) {
e.printStackTrace();
throw new DspException("文件上传失败！");
}
}
/**
* 下载文件
*
* @param keyName
* filePath
* @param fullName
* 文件名称
* @param response
* @throws IOException
*/
public static void downloadFile(HttpServletResponse response, String fullName, String keyName) {
logger.info("Downloading an object");
S3Object fullObject = null;
OutputStream out = null;
InputStream in = null;
// 获取对象
try {
fullObject = client.getObject(new GetObjectRequest(bucketName, keyName));
in = fullObject.getObjectContent();
response.setHeader("Accept-Ranges", "bytes");
response.setHeader("Access-Control-Allow-Origin", "*");
response.setContentType("application/octet-stream");
// 文件名
response.setHeader("Content-Disposition",
"attachment; filename=" + new String(fullName.getBytes("GBK"), "ISO-8859-1"));
out = response.getOutputStream();
byte[] bt = new byte[10240];
int length;
while ((length = in.read(bt)) > 0) {
out.write(bt, 0, length);
}
out.close();
in.close();
} catch (IOException e) {
e.printStackTrace();
throw new DspException("文件下载异常");
} finally {
try {
if (out != null) {
out.close();
}
if (in != null) {
in.close();
}
// 为了确保网络连接断开，请关闭任何打开的输入流
if (fullObject != null) {
fullObject.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
/**
* 获取文件流
*
* @param keyName
* @return
*/
public static InputStream getFileInputStream(String keyName) {
// 获取对象
try {
return client.getObject(new GetObjectRequest(bucketName, keyName)).getObjectContent();
} catch (Exception e) {
e.printStackTrace();
throw new DspException("从对象存储上获取文件流异常");
}
}
/**
* 下载文件
*
* @param dspFiles
* dspFile
* @param fullName
* 文件名称
* @param response
* @throws IOException
*/
public static void downloadFileZIP(HttpServletResponse response, String fullName, List<DspFile> dspFiles)
throws IOException {
logger.info("Downloading an object");
if (dspFiles == null || dspFiles.size() == 0) {
logger.debug("需要压缩的文件个数为0");
return;
}
S3Object fullObject = null;
OutputStream sos = null;
InputStream in = null;
ZipOutputStream zos = null;
int i = 1;
// 文件名
response.setHeader("Content-Disposition",
"attachment; filename=" + new String(fullName.getBytes("GBK"), "ISO-8859-1"));
// 获取对象
try {
sos = response.getOutputStream();
zos = new ZipOutputStream(sos);
BufferedInputStream bis = null;
for (DspFile file : dspFiles) {
try {
File destFile = new File(file.getFilePath());
if (!destFile.exists()) {
throw new DspException("目标文件不存在");
}
if (!destFile.canRead()) {
throw new DspException("目标文件不可读");
}
fullObject = client.getObject(new GetObjectRequest(bucketName, file.getFilePath()));
// 为了防止文件名重复,在文件前面添加序号
String fileName = file.getDisplayFileName();
fileName = fileName.replaceAll("[/\\\\:\\*\\?\\<\\>\\|\"]", "_");
zos.putNextEntry(new ZipEntry("(" + i + ")" + fileName));
i++;
in = fullObject.getObjectContent();
bis = new BufferedInputStream(in);
byte[] buffer = new byte[2048];
int length = 0;
while ((length = bis.read(buffer)) != -1) {
zos.write(buffer, 0, length);
}
} catch (Exception e) {
logger.error("生成压缩文件失败", e);
e.printStackTrace();
throw new DspException("生成压缩文件失败");
} finally {
try {
if (bis != null) {
bis.close();
}
if (in != null) {
in.close();
}
if (sos != null) {
sos.flush();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
} catch (Exception e) {
if (sos != null) {
try {
sos.close();
} catch (Exception ex2) {
throw new RuntimeException(ex2.getMessage());
}
}
if (in != null) {
try {
in.close();
} catch (Exception ex3) {
throw new RuntimeException(ex3.getMessage());
}
}
if (zos != null) {
try {
zos.close();
} catch (Exception ex3) {
throw new RuntimeException(ex3.getMessage());
}
}
} finally {
// 为了确保网络连接断开，请关闭任何打开的输入流
if (fullObject != null) {
fullObject.close();
}
if (zos != null) {
zos.flush();
}
}
}
/**
* 分段上传
*
* @param file
* @param keyName
* 新文件名称
*/
public static void multipartUpload(File file, String keyName) {
try {
long contentLength = file.length();
// etag 列表传递给 CompleteMultipartUploadRequest 请求
List<PartETag> partETags = new ArrayList<>();
long partSize = 5L * 1024 * 1024; // 设置分片大小为 5 MB.
InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName,
keyName);
InitiateMultipartUploadResult initResponse = client.initiateMultipartUpload(initRequest);
// 上传分段
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
// 因为最后的部分可能小于 5 MB，根据需要调整分片大小。
partSize = Math.min(partSize, (contentLength - filePosition));
// 创建一个上传分段的请求
UploadPartRequest uploadRequest = new UploadPartRequest().withBucketName(bucketName)
.withKey(keyName).withUploadId(initResponse.getUploadId()).withPartNumber(i)
.withFileOffset(filePosition).withFile(file).withPartSize(partSize);
// 上传分片，并将 etag 保存到列表中
UploadPartResult uploadResult = client.uploadPart(uploadRequest);
partETags.add(uploadResult.getPartETag());
filePosition += partSize;
}
// 分段上传完成
CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName,
keyName, initResponse.getUploadId(), partETags);
client.completeMultipartUpload(compRequest);
} catch (AmazonServiceException e) {
// 服务端错误
e.printStackTrace();
} catch (AmazonClientException e) {
// 客户端错误
e.printStackTrace();
}
}
/**
* 分段上传列表
*
* @return
*/
public static List<MultipartUpload> listMultipartUploads() {
logger.info("分段上传列表");
List<MultipartUpload> uploads = null;
try {
// 检索正在进行的分段上传列表
ListMultipartUploadsRequest allMultipartUploadsRequest = new ListMultipartUploadsRequest(
bucketName);
MultipartUploadListing multipartUploadListing = client.listMultipartUploads(allMultipartUploadsRequest);
uploads = multipartUploadListing.getMultipartUploads();
// 显示正在进行的分段上传信息
logger.info(uploads.size() + " multipart upload(s) in progress.");
/*
* for (MultipartUpload u : uploads) {
* logger.info("Upload in progress: Key = \"" + u.getKey() +
* "\", id = " + u.getUploadId()); }
*/
} catch (AmazonServiceException e) {
// 服务端错误
e.printStackTrace();
} catch (AmazonClientException e) {
// 客户端错误
e.printStackTrace();
}
return uploads;
}
/**
* 复制文件
*
* @param sourceKey
* 源文件
* @param destinationKey
* 目的地文件
*/
public static void copyFile(String sourceKey, String destinationKey) {
// 在同一个桶中拷贝一个新对象
CopyObjectRequest copyObjRequest = new CopyObjectRequest(bucketName, sourceKey, bucketName, destinationKey);
client.copyObject(copyObjRequest);
}
/**
* 移动文件
*
* @param sourceKey
* 源文件
* @param destinationKey
* 目的地文件
*/
public static void moveFile(String sourceKey, String destinationKey) {
// Make a copy of the object and use server-side encryption when
// storing the copy.
CopyObjectRequest copyObjRequest = new CopyObjectRequest(bucketName, sourceKey, bucketName, destinationKey);
CopyObjectResult result = client.copyObject(copyObjRequest);
logger.info(result.getSSEAlgorithm());
getEncryptionStatus(result);
// Delete the original
client.deleteObject(bucketName, sourceKey);
logger.info("Unencrypted object \"" + sourceKey + "\" deleted.");
}
private static void getEncryptionStatus(SSEResultBase response) {
String encryptionStatus = response.getSSEAlgorithm();
if (encryptionStatus == null) {
encryptionStatus = "Not encrypted with SSE";
}
logger.info("Object encryption status is: " + encryptionStatus);
}
/**
* 批量删除
*
* @param keyNames
* 集合文件keyName
* @return
*/
public static int batchDeleteFile(List<KeyVersion> keyNames) {
logger.info("批量删除文件");
// 删除对象
DeleteObjectsRequest multiObjectDeleteRequest = new DeleteObjectsRequest(bucketName).withKeys(keyNames)
.withQuiet(false);
// 确认对象是否已经删除
DeleteObjectsResult delObjRes = client.deleteObjects(multiObjectDeleteRequest);
int successfulDeletes = delObjRes.getDeletedObjects().size();
logger.info(successfulDeletes + " objects successfully deleted.");
return successfulDeletes;
}
/**
* 删除文件
*
* @param keyName
* 文件名称
*/
public static void deleteFile(String keyName) {
logger.info("删除文件 keyName={}", keyName);
client.deleteObject(bucketName, keyName);
}
/**
* 查找prefix目录下的文件
*
* @param prefix
* 例：text/
* @return
*/
public static List<S3ObjectSummary> listObjects(String prefix) {
logger.info("查找前缀" + prefix + "的文件");
return client.listObjectsV2(bucketName, prefix).getObjectSummaries();
}
public static void uploadFile(byte[] bytes, String keyName) {
try {
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(bytes.length);
PutObjectRequest putRequest = new PutObjectRequest(bucketName, keyName,
new ByteArrayInputStream(bytes), objectMetadata);
putRequest.withCannedAcl(CannedAccessControlList.BucketOwnerFullControl);
PutObjectResult putResult = client.putObject(putRequest);
logger.info("上传文件eTag：" + putResult.getETag() + " contentMD5：" + putResult.getContentMd5());
} catch (AmazonServiceException ase) {
logger.error("Caught an AmazonServiceException when put object_acl of \"" + keyName + "\".");
logger.error("Error Message: " + ase.getMessage() + " AWS Error Code: " + ase.getErrorCode()
+ " Error Type: " + ase.getErrorType());
throw new DspException("文件上传失败！");
}
}
/**
* 导出一个文件到本地存储上
*
* @param sourceFilePath
* @param destFilePath
*/
public static void exportFile(String sourceFilePath, String destFilePath) {
try {
InputStream in = getFileInputStream(sourceFilePath);
writeToLocal(in, destFilePath);
} catch (Exception e) {
e.printStackTrace();
throw new DspException("从对象存储导出文件到本地异常");
}
}
/**
* 设置生命周期
*
* @param lifecycle 需要设置的目录
*/
public static void setLifecycle(Map<String,Integer> lifecycle) {
//设置多个
List<BucketLifecycleConfiguration.Rule> listRules = new ArrayList<>();
for(String key : lifecycle.keySet()){
BucketLifecycleConfiguration.Rule rule = new BucketLifecycleConfiguration.Rule();
rule.setStatus(BucketLifecycleConfiguration.ENABLED);
//value当做过期时间
rule.setExpirationInDays(lifecycle.get(key));
//key为过期目录
rule.setFilter(new LifecycleFilter().withPredicate(new LifecyclePrefixPredicate(key)));
listRules.add(rule);
}
//BucketLifecycleConfiguration withRules = new BucketLifecycleConfiguration().withRules(rule);
//设置多个
BucketLifecycleConfiguration withRules = new BucketLifecycleConfiguration().withRules(listRules);
SetBucketLifecycleConfigurationRequest request = new SetBucketLifecycleConfigurationRequest(bucketName, withRules);
client.setBucketLifecycleConfiguration(request);
}
/**
* 获取设置的生命周期
*/
public static List<BucketLifecycleConfiguration.Rule> getLifecycle() {
logger.info("获取生命周期");
BucketLifecycleConfiguration configuration = client.getBucketLifecycleConfiguration(bucketName);
List<BucketLifecycleConfiguration.Rule> rules = configuration.getRules();
return rules;
}
/**
* 将文件从对象存储写到本地目录
*
* @param in
* @param destFilePath
* @throws IOException
* @throws FileNotFoundException
*/
private static void writeToLocal(InputStream in, String destFilePath) throws FileNotFoundException, IOException {
FileCopyUtils.copy(in, new FileOutputStream(destFilePath));
}
/**
* 读取文件
*
* @param filePath
* @return
*/
public static byte[] getFileContent(String filePath) {
ByteArrayOutputStream out = new ByteArrayOutputStream();
InputStream input = null;
try {
int index;
input = getFileInputStream(filePath);
byte[] buffer = new byte[2048];
while((index = input.read(buffer))!= -1){
out.write(buffer, 0, index);;
}
} catch (Exception e) {
e.printStackTrace();
throw new DspException("从对象存储加载文件异常");
} finally {
if (input != null) {
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return out.toByteArray();
}
/**
* 校验文件是否存在
* @param filePath
* @return
*/
public static boolean checkFileExists(String filePath) {
return client.doesObjectExist(bucketName, filePath);
}
}
SDS配置类:
package cn.com.sinosure.dsp.storage.impl;
import javax.annotation.PostConstruct;
import javax.inject.Named;
import org.slf4j.Logger;
import org.springframework.beans.factory.annotation.Value;
import cn.com.sinosure.common.log.LOG;
import cn.com.sinosure.dsp.storage.utils.XSKYObjectStroageUtils;
@Named
public class XSKYObjectStorge2 {
private static final Logger log = LOG.get(XSKYObjectStorge2.class);
@Value("${dsp.akey}")
public String akey;
@Value("${dsp.skey}")
public String skey;
@Value("${dsp.endpoint}")
public String endpoint;
@Value("${dsp.bucket.name}")
public String bucketName;
/**
* 获取aws实例
*
* @return
*/
@PostConstruct
private void initAmazonClient() {
XSKYObjectStroageUtils.initAmazonS3Client(akey, skey, endpoint, bucketName);
}
}
Finance代码
package com.tencent.wework;
/* sdk返回数据
typedef struct Slice_t {
char* buf;
int len;
} Slice_t;
typedef struct MediaData {
char* outindexbuf;
int out_len;
char* data;
int data_len;
int is_finish;
} MediaData_t;
*/
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLDecoder;
import java.nio.file.Files;
public class Finance {
public native static long NewSdk();
/**
* 初始化函数
* Return值=0表示该API调用成功
*
* @param [in] sdk NewSdk返回的sdk指针
* @param [in] corpid 调用企业的企业id，例如：wwd08c8exxxx5ab44d，可以在企业微信管理端--我的企业--企业信息查看
* @param [in] secret 聊天内容存档的Secret，可以在企业微信管理端--管理工具--聊天内容存档查看
*
*
* @return 返回是否初始化成功
* 0 - 成功
* !=0 - 失败
*/
public native static int Init(long sdk, String corpid, String secret);
/**
* 拉取聊天记录函数
* Return值=0表示该API调用成功
*
*
* @param [in] sdk NewSdk返回的sdk指针
* @param [in] seq 从指定的seq开始拉取消息，注意的是返回的消息从seq+1开始返回，seq为之前接口返回的最大seq值。首次使用请使用seq:0
* @param [in] limit 一次拉取的消息条数，最大值1000条，超过1000条会返回错误
* @param [in] proxy 使用代理的请求，需要传入代理的链接。如：socks5://10.0.0.1:8081 或者 http://10.0.0.1:8081
* @param [in] passwd 代理账号密码，需要传入代理的账号密码。如 user_name:passwd_123
* @param [out] chatDatas 返回本次拉取消息的数据，slice结构体.内容包括errcode/errmsg，以及每条消息内容。
*
* @return 返回是否调用成功
* 0 - 成功
* !=0 - 失败
*/
public native static int GetChatData(long sdk, long seq, long limit, String proxy, String passwd, long timeout, long chatData);
/**
* 拉取媒体消息函数
* Return值=0表示该API调用成功
*
*
* @param [in] sdk NewSdk返回的sdk指针
* @param [in] sdkFileid 从GetChatData返回的聊天消息中，媒体消息包括的sdkfileid
* @param [in] proxy 使用代理的请求，需要传入代理的链接。如：socks5://10.0.0.1:8081 或者 http://10.0.0.1:8081
* @param [in] passwd 代理账号密码，需要传入代理的账号密码。如 user_name:passwd_123
* @param [in] indexbuf 媒体消息分片拉取，需要填入每次拉取的索引信息。首次不需要填写，默认拉取512k，后续每次调用只需要将上次调用返回的outindexbuf填入即可。
* @param [out] media_data 返回本次拉取的媒体数据.MediaData结构体.内容包括data(数据内容)/outindexbuf(下次索引)/is_finish(拉取完成标记)
*
* @return 返回是否调用成功
* 0 - 成功
* !=0 - 失败
*/
public native static int GetMediaData(long sdk, String indexbuf, String sdkField, String proxy, String passwd, long timeout, long mediaData);
/**
* @brief 解析密文
* @param [in] encrypt_key, getchatdata返回的encrypt_key
* @param [in] encrypt_msg, getchatdata返回的content
* @param [out] msg, 解密的消息明文
* @return 返回是否调用成功
* 0 - 成功
* !=0 - 失败
*/
public native static int DecryptData(long sdk, String encrypt_key, String encrypt_msg, long msg);
public native static void DestroySdk(long sdk);
public native static long NewSlice();
/**
* @brief 释放slice，和NewSlice成对使用
* @return
*/
public native static void FreeSlice(long slice);
/**
* @brief 获取slice内容
* @return 内容
*/
public native static String GetContentFromSlice(long slice);
/**
* @brief 获取slice内容长度
* @return 内容
*/
public native static int GetSliceLen(long slice);
public native static long NewMediaData();
public native static void FreeMediaData(long mediaData);
/**
* @brief 获取mediadata outindex
* @return outindex
*/
public native static String GetOutIndexBuf(long mediaData);
/**
* @brief 获取mediadata data数据
* @return data
*/
public native static byte[] GetData(long mediaData);
public native static int GetIndexLen(long mediaData);
public native static int GetDataLen(long mediaData);
/**
* @brief 判断mediadata是否结束
* @return 1完成、0未完成
*/
public native static int IsMediaDataFinish(long mediaData);
/*static {
String OS = System.getProperty("os.name").toUpperCase();
if(OS.contains("WIN"))
{
System.out.println("OS"+OS);
System.loadLibrary("WeWorkFinanceSdk");
}
else
{
System.out.println("OS"+OS);
System.loadLibrary("WeWorkFinanceSdk_Java");
}
}*/
public static boolean isWindows() {
String osName = System.getProperties().getProperty("os.name");
System.out.println("current system is : " + osName);
return osName.toUpperCase().indexOf("WINDOWS") != -1;
}
static {
if(isWindows()) {
/*String path = Finance.class.getResource("").getPath().replaceAll("%20"," ")
.replaceFirst("/","").replace("/","\\\\");
System.out.println(path.concat("libeay32.dll"));
System.load(path.concat("libeay32.dll"));
System.load(path.concat("libprotobuf.dll"));
System.load(path.concat("ssleay32.dll"));
System.load(path.concat("libcurl.dll"));
System.load(path.concat("WeWorkFinanceSdk.dll"));*/
//System.out.println("OS"+OS);
System.loadLibrary("WeWorkFinanceSdk");
}else {
//如果使用linux系统,打成jar包后无法使用包内的so文件需要先将.so文件复制到linux目录下,然后再加载
try {
String newSoPath = loadlib("/usr/lib/libWeWorkFinanceSdk_Java.so","libWeWorkFinanceSdk_Java.so");
System.load(newSoPath +"/" + "libWeWorkFinanceSdk_Java.so");
} catch (IOException e) {
e.printStackTrace();
}
}
}
public static String loadlib(String fileName,String sourcePath) throws IOException {
InputStream in = Finance.class.getResourceAsStream(sourcePath);
byte[] buffer = new byte[1024];
File temp = new File(fileName);
Files.deleteIfExists(temp.toPath());
FileOutputStream fos = new FileOutputStream(temp);
int read = -1;
while((read = in.read(buffer)) != -1) {
fos.write(buffer, 0, read);
}
fos.close();
in.close();
String abPath = temp.getAbsolutePath();
return abPath.substring(0,abPath.lastIndexOf("/"));
}
}
pom.xml文件
<!--引入对象存储依赖 开始-->
<!-- <dependency>
<groupId>aws-java-sdk-s3</groupId>
<artifactId>com.amazonaws</artifactId>
<version>1.11.769</version>
</dependency>-->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.9</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>fluent-hc</artifactId>
<version>4.5.9</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.4.11</version>
</dependency>
<!--引入对象存储依赖 结束-->
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
<resources>
<resource>
<directory>src/main/java</directory>
<includes>
<include>**/*.dll</include>
<include>**/*.so</include>
</includes>
<filtering>false</filtering>
</resource>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</resource>
</resources>
</build>
常见问题
- 问题一: 2023-10-31 15:57:37.694 [task-1] ERROR
o.s.a.i.SimpleAsyncUncaughtExceptionHandler - Unexpected exception
occurred invoking async method: public void
com.vocust.qywx.demo.service.impl.ScheduledTasks.run(java.lang.String[])
throws java.lang.Exception java.lang.UnsatisfiedLinkError: no
WeWorkFinanceSdk in java.library.path
解决办法:
windows系统:
方法一:我们只需要在C:\Windows\system32 文件夹里放入我们要使用的dll文件即可。
博客: https://blog.csdn.net/wzp12321/article/details/117744159
方法二:加载代码中的.dll文件
博客: https://blog.csdn.net/weixin_42932323/article/details/118326236
linux系统:
方法一:把libWeWorkFinanceSdk_Java.so放在/usr/lib目录下.
方法二:加载代码中的so文件
方法三(无法加载jar包中的so文件):先把代码中的so文件复制到服务器指定目录下,然后加载服务器目录中的该文件.
参考博客: https://codeleading.com/article/3083329531/
解决办法: 将jar包中的.so文件复制到/usr/lib目录下,然后加载/usr/lib中的.so文件.
- 问题二:java.lang.UnsatisfiedLinkError:xxxx: Can`t load AMD 64-bit .dll
on a IA 32-bit platform
问题描述: 在企业微信存档功能中把代码中的.dll中的文件加载进来后,请求接口报这个错误.
问题原因: 因为本机JDK版本为32位的，而加载的dll为64位版本. 内网的eclipse中jre使用的是eclipse自带的jre,版本为32位的.
解决办法: 将eclipse中的jre配置成本地jre(64位)的.
博客: https://blog.csdn.net/qq_43409973/article/details/130707848
- 问题三:错误码:10001
原因1: 后台没有开启企业微信内容存档权限
开启权限后,corpid和secret错误时报41001
调用拉取数据接口成功时,错误码为 0.
原因2:网络问题–代理信息配置错误
解决办法: 使用nginx的正向代理.nginx做正向代理https需要手动给nginx添加ngx_http_proxy_connect_module模块.
参考博客: https://blog.csdn.net/qq_44659804/article/details/128997510
https://zhuanlan.zhihu.com/p/629353778?utm_id=0
server {
listen 8080;
server_name localhost;
resolver 114.114.114.114 ipv6=off;
proxy_connect;
proxy_connect_allow 443 80;
proxy_connect_connect_timeout 10s;
proxy_connect_data_timeout 10s;
access_log logs/access_proxy.log main;
location /qywechat-huihuai/ {
proxy_pass $scheme://$host$request_uri/;
}
原因3：网络策略问题
问题描述： 生产之前拉取数据100%成功，后来突然大量报错，报10001错误，尤其是媒体文件大量报错。
原因：接口向外请求时会转发到两台服务器，有一台的请求不通。
解决办法：排查网络转发问题。
- 问题四:错误码:301042
问题原因:该IP地址不在企业微信白名单内.
解决办法: 1,将服务器的公网IP配置到企业微信管理后台的可信任IP地址.即错误日志中出现的IP地址(from ip: 106.120.215.130)
2,本地电脑登陆VPN,然后将VPN在公网的IP地址配置到企业微信管理后台.
- 问题五: org.apache.ibatis.binding.BindingException Invalid boud
statement (not found):
com.sinosure.huihua.mapper.WechatChatdataEncryptMapper.selectLatestSeq
问题原因: xml文件没有打进去
解决办法: 在storage-core的xml中添加如下代码
src/main/resources
true
问题六：错误码10003
问题描述：拉取媒体文件时，报出10003错误，多次重试仍报这个错误。
问题原因： 官网回复： 这类问题一般都是客户端版本太低，消息记录上传数据不全导致的
