解决问题思路
起初是朝着寻找一个能基于当前日志输出情况，实现对现有日志进行自动日志脱敏，因为这种方式需要对所有输入日志进行过滤脱敏处理，会出现如果仅有极少数日志需要进行脱敏，采用这种方式就会让脱敏逻辑遍历所有日志，故又朝着基于注解方式，实现对特定类进行信息脱敏。
效果
基于上述思路，输出了两套日志脱敏方案，具体如下
第一种方案（基于输出日志脱敏）：控制台（通过日志文件配置）和日志文件（基于项目中使用的特定日志存储转换器，扩展底层日志处理逻辑），分开处理进行脱敏，实现对现有日志逻辑零侵入式脱敏
第二种方案（基于注解脱敏）：通过注解，做到精准对特定类进行日志脱敏
实现
3.1 第一种方案：基于输出日志脱敏
3.1.1 核心实现：
控制台脱敏：通过新增对输出日志的转换，实现对输出日志脱敏
输出日志文件脱敏：通过增强底层日志输出逻辑，实现对日志文件的脱敏
其他：控制台和日志输出底层使用同一套脱敏逻辑
3.1.2 实现过程：
控制台脱敏配置及类图：
控制台脱敏配置及类图：
通过对logback_spring.xml配置文件添加日志信息转换，核心配置：添加<conversionRule conversionWork="msg" converterClass="xxx"></conversionRule>详细配置，相关配置和类图如下：
<conversionRule conversionWord="msg" converterClass="com.card.util.SensitiveDataConverter"> </conversionRule><property name="LOG_PATTERN"value="[%d{yyyy-MM-dd HH:mm:ss.SSS}] [%property{projectName}] [%property{applicationName}] [%mdc{TRACE_ID}] [%mdc{X-B3-SpanId}] [%mdc{stringParam1}] [%mdc{stringParam2}] [%mdc{floatParam}] [%thread] [%level] [%logger{36}] - %msg%n"/><property name="LOG_PATTERN_WITH_COLOR"value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%tid] [%thread] %highlight(%-5level) %cyan(%logger{36}) - [%X{traceId} %X{spanId}] %msg%n"/><appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"><encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder"><layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout"><pattern>${LOG_PATTERN}</pattern></layout></encoder></appender>
输出文件脱敏（基于项目中使用了统一的日志文件输出实现）配置及类图：
通过增强<appender>下<encoder>能力实现脱敏，相关配置和类图如下：
<appender name="ROLLING_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"><file>${LOG_PATH}/app.log</file><rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"><fileNamePattern>${LOG_PATH}/app.log.%d{yyyy-MM-dd}.%i</fileNamePattern><maxFileSize>200MB</maxFileSize><maxHistory>3</maxHistory><totalSizeCap>800MB</totalSizeCap></rollingPolicy><encoder class="com.card.encoder.DesensitizationLoggerEncoder"><projectName>${projectName}</projectName><applicationName>${applicationName}</applicationName></encoder></appender>
控制台和输出文件脱敏相关实现类如下
// 读取替换配置(prefix = "desensitization")public class MessageReplacesProperties {/*** 脱敏正则列表*/private List<RegexReplacement> replaces = new ArrayList<>();}// 正则替换实现public class RegexReplacement {// 脱敏匹配正则private Pattern regex;// 替换正则private String replacement;public String format(final String msg) {return regex.matcher(msg).replaceAll(replacement);}public void setRegex(String regex) {this.regex = Pattern.compile(regex);}}// 脱敏具体实现public class SensitiveDataConverter extends MessageConverter {private MessageReplacesProperties messageReplacesProperties;public String convert(ILoggingEvent event){// 获取原始日志String requestLogMsg = event.getFormattedMessage();// 获取脱敏后的日志并返回return convertMsg(requestLogMsg);}/*** 处理日志字符串，返回脱敏后的字符串* @param msg* @return*/public String convertMsg(String msg){if (messageReplacesProperties == null) {messageReplacesProperties = ContextService.getBean(MessageReplacesProperties.class);}// msg 为空直接返回if (StringUtils.isBlank(msg)) {return msg;}// 配置为null直接返回if (messageReplacesProperties == null || CollectionUtils.isEmpty(messageReplacesProperties.getReplaces())) {return msg;}// 消息转换for (RegexReplacement replace : messageReplacesProperties.getReplaces()) {// 遍历脱敏正则 & 替换敏感数据msg = replace.format(msg);}return msg;}}// 日志信息脱敏public class DesensitizationJsonProvider extends MessageJsonProvider {private SensitiveDataConverter sensitiveDataConverter = new SensitiveDataConverter();public DesensitizationJsonProvider() {super();}public void writeTo(JsonGenerator generator, ILoggingEvent event) throws IOException {if (messageSplitPattern != null) {String[] multiLineMessage = messageSplitPattern.split(event.getFormattedMessage());JsonWritingUtils.writeStringArrayField(generator, getFieldName(), multiLineMessage);} else {JsonWritingUtils.writeStringField(generator, getFieldName(), sensitiveDataConverter.convertMsg(event.getFormattedMessage()));}}}// 更换文件message输出实现public class DesensitizationLoggerEncoder extends AtomeLoggerEncoder {public DesensitizationLoggerEncoder(){super();// 移除原有日志输出providerJsonProvider<ILoggingEvent> messageJsonProvider = null;for (JsonProvider<ILoggingEvent> provider : getProviders().getProviders()) {if (provider instanceof MessageJsonProvider) {messageJsonProvider = provider;break;}}if (messageJsonProvider != null) {getProviders().removeProvider(messageJsonProvider);}// 新增脱敏provideraddProvider(new DesensitizationJsonProvider());}}
其他：
控制台脱敏实现方式可作为通用方式实现日志信息脱敏。
日志文件脱敏实现方式，使用到了AtomeLoggerEncoder，其为项目内部定义日志存储转换器，这种方式适用于基于net.logstash.logback:logstash-logback-encoder.xxx 包中LogstashEncoder类的子类实现日志文件输出方式。
因为日志文件脱敏使用到了项目内部的日志存储转换器，第一种方案就不提供【验证】模块了，大家可以基于上面实现自行验证
3.2 第二种方案：基于注解脱敏
3.2.1 核心实现：
通过增强com.fasterxml.jackson.core:jackson-databind:xxx(项目中使用的jar包版本)包中的 ObjectMapper类的序列化能力，提供针对特定注解的类的序列化脱敏
3.2.2 实现过程：
扩充ObjectMapper对注解类处理
扩充ObjectMapper对注解类处理
public class DataMaskObjectMapper {private static ObjectMapper objectMapper;static {objectMapper = new ObjectMapper();AnnotationIntrospector rawAi = objectMapper.getSerializationConfig().getAnnotationIntrospector();AnnotationIntrospector newAi = AnnotationIntrospector.pair(rawAi, new DataMaskAnnotationIntrospector());objectMapper.setAnnotationIntrospector(newAi);}public static ObjectMapper getObjectMapper() {return objectMapper;}}
DataMaskAnnotationIntrospector为NopAnnotationIntrospector子类，重写了findSerializer方法，增强序列化功能，内部实现如下
public class DataMaskAnnotationIntrospector extends NopAnnotationIntrospector {public Object findSerializer(Annotated annotated) {DataMask annotation = annotated.getAnnotation(DataMask.class);if (annotation != null) {return new DataMaskSerializer(annotation.value().operation());}return null;}}
备注：DataMask注解类和DataMaskSerializer类为自定义类
DataMask注解：该注解配置在成员变量上，实现如下所示
@Target({ElementType.FIELD, ElementType.TYPE})@Retention(RetentionPolicy.RUNTIME)@Documentedpublic @interface DataMask {DataMaskEnum value() default DataMaskEnum.PART_MASK;}
备注：DataMaskEnum为自定义枚举
DataMaskEnum为需要脱敏的种类，如对邮箱、手机、名字、id等不同类型进行脱敏，如下所示
public enum DataMaskEnum {EMAIL_MASK(DataMaskEnum::emailMask),PHONE_MASK(DataMaskEnum::phoneMask),NAME_MASK(DataMaskEnum::commonMask),ID_MASK(DataMaskEnum::commonMask),ALL_MASK(DataMaskEnum::allMask),PART_MASK(DataMaskEnum::commonMask),;private final DataMaskOperation operation;DataMaskEnum(DataMaskOperation operation) {this.operation = operation;}public DataMaskOperation operation() {return this.operation;}/*** 对邮件内容脱敏* @param data* @return*/private static String emailMask(String data) {// 长度<=6不脱敏if (StringUtils.isBlank(data)) {return data;}// @符号前面字符小于4，不脱敏int emailFlagLocation = data.lastIndexOf('@');if (emailFlagLocation < DataMaskOperation.INT_4) {return data;}// @符号前展示3位+@符号及以后字符return data.substring(DataMaskOperation.INT_0, DataMaskOperation.INT_3) + DataMaskOperation.MIDDLE_MASK + data.substring(emailFlagLocation);}/*** 对手机号内容脱敏* @param data* @return*/private static String phoneMask(String data) {if (StringUtils.isBlank(data) || data.length() <= DataMaskOperation.INT_7) {return data;}if (data.startsWith("+") && data.length() <= DataMaskOperation.INT_10) {return data;}StringBuilder sb = new StringBuilder();if (data.startsWith("+")) {return sb.append(data, DataMaskOperation.INT_0, DataMaskOperation.INT_6).append(DataMaskOperation.MIDDLE_MASK).append(data.substring(data.length() - DataMaskOperation.INT_4)).toString();}return sb.append(data, DataMaskOperation.INT_0, DataMaskOperation.INT_3).append(DataMaskOperation.MIDDLE_MASK).append(data.substring(data.length() - DataMaskOperation.INT_4)).toString();}/*** 通用掩码* @param data* @return*/private static String commonMask(String data) {if (StringUtils.isBlank(data)) {return data;}int strLen = data.length();// 长度不足不脱敏if (strLen <= DataMaskOperation.INT_6) {return data;}// 首位各取三个，其余部分用"****"代替return data.substring(DataMaskOperation.INT_0, DataMaskOperation.INT_3) + DataMaskOperation.MIDDLE_MASK + data.substring(strLen - DataMaskOperation.INT_3);}/*** 对所有内容脱敏* @param data* @return*/private static String allMask(String data) {if (StringUtils.isNotBlank(data)) {StringBuilder sb = new StringBuilder();for (int i = 0; i < data.length(); i++) {sb.append(DataMaskOperation.MASK_CHAR);}return sb.toString();} else {return data;}}}
备注：DataMaskOperation为自定义函数式式接口
DataMaskOperation为函数式接口，具体实现式放在DataMaskEnum枚举类定义的类型上
public interface DataMaskOperation {int INT_0 = 0;int INT_1 = 1;int INT_2 = 2;int INT_3 = 3;int INT_4 = 4;int INT_6 = 6;int INT_7 = 7;int INT_10 = 10;String MIDDLE_MASK = "****";String MASK_CHAR = "*";String mask(String content);}
b. DataMaskSerializer类：为StdScalarSerializer子类，重写了serialize和serializeWithType方法，增强了序列化能力
public class DataMaskSerializer extends StdScalarSerializer<Object> {private final DataMaskOperation operation;public DataMaskSerializer(DataMaskOperation operation) {super(String.class, false);this.operation = operation;}public void serialize(Object value, JsonGenerator gen, SerializerProvider provider) throws IOException {if (Objects.isNull(operation)) {String content = DataMaskEnum.ALL_MASK.operation().mask((String) value);gen.writeString(content);} else {String content = operation.mask((String) value);gen.writeString(content);}}public final void serializeWithType(Object value, JsonGenerator gen, SerializerProvider provider, TypeSerializer typeSer) throws IOException {this.serialize(value, gen, provider);}}
通过上述对ObjectMapper对象能力的扩充即可完成对象序列化时，对敏感数据进行脱敏，具体使用通过下面类实现
public class DataMaskUtil {private static final ObjectMapper objectMapper = new ObjectMapper();private DataMaskUtil(){}public static String toString(Object object) {try {return DataMaskObjectMapper.getObjectMapper().writeValueAsString(object);} catch (Exception e) {log.warn("DataMaskUtil#toString data mask exception", e);return JsonUtil.toString(object);}}public static String toStringNoDataMask(Object object) {try {return objectMapper.writeValueAsString(object);} catch (Exception e) {log.warn("DataMaskUtil#toStringNoDataMask data mask exception", e);return JsonUtil.toString(object);}}}
3.2.3 验证
验证类：
验证类：
public class UserInfo {private String email;private String phone;private String idNum;private String name;private String allMask;private String partMask;}
测试类
public class DataMaskUtilTest{@Testpublic void testToString() {UserInfo userInfo = getNoInfoUser();log.info("-------------email start--------------");log.info("无值脱敏前：{}", DataMaskUtil.toStringNoDataMask(userInfo));log.info("无值脱敏后：{}", DataMaskUtil.toString(userInfo));log.info("----------------------------------------");userInfo = getShortInfo();log.info("较短长度脱敏前：{}", DataMaskUtil.toStringNoDataMask(userInfo));log.info("较短长度脱敏后：{}", DataMaskUtil.toString(userInfo));log.info("----------------------------------------");userInfo = getOrdinaryEmailInfo();log.info("正常长度脱敏前：{}", DataMaskUtil.toStringNoDataMask(userInfo));log.info("正常长度脱敏后：{}", DataMaskUtil.toString(userInfo));log.info("-------------email end----------------");}private UserInfo getNoInfoUser() {return new UserInfo();}private UserInfo getShortInfo() {UserInfo userInfo = new UserInfo();userInfo.setEmail("1@q.cn");userInfo.setPhone("1234567");userInfo.setIdNum("ic");userInfo.setName("na");userInfo.setAllMask("abcdef");userInfo.setPartMask("pa");return userInfo;}private UserInfo getOrdinaryEmailInfo() {UserInfo userInfo = new UserInfo();userInfo.setEmail("1adfwfa1@qq.com");userInfo.setPhone("+86123456789");userInfo.setIdNum("ic123456789");userInfo.setName("naqiu aff ad");userInfo.setPartMask("panamoabcdef");return userInfo;}}
测试结果
-------------email start--------------: 无值脱敏前：{}: 无值脱敏后：{}: ----------------------------------------: 较短长度脱敏前：{"email":"1@q.cn","phone":"1234567","idNum":"ic","name":"na","allMask":"abcdef","partMask":"pa"}: 较短长度脱敏后：{"email":"1@q.cn","phone":"1234567","idNum":"ic","name":"na","allMask":"******","partMask":"pa"}: ----------------------------------------: 正常长度脱敏前：{"email":"1adfwfa1@qq.com","phone":"+86123456789","idNum":"ic123456789","name":"naqiu aff ad","partMask":"panamoabcdef"}: 正常长度脱敏后：{"email":"1ad****@qq.com","phone":"+86123****6789","idNum":"ic1****789","name":"naq**** ad","partMask":"pan****def"}: -------------email end----------------
不同方式脱敏比较
基于输出日志脱敏 | 基于注解脱敏 | |
优点 |
|
|
缺点 |
|
|
其他
基于上述特性，在实际项目中使用了基于注解方式脱敏，在使用这类方式脱敏时建议：
1、可以将上述逻辑放在单独模块中，生成jar包，除了可以在该项目中使用，还可以通过引入jar包快速应用到其他项目中
2、很多项目，都会通过AOP方式拦截请求，并输出对应参数和返回信息，通过加入该脱敏工具，可以较高效对对这部分进行脱敏
关于领创集团
