java删除(替换)不可见的unicode/utf-8字符（主要是html显示不了的字符）

floydd

浏览: 100087 次
来自: ...

最近访客更多访客>>

w11h22j33

404919747

Errorize

terryhello

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

tools
java
javascript
regexp

替换不可见字符 zero width character

今天遇到一个问题，由于编辑人员从excel等7788的地方copy内容过来，其中有不可见的字符，导致输出内容看上去是对的，其实是多了一个零长度的字符（比如：0000200B ZERO WIDTH SPACE），下面的代码基本解决了以上问题。

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script>
window.onload=function(){
	var zero_len_charcode=[]
	var div = document.getElementById('ttt')
	for(var i=0;i<=0xffff;i++){//
		var c = String.fromCharCode(i);
		if (!c) continue;
		div.innerHTML = "|"+c+"|";
		if (div.offsetWidth == 16) //这里的16是我机器的2个'||'的显示长度，请自己调节
			zero_len_charcode.push(i.toString(16));
	}
	console.log('0x'+zero_len_charcode.join(",0x"))
}
</script>
</head>
<body>
<span id=ttt></span>
</body>
</html>

上面的js代码用于提取出看不见的字符（主要是在html上看不见，也就是前面说的0长度的），copy上面console输出的内容，然后去java里做服务器端处理（为什么这么麻烦，因为swing我不熟悉，而且也不一定可以做这样的长度适应判断）

private static String zerolize(String s) {
		if (s.length() < 4) {
			s = "000".substring(0, 4 - s.length()) + s;
		}
		return s;
	}

	public static void main(String[] args) throws IOException {
		int[] input = new int[] { 0x7f, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8a,
				0x8b, 0x8c, 0x8d, 0x8e, 0x8f, 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99,
				0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, 0xad, 0x483, 0x484, 0x485, 0x486, 0x487, 0x488, 0x489,
				0x559, 0x55a, 0x58a, 0x591, 0x592, 0x593, 0x594, 0x595, 0x596, 0x597, 0x598, 0x599, 0x59a,
				0x59b, 0x59c, 0x59d, 0x59e, 0x59f, 0x5a0, 0x5a1, 0x5a2, 0x5a3, 0x5a4, 0x5a5, 0x5a6, 0x5a7,
				0x5a8, 0x5a9, 0x5aa, 0x5ab, 0x5ac, 0x5ad, 0x5ae, 0x5af, 0x5b0, 0x5b1, 0x5b2, 0x5b3, 0x5b4,
				0x5b5, 0x5b6, 0x5b7, 0x5b8, 0x5b9, 0x5ba, 0x5bb, 0x5bc, 0x5bd, 0x5bf, 0x5c1, 0x5c2, 0x5c4,
				0x5c5, 0x5c6, 0x5c7, 0x606, 0x607, 0x608, 0x609, 0x60a, 0x63b, 0x63c, 0x63d, 0x63e, 0x63f,
				0x674, 0x6e5, 0x6e6, 0x70f, 0x76e, 0x76f, 0x770, 0x771, 0x772, 0x773, 0x774, 0x775, 0x776,
				0x777, 0x778, 0x779, 0x77a, 0x77b, 0x77c, 0x77d, 0x77e, 0x77f, 0xa51, 0xa75, 0xb44, 0xb62,
				0xb63, 0xc62, 0xc63, 0xce2, 0xce3, 0xd62, 0xd63, 0x135f, 0x200b, 0x200c, 0x200d, 0x200e,
				0x200f, 0x2028, 0x2029, 0x202a, 0x202b, 0x202c, 0x202d, 0x202e, 0x2044, 0x2071, 0xf701,
				0xf702, 0xf703, 0xf704, 0xf705, 0xf706, 0xf707, 0xf708, 0xf709, 0xf70a, 0xf70b, 0xf70c,
				0xf70d, 0xf70e, 0xf710, 0xf711, 0xf712, 0xf713, 0xf714, 0xf715, 0xf716, 0xf717, 0xf718,
				0xf719, 0xf71a, 0xfb1e, 0xfc5e, 0xfc5f, 0xfc60, 0xfc61, 0xfc62, 0xfeff, 0xfffc };
		StringBuilder b = new StringBuilder();
		int lastContinuous = -1;
		int span = 0;
		for (int i = 0; i < input.length; i++) {
			if (lastContinuous == -1 && i < input.length - 1 && input[i] + 1 == input[i + 1]) {
				lastContinuous = input[i];
				span = 1;
			} else {
				if (input[i] == lastContinuous + span) {
					span++;
				} else if (lastContinuous != -1) {
					if (b.length() > 0)
						b.append("|");
					b.append(String.format("[\\u%s-\\u%s]", zerolize(Integer.toHexString(lastContinuous)),
							zerolize(Integer.toHexString(lastContinuous + span - 1))));
					span = 0;
					lastContinuous = -1;
					i--;
				} else {
					b.append("|\\u" + zerolize(Integer.toHexString(input[i])));
				}
			}
		}
		if (lastContinuous != -1) {
			if (b.length() > 0)
				b.append("|");
			b.append(String.format("[\\u%s-\\u%s]", zerolize(Integer.toHexString(lastContinuous)),
					zerolize(Integer.toHexString(lastContinuous + span - 1))));
		}
		System.out.println(b.toString());
		Pattern pattern = Pattern.compile(b.toString());
		Matcher matcher = pattern.matcher("helloworld\ufffc撒范德萨分的阿萨德");
		System.out.println(matcher.replaceAll(""));
	}

最后组成的正则如下：

[\u007f-\u009f]|\u00ad|[\u0483-\u0489]|[\u0559-\u055a]|\u058a|[\u0591-\u05bd]|\u05bf|[\u05c1-\u05c2]|[\u05c4-\u05c7]|[\u0606-\u060a]|[\u063b-\u063f]|\u0674|[\u06e5-\u06e6]|\u070f|[\u076e-\u077f]|\u0a51|\u0a75|\u0b44|[\u0b62-\u0b63]|[\u0c62-\u0c63]|[\u0ce2-\u0ce3]|[\u0d62-\u0d63]|\u135f|[\u200b-\u200f]|[\u2028-\u202e]|\u2044|\u2071|[\uf701-\uf70e]|[\uf710-\uf71a]|\ufb1e|[\ufc5e-\ufc62]|\ufeff|\ufffc

分享到：

mysql 多表删除陷阱（跨库schema+别名的语 ... | myeclipse转eclipse jee的工程文件补丁（ ...

2012-03-06 15:25
浏览 19933
评论(1)
分类:编程语言
查看更多

1 楼 yrandy 2012-03-31

发表评论

您还没有登录,请您登录后再发表评论

最近访客更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论