Java String Charset Encoding

Charset

Charset is a named mapping between sequences of sixteen-bit Unicode code units and sequences of bytes.

1 byte = 8 bits, byte represent the int value from 0x00 through 0xFF. In ASC-II, we use a mapping to represent characters. For example, Charsets are named by strings composed of the following characters:

		The uppercase letters 'A' through 'Z' ('0x41' through '0x5a'), 
		The lowercase letters 'a' through 'z' ('0x61' through '0x7a'), 
		The digits '0' through '9' ('0x30' through '0x39'), 
		The dash character '-' ('0x2d', HYPHEN-MINUS), 
		The period character '.' ('0x2e', FULL STOP), 
		The colon character ':' ('0x3a', COLON), and 
		The underscore character '_' ('0x5f', LOW LINE). 

 

Unicode 4.0

The weakness for previous mapping is some special Characters cannot be represented, like Chinese, Greek cannot be represented. "unicode 4.0 standard" define basic multiple language encoding charset mapping, it is from \u0000 to \uFFFF.

 

Java save characters in unicode. 

 

    /** The value is used for character storage. */
    private final char value[];

1 char = 2 bytes. 

 

We can see following information from String javadoc:

 

A String represents a string in the UTF-16 format in which supplementary characters are represented by surrogate pairs (see the section Unicode Character Representations in the Character class for more information). Index values refer to char code units, so a supplementary character uses two positions in a String. 

The String class provides methods for dealing with Unicode code points (i.e., characters), in addition to those for dealing with Unicode code units (i.e., char values).

 

Encoding and Decoding String

String won't keep the Charset information, because all the characters are stored in unicode. Charset is only used when decoding the specified array of bytes to String characters. 

Following source snapshot will help us to understand well:

	String a = "èç";
	byte[] b_defaultEncoding = a.getBytes(); // 0xe8, 0xe7
	byte[] b_utf8 = a.getBytes("UTF-8"); // 0xc3, 0xa8, 0xc3, 0xa7
	byte[] b_ucs2 = a.getBytes("ISO-10646-UCS-2"); // 0x00, 0xe8, 0x00, 0xe7

	String a_defaultEncoding = new String(b_defaultEncoding);
	String a_mix = new String(b_ucs2);
	String a_ucs2 = new String(b_ucs2, "ISO-10646-UCS-2");
	String a_utf8 = new String(b_utf8, "UTF-8");

	System.out.println(a_defaultEncoding);//èç
	System.out.println(a_ucs2);//èç
	System.out.println(a_utf8);//èç
	System.out.println(a_mix); // è ç

We can see a_mix is not well converted, because it is encoded by Charset "ISO-10646-UCS-2" but when decode to String, default charset is used. 

Default Charset in java

You can specify default Charset with system property: "file.encoding". If not specified, normally "UTF-8" will be used. For more details please refer to Charset.defaultCharset().

Chinese Characters unicode charset

Normally, Simplified Chinese Characters unicode charset from \u4e00 to \u9fa5.