How to Encode ascent character into UTF-8 without losing its representation?

Question

Method:

public static void test() throws IOException {
 String input = "ABCÉ Ôpqr"; // charsetName = ISO-8859-1
 
 String utf8String = new String(StandardCharsets.UTF_8.encode(input).array());
 }

Required Output:

Output : "ABCÉ Ôpqr" Encoding : UTF-8

I wanted to convert String "ABCÉ Ôpqr" into its UTF-8 encoding without loosing its representation.

Current Output:

Output : "ABC� �pqr"

• Vérace: Where is there anything to do with a database in all of this?
• ankush chauhan: @Vérace, yes, I am storing this value in MongoDB. there it is displaying with replacement character.
• g00se: Current Output: Output where? The second line of your code, btw, doesn't do anything. Please also post output (the output in the same place) of System.out.println(java.nio.charset.Charset.defaultCharset());

Answer 1

Java's String is UTF-16 encoded. You are using StandardCharsets.UTF_8 to convert a UTF-16 String into a UTF-8 byte[] array. That part is fine. If you print out the byte values, you will see that you end up with 41 42 43 c3 89 20 c3 94 70 71 72 as expected for ABCÉ Ôpqr (É is c3 89 and Ô is c3 94).

Then, you are constructing a new String from that byte[] array, but you are not specifying the encoding of the array. This is where you are going wrong. Java needs to convert the bytes back to UTF-16, so it will use its default charset, which is clearly not UTF-8 in your case, so the byte[] array will be misinterpreted. This is why you are getting the wrong output.

For example, in ISO-8859-1, byte c3 is character Ã but bytes 89 and 94 have no characters defined. Whereas in Windows-1252, byte c3 is character Ã, byte 89 is character ‰, and byte 94 is character ".

If you want the byte[] array to be interpreted as UTF-8, then you have to specify that to the String constructor, eg:

public static void test() throws IOException {
 String input = "ABCÉ Ôpqr";
 Charset utf8 = StandardCharsets.UTF_8; 
 String decoded = new String(utf8.encode(input).array(), utf8);
 // do something with decoded string...
}

Otherwise, you can use Charset.decode() instead, to match your Charset.encode():

public static void test() throws IOException {
 String input = "ABCÉ Ôpqr";
 Charset utf8 = StandardCharsets.UTF_8;
 ByteBuffer buf = utf8.encode(input);
 String decoded = utf8.decode(buf).toString();
 // do something with decoded string...
}

• g00se: But in what way would that be useful if input.codePoints() == decoded.codePoints()?
• Remy Lebeau: @g00se You can't store UTF-8 bytes into a UTF-16 String. The only way for the OP to get their required output (which is the same as the input) using String is to decode the encoded bytes back to UTF-16 using the same Charset that created the bytes. Otherwise, they will have to set their terminal to UTF-8 and output the bytes as-is without going back to a String at all.
• g00se: That doesn't answer my question, which was, to slightly rephrase it: how is the code you just posted useful?
• Remy Lebeau: @g00se it is useful if the encoded bytes are used for other purposes before decoded, like if they are transmitted or saved somewhere.
• g00se: What's almost certainly going on is that the string is being viewed on something that doesn't support the encoding used. Code's not really going to help. Not at this stage at least. At this point, we seem to be a lot more interested in this than the OP