Edit - Stack Overflow

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Rev

Required fields*

How to Encode ascent character into UTF-8 without losing its representation?

Method:

public static void test() throws IOException {
 String input = "ABCÉ Ôpqr"; // charsetName = ISO-8859-1
 
 String utf8String = new String(StandardCharsets.UTF_8.encode(input).array());
 }

Required Output:

Output : "ABCÉ Ôpqr" Encoding : UTF-8

I wanted to convert String "ABCÉ Ôpqr" into its UTF-8 encoding without loosing its representation.

Current Output:

Output : "ABC� �pqr"

Answer*

Java's `String` is UTF-16 encoded. You are using `StandardCharsets.UTF_8` to convert a UTF-16 `String` into a UTF-8 `byte[]` array. That part is fine. If you print out the byte values, you will see that you end up with `41 42 43 c3 89 20 c3 94 70 71 72` as expected for `ABCÉ Ôpqr` (`É` is `c3 89` and `Ô` is `c3 94`).

Then, you are constructing a new `String` from that `byte[]` array, but you are not specifying the encoding of the array. This is where you are going wrong. Java needs to convert the bytes back to UTF-16, so it will use its default charset, which is clearly not UTF-8 in your case, so the `byte[]` array will be *misinterpreted*. This is why you are getting the wrong output.

For example, in ISO-8859-1, byte `c3` is character `Ã` but bytes `89` and `94` have no characters defined. Whereas in Windows-1252, byte `c3` is character `Ã`, byte `89` is character `‰`, and byte `94` is character `”`.

If you want the `byte[]` array to be interpreted as UTF-8, then you have to specify that to the `String` constructor, eg:

```java
public static void test() throws IOException {
 String input = "ABCÉ Ôpqr";

 Charset utf8 = StandardCharsets.UTF_8; 
 String decoded = new String(utf8.encode(input).array(), utf8);

 // do something with decoded string...
}
```

Otherwise, you can use `Charset.decode()` instead, to match your `Charset.encode()`:

```java
public static void test() throws IOException {
 String input = "ABCÉ Ôpqr";

 Charset utf8 = StandardCharsets.UTF_8;
 ByteBuffer buf = utf8.encode(input);
 String decoded = utf8.decode(buf).toString();

 // do something with decoded string...
}
```

Draft saved

Draft discarded

Edit Summary*

Cancel

But in what way would that be useful if input.codePoints() == decoded.codePoints()?

g00se
– g00se

2024年06月05日 22:16:40 +00:00
Commented Jun 5, 2024 at 22:16
@g00se You can't store UTF-8 bytes into a UTF-16 String. The only way for the OP to get their required output (which is the same as the input) using String is to decode the encoded bytes back to UTF-16 using the same Charset that created the bytes. Otherwise, they will have to set their terminal to UTF-8 and output the bytes as-is without going back to a String at all.

Remy Lebeau
– Remy Lebeau

2024年06月05日 22:51:47 +00:00
Commented Jun 5, 2024 at 22:51
That doesn't answer my question, which was, to slightly rephrase it: how is the code you just posted useful?

g00se
– g00se

2024年06月05日 22:56:37 +00:00
Commented Jun 5, 2024 at 22:56
@g00se it is useful if the encoded bytes are used for other purposes before decoded, like if they are transmitted or saved somewhere.

Remy Lebeau
– Remy Lebeau

2024年06月05日 23:18:00 +00:00
Commented Jun 5, 2024 at 23:18
1

What's almost certainly going on is that the string is being viewed on something that doesn't support the encoding used. Code's not really going to help. Not at this stage at least. At this point, we seem to be a lot more interested in this than the OP

g00se
– g00se

2024年06月05日 23:48:16 +00:00
Commented Jun 5, 2024 at 23:48

| Show 2 more comments

How to Edit

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

How to Format

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="https://example.com">example</a>

formatting help »
answering help »

How to Tag

A tag is a keyword or label that categorizes your question with other, similar questions. Choose one or more (up to 5) tags that will help answerers to find and interpret your question.

complete the sentence: my question is about...
use tags that describe things or concepts that are essential, not incidental to your question
favor using existing popular tags
read the descriptions that appear below the tag

If your question is primarily about a topic for which you can't find a tag:

combine multiple words into single-words with hyphens (e.g. python-3.x), up to a maximum of 35 characters
creating new tags is a privilege; if you can't yet create a tag you need, then post this question without it, then ask the community to create it for you

popular tags »

lang-java

CollectivesTM on Stack Overflow

How to Encode ascent character into UTF-8 without losing its representation?

Answer*