Lets see what happens when trying to convert random string into UTF8 using different methods.<\/p>\n
[code lang=”java”]
\npackage org.kari.test.string;<\/p>\n
import java.io.IOException;
\nimport java.io.OutputStreamWriter;
\nimport java.nio.charset.Charset;<\/p>\n
import org.apache.log4j.Logger;
\nimport org.kari.log.LogUtil;
\nimport org.kari.util.DirectByteArrayOutputStream;<\/p>\n
\/**
\n * Test UTF8 conversion
\n *
\n * @author kari
\n *\/
\npublic class UTF8Test {
\n public static final Logger LOG = LogUtil.getLogger(”utf8”);<\/p>\n
private static final DirectByteArrayOutputStream mOutBuffer = new DirectByteArrayOutputStream(100000);
\n private static OutputStreamWriter mWriter;<\/p>\n
static final class ByteArrayReference {
\n private byte[] mBuffer;
\n private int mOffset;
\n private int mLength;<\/p>\n
public void set(byte[] pBuffer, int pOffset, int pLength) {
\n mBuffer = pBuffer;
\n mOffset = pOffset;
\n mLength = pLength;
\n }<\/p>\n
public void clear() {
\n mBuffer = null;
\n mOffset = 0;
\n mLength = 0;
\n }<\/p>\n
public byte[] getBuffer() {
\n return mBuffer;
\n }<\/p>\n
public int getOffset() {
\n return mOffset;
\n }<\/p>\n
public int getLength() {
\n return mLength;
\n }<\/p>\n
}<\/p>\n
public static abstract class Test {
\n public abstract void convert(
\n String pStr,
\n ByteArrayReference pRef)
\n throws IOException;
\n }<\/p>\n
public static final class WriterTest extends Test{
\n @Override
\n public void convert(
\n String pStr,
\n ByteArrayReference pRef)
\n throws IOException
\n {
\n DirectByteArrayOutputStream out = mOutBuffer;
\n OutputStreamWriter writer = mWriter;<\/p>\n
out.reset();
\n writer.write(pStr);
\n writer.flush();<\/p>\n
pRef.set(out.getBuffer(), 0, out.size());<\/p>\n
\/\/ System.out.print(’.’);
\n }
\n }<\/p>\n
public static final class BasicTest extends Test {
\n @Override
\n public void convert(
\n String pStr,
\n ByteArrayReference pRef)
\n throws IOException
\n {
\n byte[] data = pStr.getBytes(”UTF-8”);
\n pRef.set(data, 0, data.length);
\n\/\/ System.out.print(’.’);
\n }
\n }<\/p>\n
private UTF8Test() throws Exception {
\n mWriter = new OutputStreamWriter(mOutBuffer, ”UTF-8”);
\n }<\/p>\n
public ByteArrayReference test(String str, Test pTest)
\n throws Exception
\n {
\n try {
\n ByteArrayReference ref = new ByteArrayReference();
\n System.out.println(”string len=” + str.length());<\/p>\n
long startTime = System.nanoTime();
\n int COUNT = 100;
\n for (int i = 0; i < COUNT; i++) {\n ref.clear();\n pTest.convert(str, ref);\n }\n long endTime = System.nanoTime();\n long diff = endTime - startTime;\n\n System.out.println();\n System.out.println(\" utf8 len=\" + ref.getLength());\n\n System.out.println(\"total nano = \" + diff + \" nanos\");\n System.out.println(\" per nano = \" + (diff \/ (double)COUNT) + \" nanos\");\n\n System.out.println(\"total time = \" + (diff \/ (1000.0 * 1000)) + \" ms\");\n System.out.println(\" per time = \" + ((diff \/ (1000.0 * 1000)) \/ (double)COUNT) + \" ms\");\n\n return ref;\n } catch (Exception e) {\n throw e;\n }\n }\n\n public static void main(String[] args) {\n try {\n test();\n } catch (Exception e) {\n LOG.error(\"Failed\", e);\n }\n }\n\n private static void test()\n throws Exception\n {\n UTF8Test test = new UTF8Test();\n ByteArrayReference ref1;\n ByteArrayReference ref2;\n {\n String str = createLongString();\n\n System.out.println(\"====================\");\n System.out.println(\"=====BASIC==========\");\n System.out.println(\"====================\");\n ref1 = test.test(str, new BasicTest());\n\n System.out.println(\"====================\");\n System.out.println(\"=====WRITER=========\");\n System.out.println(\"====================\");\n ref2 = test.test(str, new WriterTest());\n }\n\n System.out.println(\"====================\");\n System.out.println(\"equal= \" + equals(ref1, ref2));\n }\n\n public static boolean equals(\n ByteArrayReference ref1,\n ByteArrayReference ref2)\n {\n boolean result = false;\n result = ref1.getLength() == ref2.getLength();\n if (result) {\n byte[] buf1 = ref1.getBuffer();\n byte[] buf2 = ref2.getBuffer();\n int offset1 = ref1.getOffset();\n int offset2 = ref2.getOffset();\n\n for (int i = 0; result && i < ref1.getLength(); i++) {\n result = buf1[offset1 + i] == buf2[offset2 + i];\n }\n }\n return result;\n }\n\n private static String createLongString() {\n StringBuilder sb = new StringBuilder();\n for (int i = 0; i < 1000 * 1000; i++) {\n char ch = (char)(32 + (60000 * Math.random()));\n sb.append(ch);\n }\n return sb.toString();\n }\n}\n[\/code]\n\n\n\n
\nTest run with following memory settings with Sun Java 1.6.0_20-b02 (32bit):<\/p>\n
[code]
\n-Xms100M -Xmx400M
\n[\/code]<\/p>\n
\nAnd the results are:
\n[code]
\n====================
\n=====BASIC==========
\n====================
\nstring len=1000000<\/p>\n
utf8 len=2897030
\ntotal nano = 3675792784 nanos
\n per nano = 3.675792784E7 nanos
\ntotal time = 3675.792784 ms
\n per time = 36.75792784 ms
\n====================
\n=====WRITER=========
\n====================
\nstring len=1000000<\/p>\n
utf8 len=2897030
\ntotal nano = 3252002400 nanos
\n per nano = 3.2520024E7 nanos
\ntotal time = 3252.0024 ms
\n per time = 32.520024 ms
\n====================
\nequal= true
\n[\/code]<\/p>\n
It seems that using Writer for conversion is slightly faster in this test run. However, in real life I believe difference can be even greater due to memory trashing what String.getBytes() causes.<\/p>\n
\nFaster approach could be to extract Encoder from UTF_8 class (i.e. re-implement it). Caveat emptor of such is naturally the fact that re-implementation can introduce mild bugs into logic easily, since most of the internal character set encoding logic must be duplicated in order to do so.<\/p>\n
\nReferences:<\/strong> \nUpdate: 16.6.2010<\/strong> Net result is that this is much slower (over 50% slower) than String.getBytes(). Main reason for slowness is likely the fact that this API cannot use optimized logic in String, which allows direct char[]\u00a0access into original character data.<\/p>\n Notice 1: Notice 2: Notice 3: Lets see what happens when trying to convert random string into UTF8 using different methods. [code lang=”java”] package org.kari.test.string; import java.io.IOException; import java.io.OutputStreamWriter; import java.nio.charset.Charset; import org.apache.log4j.Logger; import org.kari.log.LogUtil; import org.kari.util.DirectByteArrayOutputStream; \/** * Test UTF8 conversion * * @author kari *\/ public class UTF8Test { public static final Logger LOG = LogUtil.getLogger(”utf8”); private static final…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-254","post","type-post","status-publish","format-standard","hentry","category-java"],"_links":{"self":[{"href":"https:\/\/kari.world.ikari.fi\/index.php?rest_route=\/wp\/v2\/posts\/254","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kari.world.ikari.fi\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kari.world.ikari.fi\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kari.world.ikari.fi\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kari.world.ikari.fi\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=254"}],"version-history":[{"count":0,"href":"https:\/\/kari.world.ikari.fi\/index.php?rest_route=\/wp\/v2\/posts\/254\/revisions"}],"wp:attachment":[{"href":"https:\/\/kari.world.ikari.fi\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=254"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kari.world.ikari.fi\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=254"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kari.world.ikari.fi\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=254"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nFaster string to UTF-8 encoding in Java<\/a>
\nFast alternative to java.nio.charset.Charset.decode(..)\/encode(..)<\/a><\/p>\n
\nFor completeness, I tried also what happens if CharEncoder is used
\n[code lang=”java”]
\nCharset cs = Charset.forName(”UTF-8”);
\nByteBuffer data = cs.encode(pStr);
\n[\/code]<\/p>\n
\nIt seems that speed of OutputStreamWriter comes with cost. Logic inside OSW is using StreamEncoder , which allocates temporary char[] for *whole* string contents in order to copy chars from String for fast access, if strings are large, this can cause problems (!).<\/p>\n
\nWhen changing test to use 100 char strings with 1 million iterations, it turned out that String.getBytes() was practically as fast than Writer (or faster, depending if extra gc() due to allocation in StreamEncoder is hit or not).<\/p>\n
\nIn my, not-so-new, hardware, I got String encoding speed in between 54M\/s for plain ASCII chars (random characters in range 32 – 255), and 18M\/s when ”high” UNICODE chars were included (random characters in range 32 – 60000). Not stellar performance, but what is note worthy is that for non-western users, speed is less than 50% (and encoded byte[] storage is tripled), so that needs to be taken in account when trying to ”optimize” strings.<\/p>\n","protected":false},"excerpt":{"rendered":"