Rick

Rick
Rick

Sunday, October 20, 2013

Why isn't there a ByteBuilder like there is a String Builder.....? ByteBuffer, ByteBuilder?



(I ended up writing ByteBuilder class called ByteBuf. A fast one that is easy to use.)

Question on StackOverFlow

"RE: I am constructing an array of bytes in java and I don't know how long the array will be. I want some tool like Java's StringBuffer that you can just call .append(byte b) or .append(byte[] buf) and have it buffer all my bytes and return to me a byte array when I'm done. Is there a class that does for bytes what StringBuffer does for Strings? It does not look like the ByteBuffer class is what I'm looking for. Anyone have a good solution?"


The  common answer seem to be to use ByteArrayBuffer, and I have used that on numerous occasions and it seems to work well. But... Since that answer was taken, and I have been hacking my own "language extensions for Java". I thought I would try a different approach.

Actual question on StackOverFlow

Let's see. There is the ByteBuffer class in Java.
It has bulk methods that transfer contiguous sequences of bytes from a byte array to hardware buffers. It would do the trick.
It also has absolute and relative get and put methods that read and write byte[]s and other primitives to / for the byte buffer.
It also has methods for compacting, duplicating, and slicing a byte buffer.
// Creates an empty ByteBuffer with a 1024 byte capacity
ByteBuffer buf = ByteBuffer.allocate(1024);

// Get the buffer's capacity
int capacity = buf.capacity(); // 10

buf.put((byte)0xAA); // position=0

// Set the position
buf.position(500);

buf.put((byte)0xFF);

// Read the position 501
int pos = buf.position(); 

// Get remaining byte count
int remaining = buf.remaining(); (capacity - position)
It also has a bulk put to put an array, which is pretty close to the append you were asking for:
public final ByteBuffer put(byte[] src)
See: http://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html#put(byte[])
I wrote my own little lib for manipulating byte arrays. It does more than that, but more on that later. Java Boon! 
:)
You can add them like so
byte [] a = ...
byte [] b = ...
byte [] c = ...

a = add(a, b);
a = add(a, c);
this would give you all of the contents of b, and c after the contents for a.
If you wanted to grow a by 21, you could do the following:
a = grow( letters,  21);
If you wanted to double the size of a, you could do the following:
a = grow( letters,  21);
See...
    byte[] letters =
            arrayOfByte(500);

    assertEquals(
            500,
            len(letters)
    );
Create
    byte[] letters =
            array((byte)0, (byte)1, (byte)2, (byte)3);

    assertEquals(
            4,
            len(letters)
    );
Index
    byte[] letters =
            array((byte)'a', (byte)'b', (byte)'c', (byte)'d');

    assertEquals(
            'a',
            idx(letters, 0)
    );


    assertEquals(
            'd',
            idx(letters, -1)
    );


    assertEquals(
            'd',
            idx(letters, letters.length - 1)
    );


    idx(letters, 1, (byte)'z');

    assertEquals(
            (byte)'z',
            idx(letters, 1)
    );
Contains
    byte[] letters =
            array((byte)'a',(byte) 'b', (byte)'c', (byte)'d');


    assertTrue(
            in((byte)'a', letters)
    );

    assertFalse(
            in((byte)'z', letters)
    );
Slice:
    byte[] letters =
            array((byte)'a', (byte)'b', (byte)'c', (byte)'d');


    assertArrayEquals(
            array((byte)'a', (byte)'b'),
            slc(letters, 0, 2)
    );

    assertArrayEquals(
            array((byte)'b', (byte)'c'),
            slc(letters, 1, -1)
    );

    //>>> letters[2:]
    //['c', 'd']
    //>>> letters[-2:]
    //['c', 'd']

    assertArrayEquals(
            array((byte)'c', (byte)'d'),
            slc(letters, -2)
    );


    assertArrayEquals(
            array((byte)'c', (byte)'d'),
            slc(letters, 2)
    );


    //>>> letters[:-2]
    //     ['a', 'b']
    assertArrayEquals(
            array((byte)'a', (byte)'b'),
            slcEnd(letters, -2)
    );


    //>>> letters[:-2]
    //     ['a', 'b']
    assertArrayEquals(
            array((byte)'a',(byte) 'b'),
            slcEnd(letters, 2)
    );
Grow
    byte[] letters =
            array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e');

    letters = grow( letters,  21);


    assertEquals(
            'e',
            idx(letters, 4)
    );


    assertEquals(
            'a',
            idx(letters, 0)
    );




    assertEquals(
            len(letters),
            26
    );


    assertEquals(
            '\0',
            idx(letters, 20)
    );
Shrink:
    letters =  shrink ( letters, 23 );

    assertArrayEquals(
            array((byte)'a', (byte)'b', (byte)'c'),
            letters

    );
Copy:
    assertArrayEquals(
            array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e'),
            copy(array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e'))

    );
Add:
    assertArrayEquals(
            array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f'),
            add(array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e'), (byte)'f') );
The add actually adds them together by using System.arraycopy (considering Unsafe, but not yet).
Add one array to another:
    assertArrayEquals(
            array(     (byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f'),
            add( array((byte)'a', (byte)'b', (byte)'c', (byte)'d'), array((byte)'e', (byte)'f') )

    );
Insert:
    assertArrayEquals(
            array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f', (byte)'g'),
            insert( array((byte)'a', (byte)'b', (byte)'d', (byte)'e', (byte)'f', (byte)'g'), 2, (byte)'c' )

    );

    assertArrayEquals(
            array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f', (byte)'g'),
            insert( array((byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f', (byte)'g'), 0, (byte)'a' )

    );

    assertArrayEquals(
            array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f', (byte)'g'),
            insert( array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'g'), 5, (byte)'f' )

    );
Here is a peek at a few of the methods:
public static byte[] grow(byte [] array, final int size) {
    Objects.requireNonNull(array);

    byte [] newArray  = new byte[array.length + size];
    System.arraycopy(array, 0, newArray, 0, array.length);
    return newArray;
}



public static byte[] grow(byte [] array) {
    Objects.requireNonNull(array);

    byte [] newArray  = new byte[array.length *2];
    System.arraycopy(array, 0, newArray, 0, array.length);
    return newArray;
}


public static byte[] shrink(byte[] array, int size) {
    Objects.requireNonNull(array);

    byte[] newArray = new byte[array.length - size];

    System.arraycopy(array, 0, newArray, 0, array.length-size);
    return newArray;
}




public static byte[] copy(byte[] array) {
    Objects.requireNonNull(array);
    byte[] newArray = new byte[array.length];
    System.arraycopy(array, 0, newArray, 0, array.length);
    return newArray;
}


public static byte[] add(byte[] array, byte v) {
    Objects.requireNonNull(array);
    byte[] newArray = new byte[array.length + 1];
    System.arraycopy(array, 0, newArray, 0, array.length);
    newArray[array.length] = v;
    return newArray;
}

public static byte[] add(byte[] array, byte[] array2) {
    Objects.requireNonNull(array);
    byte[] newArray = new byte[array.length + array2.length];
    System.arraycopy(array, 0, newArray, 0, array.length);
    System.arraycopy(array2, 0, newArray, array.length, array2.length);
    return newArray;
}



public static byte[] insert(final byte[] array, final int idx, final byte v) {
    Objects.requireNonNull(array);

    if (idx >= array.length) {
        return add(array, v);
    }

    final int index = calculateIndex(array, idx);

    //Object newArray = Array.newInstance(array.getClass().getComponentType(), array.length+1);
    byte [] newArray = new byte[array.length+1];

    if (index != 0) {
        /* Copy up to the location in the array before the index. */
        /*                 src     sbegin  dst       dbegin   length of copy */
        System.arraycopy( array,   0,      newArray, 0,       index );
    }


    boolean lastIndex = index == array.length -1;
    int remainingIndex = array.length - index;

    if (lastIndex ) {
        /* Copy the area after the insert. Make sure we don't write over the end. */
        /*                 src  sbegin   dst       dbegin     length of copy */
        System.arraycopy(array, index,   newArray, index + 1, remainingIndex );

    } else {
        /* Copy the area after the insert.  */
        /*                 src  sbegin   dst       dbegin     length of copy */
        System.arraycopy(array, index,   newArray, index + 1, remainingIndex );

    }

    newArray[index] = v;
    return  newArray;
}


public static byte[] insert(final byte[] array, final int fromIndex, final byte[] values) {
    Objects.requireNonNull(array);

    if (fromIndex >= array.length) {
        return add(array, values);
    }

    final int index = calculateIndex(array, fromIndex);

    //Object newArray = Array.newInstance(array.getClass().getComponentType(), array.length+1);
    byte [] newArray = new byte[array.length +  values.length];

    if (index != 0) {
        /* Copy up to the location in the array before the index. */
        /*                 src     sbegin  dst       dbegin   length of copy */
        System.arraycopy( array,   0,      newArray, 0,       index );
    }


    boolean lastIndex = index == array.length -1;

    int toIndex = index + values.length;
    int remainingIndex = newArray.length - toIndex;

    if (lastIndex ) {
        /* Copy the area after the insert. Make sure we don't write over the end. */
        /*                 src  sbegin   dst       dbegin     length of copy */
        System.arraycopy(array, index,   newArray, index + values.length, remainingIndex );

    } else {
        /* Copy the area after the insert.  */
        /*                 src  sbegin   dst       dbegin     length of copy */
        System.arraycopy(array, index,   newArray, index + values.length, remainingIndex );

    }

    for (int i = index, j=0; i < toIndex; i++, j++) {
        newArray[ i ] = values[ j ];
    }
    return  newArray;
}
More....

Comments
Add your comment


replied 1 day ago:
There is a follow up to this: http://rick-hightower.blogspot.com/2013/10/boon-byte-builder-round-2-read-and.html Many languages have slice notation (Ruby, Groovy and Python). Boon adds this to Java. Boon has three slc operators: slc, slc (start only), and slcEnd. With Boon you can slice strings, arrays (primitive and generic), lists, sets, tree sets, tree map's and more. This article explains slice notation and how it is implemented in Boon. It shows how to use slice notation with arrays, sets, tree sets, etc. You can use Boon slice notation to search TreeMaps and TreeSets easily. With Boon, slicing primitive arrays does not use auto-boxing so it is very efficient. Slice notations - a gentle introduction. Boon is "Simple opinionated Java for the novice to expert level Java Programmer". Boon is a "Low Ceremony. High Productivity" framework. It is meant to be a "real boon to Java to developers!"


replied 1 day ago:
There is another follow up to this: http://rick-hightower.blogspot.com/2013/10/introducing-boon-for-java.html A Boon to Java Development (introducing Boon!) Boon comes with helper methods that allow you to easily read files, create lists, sets, maps, concurrent maps, sorted maps, sorted sets, etc. It provides slice notation, searching, easy IO (files, http, etc.). The helper methods are safeList, list, set, sortedSet, safeSet,safeSortedSet, etc. The idea is to make Java feel more like list and maps are built-in types.


replied 1 day ago:
This one shows how to work with other primitives and Boon's byte array support.


replied 1 day ago:
http://rick-hightower.blogspot.com/2013/10/boon-byte-builder-round-2-read-and.html This one shows how to work with other primitives and Boon's byte array support.

Hell bells! I wrote after all

Have you ever wanted an easy to use buffer array that grow automatically and/or you can give it a fix size and just add stuff to it? I have. I wrote it too.
Look.. I can write strings to it (it converts them to UTF-8).
      ByteBuf buf = new ByteBuf();
      buf.add(bytes("0123456789\n"));
      buf.add("0123456789\n");
      buf.add("0123456789\n");
      buf.add("0123456789\n");
      buf.add("0123456789\n");
      buf.add("0123456END\n");


  }
Then later I can read the String out of the buffer:
      String out = new String(buf.readAndReset(), 0, buf.len());
      assertEquals(66, buf.len());
      assertTrue(out.endsWith("END\n"));
I never have to set the size of the array. It will auto-grow as needed in an efficient manner.
If I know exactly how large my data is going to be than I can save some bounds checking by using createExact.
      ByteBuf buf = ByteBuf.createExact(66);
      buf.add(bytes("0123456789\n"));
      buf.add("0123456789\n");
      buf.add("0123456789\n");
      buf.add("0123456789\n");
      buf.add("0123456789\n");
      buf.add("0123456END\n");
      assertEquals(66, buf.len());
If I use create exact, then I am saying... hey.. I know exactly how big it can grow to and it will never go over this number and if it does...you can hit me over the head with a sack of rocks!
THROWS AN EXCEPTION!!!!
      ByteBuf buf = ByteBuf.createExact(22);
      buf.add(bytes("0123456789\n"));
      buf.add("0123456789\n");
      buf.add("0123456789\n");
      buf.add("0123456789\n");
      buf.add("0123456789\n");
      buf.add("0123456END\n");
It works with doubles.
      ByteBuf buf = ByteBuf.createExact(8);

      //add the double
      buf.add(10.0000000000001);

      byte[] bytes = buf.readAndReset();
      boolean worked = true;

      worked |= idxDouble(bytes, 0) == 10.0000000000001 || die("Double worked");
It works with float.
      ByteBuf buf = ByteBuf.createExact(8);

      //add the float
      buf.add(10.001f);

      byte[] bytes = buf.readAndReset();
      boolean worked = true;

      worked |= buf.len() == 4 || die("Float worked");


      //read the float
      float flt = idxFloat(bytes, 0);

      worked |= flt == 10.001f || die("Float worked");
It works with int.
      ByteBuf buf = ByteBuf.createExact(8);

      //Add the int to the array
      buf.add(99);

      byte[] bytes = buf.readAndReset();
      boolean worked = true;


      //Read the int back
      int value = idxInt(bytes, 0);

      worked |= buf.len() == 4 || die("Int worked length = 4");
      worked |= value == 99 || die("Int worked value was 99");
It works with char.
      ByteBuf buf = ByteBuf.createExact(8);

      //Add the char to the array
      buf.add('c');

      byte[] bytes = buf.readAndReset();
      boolean worked = true;


      //Read the char back
      int value = idxChar(bytes, 0);

      worked |= buf.len() == 2 || die("char worked length = 4");
      worked |= value == 'c' || die("char worked value was 'c'");
It works with short.
      ByteBuf buf = ByteBuf.createExact(8);

      //Add the short to the array
      buf.add((short)77);

      byte[] bytes = buf.readAndReset();
      boolean worked = true;


      //Read the short back
      int value = idxShort(bytes, 0);

      worked |= buf.len() == 2 || die("short worked length = 2");
      worked |= value == 77 || die("short worked value was 77");
It even works with bytes.
      ByteBuf buf = ByteBuf.createExact(8);

      //Add the byte to the array
      buf.add( (byte)33 );

      byte[] bytes = buf.readAndReset();
      boolean worked = true;


      //Read the byte back
      int value = idx(bytes, 0);

      worked |= buf.len() == 1 || die("byte worked length = 1");
      worked |= value == 33 || die("byte worked value was 33");
You can add all sorts of primitives to your byte array.
      boolean worked = true;
      ByteBuf buf = ByteBuf.create(1);

      //Add the various to the array
      buf.add( (byte)  1 );
      buf.add( (short) 2 );
      buf.add( (char)  3 );
      buf.add(         4 );
      buf.add( (float) 5 );
      buf.add( (long)  6 );
      buf.add( (double)7 );

      worked |= buf.len() == 29 || die("length = 29");


      byte[] bytes = buf.readAndReset();

      byte myByte;
      short myShort;
      char myChar;
      int myInt;
      float myFloat;
      long myLong;
      double myDouble;

      myByte    =   idx       ( bytes, 0 );
      myShort   =   idxShort  ( bytes, 1 );
      myChar    =   idxChar   ( bytes, 3 );
      myInt     =   idxInt    ( bytes, 5 );
      myFloat   =   idxFloat  ( bytes, 9 );
      myLong   =    idxLong   ( bytes, 13 );
      myDouble  =   idxDouble ( bytes, 21 );

      worked |= myByte   == 1 || die("value was 1");
      worked |= myShort  == 2 || die("value was 2");
      worked |= myChar   == 3 || die("value was 3");
      worked |= myInt    == 4 || die("value was 4");
      worked |= myFloat  == 5 || die("value was 5");
      worked |= myLong   == 6 || die("value was 6");
      worked |= myDouble == 7 || die("value was 7");
Once you call byte[] bytes = buf.readAndReset() then you are done with buf. Once you ask for the bytes, it becomes useless as it sets the internal byte array to nothing. It is ok. Just create another!
You can even use the buffer you were just using as in
  ByteBuf buf2 = new ByteBuf.create(bytes); 
This is because no buffer gets copied. ByteBuf writes to the buffer you give it. If you want another copy to be given to ByteBuf then do this:
  ByteBuf buf2 = new ByteBuf.create( copy(bytes) ); 
This is boon after all. :)

No comments:

Post a Comment

Kafka and Cassandra support, training for AWS EC2 Cassandra 3.0 Training