In Java, receive audio data streaming from a C# app over TCP sockets and play streaming as it is received

Question

I have some raw PCM data for audio in a C# app. I need to get it into a Java application and playing as audio.

As a first step, I want the Java application to play the audio immediately as it is being streamed in (not saving a temporary file and playing that, playing it as it is received).

I am having problems formatting the PCM audio data in C# properly and sending it to the Java application. I am currently just using a simple TCP socket system to do so, calling the C# app (made in Unity) the "client" and the Java app the "server."

On the Unity C# client, I am using NatMic to get the raw PCM data for the Mic. This function is called many times per second as data comes in:

void IAudioProcessor.OnSampleBuffer(float[] sampleBuffer, int sampleRate, int channelCount, long timestamp)

I have a float array and two ints to describe the audio data.

How do I package this data up as bytes to be sent to Java?

I have a TCP set up working where I can send bytes to Java, and it seems to work fine when I test just sending simple string messages back and forth between the client and server.

But when I try to package up the audio and send it over, it never works. I always get an error like this:

javax.sound.sampled.UnsupportedAudioFileException: could not get audio input stream from input stream

Here is a snippet on the Java side of how I was trying to play the audio on the Java side as a test (when the client connects I run this once):

 inputStream = client.getInputStream();
 bInputStream = new BufferedInputStream(inputStream);
 Clip clip = AudioSystem.getClip();
 audioStream = AudioSystem.getAudioInputStream(bInputStream);
 clip.open(audioStream);
 clip.start();

How should I be packaging the float[], int, and int to send to the Java server so it will recognize it as an audio stream?

• duffymo: What does Clip listener expect? docs.oracle.com/javase/8/docs/technotes/guides/sound/…
• gpasch: so this really a c# question: how to package the ints into bytes: find out!!

Answer 1

First off, you will need to output the data as a SourceDataLine, not a Clip. The Clip cannot play until it has received the entire sound file.

The usual plan is to read the input line in whatever manner you want and convert it to the proper format in a loop calling the sdl.write() method.

Something along the following lines should work, for incoming data

while(playerRunning)
{
 readBuffer = getInputPCM();
 audioBytes = convertToFollowAudioFormat(readBuffer);
 sourceDataLine.write(audioBytes, 0, sdlBufferSize);
}

If your incoming data is floats, ranging from -1 to 1, and you are outputting in the basic "CD Quality" Java format (44100 fps, 16-bit, stereo, little endian), then the conversion might look like the following:

for (int i = 0, n = buffer.length; i < n; i++)
{
 buffer[i] *= 32767;
 audioBytes[i*2] = (byte) buffer[i];
 audioBytes[i*2 + 1] = (byte)((int)buffer[i] >> 8 );
}
return audioBytes;

Depending on the form of your PCM, you might have to do a bit more or less. For example, if you are reading ints or shorts, ranging from -32767 to 32767, then the multiplication step is not needed.

To get a SourceDataLine outputting "CD Quality" audio format on the default audio line:

AudioFormat audioFormat = new AudioFormat(AudioFormat.Encoding.PCM_SIGNED, 
 44100, 16, 2, 4, 44100, false);
Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
SourceDataLine sdl = (SourceDataLine)AudioSystem.getLine(info);

As far as your getInputPCM method, I'm assuming you are able to present only PCM data in the stream and there is no need for further parsing. If you want to do the task of converting the PCM to bytes in C# prior to shipping, that is also perfectly valid.

There is more written about various output formats on the Java Tutorial's Sampled Package trail.

For the above, I'm mostly copying/editing code from my github project AudioCue, which is a sort of souped-up Clip.

• bk999: Hi Phil, I really appreciate you taking the time to walk me through this. Based on the info you have provided, I have been able to get it to play back some sound! But the sound is just some white noise. When data is received from the client, I write to the sdl like this: byte[] audioBytes = convertAudioBytes(bytes); sourceDataLine.write(audioBytes, 0, audioBytes.length); The PCM in the client is normalized to -1 to 1, so I ran it through a new function based on your converter prior to writing it to the sdl.
• bk999: I had made a mistake with something on the C# side (turns out I had already converted the bytes there, so when I ran them through the converter on the Java side, it created the white noise, woops). I can hear the audio now! There are some small breaks (sounds kind of like static) in it, but it is the true audio being transmitted and should work for my purposes. Thanks again for taking the time to go over it.
• bk999: And 'static' was caused by using a wrong size for incoming byte array (too small to contain all of the buffer data), so that's gone too. Sounds perfect now.
• Phil Freihofner: Awesome! Congratulations!