How can I analyse large packets on an Arduino?

Question

I'm trying to send a large packet to an Arduino with an ethernet shield on it. I'm using the built-in RX buffer of the W5100 controller to store the data, but I need more memory (at least 16K of buffer space, the W5100 only has 8K of RX buffer space). I have control over the server side of things as I'm writing the server process myself, so I can take solutions on that side.

Is there some hack I can use in the TCP protocol to have the packet broken up?

Edit: I'm using the SFML 2.1 network API on the server side.

• Chris Stratton: TCP does indeed allow telling the other end about your buffer window. I'd be a bit surprised if the actual link level packets are individually too big. But an Arduino is not a good choice where you need large buffers - by the time you added the ethernet shield you could have had orders of magnitude more memory at a comparable price.
• HSchmale: @ChrisStratton The person I'm working with on this with insists on using arduino for this. On the server side I'm kind of treating the large string I need to send as one big packet.
• Chris Stratton: If you are using TCP on a conventional system, you have very little visibility into what a "packet" is. It's quite likely that your single network API call payload gets transmitted as many smaller packets. In an embedded implementation sometimes the layers collapse and you do have more detail, but the Arduino side shouldn't have to claim more than one link-level packet worth of data at a time.
• Jon: @ChrisStratton, please confirm that the Arduino is not capable of receiving the "many smaller packets" and that the the OP should just send smaller packets. He could add a memory shield and fix the overflow, but that wouldn't address his question about TCP.
• Chris Stratton: An application developer does not "send packets" over TCP, they send data which the TCP and lower level stacks packetize. If the Arduino can or cannot receive the small link level packets depends on the software stack on the Ethernet module. Getting it to properly set its TCP window should help too, as that should stop data flow until the Arduino has drained the module's buffer. Anyone proposing a "memory shield" (as in RAM) needs to have their head examined, as using the right MCU for the job is drastically faster and cheaper.

Answer 1

All you need to do is analyse the packet on-the-fly using a state machine.

http://www.gammon.com.au/statemachine

You certainly don't need to hold the entire packet at once, however you do need room for the information the packet contains.

Say, for example, that the packet, although wordy, contains 4 x 32-bit numbers of interest. All you need memory for is those 4 numbers, plus a minor extra amount for the current data you are processing (eg. the current state, and the current number being assembled).

Conceptually it is like streaming a movie to your PC. The PC doesn't need to hold the whole movie, it just needs to hold the current frame, and a few seconds in advance, just to keep things running smoothly.

• dda: As usual, Nick to the rescue. Your website is a treasure trove!