FATPIPE

I've been experimenting with fast file transfer over serial to my N8VEM Mark IV single board computer. FATPIPE is my file transfer program, still in development but already useful.

The Mark IV SBC has an RS-232 serial port for the system console, and an RS-422 serial port for higher speed and range. FATPIPE uses the RS-422 port for file transfers. In principle it should also work on the N8's second serial port but this is untested.

On my 36.864MHz Mark IV it runs at the hardware maximum of 0.576Mbps (56KByte/s) with over 98% efficiency. To transfer a 512K ROM image takes 166 seconds with XMODEM at 38400 bps on the console port, but just 9.2 seconds with FATPIPE (18 times faster).

Hardware

I used some spare parts left over from constructing my Mark IV SBC to put together a quick RS-422 to TTL serial adapter on a small piece of prototype board (pictures below). It has the same 6P4C modular jack as the SBC, connected to the same MAX3467 line driver. A six-pin header accepts an FTDI serial to USB board which provides 5V power and connects the TTL level serial to my PC. The FTDI FT232RL chip has a very flexible baud rate generator which can operate at any speed up to 1.5Mbit/sec.

The circuit to connect the components is hopefully obvious. If you're not sure, consult the MAX3467 datasheet or the Mark IV SBC schematics. I fitted a resistor to provide proper termination of the receive differential pair. It is not necessary to use biasing resistors as the MAX3467 has a fail-safe receiver. There is also a 100nF capacitor for power supply decoupling, which serves a dual purpose by preventing me from plugging the FTDI board in backwards. Some drilling was required for the staggered pins and mounting posts on the modular jack.

I have experimented with higher baud rates, including 2Mbit and 3Mbit/sec, but these are unusable as the Z180 ASCI is not fully asynchronous; when the "X1" bit is set to disable the 16X baud rate divider, it requires the incoming data synchronised to a clock pin.

Software

FATPIPE is a CP/M application. At present FATPIPE only support file transfers from the PC to the SBC, since that is all I have required. It uses a simple protocol to transfer files from a python script running on a PC. FATPIPE can transfer multiple files in a single session, sends the filenames in-band along with the file data, and uses a CRC32 for data integrity.

A Python script named "fatpipe-tx" runs on the PC and talks to FATPIPE on the SBC. The user simply runs "fatpipe-tx" on the PC, linsting the files to be sent on the command line. When FATPIPE runs on the SBC it will transfer each of these files and then quit. The "fatpipe-tx" script remains running; this allows FATPIPE to be run repeatedly on the SBC and to receive updated versions of files (I use it mainly to transfer software I am developing).

FATPIPE drives the RS422 port at the highest speed which the Z180 UART (ASCI) will support. I use an 18.432MHz crystal combined with the Z180 clock-doubler which gives a maximum baud rate of 576,000bps.

FATPIPE uses the Z180 DMA engine to transfer data from the serial port, allowing the CPU to compute the CRC32 and write received data to disk while the transfer continues. I wrote a table-driven CRC32 function in assembly. The DMA and fast CRC32 were the key to high speed transfers. It does not use interrupts.

Improvements

I just needed a tool to load software quickly into the machine. It does the minimum required to do the job well and it assumes a good quality transmission circuit. It's had only limited testing but it is now working well. It can't really go any faster but there's still lots of room left for improvement.

FATPIPE does not handle dropped characters well. This is in large part as I have not encountered any errors on the RS422 line. The CRC32 provides a strong data integrity check.

FATPIPE could be easily extended to upload and well as download files. The DMA engine supports both receive and transmit (concurrently, if you wish).

FATPIPE could be modified to speak XMODEM or YMODEM and thus speak to standard communications software instead of a Python script.

Download

Latest verison including full source code. GPL3 licensed.

You will need Python and pySerial ("python-serial" in Debian) to run the "fatpipe-tx" server script. You'll need to modify the script to set the name of your serial port device and the appropriate baud rate. The baud rate will be your CPU clock speed divided by 64.

Pictures