SBUS

SBUS

File:Sbus-diagram.png

sbus.c

sbus.h

Overview

While many CPUs contain some type of parallel interface that we can use for FPGA communication, the Cavium CNS2132 only has an SPI interface. It functions quite well, however some programming considerations must be made with this processor. Since SPI cannot be shared, we have implemented the SBUS which is a locking mechanism to communicate to the 128k of registers in the FPGA. We use this on all of our applications that communicate with the FPGA which includes sdctl, nandctl, xuartctl, daqctl, spiflashctl, dioctl, and ts7500ctl.

If you use the ctl applications as libraries, they will abstract this.

Code Example

This example shows accessing DIO directly without dioctl.

#include "sbus.h"
#include <assert.h>
#include <string.h>
#include <stdio.h>

#define DIO_Z 2

/*******************************************************************************
* Main: accept input from the command line and act accordingly.
*******************************************************************************/
int main(int argc, char **argv)
{
   int pin;
   int val;
   int returnedValue;
         
   // Check for invalid command line arguments
   if ((argc > 4) | (argc < 3))
   {
       printf("Usage: %s <get|set> <pin#> <set_value (0|1|2)>\n", argv[0]);
       return 1;
   }
   
   // We only want to get val if there are more than 3 command line arguments
   if (argc == 3)
      pin = strtoul(argv[2], NULL, 0);
   else
   {
      pin = strtoul(argv[2], NULL, 0);
      val = strtoul(argv[3], NULL, 0);
   }
   
   // If anything other than pins 5 through 40, fail program
   assert(pin <= 40 && pin >= 5);

   // Parse through the command line arguments, check for valid inputs, and exec
   if (!(strcmp(argv[1], "get")) && (argc == 3))
   {
      sbuslock();
      returnedValue = getdiopin(pin);
      sbusunlock();
      
      printf("pin#%d = %d \n", pin, returnedValue);
   }
   else if(!(strcmp(argv[1], "set")) && (argc == 4) && (val <= 2))
   {
      sbuslock();
      setdiopin(pin, val);
      sbusunlock();
      
      printf("pin#%d set to %d\n", pin, val);
   }   
   else
   {
      printf("Usage: %s <get|set> <pin#> <set_value (0|1|2)>\n", argv[0]);
      return 1;
   }
   return 0;
}

SBUS to FPGA Protocol

For the most part, the application and kernel level programmer will not be interested in the following sections as the below information is completely abstracted away from the user by the SBUS C API and the various TS-specific utilities. Even the FPGA programmer is abstracted from the details of this protocol by the spi_sbus.v Verilog module which turns the SBUS SPI traffic into WISHBONE bus cycles.

The SBUS protocol allows for 16 specific 16 bit registers (32-bytes of register space) to be read or written and has provision for the FPGA internal WISHBONE bus cycles to take any amount of time to be ack'ed through the mechanism of SBUS retries. SPI bandwidth efficient burst register reads/writes are also possible when writing/reading continuously to the same address.

The general protocol consists of a 24-bit SPI transaction framed by the SPI CS#. Bits are clocked in on the rising edge and the first MOSI (input) bit signifies whether the bus cycle is a read or write. This is followed by the 4 address bits MSB first. The remaining 19 bits depend on whether the cycle is a read or write. For reads, the WISHBONE bus cycle is started as soon as the last address bit is clocked in.

The 24-bit SPI SBUS data format - bit 23 (WE) is first on wire:

  16-bit WISHBONE READ operation
     23|22|21|20|19|18|17|16|15|14|13|12|11|10| 9| 8| 7| 6| 5| 4| 3| 2| 1| 0|
     --|--|--|--|--|--|--|--|--|--------------------------------------------|
  SI: 0|A3|A2|A1|A0|  |  |  | B|<------------should be zero---------------->|
  SO:              |X2|X1|X0|MSB<----------returned READ data----------->LSB|
  
  16-bit WISHBONE WRITE operation:
     23|22|21|20|19|18|17|16|15|14|13|12|11|10| 9| 8| 7| 6| 5| 4| 3| 2| 1| 0|
     --|--|--|--|--|-----------------------------------------------|--|--|--|
  SI: 1|A3|A2|A1|A0|MSB<-----------WRITE data------------------>LSB|N2|N1|N0|
  SO:                                                              |Y2|Y1|Y0|
  
  * A3-A0: Address bits (sent by CPU)
  
  * X2-X0: ack bits for reads (sent by FPGA). If any are 1, the following
    data is valid.  If they are all 0, the WISHBONE cycle did not complete in 
    time and must be immediately retried.
  
  * Y2-Y0: ack bits for write (sent by FPGA). If any are 1, the write
    cycle was completed.  If all are 0, the WISHBONE write did not complete
    and must be immediately retried.
  
  * B: Burst read.  Setting this to 1 starts another WISHBONE read cycle for
    the same address.  See "SBUS Burst mode" below.
  
  * N2-N0: For burst writes, this represents bits 15-13 of the next burst 
    write data, otherwise these bits are Dont-Cares.  See "SBUS Burst mode"
    below.

When the WISHBONE cycle does not assert the wb_ack_i signal in time and the transaction must be retried, it is not necessary to deassert and reassert SPI CS# in between each retry, though it is necessary to start a new bus cycle.

SBUS Burst Mode

After a normal 24-bit SBUS READ/WRITE cycle takes place, it is possible to continue reading or writing to the same WISHBONE register by keeping CS# asserted and continuing the SPI clock 16 bits at a time. However, once a burst is started, there is no provision to allow for an occasional long WISHBONE cycle and all WISHBONE cycles must be able to complete in 14 SPI clock cycles otherwise results are undefined. The only registers used in TS-7500 implementation for bursts are the SPI and SD card data registers.

For burst WRITEs, one issues a normal WRITE cycle as above with possible retries making sure that the 3 LSB (last) bits sent represent the 3 MSB bits (15-13) of the next WRITE data. Subsequent writes take place 16 bits at a time while CS# is held asserted. The burst stops once CS# is deasserted.

For burst READs, one issues a normal 24-bit READ cycle and 0 or more retries, but with the "B" bit set. Once the burst is started, bit 15 must always be set as long as the read burst is to continue. On the very last READ in the burst, the 16-bit data sent by the FPGA on SO should be all zeros.

SBUS Verilog Implementation

The spi_sbus Verilog module WISHBONE clock output is synchronous to the actual SPI clock. This also means when the SPI bus becomes quiet, there are no clocks to the WISHBONE bus.

The sel_i input is used as a byte lane select. On the TS-7500, there are 2 dedicated GPIO pins from the CPU to the FPGA that are used for this. Without the sel_i inputs, it is not possible to have 8-bit reads/writes.

On the TS-7500 implementation, there are 2 extra address lines via GPIO pins on the CPU. This allows a total address space of 4 * 32 bytes == 128 bytes == 64 individual 16-bit registers. For these GPIO lines (and the byte lane selects) it is important that they be setup in advance of the SPI transaction and remain stable during it.

The spi_sbus_resync Verilog module is similar to the spi_sbus module except that it can use an asynchronous WISHBONE clock. This is what is used in the TS-7500 since the common WISHBONE clock is 75Mhz but the SPI clock can be any rate from 32.5Mhz to 50Mhz depending on CPU speed. A side effect of this though is that clock resynchronization overhead can make even the shortest of bus cycles miss the 3 SPI clock window for acknowledgment and force the majority of bus cycles into at least one retry. For this reason, a 2nd WISHBONE master interface is provided for WISHBONE slaves that are okay with using the SPI SCK as their clock -- to use this bus, spiwbm_en_i has to be asserted before the next clock edge after spiwbm_cyc_o, spiwbm_stb_o, and spiwbm_adr_o are asserted and valid.

There is another quirk about the spi_sbus_resync module because burst writes accumulate in a 16-entry write buffer clocked from the SPI clock domain. Flushing of this buffer happens automatically before any subsequent READ or WRITE transaction, but it is possible for there to be writes not yet posted to the asynchronous WISHBONE clock domain at the completion of a WRITE burst. It therefore may be important to follow up any WRITE burst with a READ to insure the last WRITE in the burst completes 100%. This becomes especially significant on the TS-7500 which includes 2 extra GPIO address bits that must remain stable while there are writes pending in the write buffer. For example, it would not be safe to allow users to change those GPIO address bits while there is still pending WRITES in the write buffer.