cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
mahkoe
Observer
Observer
1,134 Views
Registered: ‎02-08-2019

AXI-Stream FIFO RDFO jumps from 256 to 0

Jump to solution

I am using an AXI-Stream FIFO with the following configuration parameters;

 

axisfifo_params.PNG

It is address-mapped as follows:

axisfifo_addrs.PNG

 

I am constantly reading packets from this FIFO using the following sequence:

  1. Poll RDFO until it is nonzero
  2. Read RLR
  3. Read ceil(RLR/4) words from RDFD
  4. Check ISR for error interrupts
  5. Go back to step 1

In my design, all my packets have two flits.

 

I ran a test which generated a bit more than 512 words, so for a while, the FIFO was full and backpressuring the source. Here is an abbreviated transcript of the logs from reading data from this FIFO:

RDFO = 514
Read back: 0x0 (0)
Read back: 0x12fa (4858)
ISR = 0xc100000
RDFO = 514
Read back: 0x1 (1)
Read back: 0x12fa (4858)
ISR = 0xc100000
RDFO = 514
Read back: 0x0 (0)
Read back: 0x97d (2429)
ISR = 0xc100000

.... About 50 packets where RDFO was at 514 ....

RDFO = 514
Read back: 0x0 (0)
Read back: 0x2e (46)
ISR = 0xc100000
RDFO = 514
Read back: 0x1 (1)
Read back: 0x2e (46)
ISR = 0xc100000
RDFO = 512
Read back: 0x0 (0)
Read back: 0x17 (23)
ISR = 0xc100000
RDFO = 510
Read back: 0x1 (1)
Read back: 0x17 (23)
ISR = 0xc100000
RDFO = 508
Read back: 0x0 (0)
Read back: 0x46 (70)
ISR = 0xc100000

.... 120 packets, where RDFO decreases by 2 each time ....

RDFO = 264
Read back: 0x0 (0)
Read back: 0xfb (251)
ISR = 0xc100000
RDFO = 262
Read back: 0x1 (1)
Read back: 0xfb (251)
ISR = 0xc100000
RDFO = 260
Read back: 0x0 (0)
Read back: 0x2f2 (754)
ISR = 0xc100000
RDFO = 258
Read back: 0x1 (1)
Read back: 0x2f2 (754)
ISR = 0xc100000
RDFO = 256
Read back: 0x0 (0)
Read back: 0x179 (377)
ISR = 0xc100000
RDFO = 0

... RDFO is 0 from now on ....

 

Notice that RDFO is 256 and then suddenly dies and goes to 0. This is causing me to miss packets.

Does anyone know why this is happening, or at least how to fix it?

 

Thanks

 

EDIT: By the way, I am using Vivado 2018.3

0 Kudos
1 Solution

Accepted Solutions
mahkoe
Observer
Observer
935 Views
Registered: ‎02-08-2019

I went into the AXI-Stream FIFO source code and was able to confirm the theory: the FIFO's packet capacity is one quarter its word capacity. Here is the relevant quote form the source:

 

   -- This FIFO is to store the RX packet length in bytes
--   COMP_rx_len_fifo: entity lib_fifo_v1_0_12.sync_fifo_fg
--        generic map (
--          C_FAMILY             => C_FAMILY,
--          C_DCOUNT_WIDTH       => 4 ,
--          C_ENABLE_RLOCS       => 0 , -- not supported in sync fifo
--          C_HAS_DCOUNT         => 1 ,
--          C_HAS_RD_ACK         => 0 ,
--          C_HAS_RD_ERR         => 0 ,
--          C_HAS_WR_ACK         => 0 ,
--          C_HAS_WR_ERR         => 0 ,
--          C_HAS_ALMOST_FULL    => 0 ,
--          C_MEMORY_TYPE        => 0 ,  -- 0 = distributed RAM, 1 = BRAM
--          C_PORTS_DIFFER       => 0 ,  
--          C_RD_ACK_LOW         => 0 ,
--          C_USE_EMBEDDED_REG   => 0 ,
--          C_READ_DATA_WIDTH    => RX_MAX_PKT_SIZE,
--          C_READ_DEPTH         => C_RX_FIFO_DEPTH/4,
--          C_RD_ERR_LOW         => 0 ,
--          C_WR_ACK_LOW         => 0 ,
--          C_WR_ERR_LOW         => 0 ,
--          C_PRELOAD_REGS       => 1 ,  -- 1 = first word fall through
--          C_PRELOAD_LATENCY    => 0 ,  -- 0 = first word fall through
--          C_WRITE_DATA_WIDTH   => RX_MAX_PKT_SIZE,
--          C_WRITE_DEPTH        => C_RX_FIFO_DEPTH/4
--          )
--        port map (
--          Clk                  => Bus2IP_Clk,
--          Sinit                => sig_rxd_reset,
--          Din                  => fg_rxd_wr_length,
--          Wr_en                => rx_len_wr_en,
--          Rd_en                => rx_len_rd_en,
--          Dout                 => sig_rxd_rd_length,
--          Almost_full          => open,
--          Full                 => open,
--          Empty                => rx_fg_len_empty,
--          Rd_ack               => open,
--          Wr_ack               => open,
--          Rd_err               => open,
--          Wr_err               => open,
--          Data_count           => open
--          );


   COMP_rx_len_fifo : xpm_fifo_sync
   generic map (
      DOUT_RESET_VALUE => "0", 
      ECC_MODE => "no_ecc",       
      FIFO_MEMORY_TYPE => "distributed", 
      FIFO_READ_LATENCY => 1,      
      FIFO_WRITE_DEPTH => C_RX_FIFO_DEPTH/4, 
      FULL_RESET_VALUE => 0,       
      PROG_EMPTY_THRESH => 10,     
      PROG_FULL_THRESH => 10,      
      RD_DATA_COUNT_WIDTH => 1,    
      READ_DATA_WIDTH => RX_MAX_PKT_SIZE,       
      READ_MODE => "fwft",         
      USE_ADV_FEATURES => "0", 
      WAKEUP_TIME => 0,            
      WRITE_DATA_WIDTH => RX_MAX_PKT_SIZE,    
      WR_DATA_COUNT_WIDTH => 1    
   )
   port map (
      almost_empty => open,  
      almost_full => open,      
      data_valid => open,       
      dbiterr => open,             
      dout => sig_rxd_rd_length,            
      empty => rx_fg_len_empty,                
      full => open,                   
      overflow => open,          
      prog_empty => open,       
      prog_full => open,        
      rd_data_count => open, 
      rd_rst_busy => open,      
      sbiterr => open,             
      underflow => open,         
      wr_ack => open,          
      wr_data_count => open,  
      wr_rst_busy => open,      
      din => fg_rxd_wr_length,                
      injectdbiterr => '0',  
      injectsbiterr => '0', 
      rd_en => rx_len_rd_en,                 
      rst => sig_rxd_reset,                    
      sleep => '0',                   
      wr_clk => Bus2IP_Clk,     
      wr_en => rx_len_wr_en           
   );
    rx_str_ready_i   <=  not (sig_rxd_wr_full);
    rx_str_ready     <= rx_str_ready_i;
  end generate grxd;

end beh;

View solution in original post

0 Kudos
9 Replies
dgisselq
Scholar
Scholar
1,113 Views
Registered: ‎05-21-2015

@mahkoe,

> Does anyone know why this is happening ... ?

Nope.  I'd have to see more information.  Perhaps you'd like to share what's feeding this FIFO and what's consuming the results?  Or are you in a complete simulation environment, from which I'd ask if you could share your simulation test bench?

Dan

0 Kudos
mahkoe
Observer
Observer
1,107 Views
Registered: ‎02-08-2019
Hi Dan,

Over the last few hours I did manage to find out some things. If the AXI-Stream FIFO fills up more than halfway (i.e. more than 256 words) before anything is read, then RDFO jumps to 0 after you read 256 words.

I just finished trying a new bitstream where I set the AXI-Stream FIFO depth to 1024. Sure enough, if the FIFO fills beyond 512 words before reading from it, then RDFO dies to zero after you 512 words.

More weird quirks: if you encounter this sudden RDFO problem, but then feed one new flit into the receive FIFO (from the slave RXD AXI Stream interface) RDFO jumps back up to what it should be, but then immediately dies back to 0 once you read the next packet.

More info: my packets are all exactly two flits long. Actually, I'd be thrilled to send you my design (it's an AXI-Stream debugging tool, similar to the one you recently posted on your blog; I'm actually a big fan yours) but it may be a little too large to attach here and to subsequently explain.

More more info: I will try running a minimal working example in simulation that illustrates the problem behaviour. This could easily take me a few hours, so you'll have to bear with me.

Thanks!

- Marco
0 Kudos
mahkoe
Observer
Observer
1,085 Views
Registered: ‎02-08-2019

I somehow managed to recreate the issue in simulation.

The attached archived project (for Vivado 2018.3) should show the problem. (Note: the file was too big for Xilinx forums, so I've uploaded it to my Google drive: https://drive.google.com/file/d/1nneSYAd7Fwl1DrrOw2r0m-0JqYjyiR2z/view?usp=sharing)

For posterity, here are screenshots and the code I used:

The filename of the block diagram is axis_fifo.

axisfifo_sim_bd.PNGaxisfifo_sim_addr.PNG

axis_count.v:

`timescale 1ns / 1ps

//A simple 32-bit AXI Stream counter. Obeys backpressure

module axis_count (
    input wire clk,
    input wire rst,
    
    output wire [31:0] count_TDATA,
    output wire count_TVALID,
    input wire count_TREADY,
    output wire count_TLAST
);

    reg [31:0] cnt = 0;
    
    always @(posedge clk) begin
        if (count_TREADY) begin
            cnt <= cnt + 1;
        end
        
        if (rst) begin
            cnt <= 0; //Uses last assignment wins rule
        end
    end
    
    assign count_TVALID = !rst;
    assign count_TDATA = cnt;
    assign count_TLAST = cnt[0];

endmodule

 

axis_sink.v:

`timescale 1ns / 1ps

//A simple sink slave for AXI Stream

module axis_sink(
	input wire clk, //Dummy clock input to make annoying warning go away

	input wire [31:0] sink_TDATA,
	input wire sink_TVALID,
	output wire sink_TREADY,
	input wire [3:0] sink_TKEEP,
	input wire sink_TLAST
);

	assign sink_TREADY = 1;

endmodule

tb.sv

`timescale 1ns / 1ps
`define ps axis_fifo.axis_fifo_i.PS.inst
`define EDGE(n) repeat(n) @(posedge tb_clk)
`define WRITE_AT(addr, data) `ps.write_burst_strb(addr, 4'h0, 3'b010, 2'b01, 2'b0, 4'h0, 3'b0, data, 1, 16'hF, 4, dummy)
`define READ_FROM(addr, data)      `ps.read_burst(addr, 4'h0, 3'b010, 2'b01, 2'b0, 4'h0, 3'b0, data, dummy)

module tb();

reg tb_clk = 0;

reg dummy;

reg [31:0] rd_data;

always #5 tb_clk <= ~tb_clk; //Tries to match 100 MHz frequency on pl_clk0, for what it's worth

initial begin
	
	$display("Performing power-on resets");
	
    //Reset the PL zynq_ultra_ps_e_0   Base_Zynq_MPSoC_zynq_ultra_ps_e_0_0
    `ps.por_srstb_reset(1'b0);
    `ps.fpga_soft_reset(32'h1);   
    #200;  // This delay depends on your clock frequency. It should be at least 16 clock cycles. 
    `ps.por_srstb_reset(1'b1);
    `ps.fpga_soft_reset(32'h0);
    
    // Set debug level info to off. For more info, set to 1.
    `ps.set_debug_level_info(1);
    `ps.set_stop_on_error(0);
    // Set minimum port verbosity. Change to 32'd400 for maximum.
    `ps.M_AXI_HPM0_FPD.set_verbosity(32'd400);
	
	`EDGE(800);
	
	repeat (300) begin
		//I know a priori that there are at least 300 packets and they all have two flits
		`READ_FROM(40'h00A001001C, rd_data); //RDFO
		`READ_FROM(40'h00A0010024, rd_data); //RLR
		`READ_FROM(40'h00A0010020, rd_data); //RDFD
		`READ_FROM(40'h00A0010020, rd_data); //RDFD
	end
	
	`EDGE(10);
	
end

//Dumb way to end the input data
always @(posedge tb_clk) begin
	if (axis_fifo.axis_fifo_i.axis_count_0.count_TDATA > 'd400 
	    && axis_fifo.axis_fifo_i.axis_count_0.count_TVALID
	    && axis_fifo.axis_fifo_i.axis_count_0.count_TREADY
	    && axis_fifo.axis_fifo_i.axis_count_0.count_TLAST) begin
	    
		#5 force axis_fifo.axis_fifo_i.axis_count_0.count_TVALID = 0;
		
	end
end

axis_fifo_wrapper axis_fifo();
endmodule

 

The simulation needs to be run for 137 microseconds (i.e. 137000 ns). 

This simulation spends the first 5 microseconds pushing 200 two-flit packets into the AXI-Stream FIFO. Then, over next 130ish microseconds, it uses the MPSoC VIP to read from RDFO, then RLR, then RDFD, then RDFD in a loop. 128 loop iterations go by. Then, at time 135725 ns, we try to read from RDFO and it is zero (when it should actually be 142).

This corresponds to exactly 256 words being read.

0 Kudos
dgisselq
Scholar
Scholar
1,077 Views
Registered: ‎05-21-2015

@mahkoe,

You realize that a combinatorial assignment of TVALID to !reset is invalid, right?  Following reset, TVALID must be low for at least one cycle.

Dan

0 Kudos
mahkoe
Observer
Observer
1,067 Views
Registered: ‎02-08-2019

Actually, I didn't know that. Keep in mind that this is just a Verilog core I cobbled together for the sake of a simulation; my real problem occurs in hardware.

Do you really think this could be the issue?

 

Also, I've been looking at some internal signals of the AXI Stream FIFO now that I have a simulation of it:

axisfifo_bug_sim_correct_occ.PNG

It looks like the FIFO is correctly keeping track of its occupancy. Just for some reason it doesn't put the correct value on the AXI Lite bus.

This is looking more and more like a bug in the IP to me

0 Kudos
dgisselq
Scholar
Scholar
1,048 Views
Registered: ‎05-21-2015

@mahkoe,

What happens if you hold TLAST high?  Could TLAST be interfering with its count at all?

Dan

0 Kudos
mahkoe
Observer
Observer
1,026 Views
Registered: ‎02-08-2019

I did a bunch of simulations, especially ones where I changed packet sizes.

From the results, I hypothesize that the FIFO pushes packet lengths (i.e. RLR values) into an auxiliary FIFO which is equal to 1/4 the depth in the IP configuration. Can someone who developed the IP confirm this?

Thanks @dgisselq for the help and ideas.

 

0 Kudos
mahkoe
Observer
Observer
936 Views
Registered: ‎02-08-2019

I went into the AXI-Stream FIFO source code and was able to confirm the theory: the FIFO's packet capacity is one quarter its word capacity. Here is the relevant quote form the source:

 

   -- This FIFO is to store the RX packet length in bytes
--   COMP_rx_len_fifo: entity lib_fifo_v1_0_12.sync_fifo_fg
--        generic map (
--          C_FAMILY             => C_FAMILY,
--          C_DCOUNT_WIDTH       => 4 ,
--          C_ENABLE_RLOCS       => 0 , -- not supported in sync fifo
--          C_HAS_DCOUNT         => 1 ,
--          C_HAS_RD_ACK         => 0 ,
--          C_HAS_RD_ERR         => 0 ,
--          C_HAS_WR_ACK         => 0 ,
--          C_HAS_WR_ERR         => 0 ,
--          C_HAS_ALMOST_FULL    => 0 ,
--          C_MEMORY_TYPE        => 0 ,  -- 0 = distributed RAM, 1 = BRAM
--          C_PORTS_DIFFER       => 0 ,  
--          C_RD_ACK_LOW         => 0 ,
--          C_USE_EMBEDDED_REG   => 0 ,
--          C_READ_DATA_WIDTH    => RX_MAX_PKT_SIZE,
--          C_READ_DEPTH         => C_RX_FIFO_DEPTH/4,
--          C_RD_ERR_LOW         => 0 ,
--          C_WR_ACK_LOW         => 0 ,
--          C_WR_ERR_LOW         => 0 ,
--          C_PRELOAD_REGS       => 1 ,  -- 1 = first word fall through
--          C_PRELOAD_LATENCY    => 0 ,  -- 0 = first word fall through
--          C_WRITE_DATA_WIDTH   => RX_MAX_PKT_SIZE,
--          C_WRITE_DEPTH        => C_RX_FIFO_DEPTH/4
--          )
--        port map (
--          Clk                  => Bus2IP_Clk,
--          Sinit                => sig_rxd_reset,
--          Din                  => fg_rxd_wr_length,
--          Wr_en                => rx_len_wr_en,
--          Rd_en                => rx_len_rd_en,
--          Dout                 => sig_rxd_rd_length,
--          Almost_full          => open,
--          Full                 => open,
--          Empty                => rx_fg_len_empty,
--          Rd_ack               => open,
--          Wr_ack               => open,
--          Rd_err               => open,
--          Wr_err               => open,
--          Data_count           => open
--          );


   COMP_rx_len_fifo : xpm_fifo_sync
   generic map (
      DOUT_RESET_VALUE => "0", 
      ECC_MODE => "no_ecc",       
      FIFO_MEMORY_TYPE => "distributed", 
      FIFO_READ_LATENCY => 1,      
      FIFO_WRITE_DEPTH => C_RX_FIFO_DEPTH/4, 
      FULL_RESET_VALUE => 0,       
      PROG_EMPTY_THRESH => 10,     
      PROG_FULL_THRESH => 10,      
      RD_DATA_COUNT_WIDTH => 1,    
      READ_DATA_WIDTH => RX_MAX_PKT_SIZE,       
      READ_MODE => "fwft",         
      USE_ADV_FEATURES => "0", 
      WAKEUP_TIME => 0,            
      WRITE_DATA_WIDTH => RX_MAX_PKT_SIZE,    
      WR_DATA_COUNT_WIDTH => 1    
   )
   port map (
      almost_empty => open,  
      almost_full => open,      
      data_valid => open,       
      dbiterr => open,             
      dout => sig_rxd_rd_length,            
      empty => rx_fg_len_empty,                
      full => open,                   
      overflow => open,          
      prog_empty => open,       
      prog_full => open,        
      rd_data_count => open, 
      rd_rst_busy => open,      
      sbiterr => open,             
      underflow => open,         
      wr_ack => open,          
      wr_data_count => open,  
      wr_rst_busy => open,      
      din => fg_rxd_wr_length,                
      injectdbiterr => '0',  
      injectsbiterr => '0', 
      rd_en => rx_len_rd_en,                 
      rst => sig_rxd_reset,                    
      sleep => '0',                   
      wr_clk => Bus2IP_Clk,     
      wr_en => rx_len_wr_en           
   );
    rx_str_ready_i   <=  not (sig_rxd_wr_full);
    rx_str_ready     <= rx_str_ready_i;
  end generate grxd;

end beh;

View solution in original post

0 Kudos
mahesh_maloo
Visitor
Visitor
235 Views
Registered: ‎06-28-2018

Could you find the solution for the RDFO issue?

 

 

0 Kudos