###############################################################################

                 Xilinx QDMA Software README

###############################################################################


_____________________________________________________________________________
Contents

1.   Installation
     1.1   Compiling the QDMA Software
     1.2   Installing the compiled binaries
     1.3   Loading the Kernel module
2    Configuration
     2.1   Configuring Queues
3.   Xilinx "dma-ctl" Command-line Utility
     3.1   Using dma-ctl for query the QDMA devices/functions
     3.2   Using dma-ctl for Queue control
4.   QDMA test script
     Sample scripts to run simple MM and ST transfers 
_____________________________________________________________________________

1. Installation:

  1.1 Compiling the QDMA Software:
  --------------------------------

  In order to compile the Xilinx QDMA software, a configured and compiled
  Linux kernel source tree is required. The source tree may be only compiled
  header files, or a complete tree. The source tree needs to be configured
  and the header files need to be compiled. And, the Linux kernel must be
  configured to use  modules.

  Additionally, the /lib/modules must have been set up for this particular
  kernel
  (i.e. "make modules_install" has been run with the Linux kernel source tree).

  a.	QDMA Linux Kernel Driver is available in Xilinx github at 
		https://github.com/Xilinx/dma_ip_drivers/QDMA/linux-kernel

  b.	Compile the Xilinx QDMA Linux driver:

  	[xilinx@]# make

	a sub-directory bin/ will be created as a result of running "make".

	By default, both PF driver (qdma-pf.ko) and VF driver (qdma-vf.ko) will
	be compiled along with all the sample applications


	If only PF driver needs to be compiled:
  	[xilinx@]# make driver MODULE=mod_pf

	If only VF driver needs to be compiled:
  	[xilinx@]# make driver MODULE=mod_vf

	Important Note for VF 4K queue support for CPM5 design only
    -----------------------------------------------------------

    To enable VF 4K queue driver support for CPM5 design, QDMA Linux driver need
    to compile by enabling the EQDMA_CPM5_VF_GT_256Q_SUPPORTED macro

    [xilinx@]# make EQDMA_CPM5_VF_GT_256Q_SUPPORTED=1

  1.2 Installing the compiled binaries:
  -------------------------------------

  To install the QDMA software, the installer must be the root user, then
  run "make install".

	Run "make install-mods" to install the drivers in /lib/modules/<kernel version>/qdma/

	Run "make install-apps" to install the applications in "/user/local/sbin"

  Example application software to issue DMA requests:
  dma-to-device for H2C queues and dma-from-device for C2H queues will be installed in /user/local/sbin.


  1.3 Loading the Kernel module:
  --------------------------------

  Kernel module cane be loaded in following different modes
  0 - Auto Mode, driver decides to process the request in poll or interrupt mode
  1 - Poll Mode
  2 - Direct Interrupt Mode
  3 - Interrupt Aggregation Mode or Indirect Interrupt Mode
  4 - Legacy Interrupt Mode

  Find the QDMA device bus number from lspci. Ex : 01:00.0 is the BDF of the device.
  By default, all the functions are loaded in Auto Mode.
  
  In order to pass the module parameters while loading a driver, a config file "qdma.conf" needs to be placed in /etc/modprobe.d directory. Format of the qdma.conf is as below

	------------------------------------------------------------------------------------------------------------------
	options <module_name> mode=<bus_num>:<pf_num>:<mode>,<bus_num>:<pf_num>:<mode>,<bus_num>:<pf_num>:<mode>,.....
	options <module_name> config_bar=<bus_num>:<pf_num>:<config_bar>,<bus_num>:<pf_num>:<config_bar>,<bus_num>:<pf_num>:<config_bar>,.....
	options <module_name> master_pf=<bus_num>:<master_pf>,<bus_num>:<master_pf>
	------------------------------------------------------------------------------------------------------------------

	- module_name:  Name of the mode. For PF: qdma-pf and for VF: qdma-vf
	- bus_num : Bus number of the PCIe endpoint card
	- func_num : Function number of the corressponding bus_num
	- mode: Mode in which the driver needs to be loaded
	- config_bar: Config bar number
	- master_pf: Master PF  
	- num_threads: number of threads for monitoring the writeback of completions

   Sample qdma.conf can be found below:

	------------------------------------------------------------------------------------------------------------------
	options qdma-pf mode=0x06:0:2,0x06:1:3,0x06:2:0,0x07:2:1
	options qdma-vf mode=0x06:0:2,0x06:1:3
	------------------------------------------------------------------------------------------------------------------

  An auxillary script, qdma_generate_conf_file.sh has been added to the scripts folder which helps create the qdma.conf 
  config file and copies it to the /etc/modprobe.d location. The script can be used as shown below - 
  ./scripts/qdma_generate_conf_file.sh <bus_num> <num_pfs> <mode> <config_bar> <master_pf>

  For loading the driver, execute the following command:
  PF Driver:
	modprobe qdma-pf
  VF Driver:
	modprobe qdma-vf

  Now the QDMA software is ready for use.

  Please note that having the qdma-pf.ko and qdma-vf.ko files in the /lib/modules/<kernel version>/qdma/ will cause
  automatic loading of the driver modules at boot time. To avoid this, it is recommended to have the drivers 
  blacklisted. 
  This can be done by adding the below 2 lines in the /etc/modprobe.d/blacklist.conf file -
  blacklist qdma-pf
  blacklist qdma-vf
  

2. Configuration

  2.1 Configuring Queues
  -------------------------------------

  To configure a QDMA queue, there are three minimum required parameters
  (in the form of key value pairs) needed to be configured.

       idx <N>:	        The 0-based queue index on the function.
       mode <mm | st>:  queue mode, default to "mm"
       dir <h2c | c2h>: queue direction, default to "h2c"

  - "idx" is a 0-based unique index on the function.
    *QDMA3.1, the range can be
	0 ~ 2047 on a physical function and
	0 ~ 2047 on a virtual function
	
    *QDMA4.0, the range can be
	0 ~ 2047 on a physical function and
	0 ~ 256 on a virtual function

  - "mode" is the operation mode of the queue.
	It is either memory mapped (mm) or streaming mode (st)

  - "dir" is the direction of the queue.
	It is either host to card (h2c) or card to host (c2h).

  A h2c queue and a c2h queue can share the same index. In other word, a index
  represents a pair of queues: one on h2c direction and the other on the c2h
  direction.


3. Xilinx "dma-ctl" Command-line Configuration Utility:

  The Xilinx QDMA control tool, dma-ctl, is a Command Line utility
  which is installed in /usr/local/sbin/ and allows administration of the
  Xilinx QDMA queues. It can perform the following functions:

  - query the qdma functions/devices the driver has bind into.

  - list all of the queues on a device/function
  - add/configure a new queues on a device/function
  - start an already added/configured queue (i.e., bring the queue online)
  - stop an started queue (i.e., bring the queue offline)
  - delete an already added/configured queue

  register access:
  - read a register
  - write a register
  - dump the qdma config bar and user bar registers

  debug helper
  - display a queue's configuration parameters
  - display a queue's descriptor ring entries
  - display a ch2 queue's completion ring entries
  - display the interrupt ring entries in indirect interrupt mode

  Apart from the above basic commands, there are various other commands
  available in dma-ctl to add/start/stop/del list of queues at the same time

  For help run:
    dma-ctl -h
       (or)
    man dma-ctl


  3.1 Using dma-ctl for query the QDMA devices/functions
  -------------------------------------

  Please refer dma-ctl man page to find all options and parameters
  available.

    1. Get the list of devices the driver has bind with

      [root@]# dma-ctl dev list

	qdma06000	0000:06:00.0	max QP: 32
	qdma06001	0000:06:00.1	max QP: 32

      The above example output shows 2 QDMA functions/devices.
      one is at pci BDF: 06:00.0
      the other is at pci BDF: 06:00.1


  3.2 Using dma-ctl for Queue control
  -------------------------------------

  Please refer dma-ctl man page to find all options and parameters available.

    a. Add/Configure a queue

      To add a MM H2C queue on qdma06000 in the above example:

      [root@]# dma-ctl qdma06000 q add idx 0 mode mm dir h2c

      *A character device /dev/qdma06000-MM-0 would be created.

      To add a MM C2H queue on qdma06000:

      [root@]# dma-ctl qdma06000 q add idx 0 mode mm dir c2h

      *A character device /dev/qdma06000-MM-0 would be created.

    b. Start an added queue

      To start the MM H2C queue on qdma06000 added in the previous example:

      [root@]# dma-ctl qdma06000 q start idx 0 dir h2c

      *After the queue is started the normal read and write operation can be
       performed on the character device /dev/qdma06000-MM-0.

      To start the MM C2H queue on qdma06000 added in the previous example:

      [root@]# dma-ctl qdma06000 q start idx 0 dir c2h

      *After the queue is started the normal read and write operation can be
       performed on the character device /dev/qdma06000-MM-0.

    c. Stop a queue

      [root@]# dma-ctl qdma06000 q stop idx 0 dir h2c
      [root@]# dma-ctl qdma06000 q stop idx 0 dir c2h


    d. Delete a queue

      [root@]# dma-ctl qdma06000 q del idx 0 dir h2c
      [root@]# dma-ctl qdma06000 q del idx 0 dir c2h


4. QDMA test script

    Test scripts are in scripts/ directory

    To use the scripts in this directory, "make" and "make install" must be run
    install the dma-from-device and dma-to-device applications.

    qdma_run_test.sh 
	The script does the following dma operations:

    	PF AXI-MM :     H2C/C2H AXI-MM transfer and check for data correctness.
	PF AXI-ST-H2C : H2C AXI-ST transfer and read data register on user side
  		        to check if the data if correct.
	PF AXI-ST-C2H : C2H AXI-St transfer and check data for correctness.

	All the above transfers are done for 4 [0 to 3] Queues.

    qdma_run_test_mm_vf.sh
    	VF AXI-MM : Script runs H2C/C2H AXI-MM transfer and check for data
		    correctness.

	All the above transfers are done for 4 [0 to 3] Queues.

    qdma_run_test_st_vf.sh
	VF AXI-ST-H2C : H2C AXI-ST transfer and read data register on user side
	  	        to check if the data if correct.
	VF AXI-ST-C2H : C2H AXI-St transfer and check data for correctness.

	All the above transfers are done for 4 [0 to 3] Queues.






  /*
  * This file is part of the Xilinx DMA IP Core driver for Linux
  *
  * Copyright (c) 2017-2022, Xilinx, Inc. All rights reserved.
  * Copyright (c) 2022-2024, Advanced Micro Devices, Inc. All rights reserved.
  *
  * This source code is free software; you can redistribute it and/or modify it
  * under the terms and conditions of the GNU General Public License,
  * version 2, as published by the Free Software Foundation.
  *
  * This program is distributed in the hope that it will be useful, but WITHOUT
  * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
  * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
  * more details.
  *
  * The full GNU General Public License is included in this distribution in
  * the file called "COPYING".
  */




