[PATCH v3] app/testpmd: fix secondary process not forwarding

[He, ShiyangX]

>-----Original Message-----
>From: Ferruh Yigit <ferruh.yigit at amd.com>
>Sent: Tuesday, March 7, 2023 7:41 PM
>To: He, ShiyangX <shiyangx.he at intel.com>; dev at dpdk.org
>Cc: Zhou, YidingX <yidingx.zhou at intel.com>; stable at dpdk.org; Zhang, Yuying
><yuying.zhang at intel.com>; Singh, Aman Deep
><aman.deep.singh at intel.com>; Burakov, Anatoly
><anatoly.burakov at intel.com>; Matan Azrad <matan at nvidia.com>; Dmitry
>Kozlyuk <dmitry.kozliuk at gmail.com>
>Subject: Re: [PATCH v3] app/testpmd: fix secondary process not forwarding
>
>On 3/7/2023 3:25 AM, He, ShiyangX wrote:
>>
>>
>>> -----Original Message-----
>>> From: Ferruh Yigit <ferruh.yigit at amd.com>
>>> Sent: Monday, March 6, 2023 11:06 PM
>>> To: He, ShiyangX <shiyangx.he at intel.com>; dev at dpdk.org
>>> Cc: Zhou, YidingX <yidingx.zhou at intel.com>; stable at dpdk.org; Zhang,
>>> Yuying <yuying.zhang at intel.com>; Singh, Aman Deep
>>> <aman.deep.singh at intel.com>; Burakov, Anatoly
>>> <anatoly.burakov at intel.com>; Matan Azrad <matan at nvidia.com>; Dmitry
>>> Kozlyuk <dmitry.kozliuk at gmail.com>
>>> Subject: Re: [PATCH v3] app/testpmd: fix secondary process not
>>> forwarding
>>>
>>> On 2/23/2023 2:41 PM, Shiyang He wrote:
>>>> Under multi-process scenario, the secondary process gets queue state
>>>> from the wrong location (the global variable 'ports'). Therefore,
>>>> the secondary process can not forward since "stream_init" is not called.
>>>>
>>>> This commit fixes the issue by calling 'rte_eth_rx/tx_queue_info_get'
>>>> to get queue state from shared memory.
>>>>
>>>> Fixes: 3c4426db54fc ("app/testpmd: do not poll stopped queues")
>>>> Cc: stable at dpdk.org
>>>>
>>>> Signed-off-by: Shiyang He <shiyangx.he at intel.com>
>>>> Acked-by: Yuying Zhang <yuying.zhang at intel.com>
>>>>
>>>> v3: Add return value description
>>>> ---
>>>>  app/test-pmd/testpmd.c | 45
>>>> ++++++++++++++++++++++++++++++++++++++++--
>>>>  1 file changed, 43 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index
>>>> 0c14325b8d..a050472aea 100644
>>>> --- a/app/test-pmd/testpmd.c
>>>> +++ b/app/test-pmd/testpmd.c
>>>> @@ -2418,9 +2418,50 @@ start_packet_forwarding(int with_tx_first)
>>>>  	if (!pkt_fwd_shared_rxq_check())
>>>>  		return;
>>>>
>>>> -	if (stream_init != NULL)
>>>> -		for (i = 0; i < cur_fwd_config.nb_fwd_streams; i++)
>>>> +	if (stream_init != NULL) {
>>>> +		for (i = 0; i < cur_fwd_config.nb_fwd_streams; i++) {
>>>> +			if (rte_eal_process_type() == RTE_PROC_SECONDARY)
>>> {
>>>> +				struct fwd_stream *fs = fwd_streams[i];
>>>> +				struct rte_eth_rxq_info rx_qinfo;
>>>> +				struct rte_eth_txq_info tx_qinfo;
>>>> +				int32_t rc;
>>>> +				rc = rte_eth_rx_queue_info_get(fs->rx_port,
>>>> +						fs->rx_queue, &rx_qinfo);
>>>> +				if (rc == 0) {
>>>> +					ports[fs->rx_port].rxq[fs-
>>>> rx_queue].state =
>>>> +						rx_qinfo.queue_state;
>>>> +				} else if (rc == -ENOTSUP) {
>>>> +					/* Set the rxq state to
>>> RTE_ETH_QUEUE_STATE_STARTED
>>>> +					 * to ensure that the PMDs do not
>>> implement
>>>> +					 * rte_eth_rx_queue_info_get can
>>> forward.
>>>> +					 */
>>>> +					ports[fs->rx_port].rxq[fs-
>>>> rx_queue].state =
>>>> +
>>> 	RTE_ETH_QUEUE_STATE_STARTED;
>>>> +				} else {
>>>> +					TESTPMD_LOG(WARNING,
>>>> +						"Failed to get rx queue
>>> info\n");
>>>> +				}
>>>> +
>>>> +				rc = rte_eth_tx_queue_info_get(fs->tx_port,
>>>> +						fs->tx_queue, &tx_qinfo);
>>>> +				if (rc == 0) {
>>>> +					ports[fs->tx_port].txq[fs-
>>>> tx_queue].state =
>>>> +						tx_qinfo.queue_state;
>>>> +				} else if (rc == -ENOTSUP) {
>>>> +					/* Set the txq state to
>>> RTE_ETH_QUEUE_STATE_STARTED
>>>> +					 * to ensure that the PMDs do not
>>> implement
>>>> +					 * rte_eth_tx_queue_info_get can
>>> forward.
>>>> +					 */
>>>> +					ports[fs->tx_port].txq[fs-
>>>> tx_queue].state =
>>>> +
>>> 	RTE_ETH_QUEUE_STATE_STARTED;
>>>> +				} else {
>>>> +					TESTPMD_LOG(WARNING,
>>>> +						"Failed to get tx queue
>>> info\n");
>>>> +				}
>>>> +			}
>>>>  			stream_init(fwd_streams[i]);
>>>> +		}
>>>> +	}
>>>>
>>>
>>>
>>> Testpmd duplicates some dpdk/ethdev state/config in application
>>> level, and this can bite in multiple cases, as it is happening here.
>>>
>>> I am not sure if this was a design decision, but I think instead of
>>> testpmd storing ethdev related state/config in application level, it
>>> should store only application level state/config, and when ethdev
>>> related state/config is required app should get it directly from ethdev.
>>>
>>> It may be too late already for testpmd, there is a mixed usage, but I
>>> am for preferring this approach when there is an opportunity.
>>>
>>>
>>>
>>> For above issue, why queue state needs to be stored in application level
>'port'
>>> variable?
>>> Where is this queue state used?
>>>
>>> Can it work to get queue state directly from ethdev where this state
>>> is used, instead of storing it in the 'port' variable in advance?
>>>
>>> And perhaps testpmd 'port' variable can be updated there, both for
>>> primary and secondary, for backward compatibility (other existing
>>> users of this queue state).
>>>
>>> What do you think?
>>
>> Thanks for your comments!
>>
>> It is an effective method to get queue state directly from ethdev where this
>state is used.
>> I also don't know the design meaning of the 'ports' variable. If
>> modification is needed, a higher level of design and more work are required.
>>
>> As a bug fix, apart from extracting the code block into a function, is the
>solution feasible?
>
>Hi Shiyang,
>
>As a bug fix, this issue (testpmd stored state not being up to date for
>secondary process) looks like have potential to occur many different flavors,
>that is why what about having a central update?
>
>I think 'start_port()' can be a good place for this kind of update:
>
>start_port() {
>
>	...
>	if (secondary)
>		update_state()
>}
>
>update_state() {
>	update_queue_state()
>}
>
>update_queue_state() {
>	<your code goes here>
>}
>
>
>Having secondary checks and updates in multiple places can make code harder
>to understand.
>
>What do you think to update as above?
>
>
>

Thanks for your reply! 
It is more reasonable to update the queue state in 'start_port()'. I'll send a new patch asap.