[dpdk-dev] keepalive: fix keepalive state alignment

  The __rte_cache_aligned was applied to the whole array,
not the array elements. This leads to a false sharing between
the monitored cores.

Fixes: e70a61ad50ab ("keepalive: export states")
Cc: remy.horton@intel.com
Signed-off-by: Andriy Berestovskyy <aber@semihalf.com>
---
 lib/librte_eal/common/rte_keepalive.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)
  

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andriy Berestovskyy
> Sent: Friday, January 19, 2018 2:48 PM
> To: dev@dpdk.org
> Cc: Horton, Remy <remy.horton@intel.com>
> Subject: [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment
> 
> The __rte_cache_aligned was applied to the whole array,
> not the array elements. This leads to a false sharing between
> the monitored cores.
> 
> Fixes: e70a61ad50ab ("keepalive: export states")
> Cc: remy.horton@intel.com
> Signed-off-by: Andriy Berestovskyy <aber@semihalf.com>
> ---
>  lib/librte_eal/common/rte_keepalive.c | 25 +++++++++++++++----------
>  1 file changed, 15 insertions(+), 10 deletions(-)
> 
> diff --git a/lib/librte_eal/common/rte_keepalive.c
> b/lib/librte_eal/common/rte_keepalive.c
> index 7ddf201..a586e03 100644
> --- a/lib/librte_eal/common/rte_keepalive.c
> +++ b/lib/librte_eal/common/rte_keepalive.c
> @@ -13,8 +13,13 @@
> 
>  struct rte_keepalive {
>  	/** Core Liveness. */
> -	enum rte_keepalive_state __rte_cache_aligned state_flags[
> -		RTE_KEEPALIVE_MAXCORES];
> +	struct {
> +		/*
> +		 * Each element of the state_flags table must be cache aligned
> +		 * to prevent false sharing.
> +		 */
> +		enum rte_keepalive_state s __rte_cache_aligned;
> +	} state_flags[RTE_KEEPALIVE_MAXCORES];


By aligning each item in the array, we do reduce false-sharing of the cache lines for all cores, however we pay the cost of increasing the footprint of the rte_keepalive struct. Note that the code iterates the full MAX_CORES in various loops in the monitoring core.

Before
(gdb) p sizeof(struct rte_keepalive)
$1 = 1728    #  27 cache lines

After
(gdb) p sizeof(struct rte_keepalive)
$1 = 9408    # 147 cache lines


These changes do reduce false-sharing however is there actually a performance benefit? A lot of cache space will be taken up if each core requires its own cache line, which will reduce performance again.. it's a tradeoff.


Little fix for a v2: "s" is not a good variable name for the rte_keepalive_state, please use something more descriptive.


<snip>
  

Hey Harry,
Thanks for the review.

On Fri, Jan 19, 2018 at 6:31 PM, Van Haaren, Harry
<harry.van.haaren@intel.com> wrote:
> These changes do reduce false-sharing however is there actually a performance benefit? A lot of cache space will be taken up if each core requires its own cache line, which will reduce performance again.. it's a tradeoff.

1. False sharing is happening in the data path vs loops in control paths.

2. The original code (prior e70a61ad50ab "keepalive: export states")
had each element aligned to the cache line, not the whole array.


> Little fix for a v2: "s" is not a good variable name for the rte_keepalive_state, please use something more descriptive.

Sure, if there are no more comments, I'll change it.


Andriy
  

On 22/01/2018 18:20, Andriy Berestovskyy wrote:
[..]
> On Fri, Jan 19, 2018 at 6:31 PM, Van Haaren, Harry
> <harry.van.haaren@intel.com> wrote:
>> These changes do reduce false-sharing however is there actually a
>> performance benefit? A lot of cache space will be taken up if each
>> core requires its own cache line, which will reduce performance
>> again.. it's a tradeoff.
[..]
> 2. The original code (prior e70a61ad50ab "keepalive: export states")
> had each element aligned to the cache line, not the whole array.

Aligning each flag element was the original intention, so I see no issue 
in restoring it. The monitoring core only reads the entries within 
state_flags for which the corresponding active_core is set, so 
ultimately the trade-off in cache line usage is one made by the 
application when it decides which cores need monitoring.

..Remy
  

> From: Horton, Remy

> Sent: Tuesday, January 23, 2018 10:17 AM

> To: Andriy Berestovskyy <aber@semihalf.com>; Van Haaren, Harry

> <harry.van.haaren@intel.com>

> Cc: dev@dpdk.org

> Subject: Re: [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment

> 

> 

> On 22/01/2018 18:20, Andriy Berestovskyy wrote:

> [..]

> > On Fri, Jan 19, 2018 at 6:31 PM, Van Haaren, Harry

> > <harry.van.haaren@intel.com> wrote:

> >> These changes do reduce false-sharing however is there actually a

> >> performance benefit? A lot of cache space will be taken up if each

> >> core requires its own cache line, which will reduce performance

> >> again.. it's a tradeoff.

> [..]

> > 2. The original code (prior e70a61ad50ab "keepalive: export states")

> > had each element aligned to the cache line, not the whole array.

> 

> Aligning each flag element was the original intention, so I see no issue

> in restoring it. The monitoring core only reads the entries within

> state_flags for which the corresponding active_core is set, so

> ultimately the trade-off in cache line usage is one made by the

> application when it decides which cores need monitoring.

> 

> ..Remy


No objection here - just making sure the change was intentional
and the effects are considered.

@Remy as you're the Keepalive maintainer I'll leave Acking to you :)

-Harry