[dpdk-dev] [PATCH] eal/armv8: high-resolution cycle counter

[Jan Viktorin]

On Fri, 19 Aug 2016 17:16:12 +0530
Jerin Jacob <jerin.jacob at caviumnetworks.com> wrote:

> On Fri, Aug 19, 2016 at 09:43:36AM +0000, Nipun Gupta wrote:
> > Hi Jerin,
> >   
> 
> Hi Nipun,
> 
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Jerin Jacob
> > > Sent: Thursday, August 18, 2016 17:22
> > > To: dev at dpdk.org
> > > Cc: thomas.monjalon at 6wind.com; jianbo.liu at linaro.org;
> > > viktorin at rehivetech.com; Jerin Jacob <jerin.jacob at caviumnetworks.com>
> > > Subject: [dpdk-dev] [PATCH] eal/armv8: high-resolution cycle counter
> > > 
> > > Existing cntvct_el0 based rte_rdtsc() provides portable
> > > means to get wall clock counter at user space. Typically
> > > it runs at <= 100MHz.
> > > 
> > > The alternative method to enable rte_rdtsc() for high resolution
> > > wall clock counter is through armv8 PMU subsystem.
> > > The PMU cycle counter runs at CPU frequency, However,
> > > access to PMU cycle counter from user space is not enabled
> > > by default in the arm64 linux kernel.
> > > It is possible to enable cycle counter at user space access
> > > by configuring the PMU from the privileged mode (kernel space).
> > > 
> > > by default rte_rdtsc() implementation uses portable
> > > cntvct_el0 scheme. Application can choose the PMU based
> > > implementation with CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU
> > > 
> > > Signed-off-by: Jerin Jacob <jerin.jacob at caviumnetworks.com>
> > > ---
> > > 
> > > The PMU based scheme useful for high accuracy performance profiling.
> > > Find below the example steps to configure the PMU based cycle counter on an
> > > armv8 machine.
> > > 
> > > # git clone https://github.com/jerinjacobk/armv8_pmu_cycle_counter_el0
> > > # cd armv8_pmu_cycle_counter_el0
> > > # make
> > > # sudo insmod pmu_el0_cycle_counter.ko
> > > # cd $DPDK_DIR
> > > # make config T=arm64-armv8a-linuxapp-gcc
> > > # echo "CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU=y" >> build/.config
> > > # make -j 4  
> > 
> > Can we make this kernel module also a part of DPDK. May be in the linuxapp so that it is also compiled with DPDK?  
> 
> I thought so, Later I realized it may not be a good idea to add yet
> another out of tree module in DPDK repo and DPDK tries to get rid of
> existing out of tree modules.

This has also been my way of thinking. However, if we discover that such
kernel module would be really useful, I think we can do it.

> 
> >   
> > > 
> > > ---
> > >  .../common/include/arch/arm/rte_cycles_64.h        | 33
> > > ++++++++++++++++++++++
> > >  1 file changed, 33 insertions(+)
> > > 
> > > diff --git a/lib/librte_eal/common/include/arch/arm/rte_cycles_64.h
> > > b/lib/librte_eal/common/include/arch/arm/rte_cycles_64.h
> > > index 14f2612..867a946 100644
> > > --- a/lib/librte_eal/common/include/arch/arm/rte_cycles_64.h
> > > +++ b/lib/librte_eal/common/include/arch/arm/rte_cycles_64.h
> > > @@ -45,6 +45,11 @@ extern "C" {
> > >   * @return
> > >   *   The time base for this lcore.
> > >   */
> > > +#ifndef RTE_ARM_EAL_RDTSC_USE_PMU
> > > +/**
> > > + * This call is portable to any ARMv8 architecture, however, typically
> > > + * cntvct_el0 runs at <= 100MHz and it may be imprecise for some tasks.
> > > + */
> > >  static inline uint64_t
> > >  rte_rdtsc(void)
> > >  {
> > > @@ -53,6 +58,34 @@ rte_rdtsc(void)
> > >  	asm volatile("mrs %0, cntvct_el0" : "=r" (tsc));
> > >  	return tsc;
> > >  }
> > > +#else
> > > +/**
> > > + * This is an alternative method to enable rte_rdtsc() with high resolution
> > > + * PMU cycles counter.The cycle counter runs at cpu frequency and this scheme
> > > + * uses ARMv8 PMU subsystem to get the cycle counter at userspace, However,
> > > + * access to PMU cycle counter from user space is not enabled by default in
> > > + * arm64 linux kernel.
> > > + * It is possible to enable cycle counter at user space access by configuring
> > > + * the PMU from the privileged mode (kernel space).
> > > + *
> > > + * asm volatile("msr pmintenset_el1, %0" : : "r" ((u64)(0 << 31)));
> > > + * asm volatile("msr pmcntenset_el0, %0" :: "r" BIT(31));
> > > + * asm volatile("msr pmuserenr_el0, %0" : : "r"(BIT(0) | BIT(2)));
> > > + * asm volatile("mrs %0, pmcr_el0" : "=r" (val));
> > > + * val |= (BIT(0) | BIT(2));
> > > + * isb();
> > > + * asm volatile("msr pmcr_el0, %0" : : "r" (val));  
> > 
> > In your git repo I see that on cleanup the cycle count register is not disabled (PMCNTENCLR_EL0). It shall be better to disable the cycle count register too at module exit.  
> 
> OK

+1

I've got a private kernel driver enabling and disabling (hopefully) properly
this for ARMv7. If we'd like to merge it, I'd like to have a single module
or at least single module with 2 implementations...

I can post it if it would be helpful.

Regards
Jan

> 
> >   
> > > + *
> > > + */
> > > +static inline uint64_t
> > > +rte_rdtsc(void)
> > > +{
> > > +	uint64_t tsc;
> > > +
> > > +	asm volatile("mrs %0, pmccntr_el0" : "=r"(tsc));
> > > +	return tsc;
> > > +}
> > > +#endif
> > > 
> > >  static inline uint64_t
> > >  rte_rdtsc_precise(void)
> > > --
> > > 2.5.5  
> > 
> > Do you also plan to support performance monitor event counters?  
> 
> No. This patch was inspired by armv7 PMU scheme and its part of DPDK.
> The sole reason to add this support to catch any performance regression
> through app/test application.Other than that, I think cntvct_el0 based
> existing scheme is good enough for all the use cases.
> 
> > 
> > Regards,
> > Nipun
> >   



-- 
   Jan Viktorin                  E-mail: Viktorin at RehiveTech.com
   System Architect              Web:    www.RehiveTech.com
   RehiveTech
   Brno, Czech Republic