[dpdk-dev] [PATCH 0/2] new headroom stats library and example application

[Neil Horman]

On Fri, Jan 30, 2015 at 10:47:21AM +0000, Wodkowski, PawelX wrote:
> 
> 
> > -----Original Message-----
> > From: Neil Horman [mailto:nhorman at tuxdriver.com]
> > Sent: Thursday, January 29, 2015 8:13 PM
> > To: Wodkowski, PawelX
> > Cc: dev at dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH 0/2] new headroom stats library and example
> > application
> > 
> > On Thu, Jan 29, 2015 at 05:10:36PM +0000, Wodkowski, PawelX wrote:
> > > > -----Original Message-----
> > > > From: Neil Horman [mailto:nhorman at tuxdriver.com]
> > > > Sent: Thursday, January 29, 2015 2:25 PM
> > > > To: Wodkowski, PawelX
> > > > Cc: dev at dpdk.org
> > > > Subject: Re: [dpdk-dev] [PATCH 0/2] new headroom stats library and
> > example
> > > > application
> > > >
> > > > On Thu, Jan 29, 2015 at 12:50:04PM +0100, Pawel Wodkowski wrote:
> > > > > Hi community,
> > > > > I would like to introduce library for measuring load of some arbitrary jobs.
> > It
> > > > > can be used to profile every kind of job sets on any arbitrary execution unit.
> > > > > In provided l2fwd-headroom example I demonstrate how to use this library
> > to
> > > > > profile packet forwarding (job set is froward, flush and stats) on LCores
> > > > > (execution unit). This example does no limit possible schemes on which this
> > > > > library can be used.
> > > > >
> > > > > Pawel Wodkowski (2):
> > > > >   librte_headroom: New library for checking core/system/app load
> > > > >   examples: introduce new l2fwd-headroom example
> > > > >
> > > > >  config/common_bsdapp               |    6 +
> > > > >  config/common_linuxapp             |    6 +
> > > > >  examples/Makefile                  |    1 +
> > > > >  examples/l2fwd-headroom/Makefile   |   51 +++
> > > > >  examples/l2fwd-headroom/main.c     |  875
> > > > ++++++++++++++++++++++++++++++++++++
> > > > >  lib/Makefile                       |    1 +
> > > > >  lib/librte_headroom/Makefile       |   50 +++
> > > > >  lib/librte_headroom/rte_headroom.c |  368 +++++++++++++++
> > > > >  lib/librte_headroom/rte_headroom.h |  481 ++++++++++++++++++++
> > > > >  mk/rte.app.mk                      |    4 +
> > > > >  10 files changed, 1843 insertions(+)
> > > > >  create mode 100644 examples/l2fwd-headroom/Makefile
> > > > >  create mode 100644 examples/l2fwd-headroom/main.c
> > > > >  create mode 100644 lib/librte_headroom/Makefile
> > > > >  create mode 100644 lib/librte_headroom/rte_headroom.c
> > > > >  create mode 100644 lib/librte_headroom/rte_headroom.h
> > > > >
> > > > > --
> > > > > 1.7.9.5
> > > > >
> > > > >
> > > >
> > > > Whats the advantage of this library over the other tools to preform the same
> > > > function.
> > >
> > > Hi Neil,
> > >
> > > Good point, what is advantage over perf. Answer is: this library does not
> > > supposed to be a perf competition and is not for profiling app in the way perf
> > does.
> > > It is an small and fast extension. It's main task is to manage job list to invoke
> > > them exactly when needed and provide some basic stats about application idle
> > > time (whatever programmer will consider the idle) and busy time.
> > >
> > > For example:
> > > application might decide to add remove some jobs to/from LCore(s)
> > dynamically
> > > basing on current idle time (ex: move job from one core to another).
> > >
> > > Also application might have some information's about traffic type it handles
> > > and provide own algorithm to calculate invocation time (it can also
> > dynamically
> > > switch between those algorithms only replacing handlers).
> > >
> > > >  Perf can provide all the information in this library, and do so
> > > > without having to directly modify the source for the execution unit under
> > test
> > >
> > > Yes, perf can provide those information's  but it can't handle the case when
> > > you are poling for packets too fast or too slow and waist time getting only
> > couple
> > > of them. Library will adjust time when it execute job basing on value this job
> > > returned previously. Code modifications are not so deep, as you can see
> > comparing
> > > l2wf vs l2fwd-headroom app.
> > >
> > > For example in  application I introduced, when forward job return less than
> > > MAX_PKT_BURST execution period will be increased. If it return more it will
> > decrease
> > > execution period. Stats provided for that can be used to determine if
> > application is
> > > behaving correctly and if there is a time for handling another port (what did for
> > tests).
> > >
> > You're still re-inventing the wheel here, and I don't see any advantage to doing
> > so.  If the goal of the library is to profile the run time of a task, then you
> > have perf and systemtap for such purposes.  If the goal is to create a job
> > scheduler that allows you to track multiple parallel tasks, and adjust their
> > execution, there are several pre-existing libraries that any application
> > programmer can already leverage to do just that (Berkely UPC or libtask to name
> > just two examples).  Truthfully, on a dedicated cpu, you could just as easily
> > create multiple child processes runnnig at SCHED_RR and set their priorities
> > accordingly.
> > 
> > I don't see why we need another library to do what several other tools/libraries
> > can do quite well.
> > 
> 
> I am under impression that I am unable to express myself clearly enough.
> 
No, I think you're making yourself clear, you've created a library that allows
you to run a set of jobs, measure various metrics about them, and adjust their
execution time.  Am I correct?

> I did not meant to make competition to perf nor reinvent the tasking library.
> 
I understand thats not what your intent was, but it seems to have been your
result.

> 1. Idle and runtime statistics are provided "by the way" and you can use them
> or use perf on linux or libpmc on freebsd or whatever tool you like to get more
> sophisticated ones.
Ok, so we're in agreement that the idle and runtime statistics in this library
are simmilar to, and somewhat less sophisticated that those provided by perf of
libpmc.

> 2. You can also use tasking library what you like with a headroom object on top
> of every task you created with any scheduling you want. This way you can assign
> priorities to job sets and adjust sleep time as you like. You can also decide what
> is your idle/load time by placing task-dependet-yeld/sleep in idle callback or separate
> job callback or loop end callback.
> 
I'm a little unclear on what you're saying here.  What I think you're saying is
that we can also use other tasking libraries to achieve what you've done with
libheadroom, but I might be misreading you.  If so, please clarify.

> Both of those two points above are no covering one gap - we are running in poll
> mode and here is a lack of mutex, semaphores or other synchronization mechanisms
> provided by OS/tasking layer.
Ok, I'm not sure how thats relevant though. The OS doesn't have to be
involved here at all if you don't want it to.  Take a look at libtask:
http://swtch.com/libtask.tar.gz

Its a rudimentary tasking library that makes locking available if need be, but
in no way requires it.

UPC is simmilar:
http://upc.lbl.gov/task.shtml

Its got locking, but with proper cpu separation you don't need to use it.

 This library try to provide facility for estimating optimal
> job execution time because every mis-poll is a waste of time and can lead to
> impression that core/task is fully loaded but what it does is only polling and getting 1
> or 2 packets instead of 32 (it degrades total NIC throughput).
> 
Yes, I get that you've got a specific use case in mind (optimizing the time
spent in a specific task dynamically).  My question is: Why did you re-create
the code already available in other libraries to do it?  Theres no reason you
can't use the UPC library to create an example of a job set consisting of 4
jobs:
1) A job to record the start time of job (2)
2) A job to poll an interface
3) A job to measure the end time of job (2) and the number of packets it
recorded
4) A job to adjust the run time of job (2) based on the results obtained from
stats gathered in job (3)

I don't deny that you have a useful use case here, you definately do.  My only
concern is that you've re-invented the wheel to make it happen.  You don't
really need the library here, just the example that makes use of an existing
library.

> You can use perf (or whatever tool you like) to profile your code for performance. If
> the code good enough, you use headroom library that will estimate how often or
> when your perfect-profiled-and-optimized code need to be executed. When execution
> period estimation settles you have your headroom for other jobs on this core/task.
> You can also provide your own algorithm for estimating execution period. You are also
> able to drop using this headroom library at runtime and switch to tasking one when
>  you decide that you have enough data make decision how many jobs you can execute
> on lcore/task basing on current throughput.
> 
> I deliberately did not used 'task' but 'job' to not make an impression that I am 
> reinventing another task library. This code is to be simple and fast. Of course you can
> do all those things in every application you make but this library is provided to not
> reinvent this logic all the time.
> 
All user space task libraries are simple and fast, its the only reason they
exist :).  Theres no need to create another one here.

Neil