[dpdk-dev] Application framework vs. library

[Steffen.Bauch at rohde-schwarz.com]

Hi,

as a developer of a product that uses DPDK I inherited a custom changeset
towards DPDK 2.0 to allow better integration into our application stack. I
am currently underway of updating the used DPDK release and would like to
get rid of these local changes in the future by influencing the upstream
development to some degree.

Concerning the integration we try to cope with the circumstance that
DPDK is more comparable to an application framework and less to a
library. The main part of the changes deal with EAL and specifically with
rte_eal_init() and prune or replace functionality. The pruned features
are in the responsibility of other proprietary libraries and modules
within our application. For illustration purposes, I have included an
old changeset towards v2.0.0. for reference, but due to the nature
of the whole changeset it does not make sense to apply it directly,
as there is a total lack of generality in the specific implementation
(e.g. only the linuxapp part is customized, incomplete implemented concept
in termination behavior ...). Nonetheless I am underway of researching
concepts and approaches that are general enough for upstream inclusion.

The main reason for attaching this patchset is to be able to discuss
possible approaches of further development, as I would like to get
feedback on possible development approaches to address the named problems.

In detail the changes address the following areas:

Logging behavior

Opening a connection to the system logger for our program is within the
responsibility of our specific logging module that is also used in the
proprietary parts of the application. For this reason openlog() should
not be called in our use case. In addition, using rte_openlog_stream()
is not usable before rte_eal_init() as logging init will overwrite the
used log stream. For this reason a large part of the init logs can not
be handled by a custom log stream. Btw, after removal of the log caching,
is there a fundamental difference between early log and normal logging?

A possible approach for the future could be to initialize rte_logs.file
to NULL (in fact it is, as it is static) and only set it during
initialization if it is NULL with the goal to be able to use
rte_openlog_stream() before rte_eal_init(). As the openlog() call is
only relevant when the default log stream is used (and architecture
dependent!) I introduced a specific entry point for calling openlog. The
main complexity to this changeset is added due to two reasons. First
reason is the circumstance that rte_eal_common_log_init() is used several
times during early logging and real logging initialization. Second
aspect is that a user might revert to the use of the default_log_stream
after a custom log stream has be used straight from the beginning (even
before rte_eal_init()). A dedicated changeset towards v16.07 for this
improvement is attached for review, feedback and possible inclusion.

Process termination behavior

As the DPDK functionality is only a small part of the whole application
ownership of the running process is not with DPDK and it is not allowed
to cancel the running process as a result of an error. For this reason,
the current changeset instead of calling rte_panic or rte_exit returns
an error code and handles it in the calling function. Unfortunately this
change was only implemented in rte_eal_init() and not in other places
(drivers, libs ...), so there is currently no safety that an unknown
code part terminates the whole application on the spot. Future changes
and patches could change the error handling in a more thorough approach,
but I would like to have some feedback and opinions before I start the
heavy-lifting work in over 700 code locations. Even some interfaces
might be affected, as lots of functions return void now,

Thread creation

In our application thread creation and setting the affinity is already
handled in a proprietary module. Frames are polled and processed
in non-EAL pthreads. For this reason the lcore thread creation in
rte_eal_init() is completely removed as we do not want additional threads
to be assigned to processing CPUs. Unfortunately setting the coremask
to 0 is not feasible directly. For the non-EAL threads a custom function
within the application sets the per_thread _lcore variable to the right
value to try to enable mbuf cacheing. The changeset also disables the
interrupt handling thread as in our application it currently seems not
necessary (maybe make this optional?)

In an experiment I removed the error check for the non-zero number of
lcores and quick-fixed certain locations where rte_lcore_count() is used
for memory size computations and at least got a running application.

A possible approach could be to enhance enum rte_lcore_role_t with an
auxiliary thread type and introduce additional eal arguments to assign
lcores with these auxiliary role.

I would like to receive feedback on possible development directions.

Thank you,

Steffen Bauch




-- 

Steffen Bauch
Senior Software Engineer Platform & Technology

R&S Cybersecurity ipoque GmbH
Augustusplatz 9, D-04109 Leipzig
Phone: + 49 341 59403 0
Email: steffen.bauch at rohde-schwarz.com
Internet: www.ipoque.com

Trade register Amtsgericht Leipzig HRB21462
Gesellschaft mit beschränkter Haftung (GmbH)
Dirk Czepluch - Managing Director