[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility

[Neil Horman]

On Wed, Oct 08, 2014 at 05:57:29PM +0200, Thomas Monjalon wrote:
> Hi Neil,
> 
> 2014-10-07 17:01, Neil Horman:
> > On Wed, Oct 01, 2014 at 02:59:40PM -0400, Neil Horman wrote:
> > > On Fri, Sep 26, 2014 at 10:45:49AM -0400, Neil Horman wrote:
> > > > On Fri, Sep 26, 2014 at 12:41:33PM +0200, Thomas Monjalon wrote:
> > > > > Hi Neil,
> > > > > 
> > > > > 2014-09-24 14:19, Neil Horman:
> > > > > > Ping Thomas. I know you're busy, but I would like this to not fall off anyones
> > > > > > radar.  You alluded to concerns regarding what, for lack of a better term,
> > > > > > ABI/API lockin.  I had asked you to enuumerate/elaborate on specifics, but never
> > > > > > heard back.  Are there further specifics you wish to discuss, or are you
> > > > > > satisfied with the above answers?
> > > > > 
> > > > > Sorry for not being very reactive on this thread.
> > > > > All this discussion is very interesting but it's really not the proper
> > > > > time to apply it. As you said, it requires an extra effort. I'm not saying
> > > > > it will never be integrated. I'm just saying that we cannot change
> > > > > everything at the same time.
> > > > > 
> > > > > Let me sum up the situation. This community project has been very active
> > > > > for few months now. First, we learnt how to make some releases together
> > > > > and we are improving the process to be able to deliver a new major release
> > > > > every 4 months while having some good quality process.
> > > > > But these releases are still not complete because documentation is not
> > > > > integrated yet. Then developers should have a role in documentation updates.
> > > > > We also need to integrate and learn how to use more tools to be more
> > > > > efficient and improve quality.
> > > > > 
> > > > > So the question is "when should we care about API compatibility"?
> > > > > And the answer is: ASAP, but not now. I feel next year is a better target.
> > > > > Because the most important priority is to move together at a pace which
> > > > > allow most of us to stay in the race.
> > > > 
> > > > I'm sorry Thomas, I don't accept this.  I asked you for details as to your
> > > > concerns regarding this patch series, and you've provided more vague comments.
> > > > I need details to address
> > > > 
> > > > You say it requires extra effort, you're right it does.  Any feature that you
> > > > integreate requires some additional effort.  How is this patch any different
> > > > from adding the acl library or any other new API?  Everything requires
> > > > maintenence, thats how software works.  What specfically about this patch series
> > > > makes the effort insurmountable to you?
> > > > 
> > > > You say you're improving your process.  Great, this patch aids in that process
> > > > by ensuring backwards compatibility for a period of time.  Given that the API
> > > > and ABI can still evolve within this framework, as I've described, how is this
> > > > patch series not a significant step forward toward your goal of quality process.
> > > > 
> > > > You say documentation isn't integrated.  So, what does getting documentation
> > > > integrated have to do with this patch set, or any other?  I don't see you
> > > > holding any other patches based on documentation.  Again, nothing in this series
> > > > prevents evolution of the API or ABI.  If you're hope is to wait until
> > > > everything is perfect, then apply some control to the public facing API, and get
> > > > it all documented, none of thosse things will ever happen, I promise you.
> > > > 
> > > > You say you also need to learn to use more tools to be more efficient and
> > > > improve quality.  Great!  Thats exactly what this is. If we mandate even a short
> > > > term commitment to ABI stability (1 single relese worth of time), we will
> > > > quickly identify what API's change quickly and where we need to be cautious with
> > > > our API design.  If you just assume that developers will get better of their own
> > > > volition, it will never happen.
> > > > 
> > > > You say this should go in next year, but not now.  When exactly?  What event do
> > > > you forsee occuring in the next 12-18 months that will change everything such
> > > > that we can start supporing an ABI for more than just a few weeks at the head of
> > > > the tree?  
> > > > 
> > > > To this end, I just did a quick search through the git history for dpdk to look
> > > > at the histories of all the header files that are exposed via the makefile
> > > > SYMLINK command (given that that provides a list of header files that
> > > > applications can include, and embodies all the function symbols and data types
> > > > applications have access to.
> > > > 
> > > > There are 179 total commits in that list
> > > > Of those, a bit of spot checking suggests that about 10-15% of them actually
> > > > change ABI, and many of those came from Bruce's rework of the mbuf structure.
> > > > That about 17-20 instances over the last 2 years where an ABI update would have
> > > > been needed.  That seems pretty reasonable to me.  Where exactly is your concern
> > > > here?
> > > 
> > > Ping Thomas, I'd like to continue this debate to a conclusion.  Could you please
> > > provide specific details and/or concerns that you have with this patch series?
> > > 
> > Ping again Thomas, please lets debate this to a reasonable consensus.  Ignoring
> > it won't help anything.
> 
> I'm not ignoring the discussion, I was trying to focus on other topics.
> 
for 2 weeks?  In that time frame, it seems like this could have made 1.8, but I
suppose thats neither here nor there.

> You're right, we need a conclusion.
> This patch is an important change which needs time to be finely checked and
> tested.
Not sure I understand this.  At the moment, all this patch introduces in
infrastructure, testing has been done on it, to ensure that it allows for
multiple versions of a function to exist (See Sergios comments).  Beyond that,
this patch should do nothing, save for restrict the symbol export list to the
set of symbols you expose as an API.  That last bit was why I was so eager to
get this in weeks ago, so people could do builds and make sure no symbols were
missed).  But from a functional standpoint, this patch current does nothing (nor
should it).  Its functionality (and need for testing) comes only when the next
ABI change arrives, and we need to ensure that both old and new versions of a
given function work.

> So I plan to integrate it in version 2.0 which will be the next one
> after 1.8 release.
What do you see as the advantage to waiting longer here?  Already this patch
likely will need some fixups because of API changing patches integrated ahead of
the time I posted it.  As noted above, this series just install infrastructure,
its doesn't actively "do" anything right now, nor should it. It will only allow
developers to do something after you integrate it and ABI needs to change (in
that order).  By waiting longer, the only thing that happens is that the export
map file become more out of date.

> In the mean time you could test this patch with Fedora
> and see how it helps application packaging. Then we could be more confident
> that we are applying the right policy starting with 2.0.
> 
Again, I could add this patch to fedora, but without a subsequent change that
introduces an ABI incompatibility that we need to provide backwards
compatibility for, this patch will be functionally invisible.  I'm not sure what
the goal of doing something like that would be.  If you have a specific goal in
mind, please let me know and I gladly consider it, I just don't see it at the
moment.

> Thanks Neil, I appreciate your involvement in DPDK
And I yours.  I'm sorry for being so beligerent about this, but it seems to me
that, without raising my voice, this would go nowhere.

Neil

> -- 
> Thomas
>