[dpdk-ci] [RFC] test lab database schema

[Thomas Monjalon]

Hi,

10/11/2017 00:53, Patrick MacArthur:
> I have been working on a database schema for storing performance
> results. I am attempting to make this generic enough to store whatever
> measurements we want.

Thanks for working on it.

> I have attached an entity-relationship diagram of the schema which
> should illustrate the starting point for the schema.

I will do some comments below.

[...]
> patch: I propose that for this schema patches will be stored on the
> filesystem content-addressable by the sha256 hash of the patch file.

I don't see the need to store the full diff.
You could store the patchwork id to retrieve it.

> patchset: "branch" refers to corresponding repo (master -> dpdk,
> dpdk-next-net -> dpdk/next-net, etc.) and will be NULL until the
> patchset is applied. Tarballs stored as files named after patchset id.

You should also be able to store measurements for a given state of a git tree.
It will be used for regular tests (like daily).

> patchset_result: Result entry for a patchset. A patchset passes if
> there is a result for every measurement which is either PASS or N/A.
> 
> environment: This should represent everything about where the test was
> run and a new environment needs to be created every time this changes
> (e.g., kernel or compiler update). I gathered the list of fields by
> looking at the existing performance reports on the DPDK website. This
> can be used for verification, to allow the test environment to be
> reproducible, and to ensure that all comparisons are within an
> identical setup.

Do we want a limited set of environment fields, or something flexible?
For instance, there is a field dts_config, but we could use other test tools.

> measurement: A single measurement which can be applied to any
> patchset. We can use values like (name: “BUILD”, higherIsBetter: TRUE,
> expectedValue: 1, deltaLimit: 0) to verify non-performance conditions,
> such as the build succeeding for the given environment.

I think we should store the normal values per-environment at different time.
I mean, the normal value will evolve with time and we should keep history of it.

> The primary keys for these tables are not shown; I will likely be
> implementing the database using the data modeling framework for
> whatever Web backend we wind up selecting, which will set up primary
> keys and table join relationships automatically.
> 
> Comments/suggestions? Is there anything that this schema does not cover?

As I said above, we need to cover tests based on git state, not patchset.

Thanks :)