Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108


Ingo Molnar wrote:
that's not true, and this is the important thing that i believe you are
missing. A dynamic tracepoint is _detached_ from the normal source code
and thus is zero maintainance overhead. You dont have to maintain it
during normal development - only if you need it. You dont see the
dynamic tracepoints in the source code.

And that's actually a problem for those who maintain such dynamic
trace points.

a static tracepoint, once it's in the mainline kernel, is a nonzero
maintainance overhead _until eternity_. It is a constant visual
hindrance and a constant build-correctness and boot-correctness problem
if you happen to change the code that is being traced by a static
tracepoint. Again, I am talking out of actual experience with static
tracepoints: i frequently break my kernel via static tracepoints and i
have constant maintainance cost from them. So what i do is that i try to
minimize the number of static tracepoints to _zero_. I.e. i only add
them when i need them for a given bug.

Bzzt, wrong. This is your own personal experience with tracing. Marked
up code does not need to be active under all build conditions. In
fact trace points can be inactive by default at all times, except
when you choose to build them in.

And as I said elsewhere, the fact that your use of instrumentation is
solely for debugging ("i only add them when i need them for a given bug"),
I repeat that there are mortals out there that need this for their
applications.

static tracepoints are inferior to dynamic tracepoints in almost every
way.

Sorry, orthogonal is the word.

hundreds (or possibly thousands) of tracepoints? Have you ever tried to
maintain that? I have and it's a nightmare.

I have, and I've showed you that you're wrong. The only reason you can
make this argument is that you view these things from the point of view
of what use they are for you as a kernel developer and I will repeat
what I've said for years now: static instrumentation of the kernel
isn't meant to be useful for kernel developers. While it may indeed
be in some cases, in most cases it's likely useless, as you've been
very successfully arguing in this thread. Nevertheless there are very
legitimate uses for standardized instrumentation points.

Even assuming a rich set of hundreds of static tracepoints, it doesnt
even solve the problems at hand: people want to do much more when they
probe the kernel - and today, with DTrace under Solaris people _know_
that much better tracing _can be done_, and they _demand_ that Linux
adopts an intelligent solution. The clock is ticking for dinosaurs like
static printks and static tracepoints to debug the kernel...

Thank you, I couldn't have put it better. This paragraph, more than
any other snippet I've seen to date, clearly demonstrates why
tracing is such a contentious issue. Kernel developers use tracing
during their normal development process, and of course their gut
reaction is: why the hell would anybody need this for mainline? But
of course this misses the entire point. Kernel tracing for developers
is but a corner case of kernel tracing in general. There are very valid
and legitimate reasons for userspace to be able to obtain important
events. And of course any infrastructure developed with that in
mind should also be usable by kernel developers.

Karim
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108
    ... the problem is, LTT is not about a 'few' tracepoints: it adds a whopping 350 tracepoints, a fair portion of it is multi-line with tons of arguments. ... Static tracepoints are a maintainance PITA that will rarely get smaller, ... SystemTap should very much be integrated into the kernel proper, but i dont think the _rules_ should become part of the _source code files themselves_. ...
    (Linux-Kernel)
  • Re: [patch 4/4] KVM-trace port to tracepoints
    ... Tracepoints allow dormat instrumentation, like the kernel markers, but also ... This patch depends on the "Tracepoints" patch. ... Is it a specific property of KVM-trace that causes this LOC blow-up? ... strings into the kernel code, which, to some, looks like debugging code. ...
    (Linux-Kernel)
  • Re: [PATCH] ftrace: add an fsync tracer
    ... markers, or by tracepoints converted to markers, or by tracepoints ... The tracepoint declaration is linked to the kernel code that is meant to ... I never said it would be harder to generate the trace data. ...
    (Linux-Kernel)
  • Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108
    ... by browsing the source code of various subsystems/filesystems/etc. ... who's authors saw fit to include their own static tracepoints. ... mechanisms for adding more localized instrumentation. ... set of known-to-be-good source of key kernel events. ...
    (Linux-Kernel)
  • Re: [patch 07/17] LTTng instrumentation - timer
    ... Instrument timer activity to keep ... information about the "real time" flow within the kernel. ... Those tracepoints are used by LTTng. ... This is a very dangerous trace point - we're holding xtime lock here. ...
    (Linux-Kernel)