Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108


* Roman Zippel <zippel@xxxxxxxxxxxxxx> wrote:

Hi,

On Thu, 14 Sep 2006, Ingo Molnar wrote:

On Thu, 14 Sep 2006, Ingo Molnar wrote:

i have one very fundamental question: why should we do this
source-intrusive method of adding tracepoints instead of the dynamic,
unintrusive (and thus zero-overhead) KProbes+SystemTap method?

Could you define "zero-overhead"?

zero overhead when not used: not a single instruction added to the
kernel codepath that is to be traced, anywhere. (which will be the case
on 99% of the systems)

Using alternatives this could be near zero as well and it will likely
have less overhead when it's actually used.

if there are lots of tracepoints (and the union of _all_ useful
tracepoints that i ever encountered in my life goes into the thousands)
then the overhead is not zero at all.

also, the other disadvantages i listed very much count too. Static
tracepoints are fundamentally limited because:

- they can only be added at the source code level

- modifying them requires a reboot which is not practical in a
production environment

- there can only be a limited set of them, while many problems need
finegrained tracepoints tailored to the problem at hand

- conditional tracepoints are typically either nonexistent or very
limited.

for me these are all _independent_ grounds for rejection, as a generic
kernel infrastructure.

the key point is that we want _zero_ "static tracepoints". Firstly,
static tracepoints are fundamentally limited:

BTW I don't mind KProbes as an option, but I have huge problem with
making it the only option.

i'm not arguing for SystemTap to be the only option (KProbes is just the
infrastructure SystemTap is using - there are other uses for KProbes
too), but i'm arguing against the inclusion of static tracepoints as an
infrastructure, precisely because a much better option (SystemTap) is
already available and is usable on the stock kernel. You are of course
free to invent other, equally advantageous (or better) options.

But besides the usability problems, the most important problem is
that static tracepoints add a _constant maintainance overhead_ to
the kernel. I'm talking from first hand experience: i wrote
'iotrace' (a static tracer) in 1996 and have maintained it for many
years, and even today i'm maintaining a handful of tracepoints in
the -rt kernel. I _dont_ want static tracepoints in the mainline
kernel.

Even dynamic tracepoints have a maintainance overhead and I doubt
there is much difference. The big problem is having to maintain them
outside the mainline kernel, that's why it's so important to get them
into the mainline kernel.

i dispute that: for example kernel/sched.c has zero maintainance
overhead under SystemTap, while it's nonzero with static tracepoints. Of
course SystemTap _itself_ has maintainance overhead, but it does not
slow down any other subsystem's speed of progress.

You didn't address my main issue at all - kprobes is only available
for a few archs...

the kprobes infrastructure, despite being fairly young, is widely
available: powerpc, i386, x86_64, ia64 and sparc64. The other
architectures are free to implement them too, there's nothing
hardware-specific about kprobes and the "porting overhead" is in essence
a one-time cost - while for static tracepoints the maintainance overhead
goes on forever and scales linearly with the number of tracepoints
added.

Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108
    ... kernel infrastructure. ... Tracepoints of course need to be managed, ... and thus is zero maintainance overhead. ... i frequently break my kernel via static tracepoints and i ...
    (Linux-Kernel)
  • Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108
    ... People want to be able to insert a tracepoint anywhere and anytime - and also they want to have zero overhead from tracing if no tracepoints are used on a system. ... on the current kernel, because the 'coupling' is loose: ... you can add add-hoc tracepoints to thousands of functions, instead of having to maintain hundreds of static tracepoints all the time. ... are magically different and should be maintained out of tree somehow. ...
    (Linux-Kernel)
  • Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108
    ... source-intrusive method of adding tracepoints instead of the dynamic, ... static tracepoints add a _constant maintainance overhead_ to the kernel. ... It needs no changes at the source code level at ...
    (Linux-Kernel)
  • Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108
    ... A dynamic tracepoint is _detached_ from the normal source code ... dynamic tracepoints in the source code. ... What happen if someone need trace points in "normal kernel development" (which ... minimize the number of static tracepoints to _zero_. ...
    (Linux-Kernel)
  • Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108
    ... If they already have to go with their own tracepoints for the ... Dynamic instrumentation fills the same sort of gap ... for not having to depend on me to get a working traceable kernel. ... The static tracepoints we maintained were *the* solution for a great ...
    (Linux-Kernel)