Re: [KORG] Re: kernel.org lies about latest -mm kernel

On Mon, 2007-01-08 at 22:20 +0100, Jean Delvare wrote:
Hi JH,

On Sat, 16 Dec 2006 11:30:34 -0800, J.H. wrote:
The root cause boils down to with git, gitweb and the normal mirroring
on the frontend machines our basic working set no longer stays resident
in memory, which is forcing more and more to actively go to disk causing
a much higher I/O load. You have the added problem that one of the
frontend machines is getting hit harder than the other due to several
factors: various DNS servers not round robining, people explicitly
hitting [git|mirrors|www|etc]1 instead of 2 for whatever reason and

I am trying to be a good citizen by explicitely asking for
www2.kernel.org, unfortunately I notice that many links on the main
page point to www.kernel.org rather than www2.kernel.org. Check the
location, patchtype, full source, patch, view patch, and changeset
links for example. Fixing these links would let people really use www2
if they want to, that might help.

True - however if you look at the underlying link for those you'll
notice that most of the links will continue to use www2 instead of www.
The ones that explicitly point to www probably have a good reason for
doing so, but I'll have to check on that. Regardless the kernel.org
webpages need some work and it's on my todo list (maybe I should post
that somewhere...)


BTW, I'm no DNS expert, but isn't it possible to favor one host in the
round robin mechanism? E.g. by listing the server 2 twice, so that it
gets 2/3 of the load? This could also help if server 1 otherwise gets
more load.

Could, but the bigger problem seems to be people explicitly pointing
rsync at 1 instead of the generic name or 2. Beyond that traffic seems
to distribute as we are expecting.


So we know the problem is there, and we are working on it - we are
getting e-mails about it if not daily than every other day or so. If
there are suggestions we are willing to hear them - but the general
feeling with the admins is that we are probably hitting the biggest
problems already.

I have a few suggestions although I realize that the other things
you're working on are likely to be much more helpful:

* Shorten the www.kernel.org main page. I guess that 99% of the hits on
this page are by people who just want to know the latest versions, and
possibly download a patch or access Linus' git tree through gitweb. All
the rest could be moved to a separate page, or if you think it's
better to keep all the general info on the main page, move the array
with the versions to a separate page, which developers can bookmark.
Splitting the dynamic content (top) from the essentially static content
(bottom) of this page should help with caching, BTW.

The frontpage itself cache's pretty nicely and the upper 'dynamic'
content isn't constantly being generated on every page request so by and
large this caches and we don't have any real issue with it.


* Drop the bandwidth graphs. Most visitors certainly do not care, and
their presence generates traffic on all web servers regardless of the
one the visitor is using, as each graph is generated by the respective
server. If you really like these graphs, just move them to a separate
page for people who want to watch them. As far as I am concerned, I
find them rather confusing and uninformative - from a quick look you
just can't tell if the servers are loaded or not, you have to look at
the numbers, so what's the point of drawing a graph...

While I agree that most users don't care, they are useful. If someone
notices that 1 has an incredibly high load and moving lots of traffic in
comparison to 2, than they can manually redirect to 2 for better &
faster service on their own. Since these images aren't particularly big
they cache just fine and it's not that big of a deal, and there are much
longer poles in the tent right now.


Of course the interest of these proposals directly depends on how much
the www.kernel.org/index page accounts in the total load of the servers.


Honestly - negligible at best. We have bigger issues from trying to
service 200 seperate rsync processes on top of http, ftp, git, gitweb,
etc than worying about a couple of small, 90% static pages.

- John 'Warthog9' Hawley
Kernel.org Admin

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: [KORG] Re: kernel.org lies about latest -mm kernel
    ... on the frontend machines our basic working set no longer stays resident ... a much higher I/O load. ... their presence generates traffic on all web servers regardless of the ...
    (Linux-Kernel)
  • Re: Active Directory Question
    ... "As to whether we go smaller servers to spread the load or a large unit has ... Some applications pick a domain controller and don't spread the load. ... However, if your app is ...
    (microsoft.public.windows.server.active_directory)
  • Re: Active Directory Question
    ... > Some applications pick a domain controller and don't spread the load. ... > application you'll need to know how the app will behave before you can ... >> As far as production servers go, we tend to purchase IBM Xseries servers ...
    (microsoft.public.windows.server.active_directory)
  • Re[2]: FreeBSD Machines dieing, weve tried so much....
    ... We have five freeBSD servers in total, ... That is why I do not believe this is a hardware problem. ... Portsdb -uU ran without problems after I disabled SMP. ... these two are the ones with the highest load. ...
    (freebsd-questions)
  • Re: FreeBSD Machines dieing, weve tried so much....
    ... dmesg and hardware specs, etc. ... We have two different servers crashing. ... these two are the ones with the highest load. ... crashes next. ...
    (freebsd-questions)