Re: multicore management system



On Friday December 28 2007 10:23:26 Douglas A. Tutty wrote:
On Fri, Dec 28, 2007 at 06:58:42AM -0600, Ron Johnson wrote:
On Friday December 28 2007 00:27:24 Jerome BENOIT wrote:
[snip]

I have a long list of jobs that I want to launch on my box.
On a one core box, I can make a for loop in order to submit
then in sequence. On a multi-core box, this approach does
not work anymore. Hence my intention to use Torque: here I
envisage my multi-core box as a cluster where each core is
a node. Note that you can configure LAM in such a way.

Old school OSs like OS/MVS, DOS/VSE, OpenVMS, MPE, MCP, etc
all have sophisticated batch queue functionality.

I've never understood why Unix has never received such power.

Because nobody has written it. Ron, I've never run any of
those OSs. AIUI, you had queues: one for heavy IO, one for
heavy comp, one for a mix. The queue manager ran one heavy IO
with one heavy comp, but only one mixed one at a time. IIUC,
this would require the operator/programmer to classify the
program as to which queue it should reside.

That's not how it works on z/OS (OS/MVS), DOS/VSE & OpenVMS.

On them, you have "named batch queues". Each queue has a default
(in Unix terminology) niceness level, and "width" (like how a
bank branch has a single line feeding multiple teller windows,
the "width" defines the number of jobs that can be in the Running
state, whereas the jobs standing in "line" are Pending). Also,
there is a job priority, so that important Pending jobs jump to
the front of the line, and less important jobs fall to the rear.
Just like with "at", jobs can be scheduled to run at various
times. Of course, if the execute slots are all full at a job's
run time, it goes not to Executing but to Holding state.

You can have as many named batch queues as you want, but typically
only a few are used.

The beauty of batch queues is that they are dynamic. So, for
example, at 18:00 after all the users have logged off you
increase the queue width so that more batch jobs can run, and at
06:00 you shrink the queue width, so that by 08:00 the "excess"
batch jobs have completed and only a few can run at a time, thus
giving more resources to the on-line users.

The most impressive product, though, IMNSHO, was DEC SCHEDULER
(now CA Unicenter Job Management). Like cron it could schedule
things by the minute thru dow & month, but also at fiscal
intervals, and had pre- and post-functions and allowed for
complex dependencies. Of course, since it was a DEC product that
ran on VMS, it was cluster-aware and could run jobs on remote
nodes thru DECnet (think NFS on steroids, or SMB if it were
written not by MSFT but by a company that knew what it was
doing).

Right now, with cron, jobs get done sequentially. If you watch
it with top, at any given point, you could find the system
maxing out CPU and IO being trivial, or the system spent
waiting for IO with free comp cycles available.

It shouldn't be too hard to write program that looked at the
current load and decided which queue to select for the next
program to run. This would take into account running programes
not under controll of the queueing system (eg normal cron runs,
other users, etc). Although, IIRC, on those older OSs, the
system was either in multi-user state or batch state.

Geez. When? 1965? Certainly not since the early 1970s.

--
-----------------------------------------------------------------
Ron Johnson, Jr.
Jefferson, LA USA

"A great many open minds should be closed for repairs."
Toledo Blade Newspaper

Attachment: signature.asc
Description: This is a digitally signed message part.