Re: Linux or BSD alternative to Windows Home Server

On Mon, 03 Sep 2007 10:57:50 +0100, The Natural Philosopher wrote:

Douglas Mayne wrote:
On Sun, 02 Sep 2007 22:27:48 +0000, dh003i wrote:

My systems don't have enough memory to worry about whether they are 32 or
64-bit. Some major distributions have versions which have been
precompiled for 64-bit. There is starting to be more interest in 64-bit,
but RAM is a limiting factor for a lot of systems. 32-bit systems have
quite a bit of life remaining, IMO.

I did some research with my server..the processor is 64 bit, but it
would run either 32 or 64 bit kernels.

I chose the 32 bit one because

- many 3rd party apps are not available yet in a stable 64 bit port
- the 32 bit code was smaller.
- in many tests the 64 bit code wasn't any faster
- I wanted rock solid performance,and the 32 bit has been around a long
time.
- in a file and web server, computaional speed is almost irrelevant: wah
is needed is a fast network and disk. 32/64 bit makes sod all difference.

Thanks for this info. If I understand the memory model correctly, the
computer can have more memory than 4G, but only 4G can be paged in at
one time. This limits the amount of memory the kernel can give each
process to be at most* 2^32 bytes. (*Memory available to each process will
be less than that because the kernel uses some of the address space for
its own processes, probably along some division line (say, 3G user + 1G
kernel).) The advantage of 64-bit is its larger address space, which will
not place a 4G limit on any one process. The 64-bit address space is so
large that it is probably not possilbe to _ever_ build a computer which
has consumed its entire address space. Correct?



5. One of my foreward-looking concerns is scalability & ability to
upgrade: possibility to upgrade to numerous HD, multiple CPUs,
multiple GPUs, RAM, etc. Thus, my concerns here are the limits in
Linux and BSD on hard-drive space recognizeable, and RAM recognizable.

It would be nice. However, almost every new generation of motherboards
requires wholesale replacement of the CPU, memory, motherboard, and
possibly the case and power supply also. If cost is no object, then the
sky is the limit. Personally, cost is an object for me. I try to keep
cost in check. For example, I have just started upgrading the P-III
architecture boards which I have used up until now. The Intel Core 2
architecture offers a 10x to 100x performance boost. I think I have
probably saved a lot of money waiting for a compelling upgrade. I
forsee that the new Core 2 boards that I am rolling out will have a
long life, too.

I have an old pentium here with a few hundred Mbyte RAM that is capable
of serving files as fast as the network can deliver them.


You are a gamer, that takes CPU power. Serving web pages and files does
not.

Start thinking professionally, and gear your hardware not to the sexy
marketing, but to the requirements of the job in hand.

It's hard not to get caught up in the blitz. Some of the current
generation mohterboards/CPUs provide a very big bang to the buck. We
are literally awash in CPU power- the current generation of boards is
probably overkill for a lot of uses that they will be put. Still, a
new board may be the best choice in the long run, even if it is
overpowered, because it may have a long service life. That said, I
also believe upgrades should be rolled out only as necessary.

Actually, this remains my current upgrade strategy:
1. Workstations are upgraded to Core 2 Duo (E6600 CPU, typ), if the user
needs the power. The typical user is doing complex spread***/database
and mathematical work. They have said the upgrade provides a boost of
between 10x to 100x when compared to their old computer.

2. Motherboard, CPU, and memory from old workstations are P-III class. The
motherboard will accept 1.5G RAM. Some workstations had the full amount
installed. These are being inspected, and used to upgrade some office
servers which could benefit from newer hardware. are currently using more
obsolete hardware than that. These boards are also appropriate for use as
clones, see below.

BTW, this strategy is probably only appropriate for a small office. Larger
offices are less interested in recycling their equipment in this manner,
for a variety of reasons. Probably, it is mostly due to the fact they
like working with "heavy iron." ;-) They don't like to "piss around" on
the small stuff.

6. I'm aware that I could probably create scripts to regularly backup
certain files, and use BASH shell commands to specify a schedule for
such; but I was wondering if there were utilities for this. And also,
streaming of photos or video or music to the TV or stereo system? Is
that possible?
Making backups is a big topic in itself. It can be easy, or it can be
relatively complex. People tend to roll out solutions which "fit" their
needs. One solution that I often recommend is to take a snapshot using
an external disk, either to the storage on another network computer, or
directly to an external disk (USB 2.0). I see that a lot of people on
these groups recommend Amanda, and similar programs. The goal of these
programs is to provide a self-booting backup set.


The biggest problem with backups is that the backup media size is now
well outstripped by the disk size. You are better off with a twin
machine or twin disk setup than hoping to burn a tape once a day as we
used to do.

I'd be interested i strategies too.

I haved started using a poor-man's hot-swap on some critical systems; it's
a "warm" swap, I guess ;-). The goal is to provide a quicker recovery than
would be possible if restoring the server from absolute scratch. I
estimate this idea could save 2-4 hours of time restoring the server under
some failure modes. It only relies on one fact: hardware is cheap. In the
past, this assumption was not true. However, Moore's law has definitely
caught up. In my setup, I can afford to have another similar system
sitting beside critical servers. These warm swap servers are initially
setup by copying the source server (cloning). After the initial copy, some
minor tweaks are required to make the server and its clone non-identical
(unique IP address, etc.). That step is to ensure that there will be no
network conflicts when they are booted at the same time (next step.) Most
of the time the warm swap computer is turned off; it is only turned on
once every 2-4 weeks. When the clone is booted, a simple restore operation
is performed to bring the served data up to date. Also, OS patches which
have been issued in the interim may be applied as necessary, but this
technique is primarily a method for quickly snapshotting the _data
partition_ which is being served.

The clone is brought up to date in two steps:
1. The data from relative backups which have been made since the
last sync are restored to the clone.
2. Files are closed on the source server, then the clone begins an rsync
operation against it. Because of step 1, this step should complete quickly
and limit the amount of time which the server is effectively offline to
users. Files must be closed to ensure a clean, consistent backup. This
step should complete in a few minutes.

In the event the clone is needed due to a failure. it can be brought
online quickly. It will be powered on, restored (as above) using the most
recent snapshot, and will assume the identity (IP, name) of its source
server at boot.

BTW, there is still potential for data loss using this method. If this
potential is not acceptable, then use a true failover cluster (or
something similar.)

BTW, this method only _supplements_ other backup tools that I
use. It obviously does not protect against failure modes where both
the server and its clone are destroyed. In that case, restoration of the
backup set will require more time and rely on other backups being
available. I use external USB 2.0 hard drives for offsite backups. These
are a good bang to the buck, too. Their capacity is approaching 1TB per
disk. I did a rough calculation recently which compared LTO tape vs. USB
HD at current price levels. IIRC, tape was cheaper, if your backup set was
larger than 40TB. I have less data than that.

--
Douglas Mayne
.