The Server has 3GB of memory; Running DNS on the server

Installed two 1 GB DIMMS and booted back up; seems to work fine.

Also noted that our server CAN accept 2GB DIMMS, so we should order at least a pair of those.  So we'll have 512 MB x 2, 1 GB x 2, then 2 GB x 2, for 7 GB total.

Also managed to use Remote Desktop to set all lab machines to use ONLY northshoreview for DNS.  (Various machines had been using combinations of 10.3.30.110, 10.17.30.110, and northshoreview (10.25.40.12)).  Now if the Northshoreview DNS stops working, so will the lab machines, but at least we will know what's going on instead of just getting mysterious lag.  Remote Desktop is pretty cool.  Now if only these machines had LOM (Lights Out Management) instead of needing a kid to run around the room pressing spacebars and/or power buttons.

Remote Desktop has access to all lab machines

OK, now both instances of Remote Desktop have access to all the lab machines.  ns-lab-imac-02 and ns-lab-imac-01 were holdouts.  I don't know why they didn't show up with a name, but I created a custom Remote Desktop Installer, put that on the diradmin directory on northshoreview, mounted it from each of the two, and ran it.  After that install, both machines now show up on both Melissa's admin console and the secondary one.

I think Remote Desktop will now work for all machines, from either admin console.

Remote Desktop from Lab-Admin(2) has access to all lab machines

On the four remaining holdouts (Ch, Ck, Wd, and BWd) from Remote Desktop, I went to each machine, logged in as admin or nsvadmin, chose Sharing, Apple Remote Desktop, then Access Permissions.  I enabled at least some account (usually nsvadmin) for ARD and enabled all the ARD things (report, control, observe, etc.)  I then entered the matching login/password info in Remote Desktop on Lab-Admin(2).  That, plus a reboot, got all machines controllable from Lab-Admin(2).

I also updated the ARD client software on those machines, plus a few more throughout the network that happened to be available.

Last round of Software Updates for a while?

As far as I can tell, all the lab machines are up to date.  That took a while.

Now that the server has current and correct packages on it, I re-ran software update on all the machines until there were no new packages to install.  The main thing that seems to have been out of date was Java, although we got a couple more Safari updates (to Safari 3.2.1).

Gathered some data on server disk utilization

Melissa had a classroom full (ok, 24) of kids all launch Safari at once (as she would typically do when teaching the kids a new procedure).  I gathered iostats until all the kids were successfully at their destination web page.  I noticed that the number of transactions per second on disk1 and disk2 spikes in the 160+ range, then gradually dwindles down to the 50-ish range after a few minutes.  When it's in the 150+ range, kids definitely seem to be getting the beach ball of doom.  Also, for reference, the steady-state number of tps when the lab is idle (except for me) is in the 0 - 20 range.

This makes me think of two ways we might improve things.

1.  Move some or all of the kids' home directories to another (or multiple) file servers.

2.  Add RAM to the server so that some of the disk activity might be cached in RAM.

I still don't know how congested the local network is because I don't know how to log on to the managed(?) network switches (or how to use them, for that matter).  The outbound connection seemed OK during this experiment because we had zero packet loss while pinging yahoo.com for the duration.  I guess that also argues that the local network isn't too congested.

The other thing I noticed was that when counting net connections using netstat, it pauses frequently when it's about to display an ipv6 connection.  For example:

northshoreview:~ admin$ netstat

. . .

tcp6 0 0 fe80:4::20d:93ff.548 fe80:4::214:51ff.49229 ESTABLISHED

It doesn't pause with netstat -n, so I think we have a reverse DNS lookup problem.  I don't know much about our particular DNS setup, but I'll try to learn about it and see if I can figure out what's working and what isn't.  Poorly performing DNS setups can cause a lot of end-user slowness.

Here's the (sorry, really ugly) iostat output.

northshoreview:~ diradmin$ iostat 5
disk1 disk0 disk2 cpu

KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
northshoreview:~ diradmin$ iostat 5
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
17.63 10 0.17 17.62 10 0.17 10.42 1 0.01 5 6 89
11.26 94 1.03 11.06 93 1.00 4.39 14 0.06 20 41 39
11.83 93 1.08 11.84 97 1.12 6.22 14 0.09 17 42 42
8.50 90 0.74 9.09 81 0.72 8.86 6 0.05 18 38 43
10.43 81 0.82 10.95 75 0.80 13.17 15 0.19 18 37 46
8.96 54 0.47 9.23 61 0.55 0.00 0 0.00 20 40 40
10.02 66 0.65 10.12 67 0.66 0.00 0 0.00 15 32 53
10.79 66 0.69 10.60 66 0.69 4.44 4 0.02 19 37 44
12.29 88 1.05 11.32 94 1.04 22.24 6 0.13 13 30 57
11.15 51 0.55 10.75 50 0.53 5.01 13 0.07 20 40 40
17.58 127 2.18 17.76 126 2.18 17.00 4 0.07 16 33 51
11.32 168 1.86 11.90 168 1.95 5.40 13 0.07 15 33 51
14.47 167 2.36 15.36 162 2.44 8.92 10 0.09 15 32 53
11.66 178 2.02 11.86 173 2.00 5.10 6 0.03 14 29 58
12.72 184 2.28 12.35 185 2.23 70.86 1 0.10 15 31 54
12.41 112 1.36 12.88 116 1.46 12.44 5 0.07 13 41 45
10.11 87 0.86 10.03 91 0.89 5.30 14 0.07 19 38 43
9.93 80 0.77 9.56 82 0.76 22.62 5 0.11 16 39 45
8.56 86 0.71 8.89 85 0.74 13.85 16 0.22 18 45 37
9.57 67 0.63 10.56 66 0.68 10.87 9 0.10 20 46 34
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
9.73 80 0.76 9.29 87 0.79 12.00 2 0.03 13 25 62
12.51 113 1.38 12.93 112 1.42 8.52 11 0.09 21 34 44
11.21 75 0.82 10.57 77 0.80 11.72 12 0.13 16 41 43
10.94 80 0.86 11.09 80 0.87 22.55 7 0.15 22 45 34
11.24 83 0.91 11.48 86 0.96 15.21 12 0.18 23 52 25
11.37 93 1.03 11.17 90 0.98 5.14 1 0.01 20 42 39
11.93 80 0.93 12.90 77 0.97 0.00 0 0.00 25 50 25
13.20 147 1.89 13.33 144 1.87 0.00 0 0.00 18 38 44
9.90 107 1.04 9.83 110 1.06 0.00 0 0.00 22 43 35
11.68 89 1.02 11.39 92 1.02 0.00 0 0.00 23 42 35
10.72 86 0.90 11.20 81 0.89 0.00 0 0.00 21 47 32
11.13 85 0.93 10.35 88 0.89 0.00 0 0.00 27 47 27
13.46 64 0.84 12.28 66 0.79 0.00 0 0.00 23 41 36
14.98 133 1.94 14.98 134 1.96 0.00 0 0.00 20 40 40
8.93 89 0.78 8.88 82 0.71 0.00 0 0.00 22 37 41
14.57 71 1.00 14.88 73 1.07 0.00 0 0.00 24 48 28
11.28 82 0.90 9.99 90 0.87 0.00 0 0.00 27 52 20
10.77 82 0.86 10.72 90 0.95 0.00 0 0.00 20 48 32
12.31 88 1.06 12.82 84 1.06 0.00 0 0.00 26 58 16
19.72 113 2.17 19.67 113 2.17 0.00 0 0.00 27 47 26
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
22.28 138 3.00 22.58 135 2.99 0.00 0 0.00 27 50 23
12.59 87 1.07 11.81 92 1.06 0.00 0 0.00 24 52 23
8.86 80 0.69 9.21 73 0.66 0.00 0 0.00 12 24 64
11.81 65 0.74 11.35 65 0.72 8.00 1 0.00 26 51 23
14.91 74 1.08 15.63 69 1.06 0.00 0 0.00 17 37 45
11.21 71 0.77 10.56 73 0.75 0.00 0 0.00 27 49 24
36.60 131 4.67 35.93 134 4.70 0.00 0 0.00 17 30 53
13.47 69 0.91 13.96 64 0.88 0.00 0 0.00 21 42 37
11.87 42 0.48 11.08 48 0.51 0.00 0 0.00 17 39 44
20.21 44 0.87 19.74 48 0.93 0.00 0 0.00 24 44 32
27.93 52 1.42 26.83 55 1.45 0.00 0 0.00 24 43 34
19.50 57 1.09 18.33 63 1.12 0.00 0 0.00 19 44 37
8.63 60 0.51 8.59 61 0.51 0.00 0 0.00 21 44 35
40.79 129 5.12 40.08 132 5.15 10.12 1 0.01 19 40 41
10.75 46 0.49 11.36 46 0.51 0.00 0 0.00 25 50 25
9.21 44 0.40 8.99 43 0.38 0.00 0 0.00 21 36 43
15.88 58 0.90 16.22 55 0.88 0.00 0 0.00 17 36 47
8.02 42 0.33 8.25 42 0.34 0.00 0 0.00 15 42 43
12.74 48 0.60 12.96 48 0.60 0.00 0 0.00 24 42 35
28.19 164 4.51 28.42 162 4.50 0.00 0 0.00 14 30 57
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
10.81 39 0.42 11.00 37 0.40 0.00 0 0.00 25 45 30
11.79 43 0.49 11.80 43 0.49 0.00 0 0.00 21 41 38
18.34 44 0.78 17.42 48 0.81 0.00 0 0.00 25 61 14
50.56 29 1.41 51.30 28 1.41 0.00 0 0.00 27 53 20
9.75 51 0.48 9.82 48 0.46 0.00 0 0.00 19 41 40
33.34 83 2.70 34.94 79 2.70 0.00 0 0.00 16 32 51
22.46 107 2.34 22.41 105 2.29 0.00 0 0.00 16 36 48
10.14 43 0.42 10.01 46 0.45 0.00 0 0.00 23 44 33
12.37 49 0.59 11.97 51 0.59 0.00 0 0.00 27 46 26
18.26 41 0.73 19.18 38 0.70 0.00 0 0.00 25 48 27
22.49 39 0.87 22.95 39 0.87 0.00 0 0.00 30 46 24
26.50 43 1.11 27.07 41 1.09 0.00 0 0.00 34 56 10
42.18 114 4.71 41.53 117 4.73 0.00 0 0.00 24 48 29
50.73 26 1.27 48.87 27 1.29 0.00 0 0.00 33 57 10
19.74 47 0.90 19.68 48 0.91 0.00 0 0.00 30 58 12
45.39 33 1.47 40.46 37 1.46 0.00 0 0.00 34 57 9
21.48 61 1.27 21.54 62 1.29 8.00 1 0.00 29 55 16
16.88 43 0.72 15.98 48 0.76 0.00 0 0.00 29 55 17
29.38 179 5.13 29.19 181 5.15 0.00 0 0.00 18 44 38
11.45 75 0.84 11.44 78 0.87 0.00 0 0.00 16 34 50
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
11.94 60 0.70 12.02 60 0.71 0.00 0 0.00 24 49 27
13.55 85 1.12 13.66 84 1.11 0.00 0 0.00 19 41 40
23.75 99 2.30 23.73 99 2.30 0.00 0 0.00 24 41 34
14.14 93 1.28 14.15 93 1.28 0.00 0 0.00 18 37 45
25.56 99 2.48 27.16 91 2.42 0.00 0 0.00 25 41 34
15.42 154 2.32 15.27 159 2.37 0.00 0 0.00 22 30 48
18.73 82 1.51 18.14 86 1.52 0.00 0 0.00 30 36 34
38.96 96 3.66 38.12 99 3.67 0.00 0 0.00 23 43 34
13.43 92 1.20 13.43 86 1.13 0.00 0 0.00 30 50 20
11.72 75 0.86 12.35 70 0.85 0.00 0 0.00 24 50 26
12.81 65 0.81 12.89 64 0.81 0.00 0 0.00 24 49 26
11.89 93 1.07 12.00 94 1.10 0.00 0 0.00 26 44 30
13.69 65 0.87 13.70 63 0.85 0.00 0 0.00 31 49 20
12.55 65 0.79 12.52 66 0.80 0.00 0 0.00 26 55 19
13.23 46 0.59 13.36 46 0.61 0.00 0 0.00 33 46 21
12.24 20 0.24 12.04 21 0.24 7.09 3 0.02 28 54 18
11.36 31 0.34 11.67 28 0.32 0.00 0 0.00 33 48 20
11.43 68 0.76 11.28 70 0.77 11.72 2 0.02 31 48 21
12.78 29 0.36 13.49 27 0.35 0.00 0 0.00 26 54 20
9.81 41 0.40 9.63 44 0.41 0.00 0 0.00 21 41 38
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
9.53 52 0.48 9.49 53 0.49 0.00 0 0.00 20 31 49
13.46 46 0.61 13.80 44 0.59 0.00 0 0.00 25 53 22
13.84 43 0.58 13.56 45 0.59 0.00 0 0.00 30 50 20
10.51 61 0.63 10.69 59 0.61 0.00 0 0.00 31 52 17
10.33 59 0.59 10.79 58 0.61 0.00 0 0.00 31 54 16
10.56 56 0.58 10.33 59 0.59 0.00 0 0.00 27 49 24
12.45 27 0.33 11.81 32 0.37 0.00 0 0.00 32 55 12
6.84 9 0.06 7.51 8 0.06 0.00 0 0.00 31 54 15
13.10 35 0.45 13.48 36 0.47 0.00 0 0.00 30 59 11
11.16 65 0.70 10.94 65 0.70 0.00 0 0.00 25 49 26
10.81 90 0.95 10.93 89 0.95 0.00 0 0.00 20 43 37
8.95 53 0.47 8.87 59 0.51 0.00 0 0.00 26 40 35
9.55 64 0.59 9.54 57 0.53 0.00 0 0.00 22 48 30
11.58 72 0.81 11.52 71 0.80 0.00 0 0.00 23 37 40
11.85 60 0.69 11.96 59 0.69 0.00 0 0.00 20 37 44
9.74 53 0.51 9.95 48 0.46 0.00 0 0.00 17 40 43
9.88 86 0.83 9.89 84 0.81 0.00 0 0.00 16 39 45
12.62 64 0.79 12.83 63 0.79 0.00 0 0.00 20 45 35
11.53 63 0.71 11.70 62 0.70 0.00 0 0.00 20 37 43
11.78 68 0.79 11.79 68 0.79 0.00 0 0.00 29 41 31

Managing Software Update Server

I should also mention that using Workgroup Manager, I set the "Software Update Server To Use" preference to be managed Always, and to use "http://northshoreview.smfc.k12.ca.us:8088" on all of the groups Lab Computers, and Room 2 through Room 19.

Software Update packages were missing

It looks like we were trying to use our server to mirror software update packages so we didn't have to re-fetch them for each computer.

A problem I ran into when trying to do the software updates was that at least two of the packages, GarageBand 1.1 and J2SE 5 something wouldn't install.

I poked around on the server and found where the software update packages are mirrored (/usr/share/swupd/html, as it turns out), and deleted the two offending packages (which were of size 0).  Then, using Server Admin, told Software Update to check in with Apple, which resulted in a fresh fetch of those two packages.  Yay!  When I ran Software Update on one of the lab macs, now it was able to install the packages.  Yay!  Then, that mac was eligible for the next Java update, which was also size 0 and failed.  Boo!

So I did a find in the software update directory for all zero length packages and turned up 20 or more out of 206 that were apparently messed up.   I deleted all those and am currently refetching from Apple (just poked the "Check Now" button on Server Admin -> northshoreview -> Software Update), and it looks like we'll have a fresh set of updates shortly.  Most of those won't apply to most of the lab macs, at least (e.g., updates to FinalCutPro, DVDStudioPro, WebObjects Developer, etc.), but we still need to run software update on all of them to at least get Java current.  I'll dig on on that again Friday, probably.