The Server has 3GB of memory; Running DNS on the server

Installed two 1 GB DIMMS and booted back up; seems to work fine.

Also noted that our server CAN accept 2GB DIMMS, so we should order at least a pair of those.  So we'll have 512 MB x 2, 1 GB x 2, then 2 GB x 2, for 7 GB total.

Also managed to use Remote Desktop to set all lab machines to use ONLY northshoreview for DNS.  (Various machines had been using combinations of 10.3.30.110, 10.17.30.110, and northshoreview (10.25.40.12)).  Now if the Northshoreview DNS stops working, so will the lab machines, but at least we will know what's going on instead of just getting mysterious lag.  Remote Desktop is pretty cool.  Now if only these machines had LOM (Lights Out Management) instead of needing a kid to run around the room pressing spacebars and/or power buttons.

Remote Desktop has access to all lab machines

OK, now both instances of Remote Desktop have access to all the lab machines.  ns-lab-imac-02 and ns-lab-imac-01 were holdouts.  I don't know why they didn't show up with a name, but I created a custom Remote Desktop Installer, put that on the diradmin directory on northshoreview, mounted it from each of the two, and ran it.  After that install, both machines now show up on both Melissa's admin console and the secondary one.

I think Remote Desktop will now work for all machines, from either admin console.

Remote Desktop from Lab-Admin(2) has access to all lab machines

On the four remaining holdouts (Ch, Ck, Wd, and BWd) from Remote Desktop, I went to each machine, logged in as admin or nsvadmin, chose Sharing, Apple Remote Desktop, then Access Permissions.  I enabled at least some account (usually nsvadmin) for ARD and enabled all the ARD things (report, control, observe, etc.)  I then entered the matching login/password info in Remote Desktop on Lab-Admin(2).  That, plus a reboot, got all machines controllable from Lab-Admin(2).

I also updated the ARD client software on those machines, plus a few more throughout the network that happened to be available.

Last round of Software Updates for a while?

As far as I can tell, all the lab machines are up to date.  That took a while.

Now that the server has current and correct packages on it, I re-ran software update on all the machines until there were no new packages to install.  The main thing that seems to have been out of date was Java, although we got a couple more Safari updates (to Safari 3.2.1).

Gathered some data on server disk utilization

Melissa had a classroom full (ok, 24) of kids all launch Safari at once (as she would typically do when teaching the kids a new procedure).  I gathered iostats until all the kids were successfully at their destination web page.  I noticed that the number of transactions per second on disk1 and disk2 spikes in the 160+ range, then gradually dwindles down to the 50-ish range after a few minutes.  When it's in the 150+ range, kids definitely seem to be getting the beach ball of doom.  Also, for reference, the steady-state number of tps when the lab is idle (except for me) is in the 0 - 20 range.

This makes me think of two ways we might improve things.

1.  Move some or all of the kids' home directories to another (or multiple) file servers.

2.  Add RAM to the server so that some of the disk activity might be cached in RAM.

I still don't know how congested the local network is because I don't know how to log on to the managed(?) network switches (or how to use them, for that matter).  The outbound connection seemed OK during this experiment because we had zero packet loss while pinging yahoo.com for the duration.  I guess that also argues that the local network isn't too congested.

The other thing I noticed was that when counting net connections using netstat, it pauses frequently when it's about to display an ipv6 connection.  For example:

northshoreview:~ admin$ netstat

. . .

tcp6 0 0 fe80:4::20d:93ff.548 fe80:4::214:51ff.49229 ESTABLISHED

It doesn't pause with netstat -n, so I think we have a reverse DNS lookup problem.  I don't know much about our particular DNS setup, but I'll try to learn about it and see if I can figure out what's working and what isn't.  Poorly performing DNS setups can cause a lot of end-user slowness.

Here's the (sorry, really ugly) iostat output.

northshoreview:~ diradmin$ iostat 5
disk1 disk0 disk2 cpu

KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
northshoreview:~ diradmin$ iostat 5
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
17.63 10 0.17 17.62 10 0.17 10.42 1 0.01 5 6 89
11.26 94 1.03 11.06 93 1.00 4.39 14 0.06 20 41 39
11.83 93 1.08 11.84 97 1.12 6.22 14 0.09 17 42 42
8.50 90 0.74 9.09 81 0.72 8.86 6 0.05 18 38 43
10.43 81 0.82 10.95 75 0.80 13.17 15 0.19 18 37 46
8.96 54 0.47 9.23 61 0.55 0.00 0 0.00 20 40 40
10.02 66 0.65 10.12 67 0.66 0.00 0 0.00 15 32 53
10.79 66 0.69 10.60 66 0.69 4.44 4 0.02 19 37 44
12.29 88 1.05 11.32 94 1.04 22.24 6 0.13 13 30 57
11.15 51 0.55 10.75 50 0.53 5.01 13 0.07 20 40 40
17.58 127 2.18 17.76 126 2.18 17.00 4 0.07 16 33 51
11.32 168 1.86 11.90 168 1.95 5.40 13 0.07 15 33 51
14.47 167 2.36 15.36 162 2.44 8.92 10 0.09 15 32 53
11.66 178 2.02 11.86 173 2.00 5.10 6 0.03 14 29 58
12.72 184 2.28 12.35 185 2.23 70.86 1 0.10 15 31 54
12.41 112 1.36 12.88 116 1.46 12.44 5 0.07 13 41 45
10.11 87 0.86 10.03 91 0.89 5.30 14 0.07 19 38 43
9.93 80 0.77 9.56 82 0.76 22.62 5 0.11 16 39 45
8.56 86 0.71 8.89 85 0.74 13.85 16 0.22 18 45 37
9.57 67 0.63 10.56 66 0.68 10.87 9 0.10 20 46 34
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
9.73 80 0.76 9.29 87 0.79 12.00 2 0.03 13 25 62
12.51 113 1.38 12.93 112 1.42 8.52 11 0.09 21 34 44
11.21 75 0.82 10.57 77 0.80 11.72 12 0.13 16 41 43
10.94 80 0.86 11.09 80 0.87 22.55 7 0.15 22 45 34
11.24 83 0.91 11.48 86 0.96 15.21 12 0.18 23 52 25
11.37 93 1.03 11.17 90 0.98 5.14 1 0.01 20 42 39
11.93 80 0.93 12.90 77 0.97 0.00 0 0.00 25 50 25
13.20 147 1.89 13.33 144 1.87 0.00 0 0.00 18 38 44
9.90 107 1.04 9.83 110 1.06 0.00 0 0.00 22 43 35
11.68 89 1.02 11.39 92 1.02 0.00 0 0.00 23 42 35
10.72 86 0.90 11.20 81 0.89 0.00 0 0.00 21 47 32
11.13 85 0.93 10.35 88 0.89 0.00 0 0.00 27 47 27
13.46 64 0.84 12.28 66 0.79 0.00 0 0.00 23 41 36
14.98 133 1.94 14.98 134 1.96 0.00 0 0.00 20 40 40
8.93 89 0.78 8.88 82 0.71 0.00 0 0.00 22 37 41
14.57 71 1.00 14.88 73 1.07 0.00 0 0.00 24 48 28
11.28 82 0.90 9.99 90 0.87 0.00 0 0.00 27 52 20
10.77 82 0.86 10.72 90 0.95 0.00 0 0.00 20 48 32
12.31 88 1.06 12.82 84 1.06 0.00 0 0.00 26 58 16
19.72 113 2.17 19.67 113 2.17 0.00 0 0.00 27 47 26
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
22.28 138 3.00 22.58 135 2.99 0.00 0 0.00 27 50 23
12.59 87 1.07 11.81 92 1.06 0.00 0 0.00 24 52 23
8.86 80 0.69 9.21 73 0.66 0.00 0 0.00 12 24 64
11.81 65 0.74 11.35 65 0.72 8.00 1 0.00 26 51 23
14.91 74 1.08 15.63 69 1.06 0.00 0 0.00 17 37 45
11.21 71 0.77 10.56 73 0.75 0.00 0 0.00 27 49 24
36.60 131 4.67 35.93 134 4.70 0.00 0 0.00 17 30 53
13.47 69 0.91 13.96 64 0.88 0.00 0 0.00 21 42 37
11.87 42 0.48 11.08 48 0.51 0.00 0 0.00 17 39 44
20.21 44 0.87 19.74 48 0.93 0.00 0 0.00 24 44 32
27.93 52 1.42 26.83 55 1.45 0.00 0 0.00 24 43 34
19.50 57 1.09 18.33 63 1.12 0.00 0 0.00 19 44 37
8.63 60 0.51 8.59 61 0.51 0.00 0 0.00 21 44 35
40.79 129 5.12 40.08 132 5.15 10.12 1 0.01 19 40 41
10.75 46 0.49 11.36 46 0.51 0.00 0 0.00 25 50 25
9.21 44 0.40 8.99 43 0.38 0.00 0 0.00 21 36 43
15.88 58 0.90 16.22 55 0.88 0.00 0 0.00 17 36 47
8.02 42 0.33 8.25 42 0.34 0.00 0 0.00 15 42 43
12.74 48 0.60 12.96 48 0.60 0.00 0 0.00 24 42 35
28.19 164 4.51 28.42 162 4.50 0.00 0 0.00 14 30 57
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
10.81 39 0.42 11.00 37 0.40 0.00 0 0.00 25 45 30
11.79 43 0.49 11.80 43 0.49 0.00 0 0.00 21 41 38
18.34 44 0.78 17.42 48 0.81 0.00 0 0.00 25 61 14
50.56 29 1.41 51.30 28 1.41 0.00 0 0.00 27 53 20
9.75 51 0.48 9.82 48 0.46 0.00 0 0.00 19 41 40
33.34 83 2.70 34.94 79 2.70 0.00 0 0.00 16 32 51
22.46 107 2.34 22.41 105 2.29 0.00 0 0.00 16 36 48
10.14 43 0.42 10.01 46 0.45 0.00 0 0.00 23 44 33
12.37 49 0.59 11.97 51 0.59 0.00 0 0.00 27 46 26
18.26 41 0.73 19.18 38 0.70 0.00 0 0.00 25 48 27
22.49 39 0.87 22.95 39 0.87 0.00 0 0.00 30 46 24
26.50 43 1.11 27.07 41 1.09 0.00 0 0.00 34 56 10
42.18 114 4.71 41.53 117 4.73 0.00 0 0.00 24 48 29
50.73 26 1.27 48.87 27 1.29 0.00 0 0.00 33 57 10
19.74 47 0.90 19.68 48 0.91 0.00 0 0.00 30 58 12
45.39 33 1.47 40.46 37 1.46 0.00 0 0.00 34 57 9
21.48 61 1.27 21.54 62 1.29 8.00 1 0.00 29 55 16
16.88 43 0.72 15.98 48 0.76 0.00 0 0.00 29 55 17
29.38 179 5.13 29.19 181 5.15 0.00 0 0.00 18 44 38
11.45 75 0.84 11.44 78 0.87 0.00 0 0.00 16 34 50
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
11.94 60 0.70 12.02 60 0.71 0.00 0 0.00 24 49 27
13.55 85 1.12 13.66 84 1.11 0.00 0 0.00 19 41 40
23.75 99 2.30 23.73 99 2.30 0.00 0 0.00 24 41 34
14.14 93 1.28 14.15 93 1.28 0.00 0 0.00 18 37 45
25.56 99 2.48 27.16 91 2.42 0.00 0 0.00 25 41 34
15.42 154 2.32 15.27 159 2.37 0.00 0 0.00 22 30 48
18.73 82 1.51 18.14 86 1.52 0.00 0 0.00 30 36 34
38.96 96 3.66 38.12 99 3.67 0.00 0 0.00 23 43 34
13.43 92 1.20 13.43 86 1.13 0.00 0 0.00 30 50 20
11.72 75 0.86 12.35 70 0.85 0.00 0 0.00 24 50 26
12.81 65 0.81 12.89 64 0.81 0.00 0 0.00 24 49 26
11.89 93 1.07 12.00 94 1.10 0.00 0 0.00 26 44 30
13.69 65 0.87 13.70 63 0.85 0.00 0 0.00 31 49 20
12.55 65 0.79 12.52 66 0.80 0.00 0 0.00 26 55 19
13.23 46 0.59 13.36 46 0.61 0.00 0 0.00 33 46 21
12.24 20 0.24 12.04 21 0.24 7.09 3 0.02 28 54 18
11.36 31 0.34 11.67 28 0.32 0.00 0 0.00 33 48 20
11.43 68 0.76 11.28 70 0.77 11.72 2 0.02 31 48 21
12.78 29 0.36 13.49 27 0.35 0.00 0 0.00 26 54 20
9.81 41 0.40 9.63 44 0.41 0.00 0 0.00 21 41 38
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
9.53 52 0.48 9.49 53 0.49 0.00 0 0.00 20 31 49
13.46 46 0.61 13.80 44 0.59 0.00 0 0.00 25 53 22
13.84 43 0.58 13.56 45 0.59 0.00 0 0.00 30 50 20
10.51 61 0.63 10.69 59 0.61 0.00 0 0.00 31 52 17
10.33 59 0.59 10.79 58 0.61 0.00 0 0.00 31 54 16
10.56 56 0.58 10.33 59 0.59 0.00 0 0.00 27 49 24
12.45 27 0.33 11.81 32 0.37 0.00 0 0.00 32 55 12
6.84 9 0.06 7.51 8 0.06 0.00 0 0.00 31 54 15
13.10 35 0.45 13.48 36 0.47 0.00 0 0.00 30 59 11
11.16 65 0.70 10.94 65 0.70 0.00 0 0.00 25 49 26
10.81 90 0.95 10.93 89 0.95 0.00 0 0.00 20 43 37
8.95 53 0.47 8.87 59 0.51 0.00 0 0.00 26 40 35
9.55 64 0.59 9.54 57 0.53 0.00 0 0.00 22 48 30
11.58 72 0.81 11.52 71 0.80 0.00 0 0.00 23 37 40
11.85 60 0.69 11.96 59 0.69 0.00 0 0.00 20 37 44
9.74 53 0.51 9.95 48 0.46 0.00 0 0.00 17 40 43
9.88 86 0.83 9.89 84 0.81 0.00 0 0.00 16 39 45
12.62 64 0.79 12.83 63 0.79 0.00 0 0.00 20 45 35
11.53 63 0.71 11.70 62 0.70 0.00 0 0.00 20 37 43
11.78 68 0.79 11.79 68 0.79 0.00 0 0.00 29 41 31

Managing Software Update Server

I should also mention that using Workgroup Manager, I set the "Software Update Server To Use" preference to be managed Always, and to use "http://northshoreview.smfc.k12.ca.us:8088" on all of the groups Lab Computers, and Room 2 through Room 19.

Software Update packages were missing

It looks like we were trying to use our server to mirror software update packages so we didn't have to re-fetch them for each computer.

A problem I ran into when trying to do the software updates was that at least two of the packages, GarageBand 1.1 and J2SE 5 something wouldn't install.

I poked around on the server and found where the software update packages are mirrored (/usr/share/swupd/html, as it turns out), and deleted the two offending packages (which were of size 0).  Then, using Server Admin, told Software Update to check in with Apple, which resulted in a fresh fetch of those two packages.  Yay!  When I ran Software Update on one of the lab macs, now it was able to install the packages.  Yay!  Then, that mac was eligible for the next Java update, which was also size 0 and failed.  Boo!

So I did a find in the software update directory for all zero length packages and turned up 20 or more out of 206 that were apparently messed up.   I deleted all those and am currently refetching from Apple (just poked the "Check Now" button on Server Admin -> northshoreview -> Software Update), and it looks like we'll have a fresh set of updates shortly.  Most of those won't apply to most of the lab macs, at least (e.g., updates to FinalCutPro, DVDStudioPro, WebObjects Developer, etc.), but we still need to run software update on all of them to at least get Java current.  I'll dig on on that again Friday, probably.

Software Update, both Apple and Microsoft

The following applies to machines Ch - Cn, plus the back wall machines, BWa - BWd and imac03 (which is the last back wall machine).

Ran Software Update (multiple times with reboots, it turns out) to get the remaining macs up to date.  A few were still running Safari 3.0.4, in particular, which could have caused some stability issues.  Now they are all up to Safari 3.2.1.

Also ran Microsoft Autoupdate repeatedly to get through the Microsoft Office for Mac 11.5.3 update.

I found it helpful to leave machines with software update showing "no updates available" and with Word launched, showing the (latest) version.

The only problem I know of with those machines is that three of the machines, Cl, Cm and Cn, seem to have a bogus software update server or at least bogus software update packages for Garageband and J2SE (Java).  I checked one of them and Safari is at least up to date, so they should be fine for most purposes. 

Remaining work is to get the other machines (Ca - Cg and the left wall machines Wd, We, etc.) up to date on Microsoft software, and to verify that they are all set to auto-download updates, both microsoft and apple.

Software Updated 3 more lab macs

Software updated -Ch, Cj and Ck.  Missed Ci.

pbs.org seemed to be intermittently down, both from the lab and from Dennis' home system.  Traceroute showed that it was down after reaching pbs's gateway.

NSM-Lab-AdminConsole-2:~ admin$ traceroute www.pbs.org
traceroute to www.pbs.org (149.48.228.121), 64 hops max, 40 byte packets
1 10.25.0.1 (10.25.0.1) 1.857 ms 2.537 ms 0.891 ms
2 10.1.1.65 (10.1.1.65) 5.658 ms 1.096 ms 2.527 ms
3 10.3.0.254 (10.3.0.254) 33.146 ms 29.328 ms 24.455 ms
4 209.188.200.1 (209.188.200.1) 16.870 ms 2.119 ms 1.838 ms
5 209.188.131.193 (209.188.131.193) 5.052 ms 1.843 ms 1.979 ms
6 209.188.128.10 (209.188.128.10) 5.252 ms 4.222 ms 7.091 ms
7 209.188.128.30 (209.188.128.30) 2.754 ms 2.490 ms 2.112 ms
8 dc-svl-dc2--sanmateocoe-egm.cenic.net (137.164.32.182) 3.110 ms 3.252 ms 2.154 ms
9 dc-svl-isp--dc2-ge-1.cenic.net (137.164.22.61) 3.314 ms 5.053 ms 9.928 ms
10 63.145.224.13 (63.145.224.13) 3.601 ms 3.172 ms 2.878 ms
11 svl-core-01.inet.qwest.net (205.171.14.70) 4.324 ms 7.597 ms 3.187 ms
12 snj-core-02.inet.qwest.net (67.14.34.18) 4.404 ms 5.464 ms 3.916 ms
13 sjp-brdr-01.inet.qwest.net (205.171.233.30) 6.583 ms 3.847 ms 10.219 ms
14 0.so-5-2-0.br1.scl2.alter.net (204.255.169.177) 4.702 ms 4.864 ms 6.365 ms
15 0.so-0-1-0.xt1.scl2.alter.net (152.63.57.50) 4.467 ms 3.957 ms 4.779 ms
16 0.so-7-3-0.xt1.dca5.alter.net (152.63.0.169) 73.960 ms 73.781 ms 74.076 ms
17 pbs-gw.customer.alter.net (157.130.41.38) 76.221 ms 75.560 ms 76.921 ms
18 149.48.224.2 (149.48.224.2) 77.230 ms 76.432 ms 75.475 ms
19 * * *

Software Updated another 15 lab macs

Ran software update on the LAB-Wx macs and -Ca through -Cg.  Set them all to download updates in the background.

That leaves the back wall machines and -Ch through -Cn

Didn't run Microsoft Autoupdate yet.

Remote Desktop

Verified that both NS-LAB-ADMIN(2) and "System Admin Console" are running the latest Remote Desktop software version, 3.2.2.

From the main admin console, pushed out remote desktop client software version 3.2.2 to 19 of 74 computers.  (The rest all had some issue or other -- being offline, access denied, or not supported.)

The states are:

Current Remote Desktop Client, 33 Computers (mostly lab)

Access Denied, 2 computers

Not Supported, 4

Offline, 35 (mostly classroom computers)