Gathered some data on server disk utilization

Melissa had a classroom full (ok, 24) of kids all launch Safari at once (as she would typically do when teaching the kids a new procedure).  I gathered iostats until all the kids were successfully at their destination web page.  I noticed that the number of transactions per second on disk1 and disk2 spikes in the 160+ range, then gradually dwindles down to the 50-ish range after a few minutes.  When it's in the 150+ range, kids definitely seem to be getting the beach ball of doom.  Also, for reference, the steady-state number of tps when the lab is idle (except for me) is in the 0 - 20 range.

This makes me think of two ways we might improve things.

1.  Move some or all of the kids' home directories to another (or multiple) file servers.

2.  Add RAM to the server so that some of the disk activity might be cached in RAM.

I still don't know how congested the local network is because I don't know how to log on to the managed(?) network switches (or how to use them, for that matter).  The outbound connection seemed OK during this experiment because we had zero packet loss while pinging yahoo.com for the duration.  I guess that also argues that the local network isn't too congested.

The other thing I noticed was that when counting net connections using netstat, it pauses frequently when it's about to display an ipv6 connection.  For example:

northshoreview:~ admin$ netstat

. . .

tcp6 0 0 fe80:4::20d:93ff.548 fe80:4::214:51ff.49229 ESTABLISHED

It doesn't pause with netstat -n, so I think we have a reverse DNS lookup problem.  I don't know much about our particular DNS setup, but I'll try to learn about it and see if I can figure out what's working and what isn't.  Poorly performing DNS setups can cause a lot of end-user slowness.

Here's the (sorry, really ugly) iostat output.

northshoreview:~ diradmin$ iostat 5
disk1 disk0 disk2 cpu

KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
northshoreview:~ diradmin$ iostat 5
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
17.63 10 0.17 17.62 10 0.17 10.42 1 0.01 5 6 89
11.26 94 1.03 11.06 93 1.00 4.39 14 0.06 20 41 39
11.83 93 1.08 11.84 97 1.12 6.22 14 0.09 17 42 42
8.50 90 0.74 9.09 81 0.72 8.86 6 0.05 18 38 43
10.43 81 0.82 10.95 75 0.80 13.17 15 0.19 18 37 46
8.96 54 0.47 9.23 61 0.55 0.00 0 0.00 20 40 40
10.02 66 0.65 10.12 67 0.66 0.00 0 0.00 15 32 53
10.79 66 0.69 10.60 66 0.69 4.44 4 0.02 19 37 44
12.29 88 1.05 11.32 94 1.04 22.24 6 0.13 13 30 57
11.15 51 0.55 10.75 50 0.53 5.01 13 0.07 20 40 40
17.58 127 2.18 17.76 126 2.18 17.00 4 0.07 16 33 51
11.32 168 1.86 11.90 168 1.95 5.40 13 0.07 15 33 51
14.47 167 2.36 15.36 162 2.44 8.92 10 0.09 15 32 53
11.66 178 2.02 11.86 173 2.00 5.10 6 0.03 14 29 58
12.72 184 2.28 12.35 185 2.23 70.86 1 0.10 15 31 54
12.41 112 1.36 12.88 116 1.46 12.44 5 0.07 13 41 45
10.11 87 0.86 10.03 91 0.89 5.30 14 0.07 19 38 43
9.93 80 0.77 9.56 82 0.76 22.62 5 0.11 16 39 45
8.56 86 0.71 8.89 85 0.74 13.85 16 0.22 18 45 37
9.57 67 0.63 10.56 66 0.68 10.87 9 0.10 20 46 34
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
9.73 80 0.76 9.29 87 0.79 12.00 2 0.03 13 25 62
12.51 113 1.38 12.93 112 1.42 8.52 11 0.09 21 34 44
11.21 75 0.82 10.57 77 0.80 11.72 12 0.13 16 41 43
10.94 80 0.86 11.09 80 0.87 22.55 7 0.15 22 45 34
11.24 83 0.91 11.48 86 0.96 15.21 12 0.18 23 52 25
11.37 93 1.03 11.17 90 0.98 5.14 1 0.01 20 42 39
11.93 80 0.93 12.90 77 0.97 0.00 0 0.00 25 50 25
13.20 147 1.89 13.33 144 1.87 0.00 0 0.00 18 38 44
9.90 107 1.04 9.83 110 1.06 0.00 0 0.00 22 43 35
11.68 89 1.02 11.39 92 1.02 0.00 0 0.00 23 42 35
10.72 86 0.90 11.20 81 0.89 0.00 0 0.00 21 47 32
11.13 85 0.93 10.35 88 0.89 0.00 0 0.00 27 47 27
13.46 64 0.84 12.28 66 0.79 0.00 0 0.00 23 41 36
14.98 133 1.94 14.98 134 1.96 0.00 0 0.00 20 40 40
8.93 89 0.78 8.88 82 0.71 0.00 0 0.00 22 37 41
14.57 71 1.00 14.88 73 1.07 0.00 0 0.00 24 48 28
11.28 82 0.90 9.99 90 0.87 0.00 0 0.00 27 52 20
10.77 82 0.86 10.72 90 0.95 0.00 0 0.00 20 48 32
12.31 88 1.06 12.82 84 1.06 0.00 0 0.00 26 58 16
19.72 113 2.17 19.67 113 2.17 0.00 0 0.00 27 47 26
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
22.28 138 3.00 22.58 135 2.99 0.00 0 0.00 27 50 23
12.59 87 1.07 11.81 92 1.06 0.00 0 0.00 24 52 23
8.86 80 0.69 9.21 73 0.66 0.00 0 0.00 12 24 64
11.81 65 0.74 11.35 65 0.72 8.00 1 0.00 26 51 23
14.91 74 1.08 15.63 69 1.06 0.00 0 0.00 17 37 45
11.21 71 0.77 10.56 73 0.75 0.00 0 0.00 27 49 24
36.60 131 4.67 35.93 134 4.70 0.00 0 0.00 17 30 53
13.47 69 0.91 13.96 64 0.88 0.00 0 0.00 21 42 37
11.87 42 0.48 11.08 48 0.51 0.00 0 0.00 17 39 44
20.21 44 0.87 19.74 48 0.93 0.00 0 0.00 24 44 32
27.93 52 1.42 26.83 55 1.45 0.00 0 0.00 24 43 34
19.50 57 1.09 18.33 63 1.12 0.00 0 0.00 19 44 37
8.63 60 0.51 8.59 61 0.51 0.00 0 0.00 21 44 35
40.79 129 5.12 40.08 132 5.15 10.12 1 0.01 19 40 41
10.75 46 0.49 11.36 46 0.51 0.00 0 0.00 25 50 25
9.21 44 0.40 8.99 43 0.38 0.00 0 0.00 21 36 43
15.88 58 0.90 16.22 55 0.88 0.00 0 0.00 17 36 47
8.02 42 0.33 8.25 42 0.34 0.00 0 0.00 15 42 43
12.74 48 0.60 12.96 48 0.60 0.00 0 0.00 24 42 35
28.19 164 4.51 28.42 162 4.50 0.00 0 0.00 14 30 57
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
10.81 39 0.42 11.00 37 0.40 0.00 0 0.00 25 45 30
11.79 43 0.49 11.80 43 0.49 0.00 0 0.00 21 41 38
18.34 44 0.78 17.42 48 0.81 0.00 0 0.00 25 61 14
50.56 29 1.41 51.30 28 1.41 0.00 0 0.00 27 53 20
9.75 51 0.48 9.82 48 0.46 0.00 0 0.00 19 41 40
33.34 83 2.70 34.94 79 2.70 0.00 0 0.00 16 32 51
22.46 107 2.34 22.41 105 2.29 0.00 0 0.00 16 36 48
10.14 43 0.42 10.01 46 0.45 0.00 0 0.00 23 44 33
12.37 49 0.59 11.97 51 0.59 0.00 0 0.00 27 46 26
18.26 41 0.73 19.18 38 0.70 0.00 0 0.00 25 48 27
22.49 39 0.87 22.95 39 0.87 0.00 0 0.00 30 46 24
26.50 43 1.11 27.07 41 1.09 0.00 0 0.00 34 56 10
42.18 114 4.71 41.53 117 4.73 0.00 0 0.00 24 48 29
50.73 26 1.27 48.87 27 1.29 0.00 0 0.00 33 57 10
19.74 47 0.90 19.68 48 0.91 0.00 0 0.00 30 58 12
45.39 33 1.47 40.46 37 1.46 0.00 0 0.00 34 57 9
21.48 61 1.27 21.54 62 1.29 8.00 1 0.00 29 55 16
16.88 43 0.72 15.98 48 0.76 0.00 0 0.00 29 55 17
29.38 179 5.13 29.19 181 5.15 0.00 0 0.00 18 44 38
11.45 75 0.84 11.44 78 0.87 0.00 0 0.00 16 34 50
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
11.94 60 0.70 12.02 60 0.71 0.00 0 0.00 24 49 27
13.55 85 1.12 13.66 84 1.11 0.00 0 0.00 19 41 40
23.75 99 2.30 23.73 99 2.30 0.00 0 0.00 24 41 34
14.14 93 1.28 14.15 93 1.28 0.00 0 0.00 18 37 45
25.56 99 2.48 27.16 91 2.42 0.00 0 0.00 25 41 34
15.42 154 2.32 15.27 159 2.37 0.00 0 0.00 22 30 48
18.73 82 1.51 18.14 86 1.52 0.00 0 0.00 30 36 34
38.96 96 3.66 38.12 99 3.67 0.00 0 0.00 23 43 34
13.43 92 1.20 13.43 86 1.13 0.00 0 0.00 30 50 20
11.72 75 0.86 12.35 70 0.85 0.00 0 0.00 24 50 26
12.81 65 0.81 12.89 64 0.81 0.00 0 0.00 24 49 26
11.89 93 1.07 12.00 94 1.10 0.00 0 0.00 26 44 30
13.69 65 0.87 13.70 63 0.85 0.00 0 0.00 31 49 20
12.55 65 0.79 12.52 66 0.80 0.00 0 0.00 26 55 19
13.23 46 0.59 13.36 46 0.61 0.00 0 0.00 33 46 21
12.24 20 0.24 12.04 21 0.24 7.09 3 0.02 28 54 18
11.36 31 0.34 11.67 28 0.32 0.00 0 0.00 33 48 20
11.43 68 0.76 11.28 70 0.77 11.72 2 0.02 31 48 21
12.78 29 0.36 13.49 27 0.35 0.00 0 0.00 26 54 20
9.81 41 0.40 9.63 44 0.41 0.00 0 0.00 21 41 38
disk1 disk0 disk2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id
9.53 52 0.48 9.49 53 0.49 0.00 0 0.00 20 31 49
13.46 46 0.61 13.80 44 0.59 0.00 0 0.00 25 53 22
13.84 43 0.58 13.56 45 0.59 0.00 0 0.00 30 50 20
10.51 61 0.63 10.69 59 0.61 0.00 0 0.00 31 52 17
10.33 59 0.59 10.79 58 0.61 0.00 0 0.00 31 54 16
10.56 56 0.58 10.33 59 0.59 0.00 0 0.00 27 49 24
12.45 27 0.33 11.81 32 0.37 0.00 0 0.00 32 55 12
6.84 9 0.06 7.51 8 0.06 0.00 0 0.00 31 54 15
13.10 35 0.45 13.48 36 0.47 0.00 0 0.00 30 59 11
11.16 65 0.70 10.94 65 0.70 0.00 0 0.00 25 49 26
10.81 90 0.95 10.93 89 0.95 0.00 0 0.00 20 43 37
8.95 53 0.47 8.87 59 0.51 0.00 0 0.00 26 40 35
9.55 64 0.59 9.54 57 0.53 0.00 0 0.00 22 48 30
11.58 72 0.81 11.52 71 0.80 0.00 0 0.00 23 37 40
11.85 60 0.69 11.96 59 0.69 0.00 0 0.00 20 37 44
9.74 53 0.51 9.95 48 0.46 0.00 0 0.00 17 40 43
9.88 86 0.83 9.89 84 0.81 0.00 0 0.00 16 39 45
12.62 64 0.79 12.83 63 0.79 0.00 0 0.00 20 45 35
11.53 63 0.71 11.70 62 0.70 0.00 0 0.00 20 37 43
11.78 68 0.79 11.79 68 0.79 0.00 0 0.00 29 41 31