Open Directory Replica doesn’t connect to Open Directory Master in Leopard 10.5 server

leopardserver

Had a major issue with one of our Open Directory replicas.

It had always been somewhat problematic even though it was getting updates from the Open Directory master server. After a reboot, the server came back and the kerberos service was not running and I was unable to access Workgroup Manager users and unable to connect to the kerberos realm at all. This was only an issue for a specfic replica server. Another Open Directory replica server was fine. Unfortunately, this server was the one that acted as a password server to a major network service, so we were offline until I fixed it.

Typical fix for Open Directory replicas seems to be that when you have something happen, you should demote the Open Directory replica to being a standalone which should break the bad stuff happening. After you do this, you should then promote the server back into its Open Directory replica role. But, Apple’s Server Admin GUI seems to be pretty fickle on this. When you do this via Server Admin, unless there is a connection happening between the replica and master, it really doesn’t do the switch. Just like in the Active Directory world, when you want to demote or remove a server, you need to basically check it out of the network BUT you need a viable connection to do so. If you do not have that, you are SOL.

I was pretty sure that a nuke of the replica database was in order but the demotion through the Server Admin application and switch to Standalone was NOT happening. I even disabled the service entirely, but when I turned it back on through Server Admin, it still was unchanged. I have seen issues with Apple’s Server Admin as it relates to the status of the Open Directory settings from server to server. Server Admin often, from what I have seen, provides incorrect or incomplete information on what is happening with the Open Directory / LDAP server. In this emergency situation where I had a server that was dishing passwords to critical network services, something had to happen quickly.

Apple Support when you contact them, will try to walk you down a road that focuses on DNS issues. Each time I have talked with them they seem to really focus on hostname resolution issues. I suppose they get a lot of people calling them with issues after they screw something up with DNS or something but that was NOT my issue. Of course, you need to make sure your DNS / hostname resolution is happening and your network is viable. We were way past that. After a little round and round, we finally got to the main issue – something freaked-out with kerberos and we could either try to fix that or rebuild the replica. I opted for the latter and since Server Admin on the replica wasn’t helping me, we had to do something else. Here is what we did.

1. Quit Server Admin on the damaged replica server
2. Run an Archive backup of the Open Directory Master just in case through Server Admin on the Master Open Directory server and confirm backup just in case.
3. Open a terminal on the replica server
4. In the terminal on the messed-up replica server enter

$ sudo slapconfig -destroyldapserver

This does exactly what it says. Nothing nice about it. After a few seconds it nukes the LDAP server on the server.

5. Reboot the replica server.
6. After reboot, go into Server Admin on the replica and my Open Directory service was not active (grey icon rather than green) and was defaulting to ‘Standalone’ mode. This was clear that the destroy had worked.
7. From the Server Admin, go through the promotion to Open Directory Replica back to the Master again.
8. This put us back in business.

Again, your experience may differ but I opted to destroy the server replica since there was nothing of value and just rebuild it from scratch off the Master that was operating fine. Since another replica was doing fine as well, it was clear something was up with this specific server and not the overall architecture or permissions. Since Server Admin was not helping at all on that damaged replica, it was time, in my estimation to go ‘terminal’ with it.

After promotion back to replica, I got my password server back and users were authenticating again. This sort of thing may not work for you and don’t try it unless you are SOL like I was. Go through Apple Support and there might be an easier fix there for you.

New Life for Linksys WRT300N with DD-WRT

DD-WRT Logo

My Linksys WRT300N has been a good router for home but now that I needed a some more routing options, I really needed to decide what to do. I don’t need a full-blown router since I am not dishing a routable class C or anything, just a couple of IPs to for servers.

I have known about dd-wrt and openwrt for quite a while, but thought those only supported the ubiquitus Linksys WRT54G models. Linksys has really left the device for dead and not provided an updated firmware version since 2007. So, I gave DD-WRT another look and turns out someone had incorporated a build for the model I had (a Linksys WRT300N version 1.1).

Here is the link to the files needed. I was able to easily upgrade the firmware to the ‘mega’ build and I am off to the races doing some great stuff recycling the hardware that is more than capable of handling vlans, etc with the right feature-base in the firmware. This is a realy great way to give old Linksys hardware new life. If I didn’t go down this road, I would have had to plug in more hardware like ethernet switches and take up more power and make even more of a mess in my garage. Have to thank BrainSlayer and all the others who contribute to DD-WRT. I hope I can help now as well.

DISCLAIMER: You are totally on your own if you want to mess with non-Linksys firmware on your devices. If you hose or brick your router it is ***NOT*** DD-WRT.com’s fault (or mine for that matter). If you are great with the way your WRT300N or other Linksys WRT* router works and/or you are not comfortable with messing with the internals of these devices, then don’t.

Zabbix 1.6.1 on Ubuntu Hardy Heron 8.04

After working with other Open Source monitoring systems over the years, I decided to give Zabbix a try. Net-Saint then Nagios was always good, but getting it going initially and then adding the trending graphing was always a pain. Big Brother was another good one, but now that Quest Software owns it, the BTF version gets less and less appealing. Zenoss is another that I have used recently, but not super-impressed with the ability to create a nice services dashboard that I want to monitor a bunch of hosts and services on one screen. So, Zabbix, I thought, was worth a shot. Also, just getting the monitoring then having to do some hokey, non-documented grapher for Nagios was not going to be fun again so time to try something new.

Good news is the manual is pretty good. My recommendation is to go with Ubuntu for the host OS if possible. I was always a RedHat/Centos guy, but just not really into hunting for dependencies and RPMs anymore. Apt-get is just too great to handle and when you want to get something done quickly, it is just really nice.

Some of the following has been lifted from other places on the net but here goes what worked for me…

1. Hit the basics that need to be in-place. When you do your Heron install, you can hit them then or after with apt-get. Of course, you need;

Apache
PHP5
MySQL-Server

and a few other ones you probably don’t have

php5-gd
snmp
libsnmp9-dev   (this is important for NET-SNMP support for Zabbix at config time)
snmpd

You can issue this to get the ones that are not so standard:

$ sudo apt-get install php5-gd snmp libsnmp9-dev snmpd

(wait for a while)

2. Create zabbix user/group

$ sudo adduser zabbix

(it will walk you through the user info q&a) then when done and he user to the ‘admin’ group as well

$ sudo adduser zabbix admin

3. Create the mysql db for the backend (make sure mysqld is running prior of course). There is no password for the default root user for new mysqld installs, so no need to enter a password. You should change that and secure mysql of course later…

$ mysql -uroot

mysql> create database zabbix;
mysql> quit;

4. Download zabbix from their site. The version I am dealing with now is 1.6.1 http://www.zabbix.com/download.php and place it somewhere on the server and unzip/untar it so you have a directory with the release contents.You probably would be served best to place it into the home directory of the zabbix user (i.e – /home/zabbix/zabbix-1.6.1). By this time, I am just staying as root user throughout after getting sick of sudo.

5. Setting up the mysql db with the sql files from the release –

cd into the zabbix release directory (I will use /home/zabbix/zabbix-1.6.1 from now on) and run

$ cat create/schema/mysql.sql | mysql -uroot zabbix
$ cd create/data
$ cat data.sql | mysql -uroot zabbix
$ cat images_mysql.sql | mysql -uroot zabbix

That should get the db populated. To check mysql later and manage other mysql dbs with a nice GUI, phpmyadmin is what I would recommend. But onward!

6. While sitting in the release directory (/home/zabbix/zabbix-1.6.1) run the configure as root or sudo:

#./configure –enable-proxy –with-mysql –with-net-snmp –with-libcurl –enable-server –enable-agent

If this fails, stop and make sure you have everything installed from step #1. I missed a few things initially but if you have everything installed, it should go. Then

# make

# make install

7. Create config directory and copy the defaults over;

$ sudo mkdir /etc/zabbix
$ sudo chown -R zabbix.zabbix /etc/zabbix/
$ cp misc/conf/zabbix_* /etc/zabbix/

I would not mess with the configs just yet. I always over think settings so I tend to see if it can run then tweak after. These ran fine on Hardy Heron without modification initially.

8. Get the web interface files placed:

mkdir /home/zabbix/web
cp -R /home/zabbix/zabbix-1.6.1/frontends/php/* /home/zabbix/web

You should chown and chmod the directory for the initial config to go smoothly later:

$ sudo chown -R zabbix /home/zabbix/web
$ sudo chgrp -R <your webserver user> /home/zabbix/web
$ sudo chmod -R 775 /home/zabbix/web

and add the alias to the apache default site

$ sudo vi /etc/apache2/sites-enabled/000-default

When in vi, paste something like this prior to the termination of the virtualhost:

Alias /zabbix/ /home/zabbix/web/
<Directory /home/zabbix/web>
AllowOverride FileInfo AuthConfig Limit Indexes
Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
<Limit GET POST OPTIONS PROPFIND>
Order allow,deny
Allow from all
</Limit>
<LimitExcept GET POST OPTIONS PROPFIND>
Order deny,allow
Deny from all
</LimitExcept>
</Directory>

9. Start zabbix server and the agent:

$ sudo zabbix_server
$ sudo zabbix_agentd &

10. Check the php.ini and change these (the max_execution_time for me was 30 seconds initially)

max_execution_time = 300 ; Maximum execution time of each script, in seconds
date.timezone = <your timezone>

Timezone reference for php @

http://us3.php.net/manual/en/timezones.php

Mine is —->      date.timezone = America/Los_Angeles
11. Restart apache to kick-in the site changes alias and the php.ini changes if they were needed.

12. Go to http://<hostname / ip>/zabbix/

13. The initial login should be

Admin
zabbix

14. Walk through the install and because of what you did in step 8 above, you should be able to have it save the config to the directory. If not, you can save it and scp or sftp it up to the /home/zabbix/web/conf directory.

15. Add hosts and play with graphs now. You shoudl go add yourself as a separate user and change the main admin password, etc.

I am sure I missed something or could have done things in a better order, but this is what I was able to piece together from older version install guides on the web and what worked for me. There are addition setups that should be done so you HAVE to check the install guide, but this is the basic way to get operational at least on Ubuntu Hardy Heron.

Hope this helps,

Jonathan

UPDATE: For Ubuntu 8.10 I needed to also install

libcurl4-gnutls-dev
and I just installed all libmysql

$ sudo apt-get install libmysql*
$ sudo apt-get install libcurl4-gnutls-dev

Seagate Momentus ST96812AS = Nightmare

bunch of dead Seagate Momentus ST96812ASbunch of dead Seagate Momentus ST96812AS

Wow – these drives are frying themselves at my work like crazy. The Seagate Momentus 60gb ST96812AS are a complete disaster. We have lost 8 in the last 3 months not to mention the ones that were lost prior to my arrival. These drives were installed on the first generation MacBooks and when they crash, they crash hard.

Stay away from these drives.