A while ago I was given the responsibility of server maintenance. Not just any servers, server clusters. We are using DRBD 8 and Heartbeat for our clusters. When it's working, it's a thing of beauty, setting it up however, can be challenging.
I have been working with one of our new clients who had wanted to do their own server setup but couldn't get the DRBD disk syncing to work. I had initially helped them get it to the point where it was syncronizing over eth0 (eth1 was preferred) and then a couple days later nothing was working.
What I found out was that DRBD had become split-brained at one point and now both nodes would default to Secondary and disconnected states. The trick was to get them back together again. Fortunately, this was a new client so there was no worry about losing any data.
Through the usual Google searches I discovered that I needed to do the following:
1. Stop DRBD on the secondary
# /etc/init.d/drbd stop
2. Restart DRBD on the primary
# /etc/init.d/drbd restart
3. Make sure the primary is primary. In this case it would start as primary and then switch to secondary when starting.
# drbdadm primary [resource]
4. Invalidate the data on the secondary. This will force it do re-sync with the primary.
# modprobe drbd -- we need the drbd module to be loaded
# drbdadm invalidate [resource]
# drbdadm adjust [resource]
At this point, /proc/drbd should give you a syncronization progress
[user@Primary ~]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:43:01
0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
ns:78160576 nr:0 dw:0 dr:78160576 al:0 bm:4770 lo:0 pe:0 ua:0 ap:0 oos:395131820
[==>.................] sync'ed: 16.6% (385870/462199)M
finish: 10:28:23 speed: 10,436 (10,232) K/sec
That's what it took for me to fix my situation. It's possible you might not need the exact same solution, but maybe it will help.
Tags
motorcycle (8) cars (4) work (11) web development (7) porsche (2) life (26) holidays (1) new year (1) photography (3) travel (3) london (1) ubuntu (12) linux (21) birthday (2) weather (1) bow island (2) parenthood (1) qos (1) firewall (1) css (3) fire (1) diy (3) laptop (2) chart (1) test (2) cooling (1) politics (2) harper (1) media (1) modx (1) development (18) pie (1) microsoft (3) sharepoint (1) wine (1) video editor (1) canada (1) abby (1) wii (1) parenting (3) shoelaces (1) christmas (1) pictures (2) unix (1) time (1) google (2) picasa (1) windows (2) joel (1) twitter (1) technology (1) deadline (1) triumph (3) tiger (1) holiday (1) apache (1) mysql (10) tip (1) mechanics (1) server (1) smartphone (1) android (7) vm (1) ted (1) music (1) sql (1) postgres (1) svn (1) vmware (1) house (2) camping (1) replication (2) mechanic (1) oracle (2) databases (1) debian (1) sqlserver (1) sql server (1) datapump (1) blog (1) C# (1) MVC (1)