[Gllug] More Xen problems: Unable to start xend

[- Tethys]

So, our newly built production server was due to go live this
afternoon. I was just doing some final preparations before the launch,
when I noticed that I couldn't get to one of the Xen guests. It gave
me an error claiming that the xenstored tdb was corrupt (sorry, I
don't have the original message any more).

After much scratching of heads, and wondering what on earth had
happened, I decided to try removing the tdb, and rebooting, in the
hopes that it would rebuild itself, and we could go from there. But
xend won't start. Even if I restore the old tdb, I still get the same
error:

[2007-01-17 14:34:55 xend 8033] ERROR (SrvDaemon:297) Exception
starting xend ((28, 'No space left on device, while writing
/local/domain/0/name : Domain-0'))

It's lying. There's plenty of space on every filesystem on this
machine. In fact, this same problem was logged almost a year ago at:

   http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=487

In comment #8, you can see why it's claiming there's no space left.
But no one at Xen has done anything about fixing it (either the
incorrect errno setting, or more importantly, the bug itself).

I'm more than a bit stumped here. Is there anything I can try? I've
even tried asking xensource about getting a support contract, but
they're not very responsive.

If it helps any, here's the stack trace from xend:

Traceback (most recent call last):
  File "//usr/lib64/python/xen/xend/server/SrvDaemon.py", line 291, in run
    servers = SrvServer.create()
  File "//usr/lib64/python/xen/xend/server/SrvServer.py", line 108, in create
    root.putChild('xend', SrvRoot())
  File "//usr/lib64/python/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "//usr/lib64/python/xen/web/SrvDir.py", line 82, in get
    val = val.getobj()
  File "//usr/lib64/python/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "//usr/lib64/python/xen/xend/server/SrvDomainDir.py", line 39,
in __init__
    self.xd = XendDomain.instance()
  File "//usr/lib64/python/xen/xend/XendDomain.py", line 655, in instance
    inst.init()
  File "//usr/lib64/python/xen/xend/XendDomain.py", line 78, in init
    True))
  File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 252, in recreate
    vm.storeDomDetails()
  File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 749, in
storeDomDetails
    self.writeDom(to_store)
  File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 645, in writeDom
    return xstransact.Write(self.dompath, *args)
  File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 301, in Write
    complete(path, lambda t: t.write(*args))
  File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 353,
in complete
    result = f(t)
  File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 301,
in <lambda>
    complete(path, lambda t: t.write(*args))
  File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 86, in write
    self._write(key, d[key])
  File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 74, in _write
    ('%s, while writing %s : %s' %
RuntimeError: (28, 'No space left on device, while writing
/local/domain/0/name : Domain-0')

Does anyone know of any tools for querying the xenstored tdb database,
and ideally rebuilding it? Any ideas *very* welcome.

Tet
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug