Discussion:
[Lxc-users] [systemd-devel] Unable to run systemd in an LXC / cgroup container.
(too old to reply)
Michael H. Warfield
2012-10-26 13:32:17 UTC
Permalink
Adding in the lxc-devel list.
Hey Serge,
...
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.
Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?
http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.
Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html
This isn't working...
Based on what was in both of those articles, I added this entry to
another container (Plover) to test...
lxc.hook.mount = /var/lib/lxc/Plover/mount
[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file
I'm running the latest rc...
[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64
Is it something in git that hasn't made it to a release yet?
nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?
While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?
Note that I'm thinking that having lxc-start guess how to fill in /dev
is wrong, because different distros and even different releases of the
same distros have different expectations. For instance ubuntu lucid
wants /dev/shm to be a directory, while precise+ wants a symlink. So
somehow the template should get involved, be it by adding a hook, or
simply specifying a configuration file which lxc uses internally to
decide how to create /dev.
I agree this needs to be by some sort of convention or template that we
can adjust.
Personally I'd prefer if /dev were always populated by the templates,
and containers (i.e. userspace) didn't mount a fresh tmpfs for /dev.
But that does complicate userspace, and we've seen it in debian/ubuntu
as well (i.e. at certain package upgrades which rely on /dev being
cleared after a reboot).
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121026/a9c90afb/attachment.pgp>
Serge Hallyn
2012-10-26 14:07:28 UTC
Permalink
Post by Michael H. Warfield
Adding in the lxc-devel list.
Hey Serge,
...
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.
Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?
http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.
Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html
This isn't working...
Based on what was in both of those articles, I added this entry to
another container (Plover) to test...
lxc.hook.mount = /var/lib/lxc/Plover/mount
[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file
I'm running the latest rc...
[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64
Is it something in git that hasn't made it to a release yet?
nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?
While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?
Daniel has simply been too busy. St?phane has made a new branch which
cherrypicks 50 bugfixes for 0.8.0, with the remaining patches (about
twice as many) left for 0.9.0. I'm hoping we get 0.8.0 next week :)
Michael H. Warfield
2012-10-26 14:58:25 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?
While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?
Daniel has simply been too busy.
Don't I know THAT feeling all too well. Over on the Samba Team (where
I'm the chief security consultant on the team) we're all too busy with
juggling our domain and our web cert. On top of that, I've got my day
job (of course). On top of that, I've got about six other OpenSource
projects I'm juggling (including this one). On top of that, I've got a
consulting customer that's going through fits. And the beat goes on.

I'll test out things as fast as I can. I need this. This suddenly got
very interesting as soon as we had a thread to pick at on the systemd
ball of yarn.
Post by Serge Hallyn
St?phane has made a new branch which
cherrypicks 50 bugfixes for 0.8.0, with the remaining patches (about
twice as many) left for 0.9.0. I'm hoping we get 0.8.0 next week :)
I'm hoping the hook patches are in that cherry picked basket. We really
need them if that's what it takes to make this work. Looking forward to
it. :-)=)

I'm going to look further into this whole redirect /dev/console to a log
hang thing. That's not good and may need to be resolved soon as well.
I can live with losing the vty's although I disagree with St?phan's
arguments. They (systemd) are behaving significantly different from
sysvinit and upstart and they claim they want to be transparent? Not.
No matter. We need to make that work properly as well, agree with them
or disagree with them.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121026/fcdb0d89/attachment.pgp>
Michael H. Warfield
2012-10-27 16:45:24 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Adding in the lxc-devel list.
Hey Serge,
...
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.
Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?
http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.
Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html
This isn't working...
Based on what was in both of those articles, I added this entry to
another container (Plover) to test...
lxc.hook.mount = /var/lib/lxc/Plover/mount
[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file
I'm running the latest rc...
[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64
Is it something in git that hasn't made it to a release yet?
nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?
While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?
Daniel has simply been too busy. St?phane has made a new branch which
cherrypicks 50 bugfixes for 0.8.0, with the remaining patches (about
twice as many) left for 0.9.0. I'm hoping we get 0.8.0 next week :)
Trying to build latest from git. This is not good...

checking sys/apparmor.h usability... no
checking sys/apparmor.h presence... no
checking for sys/apparmor.h... no
configure: error: You must install the AppArmor development package in
order to compile lxc

What am I suppose to do on Fedora where we don't have that package? Is
it available in another repo somewhere? I'm looking and not finding.

Regards,
Mike
Post by Serge Hallyn
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121027/a5e23660/attachment.pgp>
Michael H. Warfield
2012-10-27 16:53:22 UTC
Permalink
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Adding in the lxc-devel list.
Hey Serge,
...
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.
Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?
http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.
Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html
This isn't working...
Based on what was in both of those articles, I added this entry to
another container (Plover) to test...
lxc.hook.mount = /var/lib/lxc/Plover/mount
[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file
I'm running the latest rc...
[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64
Is it something in git that hasn't made it to a release yet?
nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?
While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?
Daniel has simply been too busy. St?phane has made a new branch which
cherrypicks 50 bugfixes for 0.8.0, with the remaining patches (about
twice as many) left for 0.9.0. I'm hoping we get 0.8.0 next week :)
Trying to build latest from git. This is not good...
checking sys/apparmor.h usability... no
checking sys/apparmor.h presence... no
checking for sys/apparmor.h... no
configure: error: You must install the AppArmor development package in
order to compile lxc
What am I suppose to do on Fedora where we don't have that package? Is
it available in another repo somewhere? I'm looking and not finding.
nm... I see that --enable-apparmor is defaulted to on. I just had to
add an option to --disable-apparmor. Sorry for the noise.
Post by Michael H. Warfield
Regards,
Mike
Mike
Post by Michael H. Warfield
Post by Serge Hallyn
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
------------------------------------------------------------------------------
WINDOWS 8 is here.
Millions of people. Your app in 30 days.
Visit The Windows 8 Center at Sourceforge for all your go to resources.
http://windows8center.sourceforge.net/
join-generation-app-and-make-money-coding-fast/
_______________________________________________ Lxc-users mailing list Lxc-users at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121027/32a059dd/attachment.pgp>
Michael H. Warfield
2012-10-27 17:40:55 UTC
Permalink
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...

Testing the hook feature out using git rev (finally got it built)...

I added this line to my config...

lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount

In /var/lib/lxc/Plover/mount I have this:
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.

If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.

Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...

Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121027/01313e8b/attachment.pgp>
Michael H. Warfield
2012-10-27 17:51:48 UTC
Permalink
Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...
I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.

I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.
Post by Michael H. Warfield
Regards,
Mike
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121027/af8a8939/attachment.pgp>
Michael H. Warfield
2012-10-27 18:09:53 UTC
Permalink
Post by Michael H. Warfield
Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...
I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.
Crap. I've got a catch-22 here... This is going to take some work.

Yes, we can create the /dev directory with tmpfs from a template.
Problem is that /dev/pts does not exist at the time we need to mount the
devpts on /dev/pts for the pty's so that hurls chunks and dies. We
can't create the /dev/ directory contents prior to mounting in the
pre-mount hook because we won't have tmpfs in place at the time. We
have to get tmpfs mounted on /dev and then create /dev/pts and then
mount the ptys in there. There has to be a mkdir in between those two
mount actions. Simplest solution would seem to be to add some logic to
the mount logic that says "test if directory exists and, if not, create
it." I'm not sure of the consequences of that, though.

I don't see a way to make this happen with hooks. It's almost like we
need and on-mount per mount hook.
Post by Michael H. Warfield
Post by Michael H. Warfield
Regards,
Mike
Regards,
Mike
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121027/76dcfbec/attachment.pgp>
Serge Hallyn
2012-10-28 17:52:57 UTC
Permalink
Post by Michael H. Warfield
Post by Michael H. Warfield
Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...
I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.
Crap. I've got a catch-22 here... This is going to take some work.
Hey,

I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)
Post by Michael H. Warfield
Yes, we can create the /dev directory with tmpfs from a template.
Problem is that /dev/pts does not exist at the time we need to mount the
devpts on /dev/pts for the pty's so that hurls chunks and dies. We
can't create the /dev/ directory contents prior to mounting in the
pre-mount hook because we won't have tmpfs in place at the time. We
have to get tmpfs mounted on /dev and then create /dev/pts and then
mount the ptys in there. There has to be a mkdir in between those two
mount actions. Simplest solution would seem to be to add some logic to
the mount logic that says "test if directory exists and, if not, create
it." I'm not sure of the consequences of that, though.
I don't see a way to make this happen with hooks. It's almost like we
need and on-mount per mount hook.
Should be moot given my patch, which I intend to push this week, but why
couldn't a lxc.hook.mount do the whole thing, mount /dev and and populate
it? I wasn't thinking a lxc.hook.start, for the reasons you encountered,
but I assume you tried lxc.hook.mount and it failed?

Patch below:

Index: lxc-qp/src/lxc/conf.c
===================================================================
--- lxc-qp.orig/src/lxc/conf.c 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/conf.c 2012-10-28 05:44:07.871228322 -0500
@@ -619,7 +619,7 @@
}

if (mount(pty_info->name, lxcpath, "none", MS_BIND, 0)) {
- WARN("failed to mount '%s'->'%s'",
+ SYSERROR("failed to mount '%s'->'%s'",
pty_info->name, path);
continue;
}
@@ -636,7 +636,7 @@
}
} else {
if (mount(pty_info->name, path, "none", MS_BIND, 0)) {
- WARN("failed to mount '%s'->'%s'",
+ SYSERROR("failed to mount '%s'->'%s'",
pty_info->name, path);
continue;
}
@@ -842,6 +842,67 @@
return 0;
}

+struct lxc_devs {
+ char *name;
+ mode_t mode;
+ int maj;
+ int min;
+};
+
+struct lxc_devs lxc_devs[] = {
+ { "null", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 3 },
+ { "zero", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 5 },
+ { "full", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 7 },
+ { "urandom", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 9 },
+ { "random", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 8 },
+ { "tty", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 5, 0 },
+ { "console", S_IFCHR | S_IRUSR | S_IWUSR, 5, 1 },
+};
+
+/*
+ * Do we want to add options for max size of /dev and a file to
+ * specify which devices to create?
+ */
+static int setup_autodev(char *root)
+{
+ int ret;
+ struct lxc_devs *d;
+ char path[MAXPATHLEN];
+ int i;
+
+ INFO("Creating and populating /dev under %s\n", root);
+ ret = snprintf(path, MAXPATHLEN, "%s/dev", root);
+ if (ret < 0 || ret > MAXPATHLEN)
+ return -1;
+ ret = mount("none", path, "tmpfs", 0, "size=100000");
+ if (ret) {
+ SYSERROR("Failed to mount /dev at %s\n", root);
+ return -1;
+ }
+ for (i = 0; i < sizeof(lxc_devs) / sizeof(lxc_devs[0]); i++) {
+ d = &lxc_devs[i];
+ ret = snprintf(path, MAXPATHLEN, "%s/dev/%s", root, d->name);
+ if (ret < 0 || ret >= MAXPATHLEN)
+ return -1;
+ ret = mknod(path, d->mode, makedev(d->maj, d->min));
+ if (ret) {
+ SYSERROR("Error creating %s\n", d->name);
+ return -1;
+ }
+ }
+ ret = snprintf(path, MAXPATHLEN, "%s/dev/pts", root);
+ if (ret < 0 || ret >= MAXPATHLEN)
+ return -1;
+ ret = mkdir(path, S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH);
+ if (ret) {
+ SYSERROR("Failed to create /dev/pts in container");
+ return -1;
+ }
+
+ INFO("Populated /dev under %s\n", root);
+ return 0;
+}
+
static int setup_rootfs(const struct lxc_rootfs *rootfs)
{
if (!rootfs->path)
@@ -2208,6 +2269,13 @@
return -1;
}

+ if (lxc_conf->autodev) {
+ if (setup_autodev(lxc_conf->rootfs.mount)) {
+ ERROR("failed to set up /dev in the container");
+ return -1;
+ }
+ }
+
if (setup_mount(&lxc_conf->rootfs, lxc_conf->fstab, name)) {
ERROR("failed to setup the mounts for '%s'", name);
return -1;
Index: lxc-qp/src/lxc/conf.h
===================================================================
--- lxc-qp.orig/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
@@ -227,6 +227,7 @@
struct lxc_list hooks[NUM_LXC_HOOKS];
char *seccomp; // filename with the seccomp rules
int maincmd_fd;
+ int autodev; // if 1, mount and fill a /dev at start
};

int run_lxc_hooks(const char *name, char *hook, struct lxc_conf *conf);
Index: lxc-qp/src/lxc/confile.c
===================================================================
--- lxc-qp.orig/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
@@ -77,6 +77,7 @@
static int config_seccomp(const char *, char *, struct lxc_conf *);
static int config_includefile(const char *, char *, struct lxc_conf *);
static int config_network_nic(const char *, char *, struct lxc_conf *);
+static int config_autodev(const char *, char *, struct lxc_conf *);

typedef int (*config_cb)(const char *, char *, struct lxc_conf *);

@@ -118,6 +119,7 @@
{ "lxc.console", config_console },
{ "lxc.seccomp", config_seccomp },
{ "lxc.include", config_includefile },
+ { "lxc.autodev", config_autodev },
};

static const size_t config_size = sizeof(config)/sizeof(struct lxc_config_t);
@@ -853,6 +855,16 @@

return 0;
}
+
+static int config_autodev(const char *key, char *value,
+ struct lxc_conf *lxc_conf)
+{
+ int v = atoi(value);
+
+ lxc_conf->autodev = v;
+
+ return 0;
+}

static int config_aa_profile(const char *key, char *value,
struct lxc_conf *lxc_conf)
Michael H. Warfield
2012-10-28 18:06:52 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Michael H. Warfield
Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...
I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.
Crap. I've got a catch-22 here... This is going to take some work.
Hey,
I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)
Post by Michael H. Warfield
Yes, we can create the /dev directory with tmpfs from a template.
Problem is that /dev/pts does not exist at the time we need to mount the
devpts on /dev/pts for the pty's so that hurls chunks and dies. We
can't create the /dev/ directory contents prior to mounting in the
pre-mount hook because we won't have tmpfs in place at the time. We
have to get tmpfs mounted on /dev and then create /dev/pts and then
mount the ptys in there. There has to be a mkdir in between those two
mount actions. Simplest solution would seem to be to add some logic to
the mount logic that says "test if directory exists and, if not, create
it." I'm not sure of the consequences of that, though.
I don't see a way to make this happen with hooks. It's almost like we
need and on-mount per mount hook.
Should be moot given my patch, which I intend to push this week, but why
couldn't a lxc.hook.mount do the whole thing, mount /dev and and populate
it? I wasn't thinking a lxc.hook.start, for the reasons you encountered,
but I assume you tried lxc.hook.mount and it failed?
See my other comment about lxc.hook.mount. I tried using it to
populate /dev but it showed up in the parent of the mount undeneath the
tmpfs mount. It was like it ran pre-mount. I tried it for several
different combinations and couldn't get it to go. Would still have the
problem with mounting /dev/pts which would take place before the mount
hook at run to mount the file system and populate it. That actually
MIGHT work (gotta think on this now) if I used lxc.hook.pre-mount and
mounted tmpfs over /dev, and populated it but then I run into a problem
where I was using a bind-mount for the rootfs. Might still work. I'll
test your patch out first though.
<snip>

Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121028/d1006bd3/attachment.pgp>
Michael H. Warfield
2012-10-28 20:40:25 UTC
Permalink
Post by Serge Hallyn
Should be moot given my patch, which I intend to push this week, but why
couldn't a lxc.hook.mount do the whole thing, mount /dev and and populate
it? I wasn't thinking a lxc.hook.start, for the reasons you
encountered,
but I assume you tried lxc.hook.mount and it failed?
Patch failed against 0.8.0rc2 and git root. Even with loose patching
for whitespace, had failures...

This was against git:

[mhw at forest lxc]$ patch -p1 -l < ../lxc-autodev.patch
patching file src/lxc/conf.c
Hunk #1 succeeded at 616 (offset -3 lines).
Hunk #2 succeeded at 633 (offset -3 lines).
Hunk #3 succeeded at 839 (offset -3 lines).
Hunk #4 succeeded at 2203 (offset -66 lines).
patching file src/lxc/conf.h
Hunk #1 FAILED at 227.
1 out of 1 hunk FAILED -- saving rejects to file src/lxc/conf.h.rej
patching file src/lxc/confile.c
Hunk #1 FAILED at 77.
Hunk #2 FAILED at 118.
Hunk #3 succeeded at 854 with fuzz 2 (offset 1 line).
2 out of 3 hunks FAILED -- saving rejects to file src/lxc/confile

Version to patch to?

I'm trying to manually integrate those failed hunks now. Shouldn't be
too difficult for only three.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121028/b900e802/attachment.pgp>
Michael H. Warfield
2012-10-28 21:33:55 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Michael H. Warfield
Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...
I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.
Crap. I've got a catch-22 here... This is going to take some work.
Hey,
I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)
I am happy to report that, after manually editing my git head branch to
patch in the failed hunks, I was able to build it and test it and my
Fedora 17 systemd based container fired right up after adding the
lxc.autodev = 1 parameter to the config file. Yeah!!!!

I did run into one gotcha, but one I can live with. I had been bind
mounting the private root file system to another directory and then
using that as the rootfs like this:
--
lxc.rootfs = /srv/lxc/rootfs
lxc.mount.entry=/srv/lxc/private/Alcove /srv/lxc/rootfs none bind.shared 0 0
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0

lxc.autodev = 1
--
This did not work and I got the startup error that it can not mount
to /dev because it doesn't exist.

I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.

Drop that bind mount for the rootfs and this then worked like a charm:
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0

lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.

I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)

Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.

Regards,
Mike
Post by Serge Hallyn
Post by Michael H. Warfield
Yes, we can create the /dev directory with tmpfs from a template.
Problem is that /dev/pts does not exist at the time we need to mount the
devpts on /dev/pts for the pty's so that hurls chunks and dies. We
can't create the /dev/ directory contents prior to mounting in the
pre-mount hook because we won't have tmpfs in place at the time. We
have to get tmpfs mounted on /dev and then create /dev/pts and then
mount the ptys in there. There has to be a mkdir in between those two
mount actions. Simplest solution would seem to be to add some logic to
the mount logic that says "test if directory exists and, if not, create
it." I'm not sure of the consequences of that, though.
I don't see a way to make this happen with hooks. It's almost like we
need and on-mount per mount hook.
Should be moot given my patch, which I intend to push this week, but why
couldn't a lxc.hook.mount do the whole thing, mount /dev and and populate
it? I wasn't thinking a lxc.hook.start, for the reasons you encountered,
but I assume you tried lxc.hook.mount and it failed?
Index: lxc-qp/src/lxc/conf.c
===================================================================
--- lxc-qp.orig/src/lxc/conf.c 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/conf.c 2012-10-28 05:44:07.871228322 -0500
@@ -619,7 +619,7 @@
}
if (mount(pty_info->name, lxcpath, "none", MS_BIND, 0)) {
- WARN("failed to mount '%s'->'%s'",
+ SYSERROR("failed to mount '%s'->'%s'",
pty_info->name, path);
continue;
}
@@ -636,7 +636,7 @@
}
} else {
if (mount(pty_info->name, path, "none", MS_BIND, 0)) {
- WARN("failed to mount '%s'->'%s'",
+ SYSERROR("failed to mount '%s'->'%s'",
pty_info->name, path);
continue;
}
@@ -842,6 +842,67 @@
return 0;
}
+struct lxc_devs {
+ char *name;
+ mode_t mode;
+ int maj;
+ int min;
+};
+
+struct lxc_devs lxc_devs[] = {
+ { "null", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 3 },
+ { "zero", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 5 },
+ { "full", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 7 },
+ { "urandom", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 9 },
+ { "random", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 8 },
+ { "tty", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 5, 0 },
+ { "console", S_IFCHR | S_IRUSR | S_IWUSR, 5, 1 },
+};
+
+/*
+ * Do we want to add options for max size of /dev and a file to
+ * specify which devices to create?
+ */
+static int setup_autodev(char *root)
+{
+ int ret;
+ struct lxc_devs *d;
+ char path[MAXPATHLEN];
+ int i;
+
+ INFO("Creating and populating /dev under %s\n", root);
+ ret = snprintf(path, MAXPATHLEN, "%s/dev", root);
+ if (ret < 0 || ret > MAXPATHLEN)
+ return -1;
+ ret = mount("none", path, "tmpfs", 0, "size=100000");
+ if (ret) {
+ SYSERROR("Failed to mount /dev at %s\n", root);
+ return -1;
+ }
+ for (i = 0; i < sizeof(lxc_devs) / sizeof(lxc_devs[0]); i++) {
+ d = &lxc_devs[i];
+ ret = snprintf(path, MAXPATHLEN, "%s/dev/%s", root, d->name);
+ if (ret < 0 || ret >= MAXPATHLEN)
+ return -1;
+ ret = mknod(path, d->mode, makedev(d->maj, d->min));
+ if (ret) {
+ SYSERROR("Error creating %s\n", d->name);
+ return -1;
+ }
+ }
+ ret = snprintf(path, MAXPATHLEN, "%s/dev/pts", root);
+ if (ret < 0 || ret >= MAXPATHLEN)
+ return -1;
+ ret = mkdir(path, S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH);
+ if (ret) {
+ SYSERROR("Failed to create /dev/pts in container");
+ return -1;
+ }
+
+ INFO("Populated /dev under %s\n", root);
+ return 0;
+}
+
static int setup_rootfs(const struct lxc_rootfs *rootfs)
{
if (!rootfs->path)
@@ -2208,6 +2269,13 @@
return -1;
}
+ if (lxc_conf->autodev) {
+ if (setup_autodev(lxc_conf->rootfs.mount)) {
+ ERROR("failed to set up /dev in the container");
+ return -1;
+ }
+ }
+
if (setup_mount(&lxc_conf->rootfs, lxc_conf->fstab, name)) {
ERROR("failed to setup the mounts for '%s'", name);
return -1;
Index: lxc-qp/src/lxc/conf.h
===================================================================
--- lxc-qp.orig/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
@@ -227,6 +227,7 @@
struct lxc_list hooks[NUM_LXC_HOOKS];
char *seccomp; // filename with the seccomp rules
int maincmd_fd;
+ int autodev; // if 1, mount and fill a /dev at start
};
int run_lxc_hooks(const char *name, char *hook, struct lxc_conf *conf);
Index: lxc-qp/src/lxc/confile.c
===================================================================
--- lxc-qp.orig/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
@@ -77,6 +77,7 @@
static int config_seccomp(const char *, char *, struct lxc_conf *);
static int config_includefile(const char *, char *, struct lxc_conf *);
static int config_network_nic(const char *, char *, struct lxc_conf *);
+static int config_autodev(const char *, char *, struct lxc_conf *);
typedef int (*config_cb)(const char *, char *, struct lxc_conf *);
@@ -118,6 +119,7 @@
{ "lxc.console", config_console },
{ "lxc.seccomp", config_seccomp },
{ "lxc.include", config_includefile },
+ { "lxc.autodev", config_autodev },
};
static const size_t config_size = sizeof(config)/sizeof(struct lxc_config_t);
@@ -853,6 +855,16 @@
return 0;
}
+
+static int config_autodev(const char *key, char *value,
+ struct lxc_conf *lxc_conf)
+{
+ int v = atoi(value);
+
+ lxc_conf->autodev = v;
+
+ return 0;
+}
static int config_aa_profile(const char *key, char *value,
struct lxc_conf *lxc_conf)
------------------------------------------------------------------------------
WINDOWS 8 is here.
Millions of people. Your app in 30 days.
Visit The Windows 8 Center at Sourceforge for all your go to resources.
http://windows8center.sourceforge.net/
join-generation-app-and-make-money-coding-fast/
_______________________________________________
Lxc-devel mailing list
Lxc-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-devel
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121028/0e803b9a/attachment.pgp>
Serge Hallyn
2012-10-28 22:02:32 UTC
Permalink
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Michael H. Warfield
Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...
I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.
Crap. I've got a catch-22 here... This is going to take some work.
Hey,
I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)
I am happy to report that, after manually editing my git head branch to
Sorry, it was against the ubuntu quantal package. I've been in the air
without onboard wifi, so working with what i had at hand.
Post by Michael H. Warfield
patch in the failed hunks, I was able to build it and test it and my
Fedora 17 systemd based container fired right up after adding the
lxc.autodev = 1 parameter to the config file. Yeah!!!!
I did run into one gotcha, but one I can live with. I had been bind
mounting the private root file system to another directory and then
--
lxc.rootfs = /srv/lxc/rootfs
lxc.mount.entry=/srv/lxc/private/Alcove /srv/lxc/rootfs none bind.shared 0 0
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
This did not work and I got the startup error that it can not mount
to /dev because it doesn't exist.
Hm, yeah. If you do need to play a game like this, you might be best
off using a pre-mount hook for that.
Post by Michael H. Warfield
I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.
Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.

Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?
Michael H. Warfield
2012-10-29 01:35:50 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)
I am happy to report that, after manually editing my git head branch to
Sorry, it was against the ubuntu quantal package. I've been in the air
without onboard wifi, so working with what i had at hand.
Oh, I figured it was a package mismatch. Wasn't too terribly difficult
to patch those hunks in and kick out a diff against git.
Post by Serge Hallyn
Post by Michael H. Warfield
patch in the failed hunks, I was able to build it and test it and my
Fedora 17 systemd based container fired right up after adding the
lxc.autodev = 1 parameter to the config file. Yeah!!!!
I did run into one gotcha, but one I can live with. I had been bind
mounting the private root file system to another directory and then
--
lxc.rootfs = /srv/lxc/rootfs
lxc.mount.entry=/srv/lxc/private/Alcove /srv/lxc/rootfs none bind.shared 0 0
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
This did not work and I got the startup error that it can not mount
to /dev because it doesn't exist.
Hm, yeah. If you do need to play a game like this, you might be best
off using a pre-mount hook for that.
Yeah, I don't think I "need to play a game like this" anymore. I'd have
to go back through some old old E-Mails to see why I did that before. I
seem to recall we were playing with all sorts of bind mount options for
some PRIVATE thing or another. It may not be necessary at all any
longer. IAC, it's minor to switch it back. I seem to recall switching
back and forth using bind mounts several times back when that got done
that way.

I may play with the pre-mount hook just for giggles and see how that
might work as well. Any idea why I was experiencing the problem with
the mount hook when trying to populate /dev? I know it wouldn't have
worked because of the /dev/pts mount but I have more heartburn in that
it looks like it ran too early and the mount on /dev had not even taken
place at that time.
Post by Serge Hallyn
Post by Michael H. Warfield
I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.
Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.
Cool. Once again... Looks like we got some real progress here with
this one. I've still got more testing to do, undoing some of my changes
in the container itself and making sure it all still works.

Also looks like I can stop and restart one of these containers now
without the hung cgroup directory. I suspected it was some stray
devices behind that. This is good.
Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?
Yeah. I was taking it one step at a time. I wish they WOULD fire up
some getty's on those tty's since that basically makes lxc-console kinda
useless and the one on /dev/console is kinda useless in disconnected
mode with the console redirected to a file. Maybe we need some what for
lxc-console to be able to grab that? I'll have to look deeper at
systemd and figure out if it can be parameterized or conditionalized in
some way. ITMT, I probably will just turn them off.

Many thanks!

Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121028/d93e415b/attachment.pgp>
Serge Hallyn
2012-10-29 09:18:45 UTC
Permalink
Quoting Michael H. Warfield (mhw at WittsEnd.com):
...
Post by Michael H. Warfield
Yeah, I don't think I "need to play a game like this" anymore. I'd have
to go back through some old old E-Mails to see why I did that before. I
seem to recall we were playing with all sorts of bind mount options for
some PRIVATE thing or another. It may not be necessary at all any
longer. IAC, it's minor to switch it back. I seem to recall switching
back and forth using bind mounts several times back when that got done
that way.
I may play with the pre-mount hook just for giggles and see how that
might work as well. Any idea why I was experiencing the problem with
the mount hook when trying to populate /dev? I know it wouldn't have
The only idea I have is that perhaps your root is MS_SHARED by default?
Can you post the script you were using and the container config?
Post by Michael H. Warfield
worked because of the /dev/pts mount but I have more heartburn in that
it looks like it ran too early and the mount on /dev had not even taken
place at that time.
Post by Serge Hallyn
Post by Michael H. Warfield
I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.
Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.
Cool. Once again... Looks like we got some real progress here with
this one. I've still got more testing to do, undoing some of my changes
in the container itself and making sure it all still works.
Also looks like I can stop and restart one of these containers now
without the hung cgroup directory. I suspected it was some stray
devices behind that. This is good.
Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?
Yeah. I was taking it one step at a time. I wish they WOULD fire up
some getty's on those tty's since that basically makes lxc-console kinda
As I recall, you can specify gettys to start on any tty by creating a
magical symlink, something like
Post by Michael H. Warfield
useless and the one on /dev/console is kinda useless in disconnected
mode with the console redirected to a file. Maybe we need some what for
lxc-console to be able to grab that? I'll have to look deeper at
systemd and figure out if it can be parameterized or conditionalized in
some way. ITMT, I probably will just turn them off.
Many thanks!
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
Michael H. Warfield
2012-10-30 20:59:14 UTC
Permalink
Post by Serge Hallyn
...
Post by Michael H. Warfield
Yeah, I don't think I "need to play a game like this" anymore. I'd have
to go back through some old old E-Mails to see why I did that before. I
seem to recall we were playing with all sorts of bind mount options for
some PRIVATE thing or another. It may not be necessary at all any
longer. IAC, it's minor to switch it back. I seem to recall switching
back and forth using bind mounts several times back when that got done
that way.
I may play with the pre-mount hook just for giggles and see how that
might work as well. Any idea why I was experiencing the problem with
the mount hook when trying to populate /dev? I know it wouldn't have
The only idea I have is that perhaps your root is MS_SHARED by default?
Can you post the script you were using and the container config?
Another point on the curve... The documentation says that "pre-mount"
takes place before the mount occurs and "mount takes place after the
mount occurs. Only problem is that "pre-mount" is not being recognized.

lxc-start 1351627853.032 ERROR lxc_confile - unknown key lxc.hook.pre-mount

This is the same binaries from git that recognize lxc.hook.mount so I'm
assuming the doco and the code don't match at this point.

Even without my original bind mount, if I have a mount hook that does
something in a newly mounted tmpfs directory, it doesn't show up in that
directory it shows up in the parent directory as if it ran before the
mounts took place. I could put the mount in the hook.mount script and
then do it but it's seriously acting like the pre-mount hook isn't even
there (parameter unknown) and the mount hook is running before the
mounts are complete.

Simple exerts from some test scripts doing, really, nothing but testing
sequencing...

Ok... Lets try this. I won't post entire configs but... For machine
Alcove (my testbed)...
--
lxc.tty = 6
lxc.pts = 64

lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=none /srv/lxc/private/Alcove/dev.tmp tmpfs defaults 0 0
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0

lxc.autodev = 1

# lxc.hook.pre-mount = /var/lib/lxc/Alcove/pre-mount
lxc.hook.mount = /var/lib/lxc/Alcove/mount

lxc.mount.entry=shmfs /var/lib/lxc/private/Alcove/dev/shm tmpfs mode=0644 0 0
--

Now /var/lib/lxc/Alcove/mount:

--

#!/bin/sh -

touch /srv/lxc/private/Alcove/dev.tmp/mounted

--

In that directory on the host fs I have this:

[root at forest mhw]# touch /srv/lxc/private/Alcove/dev.tmp/no-mounted
[root at forest mhw]# ls /srv/lxc/private/Alcove/dev.tmp/
no-mounted

Now, when I start the container, the tmpfs should get mounted
on /dev.tmp in the container (relative to the container rootfs) and
should have the single file "mounted" in it while the parent file system
back on the host should have the single file "not-mounted" in it.

Let's see... lxc-start -n Alcove...

In the container...

[mhw at alcove ~]$ mount | grep tmpfs
none on /dev type tmpfs (rw,relatime,seclabel,size=100k)
none on /dev.tmp type tmpfs (rw,relatime,seclabel)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)
tmpfs on /media type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,mode=755)

Looks like the mount took place. I have tmpfs on /dev.tmp

[mhw at alcove ~]$ ls /dev.tmp/
[mhw at alcove ~]$

Opps... Where did the file end up?

Let's look on the host...

[mhw at forest ~]$ ls /srv/lxc/private/Alcove/dev.tmp/
mounted no-mounted

Arg... Wrong answer. It ended up in the parent file system before
tmpfs was mounted. But, the documentation says hook.mount runs after
the mounts have completed.

There's something wrong here or I am badly mistaken in my
understanding... (Probably the later, I'll admit.)

Regards,
Mike
Post by Serge Hallyn
Post by Michael H. Warfield
worked because of the /dev/pts mount but I have more heartburn in that
it looks like it ran too early and the mount on /dev had not even taken
place at that time.
Post by Serge Hallyn
Post by Michael H. Warfield
I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.
Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.
Cool. Once again... Looks like we got some real progress here with
this one. I've still got more testing to do, undoing some of my changes
in the container itself and making sure it all still works.
Also looks like I can stop and restart one of these containers now
without the hung cgroup directory. I suspected it was some stray
devices behind that. This is good.
Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?
Yeah. I was taking it one step at a time. I wish they WOULD fire up
some getty's on those tty's since that basically makes lxc-console kinda
As I recall, you can specify gettys to start on any tty by creating a
magical symlink, something like
Post by Michael H. Warfield
useless and the one on /dev/console is kinda useless in disconnected
mode with the console redirected to a file. Maybe we need some what for
lxc-console to be able to grab that? I'll have to look deeper at
systemd and figure out if it can be parameterized or conditionalized in
some way. ITMT, I probably will just turn them off.
Many thanks!
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121030/617a07e5/attachment.pgp>
Serge Hallyn
2012-10-31 17:15:39 UTC
Permalink
Can you tell me the exact git tree and branch you are using?

The results you're getting don't make sense to me... Hoping I can find
a simple answer.

-serge
Michael H. Warfield
2012-10-31 17:17:54 UTC
Permalink
Post by Serge Hallyn
Can you tell me the exact git tree and branch you are using?
I'm using head. I'm not specifying a tree.
Post by Serge Hallyn
The results you're getting don't make sense to me... Hoping I can find
a simple answer.
Me too.
Post by Serge Hallyn
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121031/9b34ef70/attachment.pgp>
Serge Hallyn
2012-10-31 17:30:46 UTC
Permalink
Post by Michael H. Warfield
Post by Serge Hallyn
Can you tell me the exact git tree and branch you are using?
I'm using head. I'm not specifying a tree.
?

I'm not sure what you mean - are you using git://github.com/lxc/lxc, or
the tree on lxc.sf.net?
Post by Michael H. Warfield
Post by Serge Hallyn
The results you're getting don't make sense to me... Hoping I can find
a simple answer.
Me too.
Post by Serge Hallyn
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
Michael H. Warfield
2012-10-31 17:33:43 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Can you tell me the exact git tree and branch you are using?
I'm using head. I'm not specifying a tree.
?
I'm not sure what you mean - are you using git://github.com/lxc/lxc, or
the tree on lxc.sf.net?
IOW, I'm not using a branch in the tree. I'm using the main trunk.

Created my tree with -

git clone git://github.com/lxc/lxc

So the former, not the later.
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
The results you're getting don't make sense to me... Hoping I can find
a simple answer.
Me too.
Post by Serge Hallyn
-serge
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121031/a24a4f72/attachment.pgp>
Serge Hallyn
2012-10-31 17:34:45 UTC
Permalink
Post by Michael H. Warfield
Post by Serge Hallyn
...
Post by Michael H. Warfield
Yeah, I don't think I "need to play a game like this" anymore. I'd have
to go back through some old old E-Mails to see why I did that before. I
seem to recall we were playing with all sorts of bind mount options for
some PRIVATE thing or another. It may not be necessary at all any
longer. IAC, it's minor to switch it back. I seem to recall switching
back and forth using bind mounts several times back when that got done
that way.
I may play with the pre-mount hook just for giggles and see how that
might work as well. Any idea why I was experiencing the problem with
the mount hook when trying to populate /dev? I know it wouldn't have
The only idea I have is that perhaps your root is MS_SHARED by default?
Can you post the script you were using and the container config?
Another point on the curve... The documentation says that "pre-mount"
takes place before the mount occurs and "mount takes place after the
mount occurs. Only problem is that "pre-mount" is not being recognized.
I have
serge at serge-ThinkPad-X130e:~$ cat /var/lib/lxc/premount
#!/bin/bash
echo 'hi there' > /tmp/hw
serge at serge-ThinkPad-X130e:~$ head -2 /var/lib/lxc/q4/config
lxc.hook.pre-mount = /var/lib/lxc/premount
lxc.network.type=veth
serge at serge-ThinkPad-X130e:~$ sudo lxc-start -n q4 -d
serge at serge-ThinkPad-X130e:~$ cat /tmp/hw
hi there

(This is using lxc 0.8.0~rc1-4ubuntu39 in ubuntu quantal)

-serge
Michael H. Warfield
2012-10-30 17:23:41 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.
Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.
Well... I'm not sure I understand what you mean by that.
The /dev/pts/* entries do does exist in the host system. But the bind
mount fails saying they do not exist. Not sure I understand what the
problem is here but I would like them connected even if systemd doesn't
start getty's on them. I've used them for other purposes in the past.
Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?
Actually, I've decided this is worthy of debugging and there may be
other ways to start a getty (or something else) on that tty. It really
should work.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121030/76cfb0a5/attachment.pgp>
Serge Hallyn
2012-10-30 18:35:45 UTC
Permalink
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.
Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.
Well... I'm not sure I understand what you mean by that.
The /dev/pts/* entries do does exist in the host system. But the bind
mount fails saying they do not exist. Not sure I understand what the
In Ubuntu, we use lxc.ttydir = lxc. That means the actual /dev/ttyN in the
container is a symlink to /dev/lxc/ttyN (to allow package upgrades which
insist on removing /dev/ttyN to succeed - they will fail if /dev/ttyN is
mounted over).

When /dev was pre-populated, /dev/ttyN existed. But when we are populating
it, it does not. So before we try tomount /dev/pts/NN from the host onto
/dev/ttyN in the container, we have to create a file to bind mount over.
I didn't put that in the patch. Yet.
Post by Michael H. Warfield
problem is here but I would like them connected even if systemd doesn't
start getty's on them. I've used them for other purposes in the past.
Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?
Actually, I've decided this is worthy of debugging and there may be
other ways to start a getty (or something else) on that tty. It really
should work.
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Michael H. Warfield
2012-10-30 18:39:48 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.
Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.
Well... I'm not sure I understand what you mean by that.
The /dev/pts/* entries do does exist in the host system. But the bind
mount fails saying they do not exist. Not sure I understand what the
In Ubuntu, we use lxc.ttydir = lxc. That means the actual /dev/ttyN in the
container is a symlink to /dev/lxc/ttyN (to allow package upgrades which
insist on removing /dev/ttyN to succeed - they will fail if /dev/ttyN is
mounted over).
When /dev was pre-populated, /dev/ttyN existed. But when we are populating
it, it does not. So before we try tomount /dev/pts/NN from the host onto
/dev/ttyN in the container, we have to create a file to bind mount over.
I didn't put that in the patch. Yet.
Got it! You and I both did the same thing. I'm thinking from the view
of Fedora and you're thinking from the view of Ubuntu. Got it. We'll
get this right yet. :-)=)
Post by Serge Hallyn
Post by Michael H. Warfield
problem is here but I would like them connected even if systemd doesn't
start getty's on them. I've used them for other purposes in the past.
Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?
Actually, I've decided this is worthy of debugging and there may be
other ways to start a getty (or something else) on that tty. It really
should work.
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121030/3132ab24/attachment.pgp>
Michael H. Warfield
2012-11-06 16:07:10 UTC
Permalink
Note that there are reports that LXC has issues with the fact that newer
systemd enables shared mount propagation for all mounts by default (this
should actually be beneficial for containers as this ensures that new
mounts appear in the containers). LXC when run on such a system fails as
soon as it tries to use pivot_root(), as that is incompatible with
shared mount propagation. The needs fixing in LXC: it should use MS_MOVE
or MS_BIND to place the new root dir in / instead. A short term
work-around is to simply remount the root tree to private before
invoking LXC.
In another thread, Serge had some heartburn over this shared mount
propagation which then rang a bell in my head about past problems we
have seen.
...
This was from another threat with the systemd guys.
Note that there are reports that LXC has issues with the fact that
newer
systemd enables shared mount propagation for all mounts by default
(this
should actually be beneficial for containers as this ensures that new
mounts appear in the containers). LXC when run on such a system fails
MS_SLAVE does this as well. MS_SHARED means container mounts also
propagate into the host, which is less desirable in most cases.
Here's where we've seen some problems in the past. It's not just mounts
that are propagated but remounts as well. The problem arose that some
of us had our containers on a separate partition. When we would shut a
container down, that container tried to remount its file systems ro
which then propagated back into the host causing the hosts file system
to be ro (doesn't happen if you are running on the host's root fs for
the containers) and from there across into the other containers.

Are you using MS_SHARED or MS_SLAVE for this? If you are using
MS_SHARED do you create a potential security problem where actions in
the container can bleed into the state of the host and into other
containers. That's highly undesirable. If a mount in a propagates back
into the host and is then reflected to another container sharing that
same mount tree (I have shared partitions specific to that sort of
thing) does that create an information disclosure situation of one
container mounts a new file system and the other container sees the new
mount? I don't know if the mount propagation would reflect back up the
shared tree or not but I have certainly seen remounts do this. I don't
see that as desirable. Maybe I'm misunderstand how this is suppose to
work but I intend to test out those scenarios when I have a chance. I
do know that, when testing that ro problem, I was able to remount a
partition ro in one container and it would switch in the host and the
other container and I could the remount it rw in the other container and
have it propagate back. Not good.

Can you offer any clarity on this?
as
soon as it tries to use pivot_root(), as that is incompatible with
shared mount propagation. The needs fixing in LXC: it should use
MS_MOVE
or MS_BIND to place the new root dir in / instead. A short term
Actually not quite sure how this would work. It should be possible
to set up a set of conditions to work around this, but the kernel
checks at do_pivotroot are pretty harsh - mnt->mnt_parent of both
the new root and current root have to be not shared. So perhaps
we actually first chroot into a dir whose parent is non-shared,
then pivot_root from there? :)
(Simple chroot in place of pivot_root still does not suffice, not
only because of chroot escapes, but also different results in
/proc/pid/mountinfo and friends)
Comments on Serge's points?

At this point, we see where this will become problematical in Fedora 18
but appears to already be problematical in NixOS that another user is
running and which containers systemd 195 in the host.

We've had problems with chroot in the past due to chroot escapes and
other problems years ago as Serge mentioned.
Lennart
--
Lennart Poettering - Red Hat, Inc.
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20121106/ad61fe80/attachment.pgp>
Continue reading on narkive:
Loading...