Docker for Mac support

[wamatt]

Not sure if this project is still active? Would be great though to see support for the new Docker for Mac native osx environment.

Maybe D4M will someday be fast enough for mounted volumes, but I suspect there's still have a ways to go.

https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076

[gansbrest]

I think with new version of Docker ( currently in beta? ) https://blog.docker.com/2016/06/docker-mac-windows-public-beta/, where there is no need for VM we should all start switching to docker-compose and hodor will become obsolete ( sadly? ).

[wamatt]

But that's what I'm talking about ;). The new Docker for Mac (Beta), is ironically actually much slower than using a VM. (the thread I linked to goes into more detail)

It could be a long time before we see native level speed. (could be up to a year). If it was an easy problem, they would have solved it by now (already on Beta 16 - and performance is still horrible 3 months later).

[gansbrest]

Ah, ok :) So I'm trying to think what could be done for Docker for Mac without VM to speed it up with Hodor.. Any ideas here? )

[EugenMayer]

@wamatt i am not any kind of biased, but i would tend to disagree at some particular level.

My experience is, that the overall performance / overhead of docker for mac with xhyve feels better, eventhough it seems to be common sense that it should be 10% slower in regards of overall performance. In addition, things like ports exposed on localhost of the host do ease up many things in development.

Talking about host shares, OSXFS of docker for mac is faster then vbox/fusion native shares, but still a lot slower then nfs ( about 2/3 slower ) and by far useless ( compared to rsync ). So the filesystem issue did not get addressed by docker for mac properly, as by any other solution yet ( i think i tried them all: rsync, nfs, unison, osxfs, fusion(native), vbox (native).

Currently, i build my own rsync solution and trying to look for a project to actually join efforts. Like hodor, most of them require docker-machine and a VM running, which is at least not what i have right now ( anymore ) - due to docker for mac.

I would be actually be interested in hodor with docker for mac - are there any chances or is it conceptually not what you try to do?

Projects like dinghy need that docker machine to have an intermediat, common FS to start "mounting on", while syncing happens from host to vm, not host to container. Is that what you are doing also?

[gansbrest]

@EugenMayer first of all thanks for your thoughts here.

First, let me step back a bit.. When I created hodor the original idea was to create same workflow for developers on Linux, Mac and Windows.

There was no native Docker for Mac or Windows at that point so the project became pretty useful.

Now with native Docker for Mac that works without VM I'm kind of torn here, thinking that we may need to start switching back to docker-compose and, while I don't agree with some commands there that impose dependencies ( like linking outside of the project ), it seems like that is the way to go in the future.

They also stated that performance issues with Docker for Mac is not highest priority at the moment but will be addressed eventually.

You expressed interest in "hodor with native docker for mac" but I'm not really sure how we could implement speedup here, since we don't have any VMs anymore.. Can you explain your optimization strategy so we could understand more deeply if it's even possible without modifying docker core?

Thanks

[EugenMayer]

@gansbrest Well the integration is doable - not theoretical but rather practical. I spawned https://github.com/EugenMayer/docker_sync

It has docker-for-mac support, but would also work with any docker-machine, does not matter.

I also mentions Hodor, but in docker-sync uses rsync instead of unison - while this should be pretty easy exchangeable. I also use fswatch of course.

You can, if you like, integrate docker-sync for the sync part in Hodor, keeping all the other tools.
You could monkey patch

https://github.com/EugenMayer/docker_sync/blob/master/lib/docker_sync/sync_process.rb#L42
and
https://github.com/EugenMayer/docker_sync/blob/master/lib/docker_sync/sync_process.rb#L82

To go for unison - but as you see, per sync-task a container. Just check what i am doing, no magic involved.

I used the term sync, not rsync, because if was thinking about at least adding unison - but i am biased towards unison because i do not like the idea of 2 way sync at all. There should be no sync from the container to the host - that does not make a lot of sense architectural wise.

Feel free to add a pull request for unison, i am happy to add it

[gansbrest]

I'll check those links a bit later.

Actually I specifically chose unison for 2 way sync :) One of the use cases that I remember was running diff commands like npm install or grunt / gulp that would generate some additional files. Some of those you may want to commit.

There was some other point of using 2 way sync but I just can remember now. Conceptually I think of it it as real volume on Linux, where you can modify stuff either on host or inside the container without loosing files.

[EugenMayer]

@gansbrest well, use cases are different i guess. I went forward since i was refactoring anyway:

https://github.com/EugenMayer/docker_sync/tree/master/lib/docker_sync/sync_strategy
created a stub here https://github.com/EugenMayer/docker_sync/blob/master/lib/docker_sync/sync_strategy/unison.rb

You can then select, per sync task, which strategy you want to use. For now rsync will be the default. So people can chose what kind of sync to use.

this might suite your needs, created an issue here EugenMayer/docker-sync#1 - feel free to drive this forward

[wamatt]

@EugenMayer @gansbrest

My interest in hodor was specifically for the Unison support. In a way I'm in the same boat as @gansbrest, in that I don't have a specific use case in mind.

That said however, it seems useful (at least in my mind) to aim for 2 way sync over the unidirectional of rsync, as it appears to increase the option space.

[EugenMayer]

It was way to easy to keep it open, i just implemented unison support, you can activate it like this https://github.com/EugenMayer/docker_sync/blob/master/test/docker-sync.yml#L29

So now you can chose either of those. Implementation was done in below 10 minutes actually.

Glad for feedback how to optimize unison for this usecase ( sync arguments / server optimizations? )