Docker and docker-compose allow you to deploy web apps quickly and easily. But for some reason, every time you build a project with docker-compose it eats into your hard drive memory, and you cannot get that space back very easily.
If you are iterating on a project and need to run docker-compose build regularly, then the amount of hard drive space used by docker just keeps going up and up, and never goes down. If it's a big project, it can add GBs every time you run that command. This is annoying, especially when you end up with error messages saying that there is no space left on your hard drive.
The most annoying part is that it is not easy to get that space back. There is no docker command to get the space back, you have to delete the files directly. But if you delete the files directly without taking precautions, you can really badly break docker on your machine.
This article will show you how to safely recover hard drive memory from Docker in Ubuntu 16.04.
Are You Serious?
Yes. That's right. Docker uses up hard drive memory and never gives it back, even if you remove the containers and images.
After a bit of investigation, I found the culprit. Docker stores images in /var/lib/dockers/aufs. But the files in here are not cleaned up properly by docker, and so they sit around forever. And these aren't small files, I was adding 1 GB every time I built an image of my web app.
The aufs folder is using 8.6GB of hard drive space. This is way too much.
And here is a shot of my system memory on my laptop. My laptop has a small 100GB SSD hard-drive so 8.6GB is a big chunk of that.
Maybe I'm using docker wrong, I'm sure there is a way to use docker without this happening. But it shouldn't happen anyway. Docker is hard enough to learn without this problem.
It's a known problem with docker. There is an issue on Docker's github, discussions in forums and a few guides online to try and fix the problem. But there is no clear answer, and the guides do not do the job. Docker commands don't fix this on their own, I've tried that.
Anyway, enough moaning, here's how to fix it.
Words Of Warning
First, a word of warning. When I first discovered this problem, I got frustrated and just deleted everything in the aufs folder. DO NOT DO THIS. Docker immediately broke. I then uninstalled and reinstalled docker - but it was still broken. Docker leaves some kind of reference file lying around, even after being uninstalled, that expects files to be in the aufs folder, and breaks if they are not. To fix that I had to spend hours working out how to fully uninstall docker, and then reinstall it.
Secondly, another word of warning. This fix will remove all docker containers and images from your machine. If you have data in those containers that you want to keep then you need to back that data up. (Hint: use Volumes to persist data).
How To Fix It
You have to use docker commands to completely remove all the docker containers and images, then stop docker, then safely delete the files in var/lib/docker/aufs, and then start docker back up again.
Phase 1 - Remove All Docker Images And Containers
Docker images and containers have a habit of sticking around. This is good in some circumstances, but we want to remove them all. Follow these steps to do so.
1. Access the shell as a superuser.
Open a terminal window, then:
2. Check how much space aufs is using.
du -hs /var/lib/docker/aufs
This is how much space you will recover after this fix.
Why oh why is it using this much space?
NOTE - If your aufs folder is not using up loads of space then you have a different problem, so you may as well stop this guide, it won't help you.
3. Stop all docker containers.
docker stop $(docker ps -a -q)
The part in brackets lists all the docker containers by id number. And the docker stop command stops them all. Don't worry about any error messages.
3. Remove all docker containers.
docker rm $(docker ps -a -q)
4. Removes all docker images.
docker rmi --force $(docker images -q)
The part in the brackets lists all the docker images by id number, and then docker rmi removes those images. The --force flag is used because docker doesn't like removing images, and will find any excuse not to do so.
5. Clean up a bit.
docker system prune --force
This removes any files lying around that are not needed anymore. In my opinion, this should remove files in aufs, and so solve our problem, but it doesn't. Are you reading docker team?
We've just removed all docker containers and images from the system, yet the docker aufs folder is still using 8.6GB. Why oh why?
6. Sanity check.
docker ps -a
This lists docker containers, it should show no containers.
This should show no images.
If either of these are showing any images or containers, then you might have missed something - go back and start again. Don't proceed to phase 2 until both these commands show nothing.
Phase 2 - Recover The Hard Drive Space
Now that there are no docker images and containers, we can safely recover the hard drive space. WARNING - do not do this without completing Phase 1.
1. Stop docker running.
systemctl stop docker
2. Clear the aufs directory.
rm -rf /var/lib/docker/aufs
This is the big one. This command completely removes all files in /var/lib/docker/aufs.
3. Clean up.
These clear out your package library. I'm not 100% sure why these help, but I've not tested this without them, and they are harmless commands.
4. Start docker again.
systemctl start docker
5. Check the aufs folder
du -hs /var/lib/docker/aufs
This should now show just a few kb of space being used
Ahhh, that's better.
6. Sanity check
To check docker is working, run one of your containers and it should work.
In my opinion, this is a serious bug in docker. The aufs folder just keeps growing and growing until it fills up your hard drive.
Maybe I'm using docker wrong, and maybe there is some way of running docker that doesn't cause this issue. But it shouldn't happen anyway.
It's hard enough to learn Docker, and then to have your comptuer freeze up because there is no free space is very annoying. And it is far from a quick fix. It took me about a full day to work out what the problem was and fix it.
Docker is really good technology, and I love it. And it's open-source, so kudos to the docker team. But this problem should be fixed.