I recently was asked, if it is possible to tell the size of a container and, speaking of disk-space, what are the costs when running multiple instances of a container.
- Limit the amount of memory your container can use, as described in the official Docker documentation. Be mindful when configuring swap on your Docker hosts. Swap is slower and less performant than memory but can provide a buffer against running out of system memory.
- Docker container images are made from various different layers and if these layers are identical, they can be shared across multiple container images in an effort to save space on your system. The shared size amount will show how much of a particular container image is being shared with another container.
Many container customers building applications use common software packages (e.g. Operating systems, databases, and application components) that are publicly distributed as container images on Docker Hub. Has announced that the Hub service will begin limiting the rate at which images are pulled under their anonymous and free plans. These limits will progressively take. Managing Resources for Containers. When you specify a Pod, you can optionally specify how much of each resource a Container needs. The most common resources to specify are CPU and memory (RAM); there are others. When you specify the resource request for Containers in a Pod, the scheduler uses this information to decide which node to place the Pod on. When you specify a resource limit for a. To see all containers on the Docker host, including stopped containers, use docker ps -a. You may be surprised how many containers exist, especially on a development system! A stopped container's writable layers still take up disk space. To clean this up, you can use the docker container.
Let's take the IBM Domino server from my previous post as an example.
You can get the SIZE of a container with the following command:
# docker ps -as -f 'name=901FP9'
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE
5f37c4d6a826 eknori/domino:domino_9_0_1_FP_9 '/docker-entrypoint.s' 2 hours ago Exited (137) 6 seconds ago 901FP9 0 B (virtual 3.296 GB)
We get a SIZE of 0 B (virtual 3.296 GB) as a result. Virtual size? What is that?
Let me try and explain:
When starting a container, the image that the container is started from is mounted read-only. On top of that, a writable layer is mounted, in which any changes made to the container are written.
The read-only layers of an image can be shared between any container that is started from the same image, whereas the 'writable' layer is unique per container (because: you don't want changes made in container 'a' to appear in container 'b' )
Back to the docker ps -s output;
- The 'size' information shows the amount of data (on disk) that is used for the writable layer of each container
- The 'virtual size' is the amount of disk-space used for the read-only image data used by the container.
So, with a 0 B container size, it does not make any difference, if we start 1 or 100 containers.
Be aware that the size shown does not include all disk space used for a container. Things that are not included currently are;
- volumes used by the container
- disk space used for the container's configuration files (hostconfig.json, config.v2.json, hosts, hostname, resolv.conf) – although these files are small
- memory written to disk (if swapping is enabled)
- checkpoints (if you're using the experimental checkpoint/restore feature)
- disk space used for log-files (if you use the json-file logging driver) – which can be quite a bit if your container generates a lot of logs, and log-rotation (max-file / max-size logging options) is not configured
So, let's see what we have to add to the 0 B to get the overall size of our container.
We are using a volume 'domino_data' for our Domino server . To get some information about this volume (1) type
# docker volume inspect domino_data
[
{
'Name': 'domino_data',
'Driver': 'local',
'Mountpoint': '/var/lib/docker/volumes/domino_data/_data',
'Labels': {},
'Scope': 'local'
}
]
This gives us the physical location of that volume. Now we can get the size of the volume, summing up the size of all files in the volume.
# du -hs /var/lib/docker/volumes/domino_data/_data
1.1G /var/lib/docker/volumes/domino_data/_data
To get the size of the container configuration (2), we need to find the location for our container.
# ls /var/lib/docker/containers/
5f37c4d6a8267246bbaff668b3437f121b0fe375d8319364bf7eb10f50d72c69
Now we have the long Id for our CONTAINER ID. Next type
# du -hs 5f37c4d6a8267246bbaff668b3437f121b0fe375d8319364bf7eb10f50d72c69/
160K 5f37c4d6a8267246bbaff668b3437f121b0fe375d8319364bf7eb10f50d72c69/
So, with a 0 B container size, it does not make any difference, if we start 1 or 100 containers.
Be aware that the size shown does not include all disk space used for a container. Things that are not included currently are;
- volumes used by the container
- disk space used for the container's configuration files (hostconfig.json, config.v2.json, hosts, hostname, resolv.conf) – although these files are small
- memory written to disk (if swapping is enabled)
- checkpoints (if you're using the experimental checkpoint/restore feature)
- disk space used for log-files (if you use the json-file logging driver) – which can be quite a bit if your container generates a lot of logs, and log-rotation (max-file / max-size logging options) is not configured
So, let's see what we have to add to the 0 B to get the overall size of our container.
We are using a volume 'domino_data' for our Domino server . To get some information about this volume (1) type
# docker volume inspect domino_data
[
{
'Name': 'domino_data',
'Driver': 'local',
'Mountpoint': '/var/lib/docker/volumes/domino_data/_data',
'Labels': {},
'Scope': 'local'
}
]
This gives us the physical location of that volume. Now we can get the size of the volume, summing up the size of all files in the volume.
# du -hs /var/lib/docker/volumes/domino_data/_data
1.1G /var/lib/docker/volumes/domino_data/_data
To get the size of the container configuration (2), we need to find the location for our container.
# ls /var/lib/docker/containers/
5f37c4d6a8267246bbaff668b3437f121b0fe375d8319364bf7eb10f50d72c69
Now we have the long Id for our CONTAINER ID. Next type
# du -hs 5f37c4d6a8267246bbaff668b3437f121b0fe375d8319364bf7eb10f50d72c69/
160K 5f37c4d6a8267246bbaff668b3437f121b0fe375d8319364bf7eb10f50d72c69/
Now do the math yourself. x = (0B + 1.1GB + 160kB ) * n .
I leave it up to you to find out the other sizes ( 3 – 4 ) .
Sizes may vary and will change during runtime; but I assume that you got the idea. Important to know is that all containers that are using the same image in the FROM command in a Dockerfile share this (readonly) image, so there is only one copy of it on disk.
Containers have changed the face of the software industry in just a few short years. Perhaps you've gotten to the pointwhere you're now running Java in a container. That's great! Unfortunately, there are still some things to be aware ofwhen it comes to CPU and memory utilization for containerized Java applications, which I'll outline below.
This article assumes some familiarity with Java and containers in general. If you need more background, check out some (or all) of the references.
Heap Space
If there's one key statement regarding running Java in a container, it's the following:
DO NOT SET THE JVM HEAP SPACE MANUALLY FOR ANY JAVA PROCESS RUNNING IN A CONTAINER. INSTEAD, SET THE CONTAINER LIMITS.
Why?
First and foremost, setting container limits accomplishes the very basic objective of containers/cgroups to begin with… that is, to isolate resource usage of a collection of processes. When you allocate the heap space manually via JVM arguments for example, you disregard that entirely.
It allows easy tuning of container resource allocation. Need more memory? Bump the container limit. Need less? Downsize it. This opens the door for automated processes to scale containers accordingly (such as the k8s vertical pod autoscaler) without having to manually tweak JVM parameters.
If running in a container orchestration environment (such as Kubernetes), the container limits become extremely important for both node health and scheduling. The scheduler will use these limits to find an appropriate node to run a container, and ensure that equal load is spread across nodes. If you are setting memory usage via JVM arguments, this information is not available to the scheduler, and therefore the scheduler has no idea how to effectively spread load for your containers.
If you do not set container limits, and Java is running in a container without any JVM memory flags explicitly set, the JVM will automatically set the max heap to 25% of the RAM on the node it is running on. For example, if your container is running on a 64 GB node, your JVM process heap space can max out at 16 GB. If you're running 10 containers on a node (a common occurrence due to auto-scaling), then all of the sudden you're on the hook for 160 GB of RAM. It's a disaster waiting to happen.1
What can you do about it?
Set the container memory (and CPU) limits.2 It is not sufficient to rely on resource requests (soft limits) only. Requests are great for helping out the scheduler, but setting hard limits allows Docker (or whatever container runtime you're using) to allocate the specified resources to the container itself, and no more. This will also allow Java (which is 'container aware' by default as of Java 8u191) to properly allocate the memory based on the resource limits placed on the container itself, instead of the node it is running on.
Docker Container List
Regarding the [Min|Max|Initial]RAMPercentage Parameters
In a fairly recent version of Java, the following JVM parameters were introduced (and back-ported to Java8u191).
-XX:MinRAMPercentage-XX:MaxRAMPercentage-XX:InitialRAMPercentage
I won't go into detail about how these exactly work,3 but the key takeaway is that they can be used to fine tune the JVM heap size without setting the heap size directly. That is, the container can still rely on the limits imposed on it.
Docker Container Space Limit Formula
So what are the correct values to use? The answer is - 'it depends'… particularly on the limits imposed on the container.
By default, the JVM heap gets 25% of the container's memory. You can tune the initial/min/max heap parameters to change this… e.g. setting -XX:MaxRAMPercentage=50 will allow the JVM to consume 50% of the container memory for the heap, instead of the default 25%. When this is safe depends largely on how much memory the container has to work with, and what processes are running in the container.
For example, if your container is running a single Java process, has 4 GB of RAM allocated to it, and you set -XX:MaxRAMPercentage=50, the JVM heap will get 2 GB. This is opposed to the 1 GB default it would normally get. In this case, 50% is almost certainly perfectly safe, and perhaps optimal, since much of the available RAM was probably under-utilized. However, imagine the same container has only 512 MB of RAM allocated to it. Now setting -XX:MaxRAMPercentage=50 gives the heap 256 MB of RAM and only leaves the other 256 MB to rest of the entire container. This memory would need to be shared by all other processes running in the container, plus the JVM Metaspace/PermGen, etc. allocations. Perhaps 50% is not so safe in this case.
Therefore, I can offer the following recommendations:
Don't touch these parameters unless you want to squeeze some extra memory headroom for your Java process. In most cases, the 25% default is a safe approach to managing memory. It might not be the most memory efficient, but RAM is cheap and it's always better to err on the side of caution, versus having your JVM process OOM-killed for some unknown reason.
If you do want to tweak these, be conservative. 50% is most likely a safe value where you can expect to avoid issues (in most cases), however this is still is largely dependent on how big (memory-wise) your container is. I don't recommend going to 75% unless your container has at least 512 MB of RAM (preferably 1 GB), and you have a good understanding of the memory usage of the application in question.
If your container is running multiple processes in addition to Java, be extra cautious when tweaking these values.The container memory is shared across all of these processes, and understanding total container memory usage in these cases is more complicated.
Anything over 90% is probably asking for trouble.
What about Metaspace/PermGen/etc?
That's beyond the scope of this article but rest assured that this can also be tweaked, but likely shouldn't. The default JVM behavior be fine for majority of use cases. If you find yourself trying to tackle an obscure memory issue, then it might be time to look into fiddling with this somewhat esoteric area of JVM memory, but otherwise I would avoid doing anything directly.
And CPU?
There's not much to do here. As of Java 8u191, the JVM is pretty 'container aware' by default and interprets CPU share allocation correctly. There are details around this that are worth understanding though, so instead of me outlining it here, I will direct you to this excellent article explaining it all in detail.
Summary
Modern Java is well suited to run in a container environment, but there are some not-so-obvious details that everybody should know to ensure they're getting the best performance from their applications. I hope that the information presented here, combined with the excellent references, helps you accomplish this goal.
References
Note In the 64 GB / 16 GB JVM example, this does not mean that the JVM process automatically consumes 16 GB of RAM for the heap… just that it can potentially grow that large before being considered out of memory. Additionally, there is no downward pressure on the garbage collector. The heap can then easily grow to consume more memory than the container can support before the GC is triggered due to miscalculated max heap space. This would inevitable cause application issues (e.g. OOM errors) or worse (e.g. OOM-killed, crash). ↩
For information on setting hard limits in Docker and Kubernetes, check out the following links: 1, 2↩
See this StackOverflow thread where this is discussed. ↩
