CGROUPS(7)		   Linux Programmer's Manual		    CGROUPS(7)



NAME
       cgroups - Linux control groups

DESCRIPTION
       Control	cgroups,  usually  referred  to as cgroups, are a Linux kernel
       feature which allow processes to be organized into hierarchical	groups
       whose usage of various types of resources can then be limited and moni-
       tored.  The kernel's cgroup interface is	 provided  through  a  pseudo-
       filesystem called cgroupfs.  Grouping is implemented in the core cgroup
       kernel code, while resource tracking and limits are  implemented	 in  a
       set of per-resource-type subsystems (memory, CPU, and so on).

   Terminology
       A cgroup is a collection of processes that are bound to a set of limits
       or parameters defined via the cgroup filesystem.

       A subsystem is a kernel component that modifies	the  behavior  of  the
       processes  in a cgroup.	Various subsystems have been implemented, mak-
       ing it possible to do things such as limiting the amount	 of  CPU  time
       and memory available to a cgroup, accounting for the CPU time used by a
       cgroup, and freezing and resuming  execution  of	 the  processes	 in  a
       cgroup.	 Subsystems  are  sometimes also known as resource controllers
       (or simply, controllers).

       The cgroups for a controller are arranged in a hierarchy.  This hierar-
       chy  is	defined	 by  creating,	removing,  and renaming subdirectories
       within  the  cgroup  filesystem.	  At  each  level  of  the  hierarchy,
       attributes  (e.g.,  limits)  can	 be defined.  The limits, control, and
       accounting provided by cgroups generally	 have  effect  throughout  the
       subhierarchy  underneath	 the  cgroup where the attributes are defined.
       Thus, for example, the limits placed on a cgroup at a higher  level  in
       the hierarchy cannot be exceeded by descendant cgroups.

   Cgroups version 1 and version 2
       The  initial release of the cgroups implementation was in Linux 2.6.24.
       Over time, various cgroup controllers have been added to allow the man-
       agement	of  various  types  of resources.  However, the development of
       these controllers was largely uncoordinated, with the result that  many
       inconsistencies	arose between controllers and management of the cgroup
       hierarchies became rather complex.   (A	longer	description  of	 these
       problems	  can	be   found   in	 the  kernel  source  file  Documenta-
       tion/cgroup-v2.txt.)

       Because	of  the	 problems  with	 the  initial  cgroups	implementation
       (cgroups	 version  1),  starting	 in  Linux  3.10, work began on a new,
       orthogonal implementation to remedy these problems.   Initially	marked
       experimental,  and  hidden  behind  the -o __DEVEL__sane_behavior mount
       option, the new version (cgroups version 2) was eventually  made	 offi-
       cial  with  the release of Linux 4.5.  Differences between the two ver-
       sions are described in the text below.

       Although cgroups v2 is intended as a replacement for  cgroups  v1,  the
       older  system  continues	 to  exist  (and  for compatibility reasons is
       unlikely to be removed).	 Currently, cgroups v2 implements only a  sub-
       set  of	the  controllers available in cgroups v1.  The two systems are
       implemented so that both v1  controllers	 and  v2  controllers  can  be
       mounted	on  the same system.  Thus, for example, it is possible to use
       those controllers that are supported under version 2, while also	 using
       version	1  controllers where version 2 does not yet support those con-
       trollers.  The only restriction here is	that  a	 controller  can't  be
       simultaneously  employed	 in  both  a  cgroups  v1 hierarchy and in the
       cgroups v2 hierarchy.

   Cgroups version 1
       Under cgroups v1, each controller may be	 mounted  against  a  separate
       cgroup  filesystem  that	 provides its own hierarchical organization of
       the processes on the system.  It is also possible comount multiple  (or
       even  all)  cgroups  v1 controllers against the same cgroup filesystem,
       meaning that the comounted controllers  manage  the  same  hierarchical
       organization of processes.

       For  each  mounted  hierarchy,  the  directory tree mirrors the control
       group hierarchy.	 Each control group is	represented  by	 a  directory,
       with  each  of  its child control cgroups represented as a child direc-
       tory.   For  instance,  /user/joe/1.session  represents	control	 group
       1.session,  which  is a child of cgroup joe, which is a child of /user.
       Under each cgroup directory is a set of files  which  can  be  read  or
       written to, reflecting resource limits and a few general cgroup proper-
       ties.

       In addition, in cgroups v1, cgroups can be mounted with no  bound  con-
       troller,	 in  which  case they serve only to track processes.  (See the
       discussion of release notification below.)  An example of this  is  the
       name=systemd  cgroup  which is used by systemd(1) to track services and
       user sessions.

   Tasks (threads) versus processes
       In cgroups v1, a distinction is drawn between processes and tasks.   In
       this  view,  a  process	can  consist  of multiple tasks (more commonly
       called threads, from a user-space perspective, and called such  in  the
       remainder of this man page).  In cgroups v1, it is possible to indepen-
       dently manipulate the cgroup memberships of the threads in  a  process.
       Because	this  ability caused certain problems, the ability to indepen-
       dently manipulate the cgroup memberships of the threads	in  a  process
       has  been  removed  in  cgroups	v2.  Cgroups v2 allows manipulation of
       cgroup membership only for processes (which has the effect of  changing
       the cgroup membership of all threads in the process).

   Mounting v1 controllers
       The  use	 of  cgroups  requires	a  kernel built with the CONFIG_CGROUP
       option.	In addition, each of the v1 controllers has an associated con-
       figuration option that must be set in order to employ that controller.

       In  order  to  use a v1 controller, it must be mounted against a cgroup
       filesystem.  The usual place  for  such	mounts	is  under  a  tmpfs(5)
       filesystem  mounted  at	/sys/fs/cgroup.	 Thus, one might mount the cpu
       controller as follows:

	   mount -t cgroup -o cpu none /sys/fs/cgroup/cpu

       It is possible to comount multiple controllers against the same hierar-
       chy.   For  example, here the cpu and cpuacct controllers are comounted
       against a single hierarchy:

	   mount -t cgroup -o cpu,cpuacct none /sys/fs/cgroup/cpu,cpuacct

       Comounting controllers has the effect that a process  is	 in  the  same
       cgroup  for all of the comounted controllers.  Separately mounting con-
       trollers allows a process to be in  cgroup  /foo1  for  one  controller
       while being in /foo2/foo3 for another.

       It  is  possible to comount all v1 controllers against the same hierar-
       chy:

	   mount -t cgroup -o all cgroup /sys/fs/cgroup

       (One can achieve the same result by omitting -o all, since  it  is  the
       default if no controllers are explicitly specified.)

       It is not possible to mount the same controller against multiple cgroup
       hierarchies.  For example, it is not possible to mount both the cpu and
       cpuacct	controllers  against  one hierarchy, and to mount the cpu con-
       troller alone against another hierarchy.	 It is possible to create mul-
       tiple  mount points with exactly the same set of comounted controllers.
       However, in this case all that results is multiple mount points provid-
       ing a view of the same hierarchy.

       Note that on many systems, the v1 controllers are automatically mounted
       under /sys/fs/cgroup; in particular, systemd(1)	automatically  creates
       such mount points.

   Cgroups version 1 controllers
       Each  of the cgroups version 1 controllers is governed by a kernel con-
       figuration option (listed below).  Additionally,	 the  availability  of
       the cgroups feature is governed by the CONFIG_CGROUPS kernel configura-
       tion option.

       cpu (since Linux 2.6.24; CONFIG_CGROUP_SCHED)
	      Cgroups can be guaranteed a minimum number of "CPU shares"  when
	      a	 system	 is busy.  This does not limit a cgroup's CPU usage if
	      the CPUs are not busy.  For further information, see  Documenta-
	      tion/scheduler/sched-design-CFS.txt.

	      In Linux 3.2, this controller was extended to provide CPU "band-
	      width"  control.	 If  the  kernel  is  configured   with	  CON-
	      FIG_CFS_BANDWIDTH,  then	within each scheduling period (defined
	      via a file in the cgroup directory), it is possible to define an
	      upper  limit  on	the  CPU  time allocated to the processes in a
	      cgroup.  This upper limit applies even if there is no other com-
	      petition	for  the CPU.  Further information can be found in the
	      kernel source file Documentation/scheduler/sched-bwc.txt.

       cpuacct (since Linux 2.6.24; CONFIG_CGROUP_CPUACCT)
	      This provides accounting for CPU usage by groups of processes.

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/cpuacct.txt.

       cpuset (since Linux 2.6.24; CONFIG_CPUSETS)
	      This  cgroup  can be used to bind the processes in a cgroup to a
	      specified set of CPUs and NUMA nodes.

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/cpusets.txt.

       memory (since Linux 2.6.25; CONFIG_MEMCG)
	      The memory controller supports reporting and limiting of process
	      memory, kernel memory, and swap used by cgroups.

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/memory.txt.

       devices (since Linux 2.6.26; CONFIG_CGROUP_DEVICE)
	      This  supports  controlling  which  processes may create (mknod)
	      devices as well as open them for reading or writing.  The	 poli-
	      cies  may	 be specified as whitelists and blacklists.  Hierarchy
	      is enforced, so new rules must not violate  existing  rules  for
	      the target or ancestor cgroups.

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/devices.txt.

       freezer (since Linux 2.6.28; CONFIG_CGROUP_FREEZER)
	      The freezer cgroup can suspend and  restore  (resume)  all  pro-
	      cesses  in a cgroup.  Freezing a cgroup /A also causes its chil-
	      dren, for example, processes in /A/B, to be frozen.

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/freezer-subsystem.txt.

       net_cls (since Linux 2.6.29; CONFIG_CGROUP_NET_CLASSID)
	      This  places  a  classid,	 specified  for the cgroup, on network
	      packets created by a cgroup.  These classids can then be used in
	      firewall	rules,	as  well as used to shape traffic using tc(8).
	      This applies only to packets leaving the cgroup, not to  traffic
	      arriving at the cgroup.

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/net_cls.txt.

       blkio (since Linux 2.6.33; CONFIG_BLK_CGROUP)
	      The blkio cgroup controls and limits access to  specified	 block
	      devices  by  applying  IO	 control in the form of throttling and
	      upper limits against leaf nodes and intermediate	nodes  in  the
	      storage hierarchy.

	      Two  policies are available.  The first is a proportional-weight
	      time-based division of disk implemented with CFQ.	  This	is  in
	      effect  for  leaf	 nodes	using CFQ.  The second is a throttling
	      policy which specifies upper I/O rate limits on a device.

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/blkio-controller.txt.

       perf_event (since Linux 2.6.39; CONFIG_CGROUP_PERF)
	      This  controller	allows perf monitoring of the set of processes
	      grouped in a cgroup.

	      Further information can be  found	 in  the  kernel  source  file
	      tools/perf/Documentation/perf-record.txt.

       net_prio (since Linux 3.3; CONFIG_CGROUP_NET_PRIO)
	      This  allows  priorities to be specified, per network interface,
	      for cgroups.

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/net_prio.txt.

       hugetlb (since Linux 3.5; CONFIG_CGROUP_HUGETLB)
	      This supports limiting the use of huge pages by cgroups.

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/hugetlb.txt.

       pids (since Linux 4.3; CONFIG_CGROUP_PIDS)
	      This controller permits limiting the number of process that  may
	      be created in a cgroup (and its descendants).

	      Further information can be found in the kernel source file Docu-
	      mentation/cgroup-v1/pids.txt.

   Creating cgroups and moving processes
       A cgroup filesystem initially contains a single root cgroup, '/', which
       all  processes belong to.  A new cgroup is created by creating a direc-
       tory in the cgroup filesystem:

	   mkdir /sys/fs/cgroup/cpu/cg1

       This creates a new empty cgroup.

       A process may be moved to this cgroup  by  writing  its	PID  into  the
       cgroup's cgroup.procs file:

	   echo $$ > /sys/fs/cgroup/cpu/cg1/cgroup.procs

       Only one PID at a time should be written to this file.

       Writing	the  value 0 to a cgroup.procs file causes the writing process
       to be moved to the corresponding cgroup.

       When writing a PID into the cgroup.procs, all threads  in  the  process
       are moved into the new cgroup at once.

       Within  a  hierarchy,  a process can be a member of exactly one cgroup.
       Writing a process's PID to a cgroup.procs file automatically removes it
       from the cgroup of which it was previously a member.

       The  cgroup.procs  file	can  be read to obtain a list of the processes
       that are members of a cgroup.  The returned list of PIDs is not guaran-
       teed  to	 be  in order.	Nor is it guaranteed to be free of duplicates.
       (For example, a PID may be recycled while reading from the list.)

       In cgroups v1 (but not cgroups v2), an individual thread can  be	 moved
       to  another cgroup by writing its thread ID (i.e., the kernel thread ID
       returned by clone(2) and gettid(2)) to  the  tasks  file	 in  a	cgroup
       directory.   This  file can be read to discover the set of threads that
       are members of the cgroup.  This file  is  not  present	in  cgroup  v2
       directories.

   Removing cgroups
       To  remove a cgroup, it must first have no child cgroups and contain no
       (nonzombie) processes.  So long as that is the  case,  one  can	simply
       remove  the  corresponding  directory  pathname.	  Note that files in a
       cgroup directory cannot and need not be removed.

   Cgroups v1 release notification
       Two files can be used to determine whether the kernel provides  notifi-
       cations	when  a	 cgroup	 becomes  empty.  A cgroup is considered to be
       empty when it contains no child cgroups and no member processes.

       A special  file	in  the	 root  directory  of  each  cgroup  hierarchy,
       release_agent,  can  be used to register the pathname of a program that
       may be invoked when a cgroup in the hierarchy becomes empty.  The path-
       name  of the newly empty cgroup (relative to the cgroup mount point) is
       provided as the sole command-line argument when the release_agent  pro-
       gram  is	 invoked.   The	 release_agent program might remove the cgroup
       directory, or perhaps repopulate with a process.

       The default value of the release_agent file is empty, meaning  that  no
       release agent is invoked.

       Whether	or  not the release_agent program is invoked when a particular
       cgroup  becomes	 empty	 is   determined   by	the   value   in   the
       notify_on_release  file in the corresponding cgroup directory.  If this
       file contains the value	0,  then  the  release_agent  program  is  not
       invoked.	  If  it  contains  the	 value 1, the release_agent program is
       invoked.	 The default value for this file in the root cgroup is 0.   At
       the time when a new cgroup is created, the value in this file is inher-
       ited from the corresponding file in the parent cgroup.

   Cgroups version 2
       In cgroups v2, all mounted controllers reside in a single unified hier-
       archy.	While  (different)  controllers	 may be simultaneously mounted
       under the v1 and v2 hierarchies, it is not possible to mount  the  same
       controller simultaneously under both the v1 and the v2 hierarchies.

       The  new behaviors in cgroups v2 are summarized here, and in some cases
       elaborated in the following subsections.

       1. Cgroups v2 provides a	 unified  hierarchy  against  which  all  con-
	  trollers are mounted.

       2. "Internal"  processes	 are not permitted.  With the exception of the
	  root cgroup, processes may reside only in leaf nodes	(cgroups  that
	  do not themselves contain child cgroups).

       3. Active  cgroups  must	 be specified via the files cgroup.controllers
	  and cgroup.subtree_control.

       4. The	tasks	file   has   been   removed.	In    addition,	   the
	  cgroup.clone_children file that is employed by the cpuset controller
	  has been removed.

       5. An improved mechanism for notification of empty cgroups is  provided
	  by the cgroup.events file.

       For  more changes, see the Documentation/cgroup-v2.txt file in the ker-
       nel source.

   Cgroups v2 unified hierarchy
       In cgroups v1, the ability to mount different controllers against  dif-
       ferent hierarchies was intended to allow great flexibility for applica-
       tion design.  In practice, though, the flexibility turned out  to  less
       useful  than  expected, and in many cases added complexity.  Therefore,
       in cgroups v2, all available controllers are mounted against  a	single
       hierarchy.   The available controllers are automatically mounted, mean-
       ing that it is not necessary (or possible) to specify  the  controllers
       when mounting the cgroup v2 filesystem using a command such as the fol-
       lowing:

	   mount -t cgroup2 none /mnt/cgroup2

       A cgroup v2 controller is available only if it is not currently in  use
       via  a  mount against a cgroup v1 hierarchy.  Or, to put things another
       way, it is not possible to employ the same controller against both a v1
       hierarchy and the unified v2 hierarchy.

   Cgroups v2 "no internal processes" rule
       With  the  exception  of	 the root cgroup, processes may reside only in
       leaf nodes (cgroups that do  not	 themselves  contain  child  cgroups).
       This  avoids the need to decide how to partition resources between pro-
       cesses which are members of cgroup A and processes in child cgroups  of
       A.

       For  instance,  if cgroup /cg1/cg2 exists, then a process may reside in
       /cg1/cg2, but not in /cg1.  This is to avoid an ambiguity in cgroups v1
       with  respect  to the delegation of resources between processes in /cg1
       and its child cgroups.  The recommended approach in cgroups  v2	is  to
       create  a  subdirectory called leaf for any nonleaf cgroup which should
       contain processes, but no child cgroups.	 Thus, processes which	previ-
       ously  would have gone into /cg1 would now go into /cg1/leaf.  This has
       the advantage of making explicit the relationship between processes  in
       /cg1/leaf and /cg1's other children.

   Cgroups v2 subtree control
       When  a cgroup A/b is created, its cgroup.controllers file contains the
       list of controllers which were active in its parent, A.	 This  is  the
       list of controllers which are available to this cgroup.	No controllers
       are active until they are enabled  through  the	cgroup.subtree_control
       file,  by writing the list of space-delimited names of the controllers,
       each preceded by '+' (to enable) or '-' (to disable).  If  the  freezer
       controller is not enabled in /A/B, then it cannot be enabled in /A/B/C.

   Cgroups v2 cgroup.events file
       With  cgroups  v2,  a  new mechanism is provided to obtain notification
       about when a cgroup becomes empty.  The cgroups	v1  release_agent  and
       notify_on_release  files	 are removed, and replaced by a new, more gen-
       eral-purpose file, cgroup.events.  This file contains  key-value	 pairs
       (delimited  by  newline characters, with the key and value separated by
       spaces) that identify events or state for a  cgroup.   Currently,  only
       one  key appears in this file, populated, which has either the value 0,
       meaning that the cgroup (and its descendants)  contain  no  (nonzombie)
       processes, or 1, meaning that the cgroup contains member processes.

       The  cgroup.events file can be monitored, in order to receive notifica-
       tion when a cgroup transitions between the  populated  and  unpopulated
       states  (or  vice  versa).  When monitoring this file using inotify(7),
       transitions generate IN_MODIFY events, and  when	 monitoring  the  file
       using poll(2), transitions generate POLLPRI events.

       The  cgroups  v2 notify_on_release mechanism offers at least two advan-
       tages over the cgroups v1 release_agent mechanism.   First,  it	allows
       for  cheaper  notification, since a single process can monitor multiple
       cgroup.events files.  By contrast, the cgroups  v1  mechanism  requires
       the  creation of a process for each notification.  Second, notification
       can be delegated to a process that lives inside a container  associated
       with the newly empty cgroup.

   /proc files
       /proc/cgroups (since Linux 2.6.24)
	      This  file  contains  information about the controllers that are
	      compiled into the kernel.	 An example of the  contents  of  this
	      file (reformatted for readability) is the following:

		  #subsys_name	  hierarchy	 num_cgroups	enabled
		  cpuset	  4		 1		1
		  cpu		  8		 1		1
		  cpuacct	  8		 1		1
		  blkio		  6		 1		1
		  memory	  3		 1		1
		  devices	  10		 84		1
		  freezer	  7		 1		1
		  net_cls	  9		 1		1
		  perf_event	  5		 1		1
		  net_prio	  9		 1		1
		  hugetlb	  0		 1		0
		  pids		  2		 1		1

	      The fields in this file are, from left to right:

	      1. The name of the controller.

	      2. The  unique  ID  of  the  cgroup hierarchy on which this con-
		 troller is mounted.  If multiple cgroups v1  controllers  are
		 bound	to  the	 same  hierarchy, then each will show the same
		 hierarchy ID in this field.  The value in this field will  be
		 0 if:

		   a) the controller is not mounted on a cgroups v1 hierarchy;

		   b) the controller is bound to the cgroups v2 single unified
		      hierarchy; or

		   c) the controller is disabled (see below).

	      3. The number of control groups in  this	hierarchy  using  this
		 controller.

	      4. This  field  contains	the  value  1  if  this	 controller is
		 enabled, or 0 if it has been disabled (via the cgroup_disable
		 kernel command-line boot parameter).

       /proc/[pid]/cgroup (since Linux 2.6.24)
	      This file describes control groups to which the process with the
	      corresponding PID belongs.  The  displayed  information  differs
	      for cgroups version 1 and version 2 hierarchies.

	      For  each	 cgroup	 hierarchy  of	which the process is a member,
	      there is one entry containing three  colon-separated  fields  of
	      the form:

		   hierarchy-ID:controller-list:cgroup-path

	      For example:

		  5:cpuacct,cpu,cpuset:/daemons

	      The colon-separated fields are, from left to right:

	      1. For  cgroups  version	1  hierarchies,	 this field contains a
		 unique hierarchy ID number that can be matched to a hierarchy
		 ID  in	 /proc/cgroups.	  For the cgroups version 2 hierarchy,
		 this field contains the value 0.

	      2. For cgroups version 1	hierarchies,  this  field  contains  a
		 comma-separated  list of the controllers bound to the hierar-
		 chy.  For the cgroups version	2  hierarchy,  this  field  is
		 empty.

	      3. This  field contains the pathname of the control group in the
		 hierarchy to which the process	 belongs.   This  pathname  is
		 relative to the mount point of the hierarchy.

ERRORS
       The following errors can occur for mount(2):

       EBUSY  An attempt to mount a cgroup version 1 filesystem specified nei-
	      ther the name= option (to mount a named hierarchy)  nor  a  con-
	      troller name (or all).

NOTES
       A  child	 process created via fork(2) inherits its parent's cgroup mem-
       berships.   A  process's	 cgroup	 memberships  are   preserved	across
       execve(2).

SEE ALSO
       prlimit(1),  systemd(1),	 clone(2),  ioprio_set(2), perf_event_open(2),
       setrlimit(2), cgroup_namespaces(7), cpuset(7), namespaces(7), sched(7),
       user_namespaces(7)

COLOPHON
       This  page  is  part of release 4.10 of the Linux man-pages project.  A
       description of the project, information about reporting bugs,  and  the
       latest	  version     of     this    page,    can    be	   found    at
       https://www.kernel.org/doc/man-pages/.



Linux				  2016-12-12			    CGROUPS(7)