how to use the cgroup feature of linux kernel to limit the i/o rate for writing in storage devices?

Control Groups (cgroups) are a powerful feature of the Linux kernel that allow you to limit, prioritize, and isolate resource usage (such as CPU, memory, I/O) of a set of processes. To limit the I/O rate for writing to storage devices, you can use cgroups with the `blkio` subsystem.

Here's a step-by-step guide to using cgroups to limit I/O bandwidth for write operations:

### Step 1: Check if cgroups are enabled

Make sure that your Linux kernel has cgroups support. You can check this by looking for the `cgroup` filesystem in `/proc/cgroups`:

```bash
cat /proc/cgroups
```

### Step 2: Create a cgroup

You will need to create a cgroup directory. This can be done by mounting the cgroup filesystem if it is not already mounted.

```bash
# Create a mount point for cgroups, if it doesn't exist
sudo mkdir /sys/fs/cgroup

# Mount the cgroup filesystem (this is usually done at boot time)
sudo mount -t cgroup none /sys/fs/cgroup
```

Then, create a directory for your cgroup:

```bash
# Create a directory for your I/O cgroup
sudo mkdir /sys/fs/cgroup/blkio/my_cgroup
```

### Step 3: Limit I/O with cgroups

To limit the I/O rate, you need to write to certain files in the `blkio` cgroup. For example, to limit the write throughput to a block device (e.g., `/dev/sda`), you can use `blkio.throttle.write_bps_device`.

```bash
# Limit write bandwidth to 1 MB/s on /dev/sda
echo "8:0 1048576" | sudo tee /sys/fs/cgroup/blkio/my_cgroup/blkio.throttle.write_bps_device
```

Here, `8:0` is the device major and minor number of `/dev/sda`. You can find these numbers using the `lsblk` command:

```bash
lsblk -o NAME,MAJ:MIN
```

### Step 4: Add processes to your cgroup

Now, you can add processes to this cgroup. Simply echo the process ID (PID) into the `cgroup.procs` file of your cgroup.

```bash
# Assume you want to limit the I/O of a process with PID 12345
echo 12345 | sudo tee /sys/fs/cgroup/blkio/my_cgroup/cgroup.procs
```

### Step 5: Verify the limits

You can check the current settings and limits in your cgroup by reading the appropriate files:

```bash
cat /sys/fs/cgroup/blkio/my_cgroup/blkio.throttle.write_bps_device
```

### Step 6: Clean up

If you want to remove the cgroup or stop limiting the processes, you can simply delete the cgroup directory (after ensuring no processes are using it):

```bash
sudo rmdir /sys/fs/cgroup/blkio/my_cgroup
```

### Additional Configuration

- You can also limit read bandwidth using `blkio.throttle.read_bps_device`.
- Consider using the `blkio.throttle.write_iops_device` if you want to limit the number of I/O operations per second instead of limiting the bandwidth.
- For persistent usage across reboots, you may want to add your cgroup settings to a startup script or systemd unit file.

Using cgroups can help manage resources and improve system stability by preventing resource hogging by certain processes. Experiment with different settings to find the right balance for your applications.