Distributed profiling and energy measurements with perun#
How to locate performance issues on your distributed application, and fix them, in three steps:
Find the problematic/slow function in your code.
Gather statistics and data about the slow function.
Fix it!
If you want more information on perun, find any issues, or questions leaves us a message on github or check the documentation.
Installation#
Perun can be installed with pip:
pip install perun
Thourgh pip, optional dependencies can be installed that target different hardware accelerators, as well as the optional MPI support.
pip install perun[mpi,nvidia]
# or
pip install perun[mpi,rocm]
Here we have already installed perun with all necessary dependencies, please select the kernel ``perun_notebook``.
[ ]:
!perun --version
Requirement already satisfied: perun[mpi,nvidia] in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (0.9.0)
Requirement already satisfied: h5py>=3.5.9 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (3.13.0)
Requirement already satisfied: numpy>=1.20.0 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (2.2.5)
Requirement already satisfied: pandas>=1.3 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (2.2.3)
Requirement already satisfied: psutil>=5.9.0 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (7.0.0)
Requirement already satisfied: py-cpuinfo>=5.0.0 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (9.0.0)
Requirement already satisfied: tabulate>=0.9 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (0.9.0)
Requirement already satisfied: mpi4py>=3.1 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (4.0.3)
Collecting nvidia-ml-py>=12.535.77 (from perun[mpi,nvidia])
Using cached nvidia_ml_py-12.575.51-py3-none-any.whl.metadata (9.3 kB)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from pandas>=1.3->perun[mpi,nvidia]) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from pandas>=1.3->perun[mpi,nvidia]) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from pandas>=1.3->perun[mpi,nvidia]) (2025.2)
Requirement already satisfied: six>=1.5 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas>=1.3->perun[mpi,nvidia]) (1.17.0)
Using cached nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB)
Installing collected packages: nvidia-ml-py
Successfully installed nvidia-ml-py-12.575.51
[notice] A new release of pip is available: 25.0.1 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
perun 0.9.0
Basic command line usage#
Perun is primarily a command line tool. The complete functionality can be accessed through the perun command. On a terminal, simply type perun and click enter to get a help dialog with the available subcommands.
[ ]:
!perun
usage: perun [-h] [-c CONFIGURATION] [-l {DEBUG,INFO,WARN,ERROR,CRITICAL}]
[--log_file LOG_FILE] [--version]
{showconf,sensors,metadata,export,monitor} ...
Distributed performance and energy monitoring tool
positional arguments:
{showconf,sensors,metadata,export,monitor}
showconf Print perun configuration in INI format.
sensors Print available sensors by host and rank.
metadata Print available metadata.
export Export existing output file to another format.
monitor Gather power consumption from hardware devices while
SCRIPT [SCRIPT_ARGS] is running. SCRIPT is a path to
the python script to monitor, run with arguments
SCRIPT_ARGS.
options:
-h, --help show this help message and exit
-c CONFIGURATION, --configuration CONFIGURATION
Path to perun configuration file.
-l {DEBUG,INFO,WARN,ERROR,CRITICAL}, --log_lvl {DEBUG,INFO,WARN,ERROR,CRITICAL}
Logging level.
--log_file LOG_FILE Path to the log file. None by default. Writting to a
file disables logging in stdout.
--version show program's version number and exit
perun can already be used after this, without any further configuration or modification of the code. perun can monitor command line scripts, and other programs from the command lines. Try running the perun monitor -b sleep 10 on a terminal, or by running the cell below.
Note: The ipcluster should NOT be running when you execute the cell below, please stop it if it is still running.
[ ]:
%%bash
pwd
srun -n 4 perun monitor -b sleep 10
/home/juanpedroghm/code/heat/doc/source/tutorials/notebooks
[2025-05-20 16:59:39,969][perun.core][backends][ERROR] - R3/4:Unknown error loading dependecy NVMLBackend
[2025-05-20 16:59:39,969][perun.core][backends][ERROR] - R3/4:NVML Shared Library Not Found
[2025-05-20 16:59:39,969][perun.core][backends][ERROR] - R1/4:Unknown error loading dependecy NVMLBackend
[2025-05-20 16:59:39,970][perun.core][backends][ERROR] - R1/4:NVML Shared Library Not Found
[2025-05-20 16:59:39,970][perun.core][backends][ERROR] - R0/4:Unknown error loading dependecy NVMLBackend
[2025-05-20 16:59:39,970][perun.core][backends][ERROR] - R0/4:NVML Shared Library Not Found
[2025-05-20 16:59:39,976][perun.core][backends][ERROR] - R2/4:Unknown error loading dependecy NVMLBackend
[2025-05-20 16:59:39,976][perun.core][backends][ERROR] - R2/4:NVML Shared Library Not Found
In the directory reported by pwd, you should see a new directory called perun_results, (might be named bench_data if the current directory is the heat root directory ) with two files, sleep.hdf5 and sleep_.txt.
The file sleep_.txt contains a summary of what was measured on the run, with the average power draw of different hardware componets, memory usage, and the total energy. The available information depends on the available sensors that perun finds. You can see a list of the available sensors by running the sensors subcommand:
[ ]:
!perun sensors
[2025-05-20 16:55:39,740][perun.core][backends][ERROR] - R0/1:Unknown error loading dependecy NVMLBackend
[2025-05-20 16:55:39,740][perun.core][backends][ERROR] - R0/1:NVML Shared Library Not Found
| Sensor | Source | Device | Unit |
|-----------------:|--------------:|----------------:|-------:|
| cpu_0_package-0 | powercap_rapl | DeviceType.CPU | J |
| CPU_FREQ_0 | psutil | DeviceType.CPU | Hz |
| CPU_FREQ_1 | psutil | DeviceType.CPU | Hz |
| CPU_FREQ_2 | psutil | DeviceType.CPU | Hz |
| CPU_FREQ_3 | psutil | DeviceType.CPU | Hz |
| CPU_FREQ_4 | psutil | DeviceType.CPU | Hz |
| CPU_FREQ_5 | psutil | DeviceType.CPU | Hz |
| CPU_FREQ_6 | psutil | DeviceType.CPU | Hz |
| CPU_FREQ_7 | psutil | DeviceType.CPU | Hz |
| CPU_USAGE | psutil | DeviceType.CPU | % |
| DISK_READ_BYTES | psutil | DeviceType.DISK | B |
| DISK_WRITE_BYTES | psutil | DeviceType.DISK | B |
| NET_READ_BYTES | psutil | DeviceType.NET | B |
| NET_WRITE_BYTES | psutil | DeviceType.NET | B |
| RAM_USAGE | psutil | DeviceType.RAM | B |
The other file, sleep.hdf5, contains all the raw data that perun collects, that can be used for later processing. To get an interactive view of the data, navigate to myhdf5, and upload the file there.
This will let you explore the data tree that perun uses to store the hardware information. More info on the data tree can be found on the data documentation.
The data that is stored on the hdf5 file can be exported to other formats. Supported formats are text (same as text report), csv, json and bench. Run the cell below to export the last run of the sleep program to csv.
[ ]:
%%bash
perun export perun_results/sleep.hdf5 csv
cat perun_results/sleep_*.csv
,run id,hostname,device_group,sensor,unit,magnitude,timestep,value
0,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,0.0,2021.14599609375
1,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,1.0068829,964.1939697265625
2,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,2.0126529,400.12799072265625
3,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,3.0183434,2600.0
4,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,4.024712,2800.0
5,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,5.0291414,2384.971923828125
6,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,6.033699,1418.0760498046875
7,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,7.0397954,2297.81298828125
8,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,8.047083,2893.419921875
9,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,9.0511675,2456.3759765625
10,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,10.060614,1828.7459716796875
11,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,10.068606,3012.5791015625
12,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,0.0,1211.6190185546875
13,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,1.0068829,2700.0
14,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,2.0126529,1569.219970703125
15,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,3.0183434,2497.64697265625
16,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,4.024712,2693.7109375
17,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,5.0291414,2240.751953125
18,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,6.033699,3000.02099609375
19,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,7.0397954,2600.0
20,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,8.047083,3100.0
21,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,9.0511675,1806.197021484375
22,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,10.060614,3102.570068359375
23,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,10.068606,2934.219970703125
24,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,0.0,2200.10595703125
25,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,1.0068829,2700.096923828125
26,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,2.0126529,2842.551025390625
27,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,3.0183434,2488.455078125
28,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,4.024712,2651.922119140625
29,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,5.0291414,2183.43310546875
30,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,6.033699,2751.02490234375
31,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,7.0397954,2544.83203125
32,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,8.047083,3044.756103515625
33,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,9.0511675,2271.235107421875
34,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,10.060614,2385.8291015625
35,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,10.068606,3200.0
36,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,0.0,2200.012939453125
37,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,1.0068829,2700.0
38,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,2.0126529,1869.530029296875
39,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,3.0183434,2600.0
40,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,4.024712,2800.0
41,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,5.0291414,2315.37109375
42,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,6.033699,2672.827880859375
43,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,7.0397954,2600.0
44,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,8.047083,2464.04296875
45,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,9.0511675,2410.884033203125
46,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,10.060614,3060.60791015625
47,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,10.068606,2562.06201171875
48,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,0.0,2156.548095703125
49,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,1.0068829,2499.455078125
50,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,2.0126529,400.62200927734375
51,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,3.0183434,2080.2919921875
52,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,4.024712,2777.10693359375
53,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,5.0291414,1521.5909423828125
54,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,6.033699,2873.384033203125
55,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,7.0397954,2195.196044921875
56,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,8.047083,2817.139892578125
57,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,9.0511675,2418.926025390625
58,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,10.060614,2187.868896484375
59,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,10.068606,2655.29296875
60,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,0.0,2137.35791015625
61,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,1.0068829,2700.0
62,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,2.0126529,769.7069702148438
63,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,3.0183434,1988.4849853515625
64,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,4.024712,2471.529052734375
65,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,5.0291414,1931.303955078125
66,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,6.033699,2886.305908203125
67,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,7.0397954,2543.840087890625
68,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,8.047083,3100.0
69,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,9.0511675,2055.845947265625
70,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,10.060614,2340.925048828125
71,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,10.068606,2812.739990234375
72,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,0.0,2176.281005859375
73,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,1.0068829,1221.010986328125
74,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,2.0126529,1433.5810546875
75,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,3.0183434,2562.242919921875
76,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,4.024712,2591.029052734375
77,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,5.0291414,2437.9990234375
78,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,6.033699,3000.0
79,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,7.0397954,2550.77392578125
80,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,8.047083,3063.29296875
81,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,9.0511675,2261.791015625
82,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,10.060614,3050.388916015625
83,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,10.068606,3017.64892578125
84,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,0.0,2199.987060546875
85,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,1.0068829,2698.6279296875
86,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,2.0126529,1597.2509765625
87,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,3.0183434,2600.0
88,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,4.024712,2800.0
89,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,5.0291414,2749.60400390625
90,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,6.033699,1021.1300048828125
91,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,7.0397954,1945.0069580078125
92,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,8.047083,3001.322998046875
93,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,9.0511675,2486.304931640625
94,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,10.060614,3200.0
95,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,10.068606,2859.821044921875
96,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,0.0,37.5
97,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,1.0068829,25.700000762939453
98,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,2.0126529,24.600000381469727
99,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,3.0183434,33.599998474121094
100,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,4.024712,31.5
101,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,5.0291414,23.100000381469727
102,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,6.033699,26.600000381469727
103,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,7.0397954,33.900001525878906
104,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,8.047083,24.700000762939453
105,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,9.0511675,23.600000381469727
106,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,10.060614,23.299999237060547
107,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,10.068606,50.0
108,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,0.0,9.068116188049316
109,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,1.0068829,9.068116188049316
110,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,2.0126529,9.29400634765625
111,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,3.0183434,10.591010093688965
112,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,4.024712,9.672627449035645
113,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,5.0291414,9.234281539916992
114,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,6.033699,10.3326416015625
115,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,7.0397954,10.53620433807373
116,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,8.047083,8.992063522338867
117,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,9.0511675,9.542298316955566
118,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,10.060614,10.295360565185547
119,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,10.068606,11.85925579071045
120,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,0.0,6371516416.0
121,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,1.0068829,6371516416.0
122,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,2.0126529,6371516416.0
123,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,3.0183434,6371516416.0
124,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,4.024712,6371516416.0
125,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,5.0291414,6371516416.0
126,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,6.033699,6371520512.0
127,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,7.0397954,6371520512.0
128,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,8.047083,6371520512.0
129,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,9.0511675,6371520512.0
130,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,10.060614,6371520512.0
131,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,10.068606,6371520512.0
132,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,0.0,35543599104.0
133,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,1.0068829,35543599104.0
134,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,2.0126529,35543599104.0
135,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,3.0183434,35543697408.0
136,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,4.024712,35556833280.0
137,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,5.0291414,35556833280.0
138,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,6.033699,35556923392.0
139,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,7.0397954,35556923392.0
140,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,8.047083,35556923392.0
141,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,9.0511675,35556923392.0
142,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,10.060614,35557033984.0
143,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,10.068606,35557033984.0
144,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,0.0,18377730529.0
145,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,1.0068829,18377732025.0
146,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,2.0126529,18377732426.0
147,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,3.0183434,18377740366.0
148,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,4.024712,18377741928.0
149,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,5.0291414,18377741994.0
150,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,6.033699,18377741994.0
151,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,7.0397954,18391531834.0
152,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,8.047083,18391531959.0
153,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,9.0511675,18391531959.0
154,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,10.060614,18391534144.0
155,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,10.068606,18391534144.0
156,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,0.0,304896333.0
157,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,1.0068829,304897829.0
158,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,2.0126529,304898025.0
159,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,3.0183434,304900338.0
160,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,4.024712,304901904.0
161,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,5.0291414,304901904.0
162,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,6.033699,304901904.0
163,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,7.0397954,304946475.0
164,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,8.047083,304946686.0
165,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,9.0511675,304946686.0
166,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,10.060614,304948698.0
167,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,10.068606,304948698.0
168,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,0.0,7110832128.0
169,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,1.0068829,7132991488.0
170,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,2.0126529,7121014784.0
171,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,3.0183434,7130132480.0
172,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,4.024712,7077158912.0
173,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,5.0291414,7070154752.0
174,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,6.033699,7081443328.0
175,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,7.0397954,7110733824.0
176,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,8.047083,7109107712.0
177,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,9.0511675,7103995904.0
178,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,10.060614,7114371072.0
179,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,10.068606,7114371072.0
Let’s move on to a slightly more interesting example, that we are going to profile in parallel inside our notebook using ipyparallel.
Setup for a notebook#
Now it’s time to start the ipcluster again. In the terminal window, run:
cd
source ./notebooks/0_setup/0_setup_jsc_perun.sh
ipcontroller &
srun -n 4 -c 12 --ntasks-per-node 4 --time 00:90:00 -A training2546 -p dc_gpu ipengine start
[ ]:
from ipyparallel import Client
rc = Client(profile="default")
rc.ids
if len(rc.ids) == 0:
print("No engines found")
else:
print(f"{len(rc.ids)} engines found")
4 engines found
Using the perun decorators#
perun offers an alternative way to start monitoring your code by using function decorators. The main goal is to isolate the region of the code that you want to monitor inside a function, and decorate it with the @perun decorator. Now, your code can be started using the normal python command, and perun will start gathering data only when that function is reached.
Carefull: For each time the perun decorator is called, it will create a new output file and a new run, which could slow down your code significantly. If the function that you want to monitor will be run more than once, it is better to use the @monitor decorator.
Let’s look at the example below.
[ ]:
%%px
from sklearn import datasets
import heat as ht
from perun import perun, monitor
@monitor()
def data_loading():
X,_ = datasets.load_digits(return_X_y=True)
return ht.array(X, split=0)
@monitor()
def fitting(X):
k = 10
kmeans = ht.cluster.KMeans(n_clusters=k, init="kmeans++")
kmeans.fit(X)
@perun(log_lvl="WARNING", data_out="perun_data", format="text", sampling_period=0.1)
def main():
data = data_loading()
fitting(data)
The example has 3 functions, the main function with the @perun decorator, fitting and data_loading with the @monitor decorator. perun will start monitoring whenever we run the main function, and will record the entry and exit time of the other two functions marked with @monitor.
[ ]:
%%px
main()
The text report will have an extra table with with all the monitored functions, outlining the average runtime, and power draw measured while the application was running, together with other metrics. The data can also be found in the hdf5 file, where the start and stop events of the functions are stored under the regions node of the individual runs.
If you want more information on perun check the documentation or check the code in github. Thanks!