{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Distributed profiling and energy measurements with perun" ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "How to locate performance issues on your distributed application, and fix them, in three steps:\n", "\n", "1. Find the problematic/slow function in your code.\n", "2. Gather statistics and data about the slow function.\n", "3. Fix it!\n", "\n", "---\n", "\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "If you want more information on perun, find any issues, or questions leaves us a message on [github](https://github.com/Helmholtz-AI-Energy/perun) or check the [documentation](https://perun.readthedocs.io/en/latest/?badge=latest)." ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "## Installation\n", "\n", "Perun can be installed with ```pip```:\n", "\n", "```shell\n", "pip install perun\n", "```\n", "\n", "Thourgh pip, optional dependencies can be installed that target different hardware accelerators, as well as the optional MPI support.\n", "\n", "\n", "```shell\n", "pip install perun[mpi,nvidia]\n", "# or\n", "pip install perun[mpi,rocm]\n", "```\n", "\n", "Here we have already installed perun with all necessary dependencies, please **select the kernel `perun_notebook`.**" ] }, { "cell_type": "code", "execution_count": null, "id": "4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: perun[mpi,nvidia] in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (0.9.0)\n", "Requirement already satisfied: h5py>=3.5.9 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (3.13.0)\n", "Requirement already satisfied: numpy>=1.20.0 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (2.2.5)\n", "Requirement already satisfied: pandas>=1.3 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (2.2.3)\n", "Requirement already satisfied: psutil>=5.9.0 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (7.0.0)\n", "Requirement already satisfied: py-cpuinfo>=5.0.0 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (9.0.0)\n", "Requirement already satisfied: tabulate>=0.9 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (0.9.0)\n", "Requirement already satisfied: mpi4py>=3.1 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from perun[mpi,nvidia]) (4.0.3)\n", "Collecting nvidia-ml-py>=12.535.77 (from perun[mpi,nvidia])\n", " Using cached nvidia_ml_py-12.575.51-py3-none-any.whl.metadata (9.3 kB)\n", "Requirement already satisfied: python-dateutil>=2.8.2 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from pandas>=1.3->perun[mpi,nvidia]) (2.9.0.post0)\n", "Requirement already satisfied: pytz>=2020.1 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from pandas>=1.3->perun[mpi,nvidia]) (2025.2)\n", "Requirement already satisfied: tzdata>=2022.7 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from pandas>=1.3->perun[mpi,nvidia]) (2025.2)\n", "Requirement already satisfied: six>=1.5 in /home/juanpedroghm/.pyenv/versions/3.11.2/envs/heat-dev311/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas>=1.3->perun[mpi,nvidia]) (1.17.0)\n", "Using cached nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB)\n", "Installing collected packages: nvidia-ml-py\n", "Successfully installed nvidia-ml-py-12.575.51\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m25.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "perun 0.9.0\n" ] } ], "source": [ "!perun --version" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## Basic command line usage\n", "\n", "Perun is primarily a command line tool. The complete functionality can be accessed through the ```perun``` command. On a terminal, simply type ```perun``` and click enter to get a help dialog with the available subcommands." ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "usage: perun [-h] [-c CONFIGURATION] [-l {DEBUG,INFO,WARN,ERROR,CRITICAL}]\n", " [--log_file LOG_FILE] [--version]\n", " {showconf,sensors,metadata,export,monitor} ...\n", "\n", "Distributed performance and energy monitoring tool\n", "\n", "positional arguments:\n", " {showconf,sensors,metadata,export,monitor}\n", " showconf Print perun configuration in INI format.\n", " sensors Print available sensors by host and rank.\n", " metadata Print available metadata.\n", " export Export existing output file to another format.\n", " monitor Gather power consumption from hardware devices while\n", " SCRIPT [SCRIPT_ARGS] is running. SCRIPT is a path to\n", " the python script to monitor, run with arguments\n", " SCRIPT_ARGS.\n", "\n", "options:\n", " -h, --help show this help message and exit\n", " -c CONFIGURATION, --configuration CONFIGURATION\n", " Path to perun configuration file.\n", " -l {DEBUG,INFO,WARN,ERROR,CRITICAL}, --log_lvl {DEBUG,INFO,WARN,ERROR,CRITICAL}\n", " Logging level.\n", " --log_file LOG_FILE Path to the log file. None by default. Writting to a\n", " file disables logging in stdout.\n", " --version show program's version number and exit\n" ] } ], "source": [ "!perun" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "**perun** can already be used after this, without any further configuration or modification of the code. perun can monitor command line scripts, and other programs from the command lines. Try running the ```perun monitor -b sleep 10``` on a terminal, or by running the cell below.\n", "\n", "
\n", "Note:\n", "The ipcluster should NOT be running when you execute the cell below, please stop it if it is still running.\n", "
\n" ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/juanpedroghm/code/heat/doc/source/tutorials/notebooks\n", "[2025-05-20 16:59:39,969][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R3/4:\u001b[1;31mUnknown error loading dependecy NVMLBackend\u001b[0m\n", "[2025-05-20 16:59:39,969][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R3/4:\u001b[1;31mNVML Shared Library Not Found\u001b[0m\n", "[2025-05-20 16:59:39,969][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R1/4:\u001b[1;31mUnknown error loading dependecy NVMLBackend\u001b[0m\n", "[2025-05-20 16:59:39,970][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R1/4:\u001b[1;31mNVML Shared Library Not Found\u001b[0m\n", "[2025-05-20 16:59:39,970][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R0/4:\u001b[1;31mUnknown error loading dependecy NVMLBackend\u001b[0m\n", "[2025-05-20 16:59:39,970][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R0/4:\u001b[1;31mNVML Shared Library Not Found\u001b[0m\n", "[2025-05-20 16:59:39,976][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R2/4:\u001b[1;31mUnknown error loading dependecy NVMLBackend\u001b[0m\n", "[2025-05-20 16:59:39,976][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R2/4:\u001b[1;31mNVML Shared Library Not Found\u001b[0m\n" ] } ], "source": [ "%%bash\n", "pwd\n", "srun -n 4 perun monitor -b sleep 10" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "In the directory reported by ```pwd```, you should see a new directory called ```perun_results```, (might be named ```bench_data``` if the current directory is the heat root directory ) with two files, **sleep.hdf5** and **sleep_.txt**. \n", "\n", "The file **sleep_.txt** contains a summary of what was measured on the run, with the average power draw of different hardware componets, memory usage, and the total energy. The available information depends on the available *sensors* that perun finds. You can see a list of the available sensors by running the sensors subcommand:" ] }, { "cell_type": "code", "execution_count": null, "id": "10", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2025-05-20 16:55:39,740][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R0/1:\u001b[1;31mUnknown error loading dependecy NVMLBackend\u001b[0m\n", "[2025-05-20 16:55:39,740][\u001b[1;36mperun.core\u001b[0m][\u001b[1;35mbackends\u001b[0m][\u001b[1;31mERROR\u001b[0m] - R0/1:\u001b[1;31mNVML Shared Library Not Found\u001b[0m\n", "| Sensor | Source | Device | Unit |\n", "|-----------------:|--------------:|----------------:|-------:|\n", "| cpu_0_package-0 | powercap_rapl | DeviceType.CPU | J |\n", "| CPU_FREQ_0 | psutil | DeviceType.CPU | Hz |\n", "| CPU_FREQ_1 | psutil | DeviceType.CPU | Hz |\n", "| CPU_FREQ_2 | psutil | DeviceType.CPU | Hz |\n", "| CPU_FREQ_3 | psutil | DeviceType.CPU | Hz |\n", "| CPU_FREQ_4 | psutil | DeviceType.CPU | Hz |\n", "| CPU_FREQ_5 | psutil | DeviceType.CPU | Hz |\n", "| CPU_FREQ_6 | psutil | DeviceType.CPU | Hz |\n", "| CPU_FREQ_7 | psutil | DeviceType.CPU | Hz |\n", "| CPU_USAGE | psutil | DeviceType.CPU | % |\n", "| DISK_READ_BYTES | psutil | DeviceType.DISK | B |\n", "| DISK_WRITE_BYTES | psutil | DeviceType.DISK | B |\n", "| NET_READ_BYTES | psutil | DeviceType.NET | B |\n", "| NET_WRITE_BYTES | psutil | DeviceType.NET | B |\n", "| RAM_USAGE | psutil | DeviceType.RAM | B |\n", "\n" ] } ], "source": [ "!perun sensors" ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "The other file, **sleep.hdf5**, contains all the raw data that perun collects, that can be used for later processing. To get an interactive view of the data, navigate to [myhdf5](https://myhdf5.hdfgroup.org), and upload the file there.\n", "\n", "This will let you explore the data tree that perun uses to store the hardware information. More info on the data tree can be found on the [data documentation](https://perun.readthedocs.io/en/latest/data.html)." ] }, { "cell_type": "markdown", "id": "12", "metadata": {}, "source": [ "The data that is stored on the hdf5 file can be exported to other formats. Supported formats are text (same as text report), csv, json and bench. Run the cell below to export the last run of the sleep program to csv." ] }, { "cell_type": "code", "execution_count": null, "id": "13", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ",run id,hostname,device_group,sensor,unit,magnitude,timestep,value\n", "0,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,0.0,2021.14599609375\n", "1,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,1.0068829,964.1939697265625\n", "2,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,2.0126529,400.12799072265625\n", "3,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,3.0183434,2600.0\n", "4,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,4.024712,2800.0\n", "5,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,5.0291414,2384.971923828125\n", "6,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,6.033699,1418.0760498046875\n", "7,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,7.0397954,2297.81298828125\n", "8,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,8.047083,2893.419921875\n", "9,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,9.0511675,2456.3759765625\n", "10,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,10.060614,1828.7459716796875\n", "11,0,juan-20w000p2ge,cpu,CPU_FREQ_0,Hz,1000000.0,10.068606,3012.5791015625\n", "12,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,0.0,1211.6190185546875\n", "13,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,1.0068829,2700.0\n", "14,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,2.0126529,1569.219970703125\n", "15,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,3.0183434,2497.64697265625\n", "16,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,4.024712,2693.7109375\n", "17,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,5.0291414,2240.751953125\n", "18,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,6.033699,3000.02099609375\n", "19,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,7.0397954,2600.0\n", "20,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,8.047083,3100.0\n", "21,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,9.0511675,1806.197021484375\n", "22,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,10.060614,3102.570068359375\n", "23,0,juan-20w000p2ge,cpu,CPU_FREQ_1,Hz,1000000.0,10.068606,2934.219970703125\n", "24,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,0.0,2200.10595703125\n", "25,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,1.0068829,2700.096923828125\n", "26,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,2.0126529,2842.551025390625\n", "27,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,3.0183434,2488.455078125\n", "28,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,4.024712,2651.922119140625\n", "29,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,5.0291414,2183.43310546875\n", "30,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,6.033699,2751.02490234375\n", "31,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,7.0397954,2544.83203125\n", "32,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,8.047083,3044.756103515625\n", "33,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,9.0511675,2271.235107421875\n", "34,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,10.060614,2385.8291015625\n", "35,0,juan-20w000p2ge,cpu,CPU_FREQ_2,Hz,1000000.0,10.068606,3200.0\n", "36,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,0.0,2200.012939453125\n", "37,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,1.0068829,2700.0\n", "38,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,2.0126529,1869.530029296875\n", "39,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,3.0183434,2600.0\n", "40,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,4.024712,2800.0\n", "41,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,5.0291414,2315.37109375\n", "42,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,6.033699,2672.827880859375\n", "43,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,7.0397954,2600.0\n", "44,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,8.047083,2464.04296875\n", "45,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,9.0511675,2410.884033203125\n", "46,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,10.060614,3060.60791015625\n", "47,0,juan-20w000p2ge,cpu,CPU_FREQ_3,Hz,1000000.0,10.068606,2562.06201171875\n", "48,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,0.0,2156.548095703125\n", "49,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,1.0068829,2499.455078125\n", "50,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,2.0126529,400.62200927734375\n", "51,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,3.0183434,2080.2919921875\n", "52,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,4.024712,2777.10693359375\n", "53,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,5.0291414,1521.5909423828125\n", "54,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,6.033699,2873.384033203125\n", "55,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,7.0397954,2195.196044921875\n", "56,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,8.047083,2817.139892578125\n", "57,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,9.0511675,2418.926025390625\n", "58,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,10.060614,2187.868896484375\n", "59,0,juan-20w000p2ge,cpu,CPU_FREQ_4,Hz,1000000.0,10.068606,2655.29296875\n", "60,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,0.0,2137.35791015625\n", "61,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,1.0068829,2700.0\n", "62,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,2.0126529,769.7069702148438\n", "63,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,3.0183434,1988.4849853515625\n", "64,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,4.024712,2471.529052734375\n", "65,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,5.0291414,1931.303955078125\n", "66,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,6.033699,2886.305908203125\n", "67,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,7.0397954,2543.840087890625\n", "68,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,8.047083,3100.0\n", "69,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,9.0511675,2055.845947265625\n", "70,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,10.060614,2340.925048828125\n", "71,0,juan-20w000p2ge,cpu,CPU_FREQ_5,Hz,1000000.0,10.068606,2812.739990234375\n", "72,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,0.0,2176.281005859375\n", "73,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,1.0068829,1221.010986328125\n", "74,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,2.0126529,1433.5810546875\n", "75,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,3.0183434,2562.242919921875\n", "76,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,4.024712,2591.029052734375\n", "77,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,5.0291414,2437.9990234375\n", "78,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,6.033699,3000.0\n", "79,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,7.0397954,2550.77392578125\n", "80,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,8.047083,3063.29296875\n", "81,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,9.0511675,2261.791015625\n", "82,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,10.060614,3050.388916015625\n", "83,0,juan-20w000p2ge,cpu,CPU_FREQ_6,Hz,1000000.0,10.068606,3017.64892578125\n", "84,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,0.0,2199.987060546875\n", "85,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,1.0068829,2698.6279296875\n", "86,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,2.0126529,1597.2509765625\n", "87,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,3.0183434,2600.0\n", "88,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,4.024712,2800.0\n", "89,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,5.0291414,2749.60400390625\n", "90,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,6.033699,1021.1300048828125\n", "91,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,7.0397954,1945.0069580078125\n", "92,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,8.047083,3001.322998046875\n", "93,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,9.0511675,2486.304931640625\n", "94,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,10.060614,3200.0\n", "95,0,juan-20w000p2ge,cpu,CPU_FREQ_7,Hz,1000000.0,10.068606,2859.821044921875\n", "96,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,0.0,37.5\n", "97,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,1.0068829,25.700000762939453\n", "98,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,2.0126529,24.600000381469727\n", "99,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,3.0183434,33.599998474121094\n", "100,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,4.024712,31.5\n", "101,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,5.0291414,23.100000381469727\n", "102,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,6.033699,26.600000381469727\n", "103,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,7.0397954,33.900001525878906\n", "104,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,8.047083,24.700000762939453\n", "105,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,9.0511675,23.600000381469727\n", "106,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,10.060614,23.299999237060547\n", "107,0,juan-20w000p2ge,cpu,CPU_USAGE,%,1.0,10.068606,50.0\n", "108,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,0.0,9.068116188049316\n", "109,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,1.0068829,9.068116188049316\n", "110,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,2.0126529,9.29400634765625\n", "111,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,3.0183434,10.591010093688965\n", "112,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,4.024712,9.672627449035645\n", "113,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,5.0291414,9.234281539916992\n", "114,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,6.033699,10.3326416015625\n", "115,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,7.0397954,10.53620433807373\n", "116,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,8.047083,8.992063522338867\n", "117,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,9.0511675,9.542298316955566\n", "118,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,10.060614,10.295360565185547\n", "119,0,juan-20w000p2ge,cpu,cpu_0_package-0,W,1.0,10.068606,11.85925579071045\n", "120,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,0.0,6371516416.0\n", "121,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,1.0068829,6371516416.0\n", "122,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,2.0126529,6371516416.0\n", "123,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,3.0183434,6371516416.0\n", "124,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,4.024712,6371516416.0\n", "125,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,5.0291414,6371516416.0\n", "126,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,6.033699,6371520512.0\n", "127,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,7.0397954,6371520512.0\n", "128,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,8.047083,6371520512.0\n", "129,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,9.0511675,6371520512.0\n", "130,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,10.060614,6371520512.0\n", "131,0,juan-20w000p2ge,disk,DISK_READ_BYTES,B,1.0,10.068606,6371520512.0\n", "132,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,0.0,35543599104.0\n", "133,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,1.0068829,35543599104.0\n", "134,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,2.0126529,35543599104.0\n", "135,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,3.0183434,35543697408.0\n", "136,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,4.024712,35556833280.0\n", "137,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,5.0291414,35556833280.0\n", "138,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,6.033699,35556923392.0\n", "139,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,7.0397954,35556923392.0\n", "140,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,8.047083,35556923392.0\n", "141,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,9.0511675,35556923392.0\n", "142,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,10.060614,35557033984.0\n", "143,0,juan-20w000p2ge,disk,DISK_WRITE_BYTES,B,1.0,10.068606,35557033984.0\n", "144,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,0.0,18377730529.0\n", "145,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,1.0068829,18377732025.0\n", "146,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,2.0126529,18377732426.0\n", "147,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,3.0183434,18377740366.0\n", "148,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,4.024712,18377741928.0\n", "149,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,5.0291414,18377741994.0\n", "150,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,6.033699,18377741994.0\n", "151,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,7.0397954,18391531834.0\n", "152,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,8.047083,18391531959.0\n", "153,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,9.0511675,18391531959.0\n", "154,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,10.060614,18391534144.0\n", "155,0,juan-20w000p2ge,net,NET_READ_BYTES,B,1.0,10.068606,18391534144.0\n", "156,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,0.0,304896333.0\n", "157,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,1.0068829,304897829.0\n", "158,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,2.0126529,304898025.0\n", "159,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,3.0183434,304900338.0\n", "160,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,4.024712,304901904.0\n", "161,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,5.0291414,304901904.0\n", "162,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,6.033699,304901904.0\n", "163,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,7.0397954,304946475.0\n", "164,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,8.047083,304946686.0\n", "165,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,9.0511675,304946686.0\n", "166,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,10.060614,304948698.0\n", "167,0,juan-20w000p2ge,net,NET_WRITE_BYTES,B,1.0,10.068606,304948698.0\n", "168,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,0.0,7110832128.0\n", "169,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,1.0068829,7132991488.0\n", "170,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,2.0126529,7121014784.0\n", "171,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,3.0183434,7130132480.0\n", "172,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,4.024712,7077158912.0\n", "173,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,5.0291414,7070154752.0\n", "174,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,6.033699,7081443328.0\n", "175,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,7.0397954,7110733824.0\n", "176,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,8.047083,7109107712.0\n", "177,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,9.0511675,7103995904.0\n", "178,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,10.060614,7114371072.0\n", "179,0,juan-20w000p2ge,ram,RAM_USAGE,B,1.0,10.068606,7114371072.0\n" ] } ], "source": [ "%%bash\n", "perun export perun_results/sleep.hdf5 csv\n", "cat perun_results/sleep_*.csv" ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "Let's move on to a slightly more interesting example, that we are going to profile in parallel inside our notebook using **ipyparallel**. " ] }, { "cell_type": "markdown", "id": "15", "metadata": {}, "source": [ "## Setup for a notebook" ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "Now it's time to start the ipcluster again. In the terminal window, run:\n", "\n", "```shell\n", "cd\n", "source ./notebooks/0_setup/0_setup_jsc_perun.sh\n", "ipcontroller &\n", "srun -n 4 -c 12 --ntasks-per-node 4 --time 00:90:00 -A training2546 -p dc_gpu ipengine start\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "17", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4 engines found\n" ] } ], "source": [ "from ipyparallel import Client\n", "rc = Client(profile=\"default\")\n", "rc.ids\n", "\n", "if len(rc.ids) == 0:\n", " print(\"No engines found\")\n", "else:\n", " print(f\"{len(rc.ids)} engines found\")" ] }, { "cell_type": "markdown", "id": "18", "metadata": {}, "source": [ "## Using the perun decorators\n", "\n", "perun offers an alternative way to start monitoring your code by using function decorators. The main goal is to isolate the region of the code that you want to monitor inside a function, and decorate it with the ```@perun``` decorator. Now, your code can be started using the normal python command, and perun will start gathering data only when that function is reached.\n", "\n", "**Carefull**: For each time the perun decorator is called, it will create a new output file and a new run, which could slow down your code significantly. If the function that you want to monitor will be run more than once, it is better to use the ```@monitor``` decorator. \n", "\n", "Let's look at the example below." ] }, { "cell_type": "code", "execution_count": null, "id": "19", "metadata": {}, "outputs": [], "source": [ "%%px\n", "from sklearn import datasets\n", "import heat as ht\n", "from perun import perun, monitor\n", "\n", "@monitor()\n", "def data_loading():\n", " X,_ = datasets.load_digits(return_X_y=True)\n", " return ht.array(X, split=0)\n", "\n", "@monitor()\n", "def fitting(X):\n", " k = 10\n", " kmeans = ht.cluster.KMeans(n_clusters=k, init=\"kmeans++\")\n", " kmeans.fit(X)\n", "\n", "@perun(log_lvl=\"WARNING\", data_out=\"perun_data\", format=\"text\", sampling_period=0.1)\n", "def main():\n", " data = data_loading()\n", " fitting(data)\n" ] }, { "cell_type": "markdown", "id": "20", "metadata": {}, "source": [ "The example has 3 functions, the ```main``` function with the ```@perun``` decorator, ```fitting``` and ```data_loading``` with the ```@monitor``` decorator. **perun** will start monitoring whenever we run the ```main``` function, and will record the entry and exit time of the other two functions marked with ```@monitor```. " ] }, { "cell_type": "code", "execution_count": null, "id": "21", "metadata": {}, "outputs": [], "source": [ "%%px\n", "main()" ] }, { "cell_type": "markdown", "id": "22", "metadata": {}, "source": [ "The text report will have an extra table with with all the monitored functions, outlining the average runtime, and power draw measured while the application was running, together with other metrics. The data can also be found in the hdf5 file, where the start and stop events of the functions are stored under the regions node of the individual runs. " ] }, { "cell_type": "markdown", "id": "23", "metadata": {}, "source": [ "If you want more information on perun check the [documentation](https://perun.readthedocs.io/en/latest/?badge=latest) or check the code in [github](https://github.com/Helmholtz-AI-Energy/perun). Thanks!" ] } ], "metadata": { "kernelspec": { "display_name": "heat-dev311", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.2" } }, "nbformat": 4, "nbformat_minor": 5 }