TotalView delivers unbeatable platform, language, and compiler support for C, C++, and mixed-language Python – C/C++ applications.

GET LATEST RELEASE    TRY FOR FREE

TotalView 2023.4

Registers View

TotalView 2023.4 adds the ability to easily examine the CPU and GPU registers for target programs as they are debugged. Display the Registers View by selecting Window > Views > Registers.

TotalView Register View

To edit the value of a register or cast it to a different type simply right click on it and select Add to Data View or drag-and-drop it into the Data View. Through the Data View the register value can be changed by double clicking on the value and entering a new one. The type can similarly be changed by double clicking on the existing type and entering a new one.

Assembly View Early Access

The ability to debug low-level machine instructions has been added to TotalView 2023.4. The new Assembly View displays machine instructions for the current function in PC scope. Instruction level stepping commands enable fine-grained control stepping across machine instructions. Instruction level breakpoints allow precise program execution control and enable debugging tough situations where it is important to understand individual program instructions and data move operations. To display the Assembler View right click in the source area and select Show Assembler.

TotalView Assembler View

New C++ Type Transformations

TotalView 2023.4 adds additional type transformation support for the following C++ collection classes and iterators for traversing through the data elements on the collections.

  • deque
  • forward_list
  • list
  • queue
  • priority_queue
  • stack
  • array
  • pair
  • tuple

CUDA 12 Support

TotalView 2023.4 adds support for CUDA 12. See the release notes for full details and recommended configurations to successfully debug on CUDA 12 systems with TotalView.

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been addressed with TotalView 2023.4

View Full Release Notes

For more details, please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

View Release NotesGet Latest Release

TotalView 2023.3

Array Slicing and Striding

TotalView 2023.3 adds the ability to define a slice and stride for arrays in the Array View.  Simply click on the configuration icon in the Array View toolbar to display the Array Configuration Options dialog.

TotalView2023.3_image1

 

Define the start and end dimensions for the array and an optional stride for the data in the dialog.

TotalView2023.3_image2

 

Use the apply button to live update the display in the Array View or Ok to close the dialog and use the defined settings.  The type of the array can also be changed through the Options dialog.

Memory Debugging Block Overwrite / Guard Block Report 

Building on the memory block overwrite detection capabilities added in TotalView 2022.2, version 2023.3 adds the ability to generate an on-demand Corrupt Guard Block Report that lists all memory blocks that have had their bounds overwritten.  

TotalView detects memory block overwrites by padding a pattern filled guard block on either side of the allocated memory block.  If the pattern is disturbed an overwrite has occurred.  To turn on overwrite detection, open the Memory Debugging Options dialog, and toggle the Guard allocated memory checkbox.

TotalView2023.3_image3

At any point during program execution, all allocated memory blocks can be checked for overwrites by opening the Corrupt Guard Block Report.  This is easily done by pressing the icon in the Memory toolbar or selecting Debug > Corrupt Memory Block Report from the main menu.

TotalView2023.3_image4

 

The Corrupt Guard Block Report shows all the memory blocks found with overwrite violations.  Locations are broken down by process, file, function, line number, and the list of individual blocks that have been overwritten.  TotalView also reports if the block was overwritten at the start (pre) or end (post) of the block.

TotalView2023.3_image5

 

Python 3.11 Debugging Support

Debugging Python 3.11 applications along with C/C++ Python modules is supported in the TotalView 2023.3 release.  Python 3.11 significantly changed how the debug and program information was stored.  TotalView has been updated to find the proper information and provide a unified Python and C/C++ debugging view of the mixed language debugging session.

Apple M1/M2 Debugging Support

TotalView 2023.3 adds platform support for Apple’s ARM based M1 and M2 architectures.  Only debugging with TotalView’s new UI and CLI are supported.  Both macOS Monterey and Ventura are supported on M1/M2 based systems.

Preliminary CUDA 12 Support

TotalView 2023.3 adds preliminary support for CUDA 12.  See the CUDA specific platform notes later in this document for full details and recommended configurations to successfully debug on CUDA 12 systems with TotalView.

Bug Fixes and Performance Improvements 

Numerous bug fixes and performance improvements have been addressed, including performance improvements for data transformations.

View Full Release Notes

For more details, please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

RELEASE NOTES  LATEST RELEASE

TotalView 2023.2

Visualize Array Data with 3D Surface Plots 

TotalView’s data visualization capabilities have been enhanced in the 2023.2 release with the ability to visualize array data with a new 3D surface plot. To view array data with the surface plot, first select Add to Array View from the context menu of a data element in the Data Window. 

TotalView 2023.2 release_image 1

Figure 1 - Add array data to the Array View 

Once the Array View is displayed, select the Surface Plot button, to visually examine the array data. The data displayed can be sliced by changing the variable expression. In the following screenshot all the data is displayed using expression v[:][:]. To display a subset of the data the expression could be altered to something like v[10:30][20:40].  Utilize the plot controls to rotate and zoom the display. 

TotalView 2023.2 release_image 2

 Figure 2 - Array View Surface Plot 

Surface plot display is only supported on the Linux-x86-64, Linux ARM64, and macOS platforms and requires OpenGL version 2.1 or greater. See the OpenGL Support release notes for the Linux and macOS platform below for more details. 

TotalView UI Share Group Target Support 

For TotalView2023.2, the debugger’s underlying ability to target a Share Group for various debugger operations has been added to the UI. As TotalView acquires processes into its debugging session it organizes them into a Control Group and Share Group based on how they were launched and their executable image. All processes launched from a common starter process, such as MPI’s mpirun, will be placed in the same Control Group. Processes sharing a similar executable image are organized into a common Share Group. For example, a Multi-Program/Multi-Data (MPMD) style application will utilize multiple executable images.  As processes of these images are started, they are placed in their own Share Groups. A Share Group of processes can then be targeted for specific debugging operations such as stepping. Allowing you to just step just the process in the Share Group. See the TotalView User Guide for more details on using Share Groups during your debugging sessions.  

TotalView 2023.2 release_image 3

 Figure 3 - TotalView Group (Share) Control Menu Item 

Memory Buffer Overwrite Detection 

TotalView 2023.2 adds the ability to detect when an allocated memory block has been overwritten beyond the ends of its bounds using guard block technology. To enable buffer overwrite detection simply turn on the “Guard allocated memory” option from the Memory Debugging Options dialog box from the TotalView toolbar or Debug menu. 

TotalView 2023.2 release_image 4

 Figure 4 - Watch Buffer Overwrites by Guarding Allocated Memory 

Once enabled, TotalView will verify memory block bounds have not been overwritten when they are freed and raise a memory event if any violations are found. The next version of TotalView will provide a guard block check report to verify all memory blocks at any time during a debugging session. 

Advanced Memory Debugging with Hoarding 

The ability to “hoard” deallocated memory blocks have been added to TotalView 2023.2. Hoarding is useful when analyzing dangling pointer bugs.  When a hoarding is turned on blocks being freed are not actually freed with the operating system but instead are “held aside” by TotalView’s memory debugging technology. This allows the contents for the memory block to remain stable since it will not be reissued as new memory is allocated. Keeping the memory contents stable eliminates sporadic failures due to inconsistent memory contents and allows the developer to continue to diagnose a dangling pointer situation. Hoarding is often combined with memory debugging painting which paints a pattern to newly allocated and deallocated memory. Painting a pattern into memory creates consistent data in the memory blocks to more easily identify read before initialize and read after free issues.   

TotalView 2023.2 release_image 5

 Figure 5 - Painting and Hoarding Memory Debugging Options 

Upcoming macOS M1/M2 Support 

TotalView 2023.2 adds support for macOS Ventura but only for the Apple Intel architecture. Active development to support Apple’s ARM based M1 and M2 chipset is underway.  An early access beta of the upcoming TotalView 2023.3 release will be available this summer.  If you are interested in participating in the beta, please send a note to the TotalView support team at techsupport@roguewave.com. We’ll reach out once the beta is available! 

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been addressed, including performance improvements for data transformations.  

View Full Release Notes

For more details, please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

RELEASE NOTES  LATEST RELEASE

TotalView 2023.1

TotalView Performance Enhancements

For TotalView 2023.1, user interface updates and debug event processing have been optimized, resulting in faster debugging session launches, especially for large scale parallel jobs.  These changes will also improve user interface updates during debugging operations, such as when quickly stepping through multiple lines of source code.

CUDA 11.8 Support

TotalView 2023.1 adds support for debugging applications using CUDA 11.8 SDK using the legacy CUDA debugger backend.  CUDA 11.8 introduced a new unified CUDA debugger backend, but several serious problems were identified while testing TotalView against the new implementation. The only viable workaround is to force CUDA to fall back to the legacy CUDA debug implementation by setting the CUDBG_USE_LEGACY_DEBUGGER environment variable to 1, e.g., export CUDBG_USE_LEGACY_DEBUGGER=1. The CUDA 12.0 r525 driver release should address the issues so that the new unified CUDA debugger backend will work correctly with TotalView. Note, because every debugger process andapplication process in the debug session must use the same setting, we recommend that the environment variable be set in your shell startup file (e.g., .cshrc or .bashrc), which is the simplest way to satisfy this requirement.

TotalView AMD GPU ROCm Initialization and ROCm Support Update

For the 2023.1 release, TotalView automatically discovers if the target program is utilizing the ROCm toolkit for running code on AMD GPUs.  As a result, developers no longer need to add the -rocm command line switch to turn on AMD GPU debugging.  This version of TotalView also supports ROCm 5.4.

Memory Debugging Painting

Finding memory access problems for uninitialized memory or when memory has been freed is a real challenge often resulting in a lot of time debugging sporadic data issues and crashes.  To make it more predictable in finding these issues, TotalView 2023.1 adds the ability to turn on memory painting for newly allocated memory blocks and freed memory blocks.  In either case, a hexadecimal paint pattern is written into the memory block giving you a consistent memory value for cases when your program tries to read from the memory blocks.  Turn on painting by simply checking the option in the Session dialog from the Start Page.

TotalView 2023.1-1.png

 

Optionally, enable the option within the memory options dialog from the memory debugging toolbar.

TotalView2023.1-2.png

 

TotalView Working Directory

Users can now specify a working directory where TotalView will executing your target program. If not provided, the default is the directory from which you invoked TotalView. 

This value can be entered into the UI or on the command line when starting TotalView. It can then be modified during a debug session using the Process > Modify Arguments menu or simply as part of setting up a new debugging session.         

TotalView2023.1-3.png

 

Specify the working directory from the command line using the new -working_directory command line argument:

totalview -working_directory /tmp 

 

Python 3.10 Mixed Language Debugging Support

TotalView 2023.1 adds support for Python 3.10 when performing mixed language C/C++ and Python debugging.

AIX 7.2 Support

TotalView 2023.1 adds support for AIX 7.2.  In our testing though we found that targets launched through a fork/exec on AIX 7.2 occasionally hang in low-level ptrace() system calls as TotalView tries to write to the target processes address space.  This has been traced to a kernel bug in the underlying AIX operating system.  This issue has been fixed by IBM and is will soon be available in a software fix from IBM.

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been addressed, including performance improvements for data transformations.

View Full Release Notes

For more details, please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

RELEASE NOTES  Get Latest Release

TotalView 2022.4

AMD GPU Debugging Support 

TotalView 2022.4 adds support for debugging AMD GPUs in addition to its ability to debug NVIDIA GPUs. TotalView supports versions 5.1 to 5.2 of the ROCm software stack for GPU programming, including support for debugging HIP (Heterogeneous Interface for Portability) and MPI code running on AMD MI50, MI100, and MI200 series of GPUs.  TotalView supports the following debugging features on AMD GPUs: 

  • Process launch, attach, and detach 
  • Viewing scalar, vector, general, and special AMD GPU registers 
  • Instruction disassembly 
  • Breakpoint creation and deletion on AMD GPU code 
  • Single-stepping and fast smart-stepping 
  • Stack unwinding, including inlined functions 
  • Navigation controls for changing the logical workgroup / work-item focus or physical agent, queue, dispatch, wave, and lane focus 
  • Variable display with the ROCm 5.1+ compilers 
  • Data watchpoints on global memory variables 

Preparing your program for debugging is simple. Use the “-O0 -g” flags with the ROCm 5.1+ compilers to generate the necessary debug symbols for TotalView. 

To debug a program on AMD GPUs, use the “-rocm” flag when you start TotalView, for example: 

totalview -rocm a.out

Once started, simply set breakpoints in the HIP kernel code by selecting the line number where you would like to stop, then begin running your program.  TotalView will automatically plant breakpoints in the HIP code once it is loaded onto the GPU.  Use TotalView’s stepping commands to control execution, and dive on variables to view their values in your GPU code.  Reverse debugging will not be supported for code running on the AMD GPUs but is planned to work with reverse debug code running on the CPU in a combined CPU/GPU debugging session.  There currently is an issue while reverse debugging when the -rocm switch is used and debugging on the GPU.  We are working to resolve this issue for an upcoming release of TotalView. 

To learn more about specific AMD GPU debugging characteristics, see help for the drocm command in TotalView’s Command Line Interface (dhelp drocm). 

Array View for Array Debugging and Visualization 

A new Array View is available in TotalView 2022.4 which enables focused debugging of array data.  Through the Array View, you can examine statistical information about an array and visualize array data. 

To bring up the Array View, either activate it from the Window > Views > Array View menu item or select Add to Array View from the context menu in the Local Variables view or Data View when selecting an array variable. 

Figure 1 - Add to Array View menu item 

The toolbar area of the Array View displays the process and thread for the array data being displayed.  The array expression is shown along with the slice of the array to debug.  In Figure 2, array expression floatArray[:] is shown and the [:] specifies to debug the whole array.  A portion of the array can be specified by providing the beginning and ending indices, such as [10:15]. 

The Array View displays array statistics information initially.  In previous versions of TotalView this was separate functionality available in the Array Statistics View.   

Figure 2 - Array Statistics 

A histogram plot of the array data is shown when the bar plot icon is clicked in the toolbar.  Adjust the number of bins used in the plot using the field in the plot. 

Figure 3 - Visualize Array in Histogram Plo

A line plot of the array data is also available when the line plot icon is selected in the toolbar. 

Figure 4 - Visualize Array in Line Plot 

Watch for more array debugging capabilities to be added to the Array View in coming versions of TotalView! 

TotalView Startup Performance Enhancements 

For TotalView 2022.4, further startup performance improvements optimize how the debugger looks up symbols and entry point values in the runtime linker.  This can have a big impact on debugger startup performance for applications built with many shared libraries. 

DEBUGGING TIP:  A further startup performance optimization customers should consider using is to instruct TotalView to defer processing of shared libraries until later in the debug session.  By adding the TotalView command line option -no_dlopen_always_recalculate the debugger will delay processing shared library symbols until they are needed later in the debug session.  Learn more about tuning TotalView’s parallel debugging performance in the “Scalability in HPC Computing Environments” section of the TotalView User Guide, Part III Parallel Debugging chapter. 

TotalView Base Compilation System and Dependency Changes 

TotalView 2022.4 changes the base compilation for the following systems: 

Linux x86-64 

  • Base build system was changed from CentOS 7.0 to CentOS 7.9, resulting in a base OS of CentOS/RHEL 7.9 and above.  Ubuntu 18.04 and 20.04 are also supported.  
     
    Note that TotalView’s user interface requires have libxkbcommon and libXcomposite libraries, typically installed as part of a desktop graphical environment. 

Linux PowerLE 

  • Base build system was changed from Ubuntu 14.04 to CentOS 7.5.  The new base supported system changes to CentOS/RHEL 7.5 and above.  Ubuntu 18.04 and 20.04 are also supported.  

Note that TotalView’s user interface requires have libxkbcommon and libXcomposite libraries, typically installed as part of a desktop graphical environment. 

macOS 

  • Base build system was changed from macOS High Sierra to macOS Big Sur.  TotalView’s base compilation support stays the same with support for macOS Big Sur and Monterey. 

Bug Fixes and Performance Improvements 

Numerous bug fixes and performance improvements have been addressed, including performance improvements for data transformations. 

View Full Release Notes

For more details please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

Release Notes  Latest Release

TotalView 2022.3 

AMD GPU Debugging Preliminary Support 

TotalView 2022.3 expands its GPU debugging capabilities with support for debugging AMD GPUs, in addition to its ability to debug NVIDIA GPUs.  TotalView supports versions 5.1 and 5.2 of the ROCm software stack for GPU programming, including support for debugging HIP (Heterogeneous Interface for Portability) and MPI code running on AMD MI50, MI100, and MI200 series of GPUs.  With this release, TotalView supports the following debugging features on AMD GPUs: 

  • Process launch, attach, and detach 
  • Viewing scalar, vector, general, and special AMD GPU registers 
  • Instruction disassembly 
  • Breakpoint creation and deletion on AMD GPU code 
  • Single-stepping and fast smart-stepping 
  • Stack unwinding, including inlined functions 
  • Navigation controls for changing the logical workgroup / work-item focus or physical agent, queue, dispatch, wave, and lane focus 
  • Variable display with the ROCm 5.1+ compilers 
  • Data watchpoints on global memory variables 

Preparing your program for debugging is simple. Use the “-O0 -g” flags with the ROCm 5.1+ compilers to generate the necessary debug symbols for TotalView. 

To debug a program on AMD GPUs, use the “-rocm” flag when you start TotalView, for example: 

totalview -rocm a.out 

Once started, simply set breakpoints in the HIP kernel code by selecting the line number where you would like to stop, then begin running your program.  TotalView will automatically plant breakpoints in the HIP code once it is loaded onto the GPU.  Use TotalView’s stepping commands to control execution, and dive on variables to view their values in your GPU code.  Reverse debugging will not be supported for code running on the AMD GPUs but is planned to work with reverse debug code running on the CPU in a combined CPU/GPU debugging session.  There currently is an issue while reverse debugging when the -rocm switch is used and debugging on the GPU.  We are working to resolve this issue for the next release of TotalView. 

To learn more about specific AMD GPU debugging characteristics, see help for the drocm command in TotalView’s Command Line Interface (dhelp drocm). 

Watch for more AMD GPU debugging features in our 2022.4 release in November, including watchpoint support!  If you are attending Supercomputing 2022 this year, make sure to attend the “Debugging the Toughest Challenges with NVIDIA and AMD GPUs” Exhibitor Forum talk for more details about debugging AMD and NVIDIA GPUs with TotalView. 

Dive Stack Expansion State 

TotalView 2022.3 expands on the Dive Stack functionality introduced in version 2022.2.  The Dive Stack enables developers to easily explore data by traversing/diving into data structures and back out.  Version 2022.3 adds functionality to maintain the expansion and scroll state before the dive so that an undive operation restores the Data View as it was. 

NVIDIA OpenACC Support 

TotalView 2022.3 adds official support for debugging NVIDIA OpenACC compiled applications. 

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been addressed, including performance improvements for data transformations. 

View Full Release Notes

For more details please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

Watch the What's New Webinar

Bill Burns, Senior Director of Product Engineering will show you all the new features. Watch now >>

 release notes   Get Latest Release

TotalView 2022.2

Data Debugging Workflow Improvements With Dive Stacks 

The latest release incorporates a popular feature from TotalView’s Classic UI, allowing users to navigate a Dive Stack of expressions when using data structures in Data View. Navigating up and down the stack allows you to move more easily through the data and streamlines workflows.

Fedora 35 and Ubuntu 22.04 Preliminary Support

There have been many Fedora 35 and Ubuntu 22.04 system level changes that impact debuggers. Our latest release offers enhancements and fixes to support debugging on these platforms. Testing for full support in the 2022.3 release is ongoing.

Solaris Base System Change and Query About System Support

Due to infrastructure changes, we’ve moved from Solaris 11u3 to 11u4.  We are interested in hearing if customers still actively use Solaris for development and debugging. Please send an email to Bill Burns, TotalView product manager, at bburns@perforce.com if Solaris is still an important platform for you.

TotalView CLI dprint Command Timeout Option

The latest release supports the ability to define a timeout via CLI dprint commands. Simply add “-timeout nSeconds” to define the number of maximum number of seconds the command should run.

TotalView's Security Updates

TotalView 2022.2 upgrades its use of zlib to the latest version to eliminate a CVE security vulnerability.

TotalView Startup Performance Improvements

TotalView’s processing of debug symbols during startup have resulted in a 42x performance improvement and reduced processing time from minutes to seconds, in one use case, as compared to the 2021.4 version. In cases where an executable or shared library contains thousands of subroutines, this new processing logic greatly improves efficiency.

Utilize Library Build ID's to Improve TotalView Processing Performance

TotalView 2022.2 incorporates advancements to processing of libraries in parallel jobs.  Checksums are used to find similar libraries loaded by parallel processes across a cluster of nodes to determine which libraries are identical. Checksumming can incur a performance penalty as each library is checked. TotalView 2022.2 attempts to extract the Build ID from shared library image files, thereby improving overall processing.

New User Interface Improvements

TotalView’s new user interface is available by default for new users and is activated through the Display Preferences panel or using the -newUI command line option for existing TotalView users.

Share Your Feedback

If you have any feedback about new features, wish to make requests for new or missing features, or want to report any problems please send an email to tv-beta@perforce.com.

View Full Release Notes

For more details please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

View Release Notes   Get Latest Release

TotalView 2022.1

Memory Debugging Block Notify

A particularly challenging memory debugging situation occurs when a specific memory block is free but a program accesses the memory later through a dangling pointer reference. Because the memory may or may not still contain valid data, sporadic crashes and data problems can occur. TotalView 2022.1 adds the ability to watch a specific allocated memory block and raise an event when it gets freed or deleted, effectively mitigating this debugging challenge.

MacOS Monterey Support

TotalView 2022.1 adds support for Apple’s latest version of macOS 12 Monterey.

CUDA 11.4 and 11.5 Support

TotalView 2022.1 adds official debugging support for CUDA 11.4 and 11.5.

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been made to TotalView.  See the release notes for the full list of bug fixes.

Share Your Feedback

If you have any feedback about new features, wish to make requests for new or missing features, or want to report any problems please send an email to tv-beta@perforce.com.

View Full Release Notes

For more details please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

Watch the What's New Webinar

Bill Burns, Senior Director of Product Engineering will show you all the new features. Watch now >>

View Release NotesGet Latest Release