Percona Monitoring and Management 2.38.0¶

Release date:	Jul 3, 2023
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is an open-source database monitoring, management, and observability solution for MySQL, PostgreSQL, and MongoDB.

Release Highlights¶

Important security updates¶

Oracle Linux 9 as new base OS for PMM¶

As security is paramount to us, PMM 2.38.0 adds a solid and modernized enterprise-grade foundation to PMM. This release migrates our base operating system on which we build our PMM image from CentOS 7 to Oracle Linux 9. Thus, starting with PMM 2.38.0, PMM is publicly available on Oracle Linux 9.

This transition brings many benefits:

a large number of CVEs automatically fixed
improved upstream response time to security-related issues
better compatibility with new technologies
modern libraries for our team to build on

Since not all users will be able to seamlessly make the move, we will be rolling this out in phases:

Automated migration paths¶

The following users will be able to take advantage of this immediately:

New installations of PMM:
- Docker containers from our Docker hub: percona/pmm-server:2
- OVF: Download directly from our website
- AMI: Loaded from AWS Marketplace
Docker/Podman users who do full container replacement upgrades to newer versions of PMM. Your existing docker pull or docker run commands will use the new image.

Manual upgrade options¶

For users upgrading AMI, OVF, or Docker-based installations via the UI, the Upgrade Now button on the Home dashboard will upgrade your PMM version to the latest PMM 2 release. This upgrade will provide access to all the new features and bug fixes available in this version. However, the PMM base operating system will remain on EL7 (CentOS 7) and will not be upgraded to EL9 (Oracle Linux 9). This means that certain CVEs related to the base operating system will remain unpatched.

While we work on providing automated migration paths for these deployment types, you have the option to manually upgrade your PMM installation to take advantage of the updated Oracle Linux 9 base operating system and associated security enhancements.

To manually upgrade your PMM installation to the new Oracle Linux 9 base, check out the instructions in the PMM is getting a modernized enterprise-grade foundation blog post.

Grafana upgrade¶

PMM now uses Grafana to 9.2.20, which fixes some critical vulnerabilities. For more information, see Grafana’s release blog.

Exporters¶

Updated the mysqld_exporter, node_exporter, postgres_exporter, and mongo_exporter to fix the CVEs.

ClickHouse plugin¶

We have migrated from the Altinity Clickhouse datasource to the Grafana Clickhouse datasource due to the vulnerabilities discovered.

Due to this migration, we have also updated our experimental Environment Overview Dashboard.

Important

Though we have removed the datasource from PMM, we are still reusing the same name (ClickHouse) to keep the changes minimal. Thus, if you are using the old datasource in your dashboards, ensure that you upgrade the underlying queries that will pull the data from the new datasource.

Example¶

The following example shows the difference in the query before and after migration.

Before migration:

// Altinity ClickHouse datasource
SELECT $timeSeries as t, 
    sum(m_query_time_sum)/sum(m_query_time_cnt) as QLatency,
    environment
FROM $table
WHERE $timeFilter and environment != ''
GROUP BY t,environment
ORDER BY t

After migration:

// Grafana ClickHouse datasource
SELECT period_start as t,
    sum(m_query_time_sum)/sum(m_query_time_cnt) as QLatency,
    environment
FROM pmm."metrics" 
WHERE $__timeFilter(period_start) AND environment != ''
GROUP BY t, environment
ORDER BY t

Altinity data sources used macros like $timeSeries, $timeFilter, and $table, which are no longer present in the Grafana datasource. As a result, the query needs to be rewritten to use the new macros instead of the old ones. You will notice that in the example above, the macro $timeFilter has changed to $__timeFilter, and now it requires a parameter, which should be a column of type datetime.

Depending on which macros you have used, your migration effort will vary. However, we recommend checking the plugin documentation to apply the new syntax correctly, including macros and template variables.

Improved nodes view on the Inventory page¶

In our continuing endeavor to enhance user experience, we further improved the Inventory page by improving the Nodes tab. With this change, you can get more context on your inventory objects, such as nodes, and more actionable information for exploring and resolving possible issues.

Here is the change as part of PMM 2.38.0:

Check the number of agents running on any particular node. When you click on any node, the UI navigates to the view of agents, which is filtered to display only agents related to that specific node.

To see the details of the agents running:

On the Nodes tab, under the Monitoring column, click OK or Failed, depending on the status of the node that you have selected. A page that provides the user with crucial information regarding the total number of agents deployed on that node is displayed.
View comprehensive information about each agent, including key attributes such as node type, IP address, and associated services. This gives you a complete overview of the nodes at a glance.
Check the health status of the agents to see if they are running or have failed.

Backup enhancements¶

Better folder management for enhanced PBM compatibility¶

We’ve added a new Folder field to the Create Backup pages. Use this to set a specific target directory within the selected local or S3-compatible location for the backup.

Organizing backups in folders makes grouping PBM backups for clusters easier, and it improves PMM-PBM (Percona Backup for MongoDB) integration workflows.

The Folder field is automatically populated with the value of the cluster label. You can change this default folder from PMM’s Advanced Settings, but make sure you understand how your custom folder will impact PBM integration workflows.

Sharded cluster MongoDB backup support¶

This release introduces support for working with backups for sharded clusters. PMM handles the backup process end-to-end, but restoring such artifacts is currently possible only via the CLI, using Percona Backup for MongoDB.

For information on restoring sharded backups, check the PBM documentation.

SQL comments support in QAN¶

Important/Caution

This feature is still in Technical Preview and is subject to change. We recommend that early adopters use this feature for testing purposes only.

You can now collect labels from the comments placed in SQL queries. You can tag the query by key value in the query comment. This helps in query management and optimization. This feature is currently supported for MySQL and PostgreSQL.

For example:

SELECT * /* cluster=’east’ */ FROM city;

You can enable this feature as follows:

CLI - Pass the parameter comments-parsing
UI - It is enabled by default. To disable, uncheck the Disable comments parsing checkbox in the Additional Options section. This is supported for PostgreSQL and MySQL.

New PMM Agent Down alert template¶

We’ve added a new default alert template to Percona Alerting.

The new PMM Agent down alert template monitors your Node status to notify if the Agent is down. Before using this template:

ensure the PMM Agent is healthy and present on the Database Node.
check that PMM Client is on version 2.38 version or later.

!image

Grafana upgrade¶

PMM now uses Grafana v9.2.20, which includes fixes for some critical vulnerabilities. For more information, see Grafana’s release blog.

New Features¶

PMM-2852 - [QAN]: Starting with PMM 2.38.0, MySQL and PostgreSQL supports collection of labels from the comments placed in the SQL queries.
PMM-11961 - [Inventory]: Starting with PMM 2.38.0, we have enhanced the Inventory page by improving the Nodes tab. With this change, you can get more context on your inventory objects, such as node, and more actionable information for exploring and resolving possible issues.
PMM-11962 - [Inventory]: Starting with PMM 2.38.0, you can filter the nodes and services on the Inventory page using the parameters such as ID, names, status, etc., instead of scrolling through an exhaustive list of nodes or services.
PMM-12087 - [Inventory]: Added tooltips to explain the meaning of the different service statuses in the Inventory view.

Improvements¶

PMM-9367 - [Backups]: Added support for working with backups for sharded clusters.
PMM-11250 - [Backup]: Added option to specify target directory for the selected backup location to improve PMM-PBM integration workflows.
PMM-12038 - [Backup]: Extended support for MongoDB backups to cover all authentication mechanisms available for MongoDB (including LDAP, x509 certs).
PMM-9544 - [Alerting]: Added new default alert template to Node status and notify if the Agent is down.
PMM-12114 - Upgraded Grafana to 9.2.18 to fix to CVEs.
PMM-12136 - We have migrated our base operating system on which we build our base image from CentOS 7 to Oracle Linux 9 to fix the CVEs.

Bugs Fixed¶

PMM-11578 - [Alerting]: The Last notified column under Alerting > Fired Alerts has been renamed to Last triggered. This better illustrates the fact that triggered alerts may not always generate a notification. This can happen when the contact points are not correctly configured. In addition, the Suppressed status in the State column has been renamed to Silenced to keep consistency with how this status is displayed on other Alerting pages.
PMM-12067 - [QAN]: Fixed vulnerability in Vitess package.
PMM-12105 - [QAN]: Fixed an issue where QAN was broken due to wrong pgsm_query_ids.
PMM-10063 - [QAN]: The QAN filter section could display the available list of services/nodes. But the load time was too much, with the UI being unresponsive. This issue has been resolved now. The entire list is loaded with the count displayed on the upper right side.
PMM-11386 - [QAN]: MongoDB QAN query time metrics was showing the wrong unit. It should be ms. The issue has been resolved now.
PMM-12024 - [QAN]: Fixed an issue where a QAN agent couldn’t start due to an expected return value type.
PMM-12091 - [QAN]: The tooltip for QAN > MongoDB > Query Details was displayed in seconds instead of decimals. The issue has been fixed now.
PMM-9844 - [QAN]: Data for a large number of partitions or possible indexes was shown in the QAN Details tab in one scrolling column making it practically impossible to read. The issue has been resolved now.
PMM-11938 - [QAN]: Fixed the problem when PMM tried to create pg_stat_monitor_settings view all over again.
PMM-11645 - Fixed mysql service detection when this is not loaded in systemd.
PMM-9541 - PostgreSQL exporter now works as expected with --socket.
PMM-10799 - Now in the EXPLAIN/TABLE tab you can get results even for queries with table alias.
PMM-11692 - On the MySQL Instances Overview dashboard, the Top MySQL Active Client Threads panel color was misleading. The issue has been resolved now.
PMM-11715 - CPU Utilization Graph for RDS instances was not matching what Cloudwatch reported. The issue has been resolved now.
PMM-11950 - Non-admin roles are calling restricted endpoints after login or page refresh.
PMM-12085 - Fixed an issue where SSH access to AMI and OVF images for Oracle Linux 9-based images did not work. The issue has been resolved now.
PMM-12118 - Addressed the Grafana ClickHouse DS plugin CVEs.
PMM-12119 - When a user successfully adds an RDS service to PMM, the RDS exporter’s status does not change from “UNKNOWN” (the initial status of all exporters) to “RUNNING.” This issue has been resolved.
PMM-9224 - Data from Arbiter nodes is now correctly displayed on the MongoDB ReplSet Summary dashboard.

Get expert help¶

If you need assistance, visit the community forum for comprehensive and free database knowledge, or contact our Percona Database Experts for professional support and services.

Community Forum Get a Percona Expert