Resolve issues¶

This section describes solutions to common problems and scenarios you might encounter while using PMM.

Troubleshooting checklist¶

The following questions might help you identify the origin of the problem while using Percona Monitoring and Management:

Are you using the latest PMM version?
Did you check the known issues section in the Release Notes for that particular PMM release?
Are you receiving any error messages?
Do the logs contain any messages about the problem? See Message logs and Trace logs for more information.
Does the problem occur while configuring PMM, such as:
- Does the problem occur while you configure a specific function?
- Does the problem occur when you perform a particular task?
Are you using the recommended authentication method?
Does your system’s firewall allow TCP traffic on the ports used by PMM?
Have you allocated enough disk space for installing PMM? If not, check the disk allocation space.
Are you using a Technical Preview feature? Technical Preview features are not production-ready and should only be used in testing environments. For more information, see the relevant Release Notes.
For installing the PMM client, are you using a package other than a binary package without root permissions?
Is your PMM Server installed and running with a known IP address accessible from the client node?
Is the PMM Client installed, and is the node registered with PMM Server?
Is PMM-client configured correctly and has access to the config file?
For monitoring MongoDB, do you have adminUserAnyDatabase or superuser role privilege to any database servers you want to monitor?
For monitoring Amazon RDS using PMM, is there too much latency between PMM Server and the Amazon RDS instance?
Have you upgraded the PMM Server before you upgraded the PMM Client? If yes, there might be configuration issues, thus leading to failure in the client-server communication, as PMM Server might not be able to identify all the parameters in the configuration.
Is the PMM Server version higher than or equal to the PMM Client version? Otherwise, there might be configuration issues, thus leading to failure in the client-server communication, as PMM Server might not be able to identify all the parameters in the configuration.

Troubleshooting areas¶

Upgrade issues¶

PMM server not updating correctly¶

If the PMM server wasn’t updated correctly, or if you have concerns about the release, you can force the update process in 2 ways:

From the UI - Home panel: click the Alt key on the reload icon in the Update panel to make the Update Button visible even if you are on the same version as available for update. Pressing this button will force the system to rerun the update so that any broken or not installed components can be installed. In this case, you’ll go through the usual update process with update logs and successful messages at the end.
By API call (if UI not available): You can call the Update API directly with:
```
curl --user admin:admin --request POST 'http://PMM_SERVER/v1/Updates/Start'
```
Replace admin:admin with your username/password, and replace PMM_SERVER with your server address.

You will not see the logs using this method.

Refresh The Home page in 2-5 minutes, and you should see that PMM was updated.
Upgrade PMM server using Docker.

PMM server not showing latest versions available with the instances created from AWS¶

For PMM versions prior to 2.33.0, in specific environments, including AWS, some EPEL repository mirrors did not respond within the time limit defined by pmm-update (currently set to 30 seconds). It was causing supervisord to kill pmm-update-checker, which determines if a newer PMM Server is available for upgrade.

Solution

Log in to the PMM Server and run the following command as a root user:

   $ yum-config-manager --setopt=epel.timeout=1 --save

PMM server fails while upgrading¶

A bug in PMM Server ansible scripts caused PMM to upgrade Nginx’s dependencies without updating Nginx itself. Due to this, PMM throws an error while upgrading and cannot upgrade to a newer version.

Important

This issue has been resolved for PMM version 2.33.0. However, the issue persists on all the versions prior to 2.33.0.

Solution

While PMM is being upgraded, log in to the PMM server and run the following command:

   sed -i 's/- nginx/- nginx*/' /usr/share/pmm-update/ansible/playbook/tasks/update.yml

PMM cannot acess admin user after upgrading¶

After upgrading PMM from version 2.39.0 to 2.40.0 (not el7) using Docker, the admin user cannot access the PMM UI.

Solution: To fix the problem and gain back admin access to the PMM interface execute the following:

# psql -U grafana
grafana=> update "user" set id='1' where login='admin';
UPDATE 1
grafana=> \q

# grafana cli --homepath=/usr/share/grafana --config=/etc/grafana/grafana.ini admin reset-admin-password <PASS>

Configuration issues¶

This section focuses on configuration issues, such as PMM-agent connection, adding and removing services for monitoring, and so on.

Client-server connections¶

There are many causes of broken network connectivity.

The container is constrained by the host-level routing and firewall rules when using using Docker. For example, your hosting provider might have default iptables rules on their hosts that block communication between PMM Server and PMM Client, resulting in DOWN targets in VictoriaMetrics. If this happens, check the firewall and routing settings on the Docker host.

PMM can also generate diagnostics data that can be examined and/or shared with our support team to help solve an issue. You can get collected logs from PMM Client using the pmm-admin summary command.

Logs obtained in this way include PMM Client logs and logs received from the PMM Server, and stored separately in the client and server folders. The server folder also contains its client subfolder with the self-monitoring client information collected on the PMM Server.

Beginning with PMM 2.4.0, there is a flag that enables the fetching of pprof debug profiles and adds them to the diagnostics data. To enable, run pmm-admin summary --pprof.

Accessing logs and PMM Server data¶

Download logs and components configuration to troubleshoot any issues with the PMM Server:

Through direct URL, by visiting https://<address-of-your-pmm-server>/logs.zip.
By calling the Logs endpoint, which enables you to customize the log content using the line-count parameter: For example:
Default 50,000 lines: https://<pmm-server>/logs.zip
Custom number of lines: https://<pmm-server>/logs.zip?line-count=10000
Unlimited, full log: https://<pmm-server>/logs.zip?line-count=-1
Through the UI, by selecting the Help > PMM Logs option from the main menu.

If you need to share logs with Percona Support via an SFTP server, you can also use the PMM Dump option from the Help menu to generate a compressed tarball file containing a comprehensive export of your PMM metrics and QAN data. For more information, see Export PMM data with PMM Dump topic in the product documentation.

Connection difficulties¶

Passwords

When adding a service, the host might not be detected if the password contains special symbols (e.g., @, %, etc.).

In such cases, you should convert any password, replacing special characters with their escape sequence equivalents.

One way to do this is to use the encodeURIComponent JavaScript function in your browser’s web console (commonly found under a Development Tools menu). Run the function with your password as the parameter. For example:

> encodeURIComponent("s3cR#tpa$$worD")

will give:

"s3cR%23tpa%24%24worD"

Password change

When adding clients to the PMM server, you use the admin user. However, if you change the password for the admin user from the PMM UI, then the clients will not be able to access PMM due to authentication issues. Also, Grafana will lock out the admin user due to multiple unsuccessful login attempts.

In such a scenario, use API key for authentication. You can use API keys as a replacement for basic authentication.

Percona Alerting¶

No Alert rule templates tab on the Alerting page¶

Percona Alerting option isn’t active.

Go to Configuration → Settings → Advanced Settings.
Enable Alerting.

Custom alert rule templates not migrated to Percona Alerting¶

If you have used Integrated Alerting in previous PMM versions, and had custom templates under /srv/ia/templates, make sure to transfer them to /srv/alerting/templates. PMM is no longer sourcing templates from the ia folder, since we have deprecated Integrated Alerting with the 2.31 release.

Unreachable external IP addresses¶

If you get an email or page from your system that the IP is not reachable from outside my organization, do the following:

To configure your PMM Server’s Public Address, select Configuration → Settings → Advanced Settings, and supply an address to use in your alert notifications.

Alert Rule Templates are disabled¶

Built-In alerts are not editable, but you can copy them and edit the copies. (In PMM 2.14.0 and above).

If you create a custom alert rule template, you will have access to edit.

QAN issues¶

This section focuses on problems with QAN, such as queries not being retrieved so on.

Missing data¶

Why don’t I see any query-related information?

There might be multiple places where the problem might come from:

Connection problem between pmm-agent and pmm-managed
PMM-agent cannot connect to the database.
Data source is not properly configured.

Why don’t I see the whole query?

Long query examples and fingerprints can be truncated to 1024 symbols to reduce space usage. In this case, the query explains section will not work.

Plugins issues¶

PMM does not allow to install, upgrade or remove plugins

Users have encountered issues with installing, updating and removing plugins from PMM. The cause of this issue is the incorrect permissions assigned to the /srv/grafana/plugins directory. These permissions are preventing the grafana component from writing to the directory.

Solution

Set the ownership on the directory/srv/grafana/plugins to grafana:grafana.

Get expert help¶

If you need assistance, visit the community forum for comprehensive and free database knowledge, or contact our Percona Database Experts for professional support and services.

Community Forum Get a Percona Expert