Skip to content

Troubleshooting issues

This section is your go-to resource for tackling common issues and finding solutions. For additional troubleshooting tips and known issues, see the Percona Everest Release Notes and known limitations section.

General troubleshooting guidelines

Before troubleshooting, it’s important to understand how Percona Everest works at a high level. Understanding how it works can help narrow down potential areas to investigate.

Important logs and commands

Depending on the specific issue, you can review different logs for additional information.

  1. Everest Core Components

    Logs Command
    Percona operator kubectl logs -f deploy/everest-operator -n everest-system
    Percona Everest server kubectl logs -f deploy/everest-server -n everest-system
  2. Database Operators (in Namespaces)

    Operator type Namespace Command
    PostgreSQL everest kubectl logs -f deploy/percona-postgresql-operator -n everest
    MongoDB everest kubectl logs -f deploy/percona-server-mongodb-operator -n everest
    PXC everest kubectl logs -f deploy/percona-xtradb-cluster-operator -n everest
  3. Monitoring

    Component Command
    VictoriaMetrics Operator kubectl logs -f deploy/vm-operator -n everest-monitoring
    VM Agent kubectl logs -f deploy/vmagent-everest-monitoring -n everest-monitoring
  4. Database Pods and Proxies

    kubectl logs -f <pod-name of database or proxy> -c <database-container>
    
  5. Kubernetes Events

    Important

    Events (Events are stored only for 60 mins, if there are any events which are older than 60 mins, it will be lost).

    kubectl get events --sort-by=".lastTimestamp"
    

Troubleshooting key areas

Installation issues

For troubleshooting Percona Everest installation issues using everestctl or the Helm chart, the following steps may be helpful:

  1. Permissions and privileges

    Appropriate privileges may be required depending on the selected components for installation. For instance, if OLM is to be installed, cluster-admin privileges are required. If any of the components fail, verify that the appropriate privileges are granted.

    Run the following command to check if the required privileges are granted:

    kubectl auth can-i
    
  2. Helm Chart validation

    • Verify the installation status of the Helm chart. A properly functioning chart should be in a Deployed status.

      To verify the values used during the chart installation, run the following command:

      helm list -n everest-system
      NAME  NAMESPACE   REVISION  UPDATED                                 STATUS      CHART           APP VERSION
      everest-core everest-system 1  2025-01-16 16:24:56.577713 +0530 
      IST deployed    everest-1.4.0   1.4.0
      
    • As there are many components in the Percona Everest installation, installation will fail if any of the subcomponent installations fail. Check the relevant namespace where components are installed, along with the logs and events.

  3. Resource availability

    When a job is created to approve the installation plan for operators, if the cluster has no available resources to run pods, the Helm installation will wait for the specified --timeout or the default of 5 minutes before failing.

API, Authentication, and frontend issues

To troubleshoot issues with the Percona Everest API, authentication, or frontend, check the everest-server deployment.

  1. Check everest-server Pod Health

    If the Percona Everest API is not working, check the status of the everest-server pod, specifically its Status and Restarts.

    kubectl get po -l app.kubernetes.io/name=everest-server -n everest-system
    
    NAME                            READY STATUS  RESTARTS AGE
    everest-server-78699679d4-kgqk5 1/1   Running  0       4d23h
    
  2. Check the everest-server logs

    kubectl logs -f deploy/everest-server -n everest-system
    
  3. RBAC validation To resolve authentication and access issues, check the Percona Everest server logs. If Role-Based Access Control (RBAC) is implemented, validate or check the permissions using everestctl.

    kubectl get configmap everest-rbac -n everest-system
    
  4. Check the everest-operator logs for any reconciliation errors.

    kubectl logs -f deploy/everest-operator -n everest-system
    
  5. Local access via Port Forwarding

    If you experience any access issues or lag in the Percona Everest frontend or API, try port-forwarding to the service and check the latency compared to accessing it via a LoadBalancer or NodePort.

    kubectl port-forward svc/everest 8080:8080
    
    Once you have set up the port-forward, access the webpage using http://localhost:8080.

Database operation issues

Here are the common issues related to the database operations:

  1. Check the everest-operator logs

    Check the everest-operator logs if the DatabaseCluster object has not been created or if there are any issues with it.

    kubectl logs -f deploy/everest-operator -n everest-system
    
  2. Check the DatabaseCluster object

    Verify the status of DatabaseCluster object and events. The status should be Ready. If it is anything other than Ready, further investigation is required. Describing the DatabaseCluster object provides details about the database configuration.

    kubectl get DatabaseCluster <DatabaseCluster-Name>
    kubectl describe DatabaseCluster <DatabaseCluster-Name>
    kubectl get DatabaseCluster <DatabaseCluster-Name> -oyaml
    
  3. Database objects

    Check the relevant database objects such as PXC, PSMDB, and PG, as well as the operator logs. For instance, check the PXC object followed by the operator logs.

    kubectl get pxc <database-name>
    kubectl describe pxc <database-name>
    kubectl get pxc <database-name> -oyaml 
    kubectl logs -f deploy/percona-xtradb-cluster-operator
    
    # Change to pxc,psmdb,pg for respective database
    
  4. Check the database pod logs

    kubectl logs -f <database-pod-name> -c <database-container-name>  
    
    # (container name could be database,pxc,mongo)
    

Get expert help

If you need assistance, visit the community forum for comprehensive and free database knowledge, or contact our Percona Database Experts for professional support and services.