Digging around I can't find a fix. I no longer understand how one is supposed to configure grafana to use an https-offloading reverse proxy

This is caused by a change made in v8.0.4 to solve #34537

By default, Grafana now checks request Origin of WebSocket Upgrade request to match root_url option in [server] section. It's also possible to set a list of allowed origin patterns using allowed_origins option in [live] section. See docs, also an ongoing pr to v8 upgrading section - #36661 (UPD. available on doc site)

There is also a known bug #36822 – I am not sure whether it's related to your specific setup though. I need a server warning line to understand, could you guys post warning from server logs which contains Request Origin is not authorized message?

But while this bug exists it's still possible to use [live] allowed_origins option to avoid problems with WebSocket connection.

Thank you! Is fixed now and working fine!

I receiving also this msg with Grafana 8.1.1, but in my case I have 400 not 403 errors in Grafana log:

t=2021-08-19T11:55:03+0200 lvl=info msg="Request Completed" logger=context userId=4 orgId=1 uname=tomas.rohlik@EXAMPLE.COM method=GET path=/api/live/ws status=400 remote_addr=50.55.50.50 time_ms=0 size=12 referer=

In Chrome console is error visible as:

WebSocket connection to 'wss://grafana-dev.na.example.com:8443/api/live/ws' failed: centrifuge.js:544

I have this in my grafana.ini:

[live]
allowed_origins = "https://grafana-dev.na.example.com:8443"

Can you tell me why this error occuring or can I ignore it? 🤔

@Rohlik hello, 400 in most cases caused by reverse proxy. Are you using Nginx? In this case you need to enable WebSocket proxy support (since Nginx can't proxy WebSocket by default) as shown here.

@FZambia Thank you for that. Yes, we are using Apache as reverse proxy. Now, because of you I know what I need to fix. Thanks once more time 👍🏽 .

Solution for Apache2 - use this in your virtualhost configuration:

    RewriteEngine On
    RewriteCond %{HTTP:Upgrade} =websocket [NC]
    RewriteRule /(.*)           ws://localhost:3000/$1 [P,L]

Hi, I'm getting the WebSocket connection to 'wss://my.hostname.com/api/live/ws' failed: centrifuge.js:544

Grafana version: OSS 8.2.2 installed through apt on Ubuntu server

I'm using nginx as reverse proxy with letsencrypt.

grafana log file:

t=2021-10-28T13:43:05+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=//ws status=404 remote_addr=$some_ip_address time_ms=2 size=30366 referer=

note: the path=//ws looks weird; maybe it's normal ?

nginx map added in nginx.conf inside http {}

map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

nginx virtualhost:

server {
  server_name my.hostname.com;
  
  root /usr/share/nginx/html;
  index index.html index.htm;

  location / {
    proxy_pass http://127.0.0.1:3000/;
  }

  location /api/live {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $http_host;
    proxy_pass http://127.0.0.1:3000/;
  }

  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/my.hostname.com/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/my.hostname.com/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
  if ($host = my.hostname.com) {
    return 301 https://$host$request_uri;
  } # managed by Certbot


  listen 80;
  server_name my.hostname.com;
  return 404; # managed by Certbot
}

Grafana config:

[server]
protocol = http

http_port = 3000

domain = my.hostname.com

root_url = https://my.hostname.com/

[live]
max_connections = 100
allowed_origins = https://my.hostname.com

what could be wrong?

@gioy hello, path=//ws seems weird to me too (path should be /api/live/ws) – looks like some URL path rewrite happens on the way to Grafana, but looking at provided Nginx conf it's not not obvious to me why this could happen.

I had the same issue, this is what finally worked for me:

  location /grafana/ {
    proxy_pass http://localhost:3000/;
  }

  location /grafana/api/live/ws {
    rewrite  ^/grafana/(.*)  /$1 break;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $http_host;
    proxy_pass http://localhost:3000/;
  }

@Niek thank you, it worked; the extended location with the rewrite made the trick. Since I'm using a subdomain url instead of a subpath (like you do) here's my working nginx configuration:

  location / {
    proxy_pass http://127.0.0.1:3000/;
  }

  location /api/live/ws {
    rewrite  ^/(.*)  /$1 break;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $http_host;
    proxy_pass http://127.0.0.1:3000/;
  }

@gioy could you try your original configuration but without trailing slash in the proxy_pass upstream, i.e.:

  location /api/live {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $http_host;
    proxy_pass http://127.0.0.1:3000;
  }

On subpath rewrite is still required, but for subdomain this should do the trick.

I'm also using Apache, but @Rohlik 's solution above didn't work for me, possibly because we have Grafana in sub path and using SSL. This worked, however:

<Location /grafana>
      # destination directory
      ProxyPreserveHost On
      ProxyPass http://127.0.0.1:3000
      ProxyPassReverse http://127.0.0.1:3000
      RequestHeader set X-Forwarded-Proto 'https'
      Header set X-Frame-Options SAMEORIGIN

      Order deny,allow
      Allow From all
</Location>
<Location /grafana/api/live/ws>
      #Deal with websocket for live updates
      ProxyPreserveHost On
      ProxyPass ws://127.0.0.1:3000/grafana/api/live/ws
      ProxyPassReverse ws://127.0.0.1:3000/grafana/api/live/ws
</Location>

I also enabled

serve_from_sub_path = true

in grafana.ini but I am not 100% certain it is necessary.

WebSocketClient.js:16 WebSocket connection to 'wss://3000-nazar2343-testproject-94kvcv2mnk8.ws-eu38.gitpod.io:3000/ws' failed: Як вирішити проблему поможіть пліз

Same issue for Grafana 8.4.4

centrifuge.js:585 WebSocket connection to 'wss://xxxx-proxy-grafana.xxxxx.com/api/live/ws' failed:

centrifuge.js:585
centrifuge.js:1029
centrifuge.js:744

@nazar2343 @SincerelyUnique hello, WebSocket error usually happens due to 2 possible reasons:

1. Origin check - this is usually 403 status code and server logs have detailed origin failure description
2. Reverse proxy server/load balancer not configured to proxy WebSocket connections. For example, Nginx configuration described here. This is usually 400 status code.

Hi.

I know this is closed, I have trying to establish connection to wss using apache httpd on Oracle Linux 8 and i keep getting the wss failed warning when I inspect and my application is running using wildfly and it is accessible securely using https but I am unable to access web socket securely with https.

If you are using nginx proxy to serve grafana then you can use below configuration to solve web socket issue.

http {
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

upstream grafana {
    server 127.0.0.1:3000;
}

server {
    listen 8000;

    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $http_host;
        proxy_pass http://grafana;
    }
}

}

I am posting my solution for a Kubernetes deployed Grafana where I got 400 for the socket connections.

I have HAProxy terminate SSL before the request gets into the Kubernetes NGINX Inc ingress controller.

I added this annotation to my ingress:

nginx.org/server-snippets: |
      location /api {
        rewrite  ^/(.*)  /$1 break;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $http_host;
        proxy_pass http://loki-grafana.monitoring.svc.cluster.local;
      }

loki-grafana.monitoring.svc.cluster.local is the internal DNS for the service.

Now I get 101.

I got same errors.