feature request: have option to use ipv4 when setting up network configuration for octavia

I am currently trying to get Octavia to work. Does Octavia not work for you when you try to use the default ipv6 subnet?

Could you please explain how you create an IPv4 subnet for Octavia?

Thanks

I personally have not tested the Octavia Charm with it's default IPv6 network setup.
I can elaborate on creating an IPv4 network with Octavia.

So by default the Octavia charm uses IPV6. We will need to delete the IPv6 and add an ipv4 network.

# delete any ipv6 related network with octavia
openstack network list
openstack subnet list
openstakc network show lb-mgmt
openstack router remove subnet lb-mgmt lb-mgmt-subnetv6
openstack subnet delete lb-mgmt-subnetv6

#create ipv4 octavia network, subnet, router, add subnet to router
openstack network create lb-mgmt-net --tag charm-octavia
openstack subnet create --tag charm-octavia --subnet-range 10.0.0.3/24 --dhcp --ip-version 4 --network lb-mgmt-net lb-mgmt-subnet
openstack router create lb-mgmt --tag charm-octavia
openstack router add subnet lb-mgmt lb-mgmt-subnet

#add security rules
openstack security group create lb-mgmt-sec-grp --tag charm-octavia
openstack security group create lb-health-mgr-sec-grp --tag charm-octavia-health
openstack security group rule create lb-mgmt-sec-grp --protocol icmp
openstack security group rule create lb-mgmt-sec-grp --protocol tcp --protocol tcp --dst-port 22
openstack security group rule create lb-mgmt-sec-grp --protocol tcp --dst-port 9443

#create loadbalancer
openstack loadbalancer create --name lb0 --vip-network-id lb-mgmt

# You may or may not need to do this part.
# create temp novarc.services and extract octavia password
touch /tmp/novarc.services

cat << EOF > /tmp/novarc.services
OS_PROJECT_DOMAIN_NAME=service_domain
OS_USERNAME=octavia
OS_PROJECT_NAME=services
OS_USER_DOMAIN_NAME=service_domain
OS_PASSWORD=$(juju run --unit octavia/0 "grep -v "auth" /etc/octavia/octavia.conf | grep password" | awk '{print $3}')
EOF
source /tmp/novarc.services

I had issues where when creating the loadbalancers it would be creating them from the admin area and not the services area so it would fail.

Thanks v.much Heather! That's very helpful. I'm going to be trying this next week.

The charm already allows you to manage the cloud resources manually by setting the `create-mgmt-network` configuration option to 'false'. Please refer to the Octavia LBaaS appendix in OpenStack Charm Deployment guide [0] and the charm documentation [1] and overview of configuration options [2].

If you have issues with the tunneled communication from the Octavia units to the cloud the root cause of that is most likely something else than the address family being used inside the tunnel.

0: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-octavia.html#resource-configuration
1: https://charmhub.io/octavia
2: https://charmhub.io/octavia/configure

We are using bind on our MAAS server as our DNS server.

If I have the line:

listen-on-v6 { any; };

Uncommented in /etc/bind/named.conf.options on our MAAS controller, juju doesn't work.

Our OS install is out of action so I can't check at the moment but did you have to create the service_domain domain with an octavia user in it or does the Octavia charm do that? I presume I can assign it to any domain/project/user.

I presume multiple projects/users can use Octavia?

The internal communication between the Octavia worker and its amphorae has nothing to do with your physical network, MAAS or Juju configuration. The octavia units participate in the overlay network in your OpenStack cloud and the internal communication is tunneled through that.

From what I remember the octavia user should be created by the charm

| I presume multiple projects/users can use Octavia?
  Do you mean the use the Octavia user? Or just Octavia in general?

In general yeah multiple projects/users can use octavia

Hi Heather and Frode

I have now redeployed the OS bionic-train telemetry bundle but this time I set

create-mgmt-network: false

and then I've used exactly the same settings as you to create an IPv4 Octavia management network but I've not been able to successfully create a LB yet.

I'm pretty sure your create a lb command above was supposed to be:

openstack loadbalancer create --name lb0 --vip-network-id lb-mgmt-net

That command errors out pretty quickly for me, as does:

lb_vip_port_id=$(openstack loadbalancer create -f value -c vip_port_id --name lb3 --vip-subnet-id lb-mgmt-subnet)

I am running these commands when authenticated as the master admin OS user. I am presuming there is no need to add any octavia policy roles to the admin user to create lbs is there?

I've not tried sourcing your novarc.services yet because I'm unsure how you intend me to use that? Most of its vars clash with those required for me to use OS. Do you source that before creating lbs? Do your source it after you source your regular OS users credentials?

In /var/log/octavia/octavia-worker.log I can see several errors like this:

2020-12-17 21:03:38.830 215189 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2020-12-17T21:03:35Z', 'message': 'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 90b77330-c889-48ed-b4dd-8f50d040a8d5. Last exception: Security group 7486748f-d19a-463b-a065-3ca48e6d31c4 not found.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 637, in build_instances\n filter_properties, instances[0].uuid)\n File "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 895, in populate_retry\n raise exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 90b77330-c889-48ed-b4dd-8f50d040a8d5. Last exception: Security group 7486748f-d19a-463b-a065-3ca48e6d31c4 not found.\n'}

Those security groups that it says don't exist do.

$ openstack security group list
+--------------------------------------+-----------------------+------------------------+----------------------------------+--------------------------+
| ID | Name | Description | Project | Tags |
+--------------------------------------+-----------------------+------------------------+----------------------------------+--------------------------+
| 065f6c3c-00ef-4bb4-9d20-8ebf835b4549 | default | Default security group | 9dcc7bbb0d93493bad74d4f9f84b6df7 | [] |
| 07bd80a2-3d01-41aa-aad0-eba32ed1d57f | default | Default security group | | [] |
| 7486748f-d19a-463b-a065-3ca48e6d31c4 | lb-mgmt-sec-grp | lb-mgmt-sec-grp | 8138d057750d4e0d8fd359965979227f | ['charm-octavia'] |
| 89600e62-dd30-463d-9859-2b053098accd | lb-health-mgr-sec-grp | lb-health-mgr-sec-grp ...

"I had issues where when creating the loadbalancers it would be creating them from the admin area and not the services area so it would fail."

When you say "admin area", what do you mean? The admin project or the admin internal network / subnet?

Which versions of OS and Ubuntu are you using Heather?

Do you source that before creating lbs? Do your source it after you source your regular OS users credentials?
Yes I do source novarc.services before creating the loadbalancers.

by "admin area" i mean the admin network.

This is your problem:
I am running these commands when authenticated as the master admin OS user.
You need to be creating the loadbalancer with an octavia user

This error:
   Last exception: Security group 7486748f-d19a-463b-a065-3ca48e6d31c4 not found.'

Tells you that the octavia network isn't being found/used by the "services" project

can you run `openstack project list` and do you see one called services? cause that's where all of the networking and configuration for octavia needs to come from, not the admin anything

What does your octavia security groups look like?
Here's an example to setup security groups with the "services"
----
openstack security group rule add 4b965a7d-b45e-4566-b35e-c9152d17b35e --protocol icmp --project services
openstack security group rule create 4b965a7d-b45e-4566-b35e-c9152d17b35e --protocol icmp --project services
openstack security group rule add 4b965a7d-b45e-4566-b35e-c9152d17b35e --protocol tcp --dst-port 443 --project services
openstack security group rule create 4b965a7d-b45e-4566-b35e-c9152d17b35e --protocol tcp --dst-port 9443 --project services
openstack security group rule create lb-mgmt-sec-grp --protocol tcp --protocol tcp --dst-port 22 --project services
juju ssh octavia/0 sudo -s
openstack server list --all-projects
openstack security group rule create 350af0bc-a59c-465c-8645-ac6c724a2ed4 --ingress --remote-ip 0.0.0.0 --project services
 openstack security group rule create e5847ca9-30c5-4349-86bf-3f9666595a0d --egress --remote-ip 0.0.0.0 --project services
openstack security group rule list 350af0bc-a59c-465c-8645-ac6c724a2ed4
 openstack security group rule create 350af0bc-a59c-465c-8645-ac6c724a2ed4 --egress --remote-ip 0.0.0.0/0 --project services
juju ssh octavia/0 sudo reboot
juju ssh octavia/0 ip a s o-hm0

distro info
----
cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

---
hope this helps

Have a happy holidays :)

Thanks for that info

After installing the telemetry bundle and telling the octavia charm not to create any networks, I have 2 projects called services and 2 called admin.

How to know which is the correct one to add the security rules to? I could add the rules to both but I'd like to know which is the one octave will use.

$ openstack project list
+----------------------------------+----------+
| ID | Name |
+----------------------------------+----------+
| 467ce649c1e24828b1f9d365509e1a08 | services |
| 8138d057750d4e0d8fd359965979227f | admin |
| 9dc00db9733049cc91c5297e41c6d522 | admin |
| 9dcc7bbb0d93493bad74d4f9f84b6df7 | services |
+----------------------------------+----------+

$ openstack project show 467ce649c1e24828b1f9d365509e1a08
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Created by Juju |
| domain_id | default |
| enabled | True |
| id | 467ce649c1e24828b1f9d365509e1a08 |
| is_domain | False |
| name | services |
| options | {} |
| parent_id | default |
| tags | [] |
+-------------+----------------------------------+
$ openstack project show 9dcc7bbb0d93493bad74d4f9f84b6df7
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Created by Juju |
| domain_id | 8d54cf927e0f40bca220a1403e857803 |
| enabled | True |
| id | 9dcc7bbb0d93493bad74d4f9f84b6df7 |
| is_domain | False |
| name | services |
| options | {} |
| parent_id | 8d54cf927e0f40bca220a1403e857803 |
| tags | [] |
+-------------+----------------------------------+

juju ssh octavia/0 ip a s o-hm0

I should have a network interface on octavia called o-hm0?

| should have a network interface on octavia called o-hm0?
- No, you don't need a network interface called o-hm0. that was a typo on my part, it's just for confirmation that an ipv4 network address was being used by octavia.

Hmm... I only have one network and one services project.

Could you run `openstack role assignment list --names`
this tells us which projects and roles are assigned

Keeping in mind that I have since tried to run:

juju run-action --wait octavia/0 configure-resources

after briefly setting:

juju config octavia create-mgmt-network=true

but, to my total lack of surprise, configure-resources failed to complete without the ipv4 networks being pre-prepared so I am back to working on getting Octavia working with IPv4.

I noticed that juju doesn't seem too aware of the octavia management networks because I deleted these ipv4 ones to try ipv6 again but juju still claimed octavia was active.

So, this output is likely to some roles created by an unsuccessful run of configure-resources

$ openstack role assignment list --names
+-------------------------------+-------------------------------------------+-------+-------------------------+--------------+--------+-----------+
| Role | User | Group | Project | Domain | System | Inherited |
+-------------------------------+-------------------------------------------+-------+-------------------------+--------------+--------+-----------+
| Admin | cinderv2_cinderv3@service_domain | | services@service_domain | | | False |
| Admin | neutron@service_domain | | services@service_domain | | | False |
| Admin | octavia@Default | | services@Default | | | False |
| load-balancer_observer | octavia@Default | | services@Default | | | False |
| load-balancer_admin | octavia@Default | | services@Default | | | False |
| load-balancer_quota_admin | octavia@Default | | services@Default | | | False |
| load-balancer_member | octavia@Default | | services@Default | | | False |
| load-balancer_global_observer | octavia@Default | | services@Default | | | False |
| Admin | gnocchi@service_domain | | services@service_domain | | | False |
| Admin | admin@admin_domain | | admin@admin_domain | | | False |
| member | admin@admin_domain | | admin@admin_domain | | | False |
| Admin | admin@admin_domain | | | admin_domain | | False |
| Admin | nova@Default | | services@Default | | | False |
| Admin | ceilometer@service_domain | | services@service_domain | | | False |
| ResellerAdmin ...

Yeah i actually got the same error. My coworker is currently looking into why configure-resources is failing

configure-resources is failing

related configure-resources error message failure

+ juju run-action octavia/leader configure-resources --wait
unit-octavia-0:
  UnitId: octavia/0
  id: "2"
  message: cannot unpack non-iterable NoneType object
  results: {}
  status: failed
  timing:
    completed: 2020-12-18 17:35:32 +0000 UTC
    enqueued: 2020-12-18 17:35:05 +0000 UTC
    started: 2020-12-18 17:35:26 +0000 UTC

I'm still scratching my head over this and wondering how anybody has ever got Octavia to work.

I'm still uncertain as to which of my two services projects is the relevant one. I tried looking at the output of various versions of `openstack role assignment list` (eg listing by project etc) but there were no give aways.

I have several questions regarding your novarc.services file which you say you need to run before creating LBs. You are the only person / guide / resource who has mentioned the need for a creating specific RC file for these purposes. Which guide or document contains this info? There is no mention of having to create such an rc file here:

https://docs.openstack.org/octavia/latest/contributor/guides/dev-quick-start.html

Nor is any such file mentioned in the docs for the Octavia charm. Additionally, there are a few issues with it so I'm wondering if you copy/pasted an old (unworking) version? The command to fetch the Octavia password with juju doesn't work but if I replace that with my octavia password then source it I get:

Missing value auth-url required for auth plugin password

So, I've tried adding

export OS_AUTH_URL=<URLTOKEYSTONESERVER>

I'm not really impressed with

https://docs.openstack.org/octavia/latest/contributor/guides/dev-quick-start.html

as its overly vague for my liking. It starts like this:

"Create Octavia User

By default Octavia will use the ‘octavia’ user for keystone authentication, and the admin user for interactions with all other services.

You must:

    Create ‘octavia’ user."

Create octavia user where? How? What sort of user (Openstack? Linux? Octavia? Keystone)? With what command? The rest of the document isn't any better.

I've tried using:

export OS_AUTH_URL=

Using the URL for Keystone and the URL for the Octavia unit in the novarc.services with no luck.

Here are extracts of the relevant parts of my notes on deploying OpenStack so far. If you can spot any issues or if you have similar notes you could share I'd appreciate your feedback:

First I had to make a few small edits to the bundle.yaml so that it would deploy correctly, then:

Presuming you are currently in the directory of the correct bundle, you can deploy it with the Octava overlay by running:

$ juju deploy ./bundle.yaml --overlay ../overlays/loadbalancer-octavia.yaml

Deploying the Openstack telemetry bundle usually takes just under 40 minutes on our current hardware. Keystone and nova-cloud-controller are two of the last units that will come up as being ready. If nova-cloud-controller fails it could be because you have specified a DNS setting in MAAS for that subnet when it should be blank. You can check the progress of a deployment by running juju status and checking for the workload status of the various apps and units in becoming active and green.

Get login details

$ source openrc
$ env | grep OS_
$ juju status openstack-dashboard

Make a note somewhere secure of OS_PASSWORD and the IP address of the dashboard.

Download an amd64 Ubuntu cloud installation image

$ curl https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img | \
    openstack image create --public --container-format=bare \
        --disk-format=qcow2 focal

Create an external network

A shared external network must be defined in addition to every user/project having their own private network, usually created by the new user script

$ openstack network create --external --provider-network-type flat \
    --provider-physical-network physnet1 ext_net
$ openstack subnet create --subnet-range 172.20.0.0/21 --no-dhcp \
    --gateway 172.20.0.50 --network ext_net \
    --allocation-pool start=172.20.1.2,end=172.20.7.254 ext

The address passed to the gateway option is the address of the MAAS controller.

Create an internal network for the admin user

$ openstack network create internal
$ openstack subnet create --network internal \
    --subnet-range 198.51.100.0/24 \
    --dns-nameserver 1.1.1.1 \
    internal_subnet
$ openstack router create provider-router
$ openstack router set --external-gateway ext_net provider-router
$ openstack router add subnet provider-router internal_subnet

Create a flavor

$ openstack flavor create --ram 2048 --disk 20 m1.small

Create security group rules to enable SSH and ping

$ PROJECT_ID=$(openstack project list -f value -c ID \
       --domain admin_domain)
$ SECGRP_ID=$(openstack security group list --project $PROJECT_ID \
    | awk '/default/{print$2}')
$ openstack security group rule create $SECGRP_ID \
    --protocol icmp --ingress --ethertype IPv4
$ openstack security group rule create $SECGRP_ID \
    --protocol icmp --ingress --ethertype IPv6
$ openstack security group rule create $SECGRP_ID \
    --protocol tcp --ingress --ethertype IPv4 --dst-port 22
$ openstack security group rule create $SECGRP_ID \
    --protocol tcp --ingress --ethertype IPv6 --dst-port 22

Fixing Ceilometer and Ceph

If you get the error Services not running that should be: haproxy for ceph-radosgw/0, simply SSH into it and...

I shouldn't have cut this bit out:

 Edit ~/openstack-bundles/development/overlays/loadbalancer-octavia.yaml to ensure that both of the openstack-origin options match the version of openstack you are deploying eg cloud:bionic-train. Also, within the same file, in the options section of the octavia charm (on the line after the second openstack-origin option ) add the line:

create-mgmt-network: false

Hence I didn't need to delete any networks.

Hi Dan,

I hope you had a relaxing holiday.

Yes setting the,
| create-mgmt-network: false

will help out a lot and not having to delete any networks.

One of my coworkers will be better to answer your questions about setting up the Octavia user, he's still on break, but will be back soon.

Hi Heather

I had a good holiday thanks. Happy new year and big thanks to you an your co-worker for helping me get Octavia to work!

I just wanted to clarify the novarc.services file content. The first thing I tried was what you pasted earlier which amounts to:

OS_PROJECT_DOMAIN_NAME=service_domain
OS_USERNAME=octavia
OS_PROJECT_NAME=services
OS_USER_DOMAIN_NAME=service_domain
OS_PASSWORD=$(juju run --unit octavia/0 "grep -v "auth" /etc/octavia/octavia.conf | grep password" | awk '{print $3}')

My guess would be it should look something more like:

export OS_AUTH_URL=http://172.20.2.174:5000/v3
export OS_PROJECT_DOMAIN_NAME=service_domain
export OS_USERNAME=octavia
export OS_PROJECT_ID=9dcc7bbb0d93493bad74d4f9f84b6df7
export OS_PROJECT_NAME=services
export OS_USER_DOMAIN_NAME=service_domain
export OS_PASSWORD=XXX

Where OS_AUTH_URL is either the address of Keystone or maybe the Octavia server, OS_PASSWORD is the auth password contained within /etc/octavia/octavia.conf on the Octavia unit and OS_PROJECT_ID will probably need to be used instead of just the project name because of the duplication of services projects caused by installing the bundle. It'd be good to know a surefire way of identifying the correct project to use.

I was looking at the wrong installation guide for Octavia. This one has more details and does mention creating a rc file for Octavia:

https://docs.openstack.org/octavia/latest/install/install-ubuntu.html#install-and-configure-components

I'm still uncertain how to identify the correct services project to use when there are multiple.

Yeah, I'm not sure on how to identify more than one services project when multiple are being used. Let me ask around and get back to you.

What does this command look like?

openstack role assignment list

openstack role assignment list --names

Just to add that I got this working eventually using the charm default (ipv6) network but using ipv4 everywhere else.

Oh man thats great!

Ah okay interesting, thanks for the update.

If you give this a try tho, openstack role assignment list --names
it should help you sort out where projects are coming from.

A coworker mentioned this too:
create a user with the member role and all the octavia roles (in the default domain) in its own project and in the default domain - then try see if setting up a load balancer as this user

Thanks,
Heather Lemon