IBM Spectrum Scale Object Multi-region Configuration

Disclaimer: The content of this post is not approved nor endorsed by IBM.

Intro:

This is a quick start guide for enabling multi-region support for IBM Spectrum Scale Object. Multiple Object regions provide both failover capability in the event that one region fails, as well as to help alleviate data latency problems that arise when object data is accessed over the WAN.  In this tutorial, we will demonstrate how to configure three different Object regions, where each region represents a different Spectrum Scale (GPFS) cluster. Ideally, each Object region will be located in a different geographical area so that object data requests are handled in their respective regions in order to provide optimal data access performance for the end user or application.

Agenda:

  • IBM Spectrum Scale Resources
  • Test Environment / Prerequisites
  • Enable Object mult-iregion support
  • Validate Object mult-iregion configuration

IBM Spectrum Scale Resources:


Test Environment:

For this tutorial, we will use three existing Spectrum Scale (GPFS) clusters in order to configure three distinct Swift Object regions.  Each cluster will have Spectrum Scale 4.2.0 with Object installed. For steps on installing and configuring IBM Spectrum Scale, you can reference Configuring Object Using IBM Spectrum Scale . You can also reference the installation and deployment guides references in the Resources section above.

The three clusters consist of the following object protocol nodes:

Primary Region:

Node  Daemon node name            IP address       CES IP address list
———————————————————————–
17   c40bbc2xn3.gpfs.net         192.168.40.73    192.168.11.5 192.168.11.7
25   c40bbc3xn7.gpfs.net         192.168.40.91    192.168.11.6 192.168.11.8

Secondary Region:

Node  Daemon node name            IP address       CES IP address list
———————————————————————–
19   c40abc1pn3.gpfs.net         192.168.40.3     192.168.6.12 192.168.6.13

Third Region:

Node  Daemon node name            IP address       CES IP address list
———————————————————————–
4   c80f3m5n15.gpfs.net         192.168.80.155   192.168.6.97 192.168.6.98
6   c80f3m5n17.gpfs.net         192.168.80.157   192.168.6.99


 

Enabling Multiregion support for primary region

To setup a multiregion environment, a primary region/cluster needs to be enabled for multiregion support. Multiregion can be enabled during installation using the installation toolkit. Multiregion can also be enabled when manually configuring object using the mmobj command. Lastly, if object is already installed and configured, then ‘mmobj multiregion enable’ can be executed to enable multiregion for the primary region. Note: only the primary cluster/region can be enabled for multiregion access after the initial installation/configuration. Secondary Spectrum Scale clusters which you wish to add to the mutiregion environment must be installed with multiregion support. Otherwise, if the secondary regions were configured without multiregion support to begin with, then Object will need to be disabled on those clusters and reconfigured with multiregion support. In this tutorial, we will assume that multiregion was not enabled during installation for the primary region/cluster and so we will enable multiregion support as follows:

c40bbc2xn3 is a ces server from the primary region

#00:49:32# c40bbc2xn3:~ # mmobj multiregion enable
mmobj multiregion: Multi-region support is enabled in this cluster.  Region number: 1

If the command succeeds, you should expect a message indicating that multregion support was enabled for Region number 1. You can confirm that multiregion support is properly enabled for the primary region by running ‘mmobj multiregion list’:

#00:49:44# c40bbc2xn3:~ # mmobj multiregion list
Region  Endpoint   Cluster Name         Cluster Id
——  ———  ——————-  ——————–
1       RegionOne  pok_x86.gpfs.net     14008063887730057712

Next, we need to export the primary region’s configuration data:

#17:28:23# c40bbc2xn3:~ # mmobj multiregion export –region-file /tmp/region1.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region1.dat
mmobj multiregion: Region 1 checksum is: 04217-13711

/tmp/region1.dat will need to be manually copied to region2 and region3. You can use scp or you can copy the file to a network location that is accessible to both regions. Passwordless ssh access across the different regions is not a requirement since the synching process is a manual step that is handled by the user.

#17:33:13# c40bbc2xn3:~ # scp /tmp/region1.dat c40abc1pn3:/tmp/region1.dat
region1.dat                                                                                                                        100%  490KB 490.0KB/s   00:00
#17:33:27# c40bbc2xn3:~ # scp /tmp/region1.dat c80f3m5n15:/tmp/region1.dat
root@c80f3m5n15’s password:
region1.dat

Note: region1.dat only needs to be copied to one of the ces nodes in the secondary regions and not all nodes.

Adding Additional Regions to the Multi-region Environment

Now that multregion support has been successfully enabled in the primary region, we can proceed to adding our two additional regions to the existing multiregion environment. We assume that Object has been installed but as mentioned in the previous section, we need to configure/reconfigure Object with multiregion support for each additional cluster using the primary region’s configuration data. So if Object is already configured, then it needs to be first disabled by running ‘mmces service disable OBJ’

Note: Only a single keystone server is supported in a multiregion environment and so this means the secondary regions must be configured to use an existing, external keystone server using the –remote-keystone-url flag when running the ‘mmobj swift base’ command

Configuring Region 2:

c40abc1pn3 is a ces server from region 2

#18:19:26# c40abc1pn3:/tmp # /usr/lpp/mmfs/bin/mmobj swift base -g /gpfs/objfs  –cluster-hostname cesobjnode –remote-keystone-url ‘http://cesobjnode:35357/v3’ –configure-remote-keystone –join-region-file /tmp/region1.dat –region-number 2 –admin-password Passw0rd
mmobj swift base: Validating execution environment.
mmobj swift base: Creating fileset /dev/objfs object_fileset.
mmobj swift base: Validating Keystone environment.
mmobj swift base: Configuring Swift services.
mmcesobjmr: Multi-region support is enabled in this cluster.  Region number: 2
mmobj swift base: Setting cluster attribute object_singleton_node=192.168.6.12.
mmobj swift base: Uploading configuration changes to the CCR.
mmobj swift base: Configuration complete.

We are now ready to enable the Object service by running ‘mmces service enable OBJ’

Next, we need to export Region 2’s configuration data and sync it with the primary region:

#21:38:40# c40abc1pn3:~ # mmobj multiregion export –region-file /tmp/region2.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region2.dat
mmobj multiregion: Region 2 checksum is: 03405-63053

#21:51:45# c40abc1pn3:~ # scp /tmp/region2.dat c40bbc2xn3:/tmp/region2.dat
region2.dat

#22:16:05# c40bbc2xn3:/tmp # mmobj multiregion import –region-file /tmp/region2.dat
mmobj multiregion: Importing region checksum 03405-63053.
[I] Building ring file: account.builder
[I] Building ring file: container.builder
[I] Building ring file: object.builder
[I] Building ring file: object-14001511240.builder
[I] The existing ring files are up-to-date and a rebuild is not necessary.
mmobj multiregion: The region config has been updated.
mmobj multiregion: Region 1 checksum is: 48377-15827
mmobj multiregion: The configuration of this region has updated information that must
be propagated to the other regions using the ‘mmobj multiregion’ command.

We should now see two regions in our multi-region environment:

#22:19:59# c40bbc2xn3:/tmp # mmobj multiregion list
Region  Endpoint   Cluster Name        Cluster Id
——  ———  ——————  ——————–
1       RegionOne  pok_x86.gpfs.net    14008063887730057712
2       Region2    pok_ppc64.gpfs.net  6012206590852568800

Now we must export the updated primary region’s configuration data and copy it to region 3:

#22:25:02# c40bbc2xn3:/tmp # mmobj multiregion export –region-file /tmp/region1.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region1.dat
mmobj multiregion: Region 1 checksum is: 48377-15827

#22:25:22# c40bbc2xn3:/tmp # scp /tmp/region1.dat c80f3m5n15:/tmp/region1.dat
root@c80f3m5n15’s password:
region1.dat

Configuring Region 3:

c80f3m5n15 is a ces server from region 3

#21:45:45# c80f3m5n15:~ # /usr/lpp/mmfs/bin/mmobj swift base -g /gpfs/objfs  –cluster-hostname cesobjnode –remote-keystone-url ‘http://cesobjnode:35357/v3’ –configure-remote-keystone –join-region-file /tmp/region1.dat –region-number 3 –admin-password Passw0rd
mmobj swift base: Validating execution environment.
mmobj swift base: Creating fileset /dev/objfs object_fileset.
mmobj swift base: Validating Keystone environment.
mmobj swift base: Configuring Swift services.
mmcesobjmr: Multi-region support is enabled in this cluster.  Region number: 3
mmobj swift base: Setting cluster attribute object_singleton_node=192.168.6.98.
mmobj swift base: Uploading configuration changes to the CCR.
mmobj swift base: Configuration complete.

We are now ready to enable the Object service by running ‘mmces service enable OBJ’

Next, we need to export Region 3’s configuration data and copy it to the primary region:

#21:56:02# c80f3m5n15:~ # mmobj multiregion export –region-file /tmp/region3.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region3.dat
mmobj multiregion: Region 3 checksum is: 38710-02214

#21:56:10# c80f3m5n15:~ # scp /tmp/region3.dat c40bbc2xn3:/tmp/region3.dat
region3.dat

At this point, if you run ‘openestack endpoint list’ from the primary region, you should see all three regions listed as shown below:

multiregion_openstack_endpoints

Sync Region 3 with Primary Region:

Since region3 has the most up to date configuration data, we need to synch it back with region1:

#22:57:03# c40bbc2xn3:/tmp # mmobj multiregion import –region-file /tmp/region3.dat
mmobj multiregion: Importing region checksum 04215-46679.
[I] Building ring file: account.builder
[I] Building ring file: container.builder
[I] Building ring file: object.builder
[I] Building ring file: object-14001511240.builder
mmobj multiregion: The region config has been updated.
mmobj multiregion: Region 1 checksum is: 04215-46679

We should now have all three regions in our multi-region environment:

#22:57:32# c40bbc2xn3:/tmp # mmobj multiregion list
Region  Endpoint   Cluster Name        Cluster Id
——  ———  ——————  ——————–
1       RegionOne  pok_x86.gpfs.net    14008063887730057712
2       Region2    pok_ppc64.gpfs.net  6012206590852568800
3       Region3    omnipath.gpfs.net   6788346196758363892

Finally, we need to synch the primary region with region2 so that region2 can become aware of region3:

#23:00:09# c40bbc2xn3:/tmp # mmobj multiregion export –region-file /tmp/region1.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region1.dat
mmobj multiregion: Region 1 checksum is: 04215-46679

#23:00:21# c40bbc2xn3:/tmp # scp /tmp/region1.dat c40abc1pn3:/tmp/region1.dat
region1.dat

#23:01:35# c40abc1pn3:~ # mmobj multiregion import –region-file /tmp/region1.dat
mmobj multiregion: Importing region checksum 04215-46679.
[I] Building ring file: account.builder
[I] Building ring file: container.builder
[I] Building ring file: object.builder
[I] Building ring file: object-14001511240.builder
mmobj multiregion: The region config has been updated.
mmobj multiregion: Region 2 checksum is: 37509-20319
mmobj multiregion: The configuration of this region has updated information that must
be propagated to the other regions using the ‘mmobj multiregion’ command.

At this point, all three regions should be synched. You should be able to list, create and download containers and objects from any of the regions.