Disclaimer: The content of this post is not approved nor endorsed by IBM.
Intro:
This is a quick start guide for enabling multi-region support for IBM Spectrum Scale Object. Multiple Object regions provide both failover capability in the event that one region fails, as well as to help alleviate data latency problems that arise when object data is accessed over the WAN. In this tutorial, we will demonstrate how to configure three different Object regions, where each region represents a different Spectrum Scale (GPFS) cluster. Ideally, each Object region will be located in a different geographical area so that object data requests are handled in their respective regions in order to provide optimal data access performance for the end user or application.
Agenda:
- IBM Spectrum Scale Resources
- Test Environment / Prerequisites
- Enable Object mult-iregion support
- Validate Object mult-iregion configuration
IBM Spectrum Scale Resources:
- Spectrum Scale Object Multi-region Overview
- IBM Spectrum Scale FAQ
- IBM Spectrum Scale Installation Guide
- IBM Spectrum Scale Deployment Guide
- IBM Spectrum Scale Installation Demo
Test Environment:
For this tutorial, we will use three existing Spectrum Scale (GPFS) clusters in order to configure three distinct Swift Object regions. Each cluster will have Spectrum Scale 4.2.0 with Object installed. For steps on installing and configuring IBM Spectrum Scale, you can reference Configuring Object Using IBM Spectrum Scale . You can also reference the installation and deployment guides references in the Resources section above.
The three clusters consist of the following object protocol nodes:
Primary Region:
Node Daemon node name IP address CES IP address list
———————————————————————–
17 c40bbc2xn3.gpfs.net 192.168.40.73 192.168.11.5 192.168.11.7
25 c40bbc3xn7.gpfs.net 192.168.40.91 192.168.11.6 192.168.11.8
Secondary Region:
Node Daemon node name IP address CES IP address list
———————————————————————–
19 c40abc1pn3.gpfs.net 192.168.40.3 192.168.6.12 192.168.6.13
Third Region:
Node Daemon node name IP address CES IP address list
———————————————————————–
4 c80f3m5n15.gpfs.net 192.168.80.155 192.168.6.97 192.168.6.98
6 c80f3m5n17.gpfs.net 192.168.80.157 192.168.6.99
Enabling Multiregion support for primary region
To setup a multiregion environment, a primary region/cluster needs to be enabled for multiregion support. Multiregion can be enabled during installation using the installation toolkit. Multiregion can also be enabled when manually configuring object using the mmobj command. Lastly, if object is already installed and configured, then ‘mmobj multiregion enable’ can be executed to enable multiregion for the primary region. Note: only the primary cluster/region can be enabled for multiregion access after the initial installation/configuration. Secondary Spectrum Scale clusters which you wish to add to the mutiregion environment must be installed with multiregion support. Otherwise, if the secondary regions were configured without multiregion support to begin with, then Object will need to be disabled on those clusters and reconfigured with multiregion support. In this tutorial, we will assume that multiregion was not enabled during installation for the primary region/cluster and so we will enable multiregion support as follows:
c40bbc2xn3 is a ces server from the primary region
#00:49:32# c40bbc2xn3:~ # mmobj multiregion enable
mmobj multiregion: Multi-region support is enabled in this cluster. Region number: 1
If the command succeeds, you should expect a message indicating that multregion support was enabled for Region number 1. You can confirm that multiregion support is properly enabled for the primary region by running ‘mmobj multiregion list’:
#00:49:44# c40bbc2xn3:~ # mmobj multiregion list
Region Endpoint Cluster Name Cluster Id
—— ——— ——————- ——————–
1 RegionOne pok_x86.gpfs.net 14008063887730057712
Next, we need to export the primary region’s configuration data:
#17:28:23# c40bbc2xn3:~ # mmobj multiregion export –region-file /tmp/region1.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region1.dat
mmobj multiregion: Region 1 checksum is: 04217-13711
/tmp/region1.dat will need to be manually copied to region2 and region3. You can use scp or you can copy the file to a network location that is accessible to both regions. Passwordless ssh access across the different regions is not a requirement since the synching process is a manual step that is handled by the user.
#17:33:13# c40bbc2xn3:~ # scp /tmp/region1.dat c40abc1pn3:/tmp/region1.dat
region1.dat 100% 490KB 490.0KB/s 00:00
#17:33:27# c40bbc2xn3:~ # scp /tmp/region1.dat c80f3m5n15:/tmp/region1.dat
root@c80f3m5n15’s password:
region1.dat
Note: region1.dat only needs to be copied to one of the ces nodes in the secondary regions and not all nodes.
Adding Additional Regions to the Multi-region Environment
Now that multregion support has been successfully enabled in the primary region, we can proceed to adding our two additional regions to the existing multiregion environment. We assume that Object has been installed but as mentioned in the previous section, we need to configure/reconfigure Object with multiregion support for each additional cluster using the primary region’s configuration data. So if Object is already configured, then it needs to be first disabled by running ‘mmces service disable OBJ’
Note: Only a single keystone server is supported in a multiregion environment and so this means the secondary regions must be configured to use an existing, external keystone server using the –remote-keystone-url flag when running the ‘mmobj swift base’ command
Configuring Region 2:
c40abc1pn3 is a ces server from region 2
#18:19:26# c40abc1pn3:/tmp # /usr/lpp/mmfs/bin/mmobj swift base -g /gpfs/objfs –cluster-hostname cesobjnode –remote-keystone-url ‘http://cesobjnode:35357/v3’ –configure-remote-keystone –join-region-file /tmp/region1.dat –region-number 2 –admin-password Passw0rd
mmobj swift base: Validating execution environment.
mmobj swift base: Creating fileset /dev/objfs object_fileset.
mmobj swift base: Validating Keystone environment.
mmobj swift base: Configuring Swift services.
mmcesobjmr: Multi-region support is enabled in this cluster. Region number: 2
mmobj swift base: Setting cluster attribute object_singleton_node=192.168.6.12.
mmobj swift base: Uploading configuration changes to the CCR.
mmobj swift base: Configuration complete.
We are now ready to enable the Object service by running ‘mmces service enable OBJ’
Next, we need to export Region 2’s configuration data and sync it with the primary region:
#21:38:40# c40abc1pn3:~ # mmobj multiregion export –region-file /tmp/region2.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region2.dat
mmobj multiregion: Region 2 checksum is: 03405-63053
#21:51:45# c40abc1pn3:~ # scp /tmp/region2.dat c40bbc2xn3:/tmp/region2.dat
region2.dat
#22:16:05# c40bbc2xn3:/tmp # mmobj multiregion import –region-file /tmp/region2.dat
mmobj multiregion: Importing region checksum 03405-63053.
[I] Building ring file: account.builder
[I] Building ring file: container.builder
[I] Building ring file: object.builder
[I] Building ring file: object-14001511240.builder
[I] The existing ring files are up-to-date and a rebuild is not necessary.
mmobj multiregion: The region config has been updated.
mmobj multiregion: Region 1 checksum is: 48377-15827
mmobj multiregion: The configuration of this region has updated information that must
be propagated to the other regions using the ‘mmobj multiregion’ command.
We should now see two regions in our multi-region environment:
#22:19:59# c40bbc2xn3:/tmp # mmobj multiregion list
Region Endpoint Cluster Name Cluster Id
—— ——— —————— ——————–
1 RegionOne pok_x86.gpfs.net 14008063887730057712
2 Region2 pok_ppc64.gpfs.net 6012206590852568800
Now we must export the updated primary region’s configuration data and copy it to region 3:
#22:25:02# c40bbc2xn3:/tmp # mmobj multiregion export –region-file /tmp/region1.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region1.dat
mmobj multiregion: Region 1 checksum is: 48377-15827
#22:25:22# c40bbc2xn3:/tmp # scp /tmp/region1.dat c80f3m5n15:/tmp/region1.dat
root@c80f3m5n15’s password:
region1.dat
Configuring Region 3:
c80f3m5n15 is a ces server from region 3
#21:45:45# c80f3m5n15:~ # /usr/lpp/mmfs/bin/mmobj swift base -g /gpfs/objfs –cluster-hostname cesobjnode –remote-keystone-url ‘http://cesobjnode:35357/v3’ –configure-remote-keystone –join-region-file /tmp/region1.dat –region-number 3 –admin-password Passw0rd
mmobj swift base: Validating execution environment.
mmobj swift base: Creating fileset /dev/objfs object_fileset.
mmobj swift base: Validating Keystone environment.
mmobj swift base: Configuring Swift services.
mmcesobjmr: Multi-region support is enabled in this cluster. Region number: 3
mmobj swift base: Setting cluster attribute object_singleton_node=192.168.6.98.
mmobj swift base: Uploading configuration changes to the CCR.
mmobj swift base: Configuration complete.
We are now ready to enable the Object service by running ‘mmces service enable OBJ’
Next, we need to export Region 3’s configuration data and copy it to the primary region:
#21:56:02# c80f3m5n15:~ # mmobj multiregion export –region-file /tmp/region3.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region3.dat
mmobj multiregion: Region 3 checksum is: 38710-02214
#21:56:10# c80f3m5n15:~ # scp /tmp/region3.dat c40bbc2xn3:/tmp/region3.dat
region3.dat
At this point, if you run ‘openestack endpoint list’ from the primary region, you should see all three regions listed as shown below:
Sync Region 3 with Primary Region:
Since region3 has the most up to date configuration data, we need to synch it back with region1:
#22:57:03# c40bbc2xn3:/tmp # mmobj multiregion import –region-file /tmp/region3.dat
mmobj multiregion: Importing region checksum 04215-46679.
[I] Building ring file: account.builder
[I] Building ring file: container.builder
[I] Building ring file: object.builder
[I] Building ring file: object-14001511240.builder
mmobj multiregion: The region config has been updated.
mmobj multiregion: Region 1 checksum is: 04215-46679
We should now have all three regions in our multi-region environment:
#22:57:32# c40bbc2xn3:/tmp # mmobj multiregion list
Region Endpoint Cluster Name Cluster Id
—— ——— —————— ——————–
1 RegionOne pok_x86.gpfs.net 14008063887730057712
2 Region2 pok_ppc64.gpfs.net 6012206590852568800
3 Region3 omnipath.gpfs.net 6788346196758363892
Finally, we need to synch the primary region with region2 so that region2 can become aware of region3:
#23:00:09# c40bbc2xn3:/tmp # mmobj multiregion export –region-file /tmp/region1.dat
mmobj multiregion: The Object multi-region export file was successfully created: /tmp/region1.dat
mmobj multiregion: Region 1 checksum is: 04215-46679
#23:00:21# c40bbc2xn3:/tmp # scp /tmp/region1.dat c40abc1pn3:/tmp/region1.dat
region1.dat
#23:01:35# c40abc1pn3:~ # mmobj multiregion import –region-file /tmp/region1.dat
mmobj multiregion: Importing region checksum 04215-46679.
[I] Building ring file: account.builder
[I] Building ring file: container.builder
[I] Building ring file: object.builder
[I] Building ring file: object-14001511240.builder
mmobj multiregion: The region config has been updated.
mmobj multiregion: Region 2 checksum is: 37509-20319
mmobj multiregion: The configuration of this region has updated information that must
be propagated to the other regions using the ‘mmobj multiregion’ command.
At this point, all three regions should be synched. You should be able to list, create and download containers and objects from any of the regions.
mkguru 