Practical Tips for Novell Cluster Services

1. Practical Tips for Novell ® Cluster Services Mark Robinson CTO Linux, NDS8 [email_address] Tim Heywood CTO, NDS8 [email_address]

2. Agenda Introduction

3. Cluster Services in OES2

4. Our Environment

5. AutoYaST

6. Cluster Build Methodology

7. Creating Resources

8. Cluster Management

9. Troubleshooting

10. Introduction

11. Introduction Mark Robinson Linux Geek

12. Working with SUSE ® since 1998

13. Working with OES since OES1 Beta 5

14. CLP, CLE, NCE ES, CNI, etc.

15. Ex-SysOp Tim Heywood Working with Novell ® since ????

16. Working with OES since OES1 Beta 5

17. CNE, MCNE, CNI(ish)

18. Novell Knowledge Partner (SysOp)

19. Introduction NDS8 Network Design and Support Ltd. Platinum Consulting Partner

20. Based in Edinburgh, work worldwide

21. Specialities: Linux

22. Workgroup

23. SRM

24. Cluster Services in OES2

25. Cluster Services in OES2 New features are Linux only

26. New from OES2 FCS on: Resource monitoring

27. XEN virtualization support

28. x86_64 platform support Including mixed 32/64 bit node support Dynamic Storage Technology

29. What's new in SP1/2? Major rewrite of cluster code for SP2 Removed NetWare ® translation layer

30. Much faster

31. Much lower system load

32. Typical load average of 0.2! New/improved clustering for: iFolder 3

33. AFP

34. …

35. … NCP virtual server for POSIX filesystem resources

36. Types of Clusters Traditional cluster Servers (nodes)

37. Resources NSS

38. GroupWise ®

39. iPrint XEN cluster Dom0 hosts (nodes)

40. XEN guests (DomU) resources

41. Each resource is a server in its own right

42. Live migration with para-virtualised DomU

43. XEN Cluster Architecture OCFS2 LUN DomU Files Cluster Node Xen Dom0 Cluster Node Xen Dom0 Cluster Node Xen Dom0 Resource DomU Linux iPrint Resource DomU Linux iPrint Resource DomU Linux iFolder Resource DomU Linux GroupWise Resource DomU NetWare pCounter Live Migrate Live Migrate

44. Our Environment

45. Our Environment VMware Workstation based

46. VMware shared disk as an alternative to iSCSI

47. Virtual Machines Resource Server

48. Node 1 (built, in the cluster)

49. Node 2 (to be joined to cluster)

50. Node 3 (to be built) SUSE ® Linux Enterprise Server 10 SP3/OES2 SP2 combined iso

51. VMware Setup Create disks standalone

52. Add config to node vmx files disk.locking = "false" diskLib.dataCacheMaxSize = "0" scsi1.present = "TRUE" scsi1.sharedBus = "none" scsi1.virtualDev = "lsilogic" scsi1.pciSlotNumber = "35" scsi1:0.present = "TRUE" scsi1:0.fileName = "cluster-lun0.vmdk" scsi1:0.mode = "independent-persistent" scsi1:0.redo = "" scsi1:1.present = "TRUE" scsi1:1.fileName = "cluster-lun1.vmdk" scsi1:1.mode = "independent-persistent" scsi1:1.redo = "" vmware-vdiskmanager -c -s 100mb -a lsilogic -t 2 cluster-lun0.vmdk vmware-vdiskmanager -c -s 1gb -a lsilogic -t 2 cluster-lun1.vmdk

53. Our Environment Resource Server (Tree master, iManager, Installation services, AutoYaST, SMT) Storage VMware Shared Disks OES2 SP2 Nodes

54. Our Environment AutoYaST Third Node Resource Server (Tree master, iManager, Installation services, AutoYaST, SMT) Storage VMware Shared Disks OES2 SP2 Nodes

55. AutoYaST

56. Why AutoYaST? Repeatable (exactly) No “human element” XML forms part of Documentation

57. Drink coffee (or suitable non-caffeinated beverage) while server builds itself!

58. Multiple simultaneous builds Stagger by at least 15 minutes Easy to expand cluster with new nodes

59. Helps with DR

60. Why AutoYaST? What will AutoYaST do? Disk partitioning

61. Software patterns

62. Network configuration (including VLAN, bonding etc)

63. OES services eDirectory ™ – new or existing tree

64. NSS

65. NCS Security lockdown

66. Scripts/Complete config files to do the rest

67. At the end of the install we will migrate an NSS resource to the new node with no additional configuration!

68. AutoYaST – New Cluster <ncs> <admin_context> cn=admin.o=novell </admin_context> <admin_password> novell </admin_password> <cluster_dn> cn=cluster,ou=resources,o=novell </cluster_dn> <cluster_ip> 10.0.0.100 </cluster_ip> <config_type> New Cluster </config_type> <ldap_ip_address> node IP,LDAP server IP </ldap_ip_address> <ldap_secure_port config:type="integer"> 636 </ldap_secure_port> <server_name> nodename </server_name> <start> Later </start> <sbd_dev> sdx </sbd_dev> <sbd_dev2> sdy </sbd_dev2> </ncs>

69. AutoYaST – Existing Cluster <ncs> <admin_context> cn=admin.o=novell </admin_context> <admin_password> novell </admin_password> <cluster_dn> cn=cluster,ou=resources,o=novell </cluster_dn> <cluster_ip></cluster_ip> <config_type> Existing Cluster </config_type> <ldap_ip_address> node IP,LDAP server IP </ldap_ip_address> <ldap_secure_port config:type="integer"> 636 </ldap_secure_port> <server_name> nodename </server_name> <start> Later </start> <sbd_dev></sbd_dev> <sbd_dev2></sbd_dev2> </ncs>

70. Demo AutoYaST

71. Cluster Build Methodology

72. Cluster Build Methodology Start with a Resource Server iManager

73. Network Installation Server – HTTP or NFS

74. AutoYaST repository (can be password protected on HTTP)

75. SMT for patching

76. Magic PiXiEs server

77. Cluster Build Methodology Create a “template node” NodeZ

78. Use it to Create AutoYaST template using “Clone this system...”

79. Test the shared disk

80. Create the cluster

81. Create the SBD

82. Creating resources This build will NOT be part of final production cluster

83. Cluster Build Methodology Copy this XML for additional nodes Modify server name

84. IP address(es) Use diffuse to compare XML files Build the other nodes Use the XML created above

85. If not 100% right, whack it, modify XML and start build again

86. Remember these are now commodity items Whack NodeZ and rebuild to complete the system

87. Cluster Build Methodology Implement NIC bonding NIC driver independent

88. 7 different methods – some require switch support

89. Link state vs arp monitoring – blades often cannot lose local link!

90. Configurable with AutoYaST Implement Multipath (MPIO) Very simple to configure – mainly autodetect

91. Wide range of SAN support

92. Friendly LUN naming

93. Configuration file can be used (put in place with autoYaST)

94. Creating Resources

95. File Sharing Resources An NSS pool Use iManager

96. Will end up as Primary for DST pair Another NSS pool Use NSSMU (just because we can)

97. Will end up as Shadow for DST pair Combine them into one resource Delete resource for shadow

98. Modify load script for primary

99. File Sharing Resources POSIX filesystem based resource with NCP Easier than Samba to access files

100. Can be used for iPrint, DHCP etc

101. Use evmsgui to create and format the volume

102. Create the resource in iManager

103. Script to create NCP virtual server

104. File Sharing Resources Add resource monitoring

105. Add NFS access LUM enablement of target users

106. NSS/POSIX rights

107. exportfs in load script rather then /etc/exports on nodes

108. Use fsid=x for NSS

109. NFS access SHARED1 Virtual Server SHARED1 Volume NFSaccess Iface UID: 1012 Mis-dweeb UID: 1004 LUM NSS Rights Dweeb Gromit Wallace NFS FPC1 FPC2 FPC3 FPC4 FPC5 fpc.server.novell Mis UID: 1010 Oracle UID: 60003 Mis UID: 1010 Oracle UID: 60003 Iface UID: 1012 Mis UID: 1010 Oracle UID: 60003 Iface UID: 1012 Mis UID: 1010 Oracle UID: 60003 Iface UID: 1012

110. iPrint Create iPrint on NSS

111. Run iprint_nss_relocate on each node with volume in place

112. NB: only one iPrint resource may run on a node

113. Need to accept certificates in iManager for each node

114. iFolder Create iFolder on POSIX /mnt/cluster/ifolder Run /opt/novell/ifolder3/bin/ifolder_cluster_setup on each node Copy /etc/sysconfig/novell/ifldr3_2_sp2 to nodes first NB: Only one iFolder resource may run on a node

115. DNS DNS must be on NSS as NCP server required for eDirectory ™ integration

116. Check NCP:NCPServer objects

117. LUM user required for NSS rights

118. DHCP Create DHCP on NSS

119. Leases file on NSS volume

120. Log file on NSS volume Syslog-ng configuration

121. Logrotate configuration

122. Default AppArmor configuration will not allow logging to here!

123. GroupWise Create PO on NSS

124. Set namespace in load script /opt=ns=long Disable atime/diratime on volume Open nsscon

125. Run /noatime=volname

126. OCFS2 Shared Storage Shared disk! Multi-mount, read/write with distributed lock management

127. /etc/ocfs2/cluster.conf automagically created by NCS

128. Fstab mounting uses /etc/init.d/ocfs2 service

129. Cluster Management

130. Cluster Management iManager The ONLY way to create/delete/edit resources

131. View event log Cluster command Same as NetWare ®

132. No cvb rebuild or device scan. These are not required on Linux as EVMS does it for you Console One

133. The cluster Command The usual suspects cluster online/offline/migrate

134. cluster join/leave

135. cluster status/resources/view/info More interesting cluster stats display – check heartbeat/SBD ticks

136. cluster pools – check NSS pools and location

137. cluster set – modify heartbeat etc

138. cluster exec – potentially very dangerous Lots of BCC commands

139. Troubleshooting

140. Useful Linux Tools ip command – manage TCP/IP on Linux ip addess show/add/del

141. ip route show ethtool – NIC settings

142. cat /proc/net/bonding/bondX

143. netstat – network communication status Check which ports services are listening on

144. Check IP based connections to node nmap – network/port scanner

145. multipath -ll

146. LUN Identification Which LUN is which?

147. lsscsi – shows LUN ID numbers

148. ls -l /dev/disk/by-id scsi-360a98000 4334616f6b5a55572d625550

149. Need to find ID on SAN. (Netapp uses ASCII!) Multipathing will show the ID as the multipath name Use friendly naming

150. Useful Tools sbdutil – create/check/modify the SBD sbdutil -f to find the SBD

151. sbdutil -v to view the current state of the SBD /opt/novell/ncs/bin/ncs-configd.py -init option to pull down load scripts, fix node names etc cifsPool.py to fix CIFS attributes (TID #7005192)

152. OES2 NCS Master Reference TID, FAQ and Troubleshooting – TID #7001433

153. NSA – Novell ® Support Advisor Many patterns for NCS

154. File Locations Cluster configuration file /etc/opt/novell/ncs/clstrlib.conf Load/Unload scripts /var/run/ncs (run from here)

155. /var/opt/novell/ncs Load script output logs /var/opt/novell/log/ncs System Log /var/log/messages

156. File Locations Admin filesystem – virtual filesystem for NCS management /admin/Novell/Cluster Proc filesystem – virtual filesystem for Linux/NCS management /proc/ncs Cluster event log iManager

157. /admin/Novell/Cluster/EventLog.xml

158. /proc/ncs Magic Enable serious debugging! echo -n "TRACE ON" > /proc/ncs/vll

159. echo -n "TRACE SBD ON" > /proc/ncs/vll

160. echo -n "TRACE GIPC ON" > /proc/ncs/vll

161. echo -n "TRACE MCAST ON" > /proc/ncs/vll

162. echo -n "TRACE CVB ON" > /proc/ncs/cluster

163. …

164. Can be made permanent by editing /opt/novell/ncs/bin/ldncs Find the SBD cat /proc/ncs/sbdlib

165. AdminFS Magic Two type of file in /admin/Novell/Cluster *.xml – contain cluster/state information

166. *.cmd – “write then read” files for issuing cluster commands

167. Known issues EVMS issue with no NSS Unpatched SP2 nodes without NSS cannot load SBD kernel moduled iFolder shutdown script Doesn't shut down components if names have been changed during configuration IP address problem Unpatched nodes can allow duplicate IP addresses on network Resources in NDS sync state Check replica rings/referrals

168. Check case of cluster DN in clstrlib.conf

169. www.nds8.co.uk

171. Unpublished Work of Novell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.

Practical Tips for Novell Cluster Services

Related slideshows

More Related Content

Practical Tips for Novell Cluster Services

Editor's Notes