piranha,水虎鱼,食人鱼

piranha,水虎鱼,食人鱼

piranha：

Summary     : Cluster administation tools
Description : Various tools to administer and configure the Linux Virtual Server as well as heartbeating and failover components.  The LVS is a dynamically adjusted kernel routing mechanism that provides load balancing primarily for web and ftp servers though other services are supported.

piranha的组件：

/usr/sbin/pulse
heartbeating daemon for monitoring the health of cluster nodes.

/usr/sbin/lvsd
daemon to control the Red Hat clustering services.

/usr/sbin/nanny
tool to monitor status of service in a cluster.

/usr/sbin/fos
failover services daemon to control the Red Hat clustering service.

/usr/sbin/send_arp
tool  to  notify network of a new IP address / MAC address mapping.
这个工具非常有用。之前有一篇《send_arp的一个角本》，我曾经简单介绍过。

On LVS router, there are three service which need to be set to activate at boot time.

• piranha-gui
• pulse
• sshd

If you are clustering multi-port services or using firewall marks, you must enable the iptables service.

CentOS Linux 5.2 (i386) 中的Clustering组件中包含如下组件：

yum groupinfo Clustering
Group: Clustering
Description: Clustering Support.
Default Packages:
clustermon
conga-devel
ricci
system-config-cluster
ipvsadm
piranha
cluster-snmp
modcluster
ricci-modcluster
cluster-cim
rgmanager
luci

实际安装过程：
Installing:
cluster-cim                          : CentOS Cluster Suite – CIM provider
cluster-snmp                       : CentOS Cluster Suite – SNMP agent
luci                                       : Remote Management System – Management Station
piranha                                 : Cluster administation tools
rgmanager                            : Open Source HA Resource Group Failover for CentOS
ricci                                      : Remote Management System – Managed Station
system-config-cluster         : system-config-cluster is a utility which allows you to manage cluster configuration in a graphical setting.
Installing for dependencies:
cman : cman – The Cluster Manager
gnome-python2-canvas : Python bindings for the GNOME Canvas.
httpd : Apache HTTP Server
ipvsadm : Utility to administer the Linux Virtual Server
lm_sensors : Hardware monitoring tools.
modcluster : CentOS Cluster Suite – remote management
net-snmp : A collection of SNMP protocol tools and libraries.
net-snmp-libs : The NET-SNMP runtime libraries.
openais : The openais Standards-Based Cluster Framework executive and APIs
perl-Net-Telnet : Net-Telnet Perl module
perl-XML-LibXML : XML-LibXML Perl module
perl-XML-LibXML-Common : XML-LibXML-Common Perl module
perl-XML-NamespaceSupport : XML-NamespaceSupport Perl module
perl-XML-SAX : XML-SAX Perl module
php : The PHP HTML-embedded scripting language. (PHP: Hypertext Preprocessor)
php-cli : Command-line interface for PHP
php-common :  Common files for PHP
pygtk2-libglade : A wrapper for the libglade library for use with PyGTK
python-imaging : Python’s own image processing library
tix ：A set of extension widgets for Tk
tk ：Tk graphical toolkit for the Tcl scripting language
tkinter ：A graphical user interface for the Python scripting language.
tog-pegasus ：OpenPegasus WBEM Services for Linux

Command Line Administration Tools
In addition to Conga and the system-config-cluster Cluster Administration GUI, command line tools are available for administering the cluster infrastructure and the high-availability service management components. The command line tools are used by the Cluster Administration GUI and init scripts supplied by Red Hat. Table 1.1, “Command Line Tools” summarizes the command line tools.

Lustre
Lustre is a scalable, secure, robust, highly-available cluster file system. It is designed, developed and maintained by Sun Microsystems, Inc.
Designed to meet the demands of the world’s largest high-performance compute clusters, the Lustre file system redefines scalability and provides groundbreaking I/O and metadata throughput. An object-based cluster, Lustre currently supports tens of thousands of nodes, petabytes of data, and billions of files — and development is underway to support one million nodes, trillions of files, and zetta to yotta bytes.
http://www.sun.com/software/products/lustre/
http://wiki.huihoo.com/index.php?title=Lustre

AFS
AFS Reference Page

OpenAFS
What is AFS?
AFS is a distributed filesystem product, pioneered at Carnegie Mellon University and supported and developed as a product by Transarc Corporation (now IBM Pittsburgh Labs). It offers a client-server architecture for file sharing, providing location independence, scalability and transparent migration capabilities for data. OpenAFS is the Transarc source code released as it looked like around AFS3.6 under IBM Public License IPL.

Arla
Arla is a free AFS implementation.
The main goal is to make a fully functional client with all capabilities of AFS as formerly sold by Transarc and today available as OpenAFS. Other stuff, such as servers and management tools are being developed, but currently not considered stable.

Coda
Coda分布式文件系统:http://www.bsdmap.com/diary/coda.php
Coda File System http://www.coda.cs.cmu.edu/
Coda is a forked of version of AFS that support disconnected and weakly connected mode better then AFS.

InterMezzo
InterMezzo is a new distributed file system with a focus on high availability. InterMezzo will be suitable for replication of servers, mobile computing, managing system software on large clusters, and for maintenance of high availability clusters.

xFS
xFS is a Serverless Network File Service.

CFS
Cluster File Systems, Inc. is the leading developer of next generation technology for scalable high-performance file systems. Our Lustre® file system redefines scalability and has been designed from the ground up to meet the demands of the world’s largest high-performance computer clusters.

GlusterFS
GlusterFS is a cluster file-system capable of scaling to several peta-bytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. GlusterFS is based on a stackable user space design without compromising performance.

Scalable File Share
HP StorageWorks Scalable File Share
A high-bandwidth, scalable storage appliance for Linux clusters
http://h20311.www2.hp.com/HPC/cache/276636-0-0-0-121.html

MogileFS
MogileFS is our open source distributed filesystem. Its properties and features include:
-1. Application level
-2. No single point of failure
-3. Autumaic file replication
-4. “Better than RAID”
-5. Flat Namespace
-6. Shared-Nothing
-7. No RAID required
-8. Local filesystem agnostic

Hadoop
The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing, including:
* Hadoop Core, our flagship sub-project, provides a distributed filesystem (HDFS) and support for the MapReduce distributed computing metaphor.
* HBase builds on Hadoop Core to provide a scalable, distributed database.
* ZooKeeper is a highly available and reliable coordination system. Distributed applications use ZooKeeper to store and mediate updates for critical shared state.

PVFS
http://www.pvfs.org/
http://www.parl.clemson.edu/pvfs/
PVFS is designed to provide high performance for parallel applications, where concurrent, large IO and many file accesses are common. PVFS provides dynamic distribution of IO and metadata, avoiding single points of contention, and allowing for scaling to high-end terascale and petascale systems.

GFS

http://en.wikipedia.org/wiki/Global_File_System
http://www.redhat.com/docs/manuals/csgfs/
GFS (Global File System) is a cluster file system. It allows a cluster of computers to simultaneously use a block device that is shared between them (with FC, iSCSI, NBD, etc…). GFS reads and writes to the block device like a local filesystem, but also uses a lock module to allow the computers coordinate their I/O so filesystem consistency is maintained. One of the nifty features of GFS is perfect consistency — changes made to the filesystem on one machine show up immediately on all other machines in the cluster.

See also

• Comparison of file systems
• GPFS, ZFS
• Lustre
• GlusterFS
• List of file systems
• Oracle cluster file system (OCFS)
• SAN file system
• Fencing

External links About GFS

• Red Hat GFS Product Page
• Red Hat Cluster Suite and GFS Documentation Page
• GFS Project Page
• DataCore’s GFS Informations
• OpenGFS Project Page
• The GFS2 development git tree

1. HP OpenVMS
————–
The first to work with a CFS is HP OpenVMS. Oracle Parallel Server and RAC always used
the OpenVMS filesystem (RMS) for its database.

2 HP Tru64
————
CFS is a layer on top of Advfs the filesystem of HP Tru64. Oracle uses
the Direct I/O feature available in CFS. Direct I/O enables Oracle to bypass
the buffer cache (no caching at filesystem level). Oracle manages the
concurrent access to the file itself; as it does on raw devices. On CFS,
without Direct I/O enabled on files – file access goes through a CFS server.
A CFS server runs on a cluster member and serves a file domain. A file
domain can be relocated from one cluster member to another cluster member
online. A file domain may contain one or more filesystems.

Direct I/O does not go through the CFS server, but file creation and resizing
is seen as metadata operation by advfs and this has to be done by the CFS
server.  The consequence is to run file creations and resizing on the node
where the CFS server is located. File operations might take longer when the
CFS server is remote.

Oracle recommends not using the tempfile option, as tempfiles might not be
allocated until the tempfile blocks are accessed and so cause
‘remote metadata operations’for advfs.

3 Veritas
———–
VERITAS Database EditionTM / Advanced Cluster for Oracle9i RAC enables Oracle
to use the CFS.  The VERITAS Cluster File System is an extension of the VERITAS
File System (VxFS).  Veritas CFS allows the same filesystem to be simultaneously
mounted on multiple nodes.  Veritas CFS is designed with a master/slave
architecture.  Any node can initiate a metadata operation (create, delete, or
resize data), the actual operation is carried out by the master node. All other
(non metadata) IO goes directly to the disk.

CFS is used in DBE/AC to manage a filesystem in a large database environment.
When used in DBE/AC for Oracle9i RAC, Oracle accesses data files stored on CFS
filesystems by bypassing the filesystem buffer and filesystem locking for data.

4 Oracle Cluster File System
——————————
Oracle Cluster File System (OCFS) is a shared filesystem designed specifically
for Oracle Real Application Clusters. OCFS eliminates the requirement for Oracle
database files to be linked to logical drives and enables all nodes to share a
single Oracle_Home (current capabilities are detailed in section 2.8) instead
of requiring each node to have its own local copy. OCFS volumes can span one
shared disk or multiple shared disks for redundancy and performance
enhancements.

5. Netapp(R) Filer
——————-
Netapp Filer offers CFS functionality via NFS to the server machines. These
filesystems are mounted using special mount options. For details please see
Netapp documentation.

Netapp certifications can be found at:

http://www.netapp.com/part…

To understand the architecture and Oracle installation please see these
documents:

Note 210889.1: RAC Installation with a NetApp Filer in Red Hat Linux Environment
and
Oracle9i RAC Installation with a NetApp Filer on Fujitsu-Siemens Primepower
(Solaris8 Operating System) at http://www.netapp.com/tech…

6 AIX
——-
IBM’s General Parallel File System (GPFS) allows users shared access to files
that may span multiple disk drives on multiple nodes. GPFS provides access to
all data from all nodes of the cluster.  It can be configured with multiple
copies of metadata allowing continued operation should the paths to a disk or
the disk itself be broken. Metadata is the filesystem data that describes
the user data.  GPFS allows the use of RAID or other hardware redundancy
capabilities to enhance reliability.

In Oracle9i GPFS is only supported with HACMP/ES in a RAC configuration.
When placing datafiles on GPFS no CRM (Concurrent Resource Manager) needs to be
installed. Starting with Oracle10g HACMP is no longer required to use GPFS.

Metalink contains certification information and information about required
patches for having a cluster database on a GPFS.

7 Sun GFS
———–
Global File Service (GFS or Cluster File System) is a filesystem that is
accessible from all nodes in the cluster. GFS is based on global devices and
has a client/server architecure. GFS provides transparent and concurrent file
access.

Note that Sun GFS is not supported for Oracle datafiles, see section 3.10.

8 Sun StorEdge QFS
——————–
QFS software is a file manager that provides a shared filesystem where mutiple
servers can read and write simultanuously to the same file in the same filesystem.

9 Other Linux Cluster Filesystems
———————————–
There are various third party cluster filesystems available on Linux.
Consult the Oracle Certify website for the policy regarding support for third party
cluster file systems on Linux. Also, consult the RAC Technology Compatibility Matrix (RTCM)
for Linux (http://www.oracle.com/tech… … generic_linux.html)
for the latest information on which third party cluster file systems are supported
by RAC release and platform.

10 Which Platforms support what?
———————————-

Platform and                         Storage for                      Storage for
[Cluster Software]                Oracle installation             datafiles

AIX [HACMP]                          LFS (1) or CFS (2)           CFS and/or Raw devices
AIX [CRS]                                  LFS or CFS            CFS and/or Raw devices
HP/UX [MC/Service Guard]             LFS or CFS (3)        CFS (3) and/or Raw Devices
HP/UX PA-Risc [Veritas DBE/AC)       LFS or CFS            CFS and/or Raw Devices
Linux [oracm, CRS]                   LFS                   OCFS (4) and/or Raw
Devices, also NFS (5)
OpenVMS                              CFS                   CFS
Sun Solaris [Fujitsu Siemens         LFS                   Raw Devices/NFS (5)
Primecluster]
Sun Solaris [Sun Cluster]            LFS or CFS        (6,7)             CFS (7) Raw Devices/NFS (5)
Sun Solaris [Veritas DBE/AC]         LFS or CFS                         CFS and/or Raw Devices
Tru64 Unix                           LFS or CFS                         CFS and/or Raw Devices
Windows NT/2000 [oracm, CRS]         LFS or CFS                   OCFS and/or Raw Devices
Windows 2003 (32/64bit) [oracm, CRS] LFS or CFS            OCFS and/or Raw Devices

(1) LFS is the abbreviation for local filesystem and is only accessible directly
by the node that mounted the disk
(2) CFS is the abbreviation for Cluster FileSystem. The implementation
depends on the operating software vendor or cluster software vendor.
(3) MC ServiceGuard 11.17 includes a CFS which is supported with Oracle 10gR2
(4) OCFS: Oracle Cluster FileSystem
(5) NFS is supported with Netapp(R) Filer, see Metalink certification
(6) Sun GFS can only be used for Oracle_Home and archivelogs.
(7) Sun StorEdge QFS

Local Filesystem means that the Oracle Universal Installer replicates the
RAC software installation automatically to every private filesystem of the
selected nodes in the cluster. The Oracle installation products
are cluster aware and will not install the Oracle software to over-write itself.

Oracm is the Oracle Cluster manager, which is available on Linux and Windows
NT/2000. No other cluster manager is needed to setup Real Application Cluster.

Cluster Ready Services (CRS) are new in Oracle10g and provide also clustermanager
functionality.

Oracle will validate cluster filesystems of other vendors when they become
available. Oracle will support the Oracle software when running on a validated
cluster filesystem.

11 Cluster File System names
——————————

PLatform or Cluster Vendor        CFS name

AIX                                                  GPFS
HP/UX MC/ServiceGuard       CFS
Linux [oracm, CRS]                OCFS
OpenVMS                                              RMS
Tru64 Unix                                    CFS
SunCluster                  GFS, QFS
Veritas DBE/AC                                CFS
Windows NT/2000                                OCFS
Windows 2003 (32/64bit)           OCFS

For more information on certified configuration please see the certification
matrix available on Metalink.  Instructions for accessing the certification
matrix can be found in the following note:

Note 184875.1
How To Check The Certification Matrix for Real Application Clusters

12 When to use CFS over raw?
——————————
This option is very dependent on the availability of a CFS on your platform.
A CFS offers:
- Simpler management
- Use of Oracle Managed Files with RAC
- Single Oracle Software installation
- Autoextend enabled on Oracle datafiles
- Uniform accessibility to archive logs in case of physical node failure
- With Oracle_Home on CFS, when you apply Oracle patches CFS guarantees that
the updated Oracle_Home is visible to all nodes in the cluster.