Today I did my first install of CEPH, which I used as backend system for the Elastic Block Storage (EBS) in Eucalyptus. The advantage of CEPH is that it is a distributed system which is going to provide you replication (persistence for data), redundancy (we are using a pool of resources, not a single target) and scalability : the easiest way to add capacity is to add nodes which will be used for storage, and those nodes can be simple machines.

I won’t go too deep into CEPH installation. The official documentation is quite good for anyone to get up to speed quickly. I personally had no troubles using CentOS7 (el7).
Also, I won’t go too deep into Eucalyptus installation, I will simply share with you the CEPH config files and my NC configuration files which have some values non-indicated in Eucalyptus docs.

We are just about to have Eucalyptus 4.1 released with VPC implementations and some new features, but I think that it is quite important to take a few time to dig into EDGE networking and networking modes in General with Eucalyptus.

For years, we had 3 mainly used modes :

MANAGED

AWS Security Groups supported and VLANs created to give L2 separation

All traffic goes via the Cluster Controller for cross-groups communication

Requires a DHCP clean-environment

MANAGED-NOVLAN

AWS SG supported but no L2 separation

All traffic goes via the Cluster Controller for cross-groups communication

Requires a DHCP clean environment

SYSTEM

Customer DHCP server will assign IP addresses to instances

No AWS SG compatibility

As we can figure out, if you needed the AWS compatibility you also had to deal with the CC handling all the instances traffic. But the problem is that you also had to dedicate a physical machine to…

Both of these features help cloud users implement cross-account bucket and object access, and define how long an object lives in a bucket. Keeping consistent with the earlier blog entries and continuing to promote Eucalyptus’s AWS compatibility, these features will be demonstrated using both python boto and s3cmd.

Both boto and s3cmd can be installed on the same client machine to make things easier. Once the tasks above have been completed, we are ready to begin.

Python Boto

As mentioned before, past blog entries have covered how to use boto with Eucalyptus. To help demonstrate bucket/object ACLs and object lifecycle management using boto with Eucalyptus, I created a script – which can be downloaded. Before using the script, make sure and change the following variables:

aws_access_key_id=“<Eucalyptus Access Key ID>”

aws_secret_access_key=“<Eucalyptus Secret Key>”

host=“<S3_URL – Eucalyptus OSG DNS Name>”

After updating the script, run the script with the –help option to see what arguments can be passed:

As you can see, the bucket and object ACL have been set to READ access by ‘account2′. Just as with AWS S3, boto can be used to manage both ACLs and object lifecycles on Eucalyptus 4.0.x.

s3cmd

For cloud users who aren’t savvy with python, s3cmd can also interact with Eucalyptus 4.0.x OSG to manage ACLs and object lifecycles. To get started, download the latest version of s3cmd from Github. As mentioned in my previous blog regarding creating a configuration file for s3cmd, run ‘./s3cmd/s3cmd –configure‘ to create the configuration file for s3cmd. After providing the Access Key ID and Secret Key for the Eucalyptus user, select not to test the credentials. Open the configuration file (.s3cfg) with your favorite editor, and change the following attributes:

host_base = s3.amazonaws.com

to

host_base = <Eucalyptus OSG DNS Name>:8773

and

host_bucket = %(bucket)s.s3.amazonaws.com

to

host_bucket = %(bucket)s.<Eucalyptus OSG DNS Name>:8773

Once the configuration has been completed, make sure everything is working by listing the buckets owned by the Eucalyptus user account:

As you can see, the ACLs and object lifecycle can be displayed without a problem using s3cmd.

Closing the Loop

As demonstrated, both python boto and s3cmd can be used with Eucalyptus 4.0.x clouds to manage bucket and object ACLs, and object lifecycles. It is very important to remember that when using this feature, please make sure that Eucalyptus DNS is enabled on the Eucalyptus 4.0.x cloud. As mentioned earlier, these features can be used for various use cases – ranging from sharing custom Cloudformation templates with other accounts on the Eucalyptus cloud to controlling how long files stay in a bucket (which is very helpful if an instance is storing logs and/or backup files to a bucket).

Hope you enjoyed this entry. Please let me know if there are any questions. As always, feedback is greatly appreciated.

Notice the ‘RESERVATION’ line that contains the security group that the instance is using. If the ‘euca-internal-‘ prefix is removed, the security group has the following format:

{Account ID}-{load balancer Name}

This information matches the Load Balancer launched by the cloud user and will be the base for the solution.

Building the Foundation

In order to get started, the solution needs to be applied from the Cloud Administrator (i.e. admin user in the ‘eucalyptus’ account) perspective. This solution can not be applied by any other type of cloud user. In addition to cloud administrator user requirement, the following is needed:

Store these credentials in a safe place. Next, customizing the load balancer instance.

Customize the Load Balancer

To begin, a Eucalyptus Load Balancer needs to be launched in order to modify it. The goal here is to build an image from this instance using euca-bundle-instance. We will start with the load balancer mentioned earlier:

Logging for HAProxy is complete. Next, rsyslog and logrotate need to be configured.

Log Management

Storing the HAProxy logs, and rotating them is very important to this solution. The script takes the rotated log, and stores it in the OSG bucket for the access logs. The purpose of this is to make sure the file is not being written to when its being sent to the OSG bucket. To start out, download the load-balancer.conf file to use with logrotate using curl:

This is the logrotate configuration file that the cronjob script will call to rotate the log file, then execute the access-log-transfer-s3.py script with a 1 day object lifecycle. To change the lifecycle, just change the value of the –lifecycle option in the load-balancer.conf file.

Next, update rsyslog to make sure the latest is running on the instance:

# yum upgrade rsyslog -y

After this has completed, add the following to the /etc/rsyslog.d/load-balancer.conf file:

local3.* /var/log/load-balancer-access.log

Follow this step up by uncommenting and adding the following lines in /etc/rsyslog.conf:

$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

To wrap up, we need to add a script that will be kicked off by the cronjob.

Cronjob Script

To kick off the log rotation, add the ‘elb-logrotate‘ script to the instance using curl:

Creating the New Eucalyptus Load Balancer EMI

After finishing with the instance customizations, the instance is ready to be bundled and registered. First, use euca-bundle-instance to bundle and upload the instance. Use euca-describe-bundle-tasks to check on the status of the bundling operation. Once the bundling operation has been completed, use euca-register to register the new ELB EMI:

Testing Out the ELB with Access Logging

To test it out, you can use either the Cloud Administrator, or a user from a ‘non-eucalyptus’ account. In the example below, a user from a ‘non-eucalyptus’ account was used. If a ‘non-eucalyptus’ account user is used, make sure the user has the appropriate IAM access policies for EC2 (Compute), S3 (OSG), and ELB (Eucalyptus Load Balancer).

Generate some traffic to the ELB using curl or some other tool to populate the HAProxy log file. Based upon how often the cronjob was set to execute, use s3cmd to see the bucket created in the ‘eucalyptus’ account (i.e. Cloud Administrator) for the access logs. For information regarding s3cmd configuration files, refer to my previous blog:

How is this ‘non-eucalyptus’ user able to see and download the contents of this bucket? This is because of the script that creates the access log bucket, and uploads the logs to the bucket. By grabbing the account ID from the instance metadata ‘security group’ category, the script adds bucket and object READ ACLs for the account ID. The only issue here is that the cloud administrator will still need to communicate the bucket that the cloud user can access for the logs. With the extra bonus of using the object lifecycle, the cloud administrator doesn’t have to worry about managing the buckets. The objects will remove themselves after the define period of time.

Conclusion

Even though the solution isn’t exactly like AWS ELB Access Logs feature, it does provide a solution that is very similar to it. The only thing missing is the service API interaction to enable/disable the access logging feature, set the interval and define the bucket that will be used. Hopefully, this will be a feature we will see in the not too distant feature. Thanks for hanging in there with me. I hope you enjoy! Feedback is always welcome.

ZFS is a filesystem designed by Sun Microsystems that focuses on data integrity. What makes this such an attractive filesystem to use in the cloud is that a cloud user can easily do the following:

set up an LVM + RAID filesystem for storing large amounts of data (e.g. database information)

expand the filesystem by adding more storage (i.e. EBS volumes)

backup the filesystem without taking the filesystem offline/unmounting

restore the filesystem

This blog entry will focus on how a cloud user can create their own Eucalyptus Machine Image (EMI) that has ZFS support. The CentOS 6.5 EMI on the Eucalyptus Machine Image Catalog will be used as the base image.

Before Starting…

Before following the steps in this blog, make sure the following is in place:

Once these requirements have been met, everything should be ready to go.

Set Up Base Image/Instance

To begin, follow the ‘Quick Start’ instructions mentioned on the Eucalyptus Machine Image Catalog page. This will install all the images provided by the catalog. When the process has finished, list the CentOS 6.5 EMI. For example:

Create the CentOS 6.6 EMI with ZFS Support

The instance is now ready to be bundled. Bundle the instance using the euca-bundle-instance command. This command is used to bundle Windows instances, however Eucalyptus extended this command to work with Linux instances as well. Use euca-describe-bundle-tasks to monitor the bundling status:

Thats it! Notice how this only required 2 commands to set up a LVM + RAID filesystem, compared to around 7 commands using mdadm, pvcreate, vgcreate, mkfs, mkdir and mount. The instance is now ready to utilize the ZFS filesystem for the MySQL server.

As mentioned in my past blog, Eucalyptus has the concept of EUARE (IAM) Roles which pretty much work the same way as AWS IAM Roles. This blog with discuss how to set up cross-account API access using EUARE (IAM) Roles in Eucalyptus 4.0.2. Cross-account API access is very useful for use cases where there are multiple Eucalyptus EUARE accounts that need to access the same resources. A great example of this can be found in the AWS IAM documentation regarding this same feature.

To establish cross-account access, in the trusting account (Account A), you create an IAM role that designates Account B as the trusted account. The role also defines permissions that allow Account B access to specific resources in Account A. Account B can then delegate this access to its own users (IAM users or federated users). Account B cannot delegate more access to its users than the permissions that it has been granted by Account A.

To demonstrate this feature, a user in Account A will set up a role to allow DescribeInstances API access for a user in Account B.

Trust Policy for Role in Account A

In order for the user in Account B (in this example, the user is ‘user12′) to be able to assume the role under Account A, a trust policy needs to be associated with the role. Below is an example of the trust policy (which can be downloaded from here):

Once this has been completed, all work is done under Account A. Next, the user under Account B needs to be allowed to assume the role. The only important information needed from Account A, is the ARN for the role – which in this use case is the following:

arn:aws:iam::290122656840:role/cross-describe-instances

Account B EUARE (IAM) Setup

Before applying the policy to allow assuming the role to the group where ‘user12′ is a member, the user needs to be able to do the following at a minimum:

As mentioned above, apply this policy to the group that has ‘user12′ as a member under Account B, using the euare-groupuploadpolicy command.

Assume Role Access

To allow ‘user12′ under Account B to assume the role under Account A, we need to upload another policy. This can be done to the group that has ‘user12′ as a member, or directly to ‘user12′ by using euare-useruploadpolicy. The key information needed in the IAM policy is the ARN of the role (arn:aws:iam::290122656840:role/cross-describe-instances). The policy should look something similar to this:

For STSConnection.DefaultRegionEndpoint, use the Eucalyptus DNS name for Eucalyptus Token service associated with your Eucalyptus 4.0.2 cloud (e.g. “tokens.future.euca-hasp.cs.prc.eucalyptus-systems.com“)

For EC2Connection.DefaultRegionEndpoint, use the Eucalyptus DNS name for Eucalyptus Compute service associated with your Eucalyptus 4.0.2 cloud (e.g. “compute.future.euca-hasp.cs.prc.eucalyptus-systems.com“)

Conclusion

Cross-account access using Eucalyptus EUARE (IAM) roles are very powerful. It allows cloud administrators and account administrators to provide access to limited resources in different accounts without having the hassle of creating/managing users. Please refer to the AWS IAM documentation for further examples and use cases. Enjoy!

Zabbix is a rich monitoring system, which support SNMP, JMX, IPMI and uses its own agents for more functionality.
The great advantage of Zabbix is that it is self-sufficient : all the tools you need are in the packages and got connected to each others, whereas on Nagios you have to add tens of plugins.

Zabbix JMX monitoring is pretty simple : using a “JMX Gateway”, it will get connected to the server and collect information which will be transferred to the Zabbix Proxy / Server.

On the first place, we have to enable the JMX monitoring in Eucalyptus, so the JVM will allow connections to its monitoring systems. Edit /etc/eucalyptus/eucalyptus.conf