Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

16.
How is it setup?app/ !fluentd/! Attached to an application instance on |-- stderr.stat (1)! startup by DEA |-- stdout.stat (2)! • (1) and (2) is a “stat file” used by the input plugin, |-- fluentd.conf (3)! which allows it to pick up from the last position it |-- startup (4)! had monitored. |-- stop (5)!fluentd_child.pid (6)! • (3) is the configuration of the plugins dynamically generated by the DEAfluentd.pid (7)!logs/! • (4) and (5) are the scripts executed to manage the |-- fluentd.log (8)! fluentd process |-- stdout.log! • (6) and (7) are the supervisor and main process |-- stderr.log! ids from Fluentdruby!run.pid! • (8) is where any logs produced by the logging daemon are storedstartup!stop! 16

19.
fluentd.conf :: Output configuration <server>! name logserver! Location of the log storage server: host 192.168.2.1! - We use a active-standby setup at the port 4224! moment. </server>! - When primary Logserver is down, logs <server>! are forwarded to the one on standby. name backup_logserver! host 192.168.2.2! port 4224! When both are down, fluentd buffers to standby! files the logs locally until one of them is </server>! available. </store>! <store>! type cf_app_logs_streaming! port 43208! </store>!</match>! Set up VCAP_LOG_STREAM port to enable tail –f like support. 19

20.
…And now that we managed to collect the logs,we also can support the following 2 commands: 20

23.
What we could do• Persisting the logs in Cloud Foundry is a must!• Support to persist the logs is a key issue in adoption of Cloud Foundry for developing application services.• Thanks to Fluentd we could enable users to have log archives in a straightforward way.• Cloud Foundry and Fluentd themselves are both written in Ruby so integration was relatively easy. 23

24.
What we could not do• It was difficult to let the user have access to the logs from Nginx (used as a Router in CF) # Sample release process! vmc push app-v1 --runtime ruby19 => app-v1.cf.rakuten.co.jp! vmc map app-v1 olympics.rakuten.co.jp => Released!! vmc push app-v2 --runtime ruby19 => app-v2.cf.rakuten.co.jp! vmc map app-v2 olympics.rakuten.co.jp => Load balancing between apps! vmc unmap app-v1 olympics.rakuten.co.jp => Release 2 complete.!• Then the app-v1.cf.rakuten.co.jp subdomain becomes available to other user.• Need to keep track that user X had Y domain at Z time for application named A in real time while storing the logs. 24

25.
Fluentd: Pros and Cons• Pros – Easy to include in Cloud Foundry compared to other solutions • Possible to include it like gem fluentd in a Ruby component – Easy to change storage technology depending on scale – Easy to extend – Growing community o/• Cons – Fluentd needs better testing tools – Better materials about Fluentd from the community? • Learning curve is already low • Just we need to share more about how we are using Fluentd • e.g. The Little Fluentd Book, Advanced Fluentd Recipes’ – Cool.io is not maintained 25

26.
Misc: Playing with the message bus• Cloud Foundry is built with Eventmachine.• Fluentd is built with Cool.io.• Cool.io and Eventmachine do not play along very well.• The message bus from Cloud Foundry named NATS is also built using Eventmachine. 26