Log Management using Logstash and Kibana on CentOS/ RHEL / Ubuntu

logstashLog files are one of the most important files where almost all useful or sometimes unnecessary information’s are stored in regard to your servers running state. Log files will help us to rescue or identify where or what went wrong if there is any security has been breached or compromised. Log files are the most valuable tools available for Linux system security. You can find so many log files for different different application on different different places. To maintain and audit these log files is a very hard work if you have so many servers running. Logstash is an open-source log management tool through which we can manage logs from various servers at one place. In this article I will show you how to use Logstach with Kibina to create a centralize log management servers.

Logstash is used to gather logging messages, convert them into json documents and store them in an Elasticsearch cluster. Kibana is used as a frontend client to search for and display messages from Elasticsearch cluster. Below is the flow how it works:

On each servers shipping logs:

  • Download and run logstash (This will ship all the configured logs to the centralized log management server)

On the Centralized Log management server collecting and indexing your logs:

  • Download and run Elasticsearch (Will act as the Storage and Search engine of the collected logs.)
  • Download and run Redis (Will act as the Broker)
  • Download and run Logstash (Will index the collected logs.)
  • Download and run Kibana (Will search and display the logs from elasticsearch)

Step: 1

Install ElasticSearch, Redis and Logstash on the centralize cerver:

You should have JAVA installed on your machine to run elasticsearch and logstash. If you haven’t JAVA on your machine then follow the below step to install JAVA on your machine. (Install on every machine)

Then Download and install ElasticSearch (Only on the machine which you want to make as Centralized Log management Server)

Add ElasticSearch to startup config and start the service.

Step: 2

Install Redis (Only on the machine which you want to make as Centralized Log management Server)

Add EPEL repository to your server if you have CentOS then issue the below commands.

For CentOS/ RHEL/ Fedora

For Ubuntu

Add to starup and start the service

Step : 3

Now we will install Logstash: (Install on every machine)

Step : 4

Let’s create the configuration files for logstash:

Create the indexer.conf to start the indexing service. (Only on the machine which you want to make as Centralized Log management Server)

Append the below lines to the above file.

Add another file named shipper.conf (Create the file on every client machine)

Append the below lines to the above configuration file

Step : 5

Now we will start the indexer and shipper by issuing below commands. Start the indexer on the Centralized server whereas the shipper on every client machine.

Step : 6

Finally we need to install Kibana to complete our task: (Only on the machine which you want to make as Centralized Log management Server)

We need to install Ruby and RubyGem to run Kibana:

Get Kibana and install required gems:

Configure Kibana (Modify the below lines if you have custom value for it.)

Start kibana

Open any browser and acces the URL: http://127.0.0.1:5601/ and enjoy.

If you want to run the logstash as a service then create a shell script as below:

Append the below code to it:

Also you can create another script for the shipper. Just enjoy with your newly created Centralize Log Server 🙂

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS
The following two tabs change content below.

Tapas Mishra

Sr. Engineer (DevOps)
Loves to work on Opensource products. Having experience on Linux environment. Knowledge on Public cloud services like AWS, Rackspace, DigitalOcean, Linode. Please don't hesitate to give a comment on the posts. Your comments are my strength.

9 thoughts on “Log Management using Logstash and Kibana on CentOS/ RHEL / Ubuntu

  1. i got this error when trying to install kibana.
    any help?

    root@server:/opt/kibana-3.0.0milestone4# gem install bundler
    Successfully installed bundler-1.3.5
    1 gem installed
    Installing ri documentation for bundler-1.3.5…
    Installing RDoc documentation for bundler-1.3.5…

    root@server:/opt/kibana-3.0.0milestone4# bundle install
    Bundler::GemfileNotFound

    installed additional libs,
    yum install gcc+ gcc-c++ ruby ruby-devel rubygems

    • Hi Bojan,

      It seems the path problem of your rubygem. Please paste output of these commands for further diagnosis “ gem env” and “ gem list”.

      Thanks,
      -Tapas

  2. hi Tapas,

    i have the same problem as well. but with this error:

    1.running build install:

    An error occurred while installing eventmachine (1.0.3), and Bundler cannot continue.
    Make sure that gem install eventmachine -v '1.0.3' succeeds before bundling.

    2. then ran gem install eventmachine -v ‘1.0.3

    [root@localhost Kibana-0.2.0]# gem install eventmachine -v ‘1.0.3’
    Building native extensions. This could take a while…
    ERROR: Error installing eventmachine:
    ERROR: Failed to build gem native extension.

    my gem env.

    RubyGems Environment:
    – RUBYGEMS VERSION: 2.0.3
    – RUBY VERSION: 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
    – INSTALLATION DIRECTORY: /usr/lib64/ruby/gems/1.8
    – RUBY EXECUTABLE: /usr/bin/ruby
    – EXECUTABLE DIRECTORY: /usr/bin
    – RUBYGEMS PLATFORMS:
    – ruby
    – x86_64-linux
    – GEM PATHS:
    – /usr/lib64/ruby/gems/1.8
    – /root/.gem/ruby/1.8
    – GEM CONFIGURATION:
    – :update_sources => true
    – :verbose => true
    – :backtrace => false
    – :bulk_threshold => 1000
    – REMOTE SOURCES:
    https://rubygems.org/

    my gem list.

    [root@localhost Kibana-0.2.0]# gem list

    *** LOCAL GEMS ***

    atomic (1.1.14)
    bundler (1.5.2)
    daemons (1.1.9)
    diff-lcs (1.2.5)
    rake (10.1.1)
    [root@localhost Kibana-0.2.0]# gem list

    *** LOCAL GEMS ***

    atomic (1.1.14)
    bundler (1.5.2)
    daemons (1.1.9)
    diff-lcs (1.2.5)
    rake (10.1.1)

    thanks in advance

    • Hi CJ,

      I think this error is due to bundler gem. Please uninstall the bundler “ gem uninstall bundler” and install it with version “ gem install bundler -v '1.3.5'”. After completing the bundler installation try to install the rest gems by issuing the command “ bundle install”. Let me know if you still have any problems.

      Thanks,
      Tapas

Leave a Reply