Note: The data collected by the unix app is by default placed into a separate index called ‘os’ so it will not be searchable within splunk unless you either go through the UNIX app, or include the following in your search query: “index=os” or “index=os OR index=main” (don’t paste doublequotes).

You also will have to install sysstatif you want to monitor your server resources.

Step 9 (Optional): Customize UNIX app configuration on forwarders:

Look at inputs.conf in /opt/splunkforwarder/etc/apps/unix/local/ and /opt/splunkforwarder/etc/apps/unix/default/ The ~default/inputs. path shows what the app can do, but everything is disabled.

The ~local/inputs.conf shows what has been enabled – if you want to change polling intervals or disable certain scripts, make the changes in ~local/inputs.conf.

Note that Splunk also has a centralized configuration management server called Deployment Server. This can be used to define server classes and push out specific apps and configurations to those classes. So you may want to have your production servers class have the unix app configured to execute those scripts listed in ~local/inputs at the default values, but maybe your QA servers only need a few of the full stack, and at longer polling intervals.

Using Deployment Server, you can configure these classes, configure the app once centrally, and push the appropriate app/configuration to the right systems.

Eclipse is a free, Java-based development platform known for its plug-ins that allow developers to develop and test code written in other programming languages. Eclipse is released under terms of the Eclipse Public License.

Eclipse got its start in 2001 when IBM donated three million lines of code from its Java tools to develop an open source integrated development environment (IDE). The IDE was initially overseen by a consortium of software vendors seeking to create and foster a new community that would complement Apache’s open source community. Rumor has it that the platform’s name was derived from a secondary goal, which was to eclipse Microsoft’s popular IDE, Visual Studio.

Today, Eclipse is managed by the Eclipse Foundation, a not-for-profit corporation whose strategic members include CA Technologies, IBM, Oracle and SAP. The foundation, which was created in 2004, supports Eclipse projects with a well-defined development process that values quality, application programming interface (API) stability and consistent release schedules. The foundation provides infrastructure and intellectual property (IP) management services to the Eclipse community and helps community members market and promote commercial software products that are based on Eclipse.

In 2016, Microsoft announced it would join the Eclipse Foundation and support the integration of Visual Studio by giving Eclipse developers full access to Visual Studio Team services. Oracle donated the Hudson continuous integration server it inherited from Sun Microsystems to Eclipse in 2011 and is expected to donate the Java 2 Platform, Enterprise Edition (Java EE) to Eclipse in the near future.

Many people talk about the need to index tweets from twitter into Splunk, that I figured I would write a post to explain just how easy it is.

Within 10 steps and a few minutes, you will be streaming real-time tweets into Splunk, with the fields all extracted and the twitter data fully searchable.

Assumptions

Splunk is installed and running.
If you don’t have Splunk, you can download it from http://splunk.com/download.
Splunk will run fine on your laptop for this exercise.
You have a working Twitter account

3. Click on the “Create New App” button and complete the box for Name, Description and Website. You don’t need a callback URL for this exercise. Once you have completed these three fields, click on the “Create Your Twitter Application” button at the bottom of the screen.

4. Your application is now completed and we now need to generate the OAuth keys. You should see a series of tabs on the screen – click on the ‘API Keys’ tab. At the bottom of the screen when in the API Keys tab, click on the “Create my access token” button.

5. Wait about 30 seconds or so then click on the ‘Test OAuth‘ button at the top right of the screen. You should see all fields completed with cryptic codes. If you don’t, hit back, then click the ‘Test OAuth’ button again after another 30 seconds or so. Keep this page handy – we will need it in a couple of minutes.

6. Ok, now log into your Splunk environment search head, where we are going to install the free REST Api modular input application. Copy the following URL and replace mysplunkserver with whatever your splunk server name is, then click on the “Install Free” button.https://mysplunkserver:8000/en-US/manager/search/apps/remote?q=rest+api.If you are not using SSL, change it to http rather than https. You can alternatively install the application from the Splunk app store here: http://apps.splunk.com/app/1546/

7. Click on the button to “Restart Splunk” after installation of the app.

8. This app adds a new data input method to Splunk called REST. Once logged back into Splunk, click on “Settings” (top right) then “Data Inputs” from the Settings menu.

9.The Data Inputs screen will be displayed and you will see a new data input method called REST. Click on this link, then click on the “New” green button to bring up a new REST input configuration screen.

10. Ok, last step! We are going to complete the configuration details to get our Twitter data. I have only included the fields you need to configure and everything else can be left blank, unless you need to enter in a proxy to get out to the internet.
> REST API Input Name: Twitter (or whatever you want to call the feed)
> Endpoint URL: https://stream.twitter.com/1.1/statuses/filter.json
> HTTP Method: GET
> Authentication Type: oauth1
> OAUTH1 Client Key, Client Secret, Access Token, Access Token Secret: Complete from your Twitter Developer configuration screen in Step 5 above.
> URL Arguments: track=#bigdata,#splunk^stall_warnings=true
The above URL arguments are examples. In this case, I am selecting to bring in tweets that contain the hashtag of #bigdata and #splunk. I am using the ‘track’ streaming API parameter to do this. At this point, you should read here: https://dev.twitter.com/docs/streaming-apis/parameters#track. Also note, that if you want to track multiple keywords, these are separated by a comma. However, the REST API configuration screen expects a comma delimeter between key=value pairs. Notice that I have used a ^ delimiter instead, as I need to use commas for my track values.
> Response Type: json
> Streaming Request: Yes (ensure the box is checked)
> Request Timeout: 86400
Here we are setting the timeout to be 86400 seconds which is the number of seconds in a day. As long as you have at least one tweet come through per day, then you will be ok. If the timeout window is less than the amount of time between tweets streaming in, then the data input will timeout and not recover without re-enabling the input or I would imagine a Splunk restart.
> Delimeter: ^ (or whatever delimeter you used in the URL arguments field)
> Set Sourcetype: Manual
> Sourcetype: Tweets (or whatever sourcetype name you want)
> More Settings: Yes (check the box). Optionally provide a host name and an index you want the tweets to go into. The default index is main.Note: For reference, the above configuration is stored in etc/system/local/inputs.conf

This is what the final screen will look like. Hit the “Save” button when everything looks good.
twitter_finalstep10

Search the Tweets!

You are all done! After hitting save, the tweets should start coming in immediately. Assuming you used a sourcetype of twitter, you can now go to the search bar in Splunk and run this query:

sourcetype=twitter earliest=-1h

You should see data coming in. You will notice that Twitter includes a TON of fields with each tweet – it is quite awesome actually. All the usernames, hashtags, users in the tweets, URLs (even translated URLs) are all extracted and searchable.

Of course, the above does simplify things. You should definitely read the the Twitter API documentation properly.

New messages dealing with General Data Protection Regulation are flooding my mailbox today warning that countdown to GDPR enforcement has begun.

GDPR is approaching fast, but don’t panic you’ve still got some time to get compliant (25th of May, 2018).

General Data Protection Regulation (GDPR) proposed by the European Commission will strengthen and unify data protection for individuals within the European Union (EU), whilst addressing the export of personal data outside the EU.

The announcement of an agreement to finalize GDPR was made in December 2015 and following a vote by the EU parliament, the compliance deadline for GDPR was set for May 2018.

The GDPR requirements as well as the amount of internal collaboration that will be needed to address them means organizations need to plan for compliance now.

For editors and IT security consulting firms, it’s one more time the right event for shaking cash machine.

The primary objective of the GDPR is to give citizens back control of their personal data. Once GDPR takes effect it will harmonize previous and other data protection regulations throughout the EU.

Gemalto

If you are still unclear on the GDPR basics, Gemalto can help clarify your questions : Visit our GDPR resource centre now!