Complex Event Processing using Spark Streaming and SparkSQL

September 11, 2014 / 0 comments

Introduction Apache Spark has come a long way in just the last year. It now boasts the ability to not only process streams of data at scale, but to “query” that data at scale using SQL-like syntax. This ability makes Spark a viable alternative to established Complex Event Processing platforms and provides advantages over other…

Read more →

Spark Streaming and Kafka Quickstart

July 31, 2014 / 0 comments

Description In this blog post, I will show how to: Get the Spark KafkaWordCount example to run Set up your environment Requirements: Zookeeper (3.4.6+) Kafka ( Spark (1.0.1+) Java JDK (1.7+) Step 1: Install and configure Zookeeper ¬†Download a release of Zookeeper (3.4.6+) ¬†and extract to your preferred location (I will refer to this as…

Read more →

Generating md5 checksums for all files in a directory

May 14, 2014 / 0 comments

Generating md5 checksum hashes for files in a directory can be a useful way to validate the integrity of files when transferring them to another machine. How though, do we go about generating a single file with ALL the md5 hashes for files in a particular directory, given a Unix machine? It’s easy: find ….

Read more →

Using Maven central repository JARs in Android projects

March 15, 2014 / 0 comments

Are you completely satisfied with the standard set of Java libraries available in Android projects? Is there a special Apache Commons library you’re dying to rewrite? Well, if not, maybe including external JARs from Maven central is for you. All you need to do is to configure your project’s ‘build.gradle’ file to use search/use Maven…

Read more →

OODT Crawler AutoDetectProductCrawler MimeType not being detected

October 31, 2013 / 0 comments

Problem: I was working with the AutoDetectProductCrawler, and had added an entry to both the crawler’s mime-extractor-map.xml as well as the mime-types.xml file which the former points to. However, every time I tried to ingest a special .csv file, the crawler was unable to find the associated mime type specification (spec). I rechecked the glob…

Read more →

Configuring the Java editor in Eclipse to use spaces in place of tabs

August 29, 2013 / 0 comments

I was attempting to switch my use of tabs (in Eclipse Juno) to pure spaces. The problem was, even though the Text Editor configuration within Eclipse was set to use “Insert spaces for tabs”, this was not having any effect on my Java code. Solution: the Java editor is using an Eclipse editor “profile” which…

Read more →

Getting OPSUI 0.6-SNAPSHOT working with OODT WM 0.3

April 22, 2013 / 0 comments

Getting OODT OPSUI 0.6-SNAPSHOT working with OODT Workflow Manager 0.3 should more or less work out of the box. Please make sure to modify the following though: Edit your opsui webapp’s context.xml to use the old workflow-lifecycle file. cd opsui/ vim src/main/webapp/META-INF/context.xml Change: “[WORKFLOW_HOME]/policy/wengine/wengine-lifecycle.xml” to “[WORKFLOW_HOME]/policy/workflow-lifecycle.xml“

Accessing data from Amazon S3

April 17, 2013 / 0 comments

The following quick tutorial guides you in how to download and access S3 data from the command-line (on *nix) machines. Requirements: An AWS S3 storage account already set up An AWS Access Key An AWS Private Key s3cmd ( Steps: Download s3cmd Mac (using homebrew): brew install s3cmd Ubuntu: sudo apt-get install s3cmd Initialize your…

Read more →

Compiling and executing a Processing sketch as a Java class

February 12, 2013 / 0 comments

Processing sketches (PDEs) are translated from the processing syntax to pure Java and are compiled and packaged into executable JARs. Is it possible to by pass the packaging step and go straight from the Java code to your own compiled class? Yes it is! NOTE: Apart from your source and compiled class file, you will…

Read more →

Using a servlet config file with a CXF JAX-RS web-service

February 6, 2013 / 0 comments

I spent some time searching for how to do this, so thought I’d share my findings. Problem: I created an Apache CXF JAX-RS web-service using the Maven2 archetype “cxf-jaxrs-service (Simple CXF JAX-RS webapp service using Spring configuration)”. Using this as a boiler-plate, I then created my own RESTful web-service endpoint. The problem was, without having…

Read more →