Company Blog

Streaming multimedia content with Apache

Wed, 4 Nov 2009 09:20:47 +0000

Although the intention of this blog is to post about business intelligence related topics the nature of a blog makes it an ideal medium to log digital content and to subsequently retrieve this information. This post is more of an aide memoir although hopefully it will prove useful to others.

Those users who have looked around the site, will have come across several multimedia clips in the form of flash videos. Well today I decided to enabled pseudo streaming of these videos on the website using an apache extension.

So what exactly is pseudo streaming ? When you visit a website such as youtube you can skip within the file to a point of interest, this is enabled by streaming the content to the client. It confers several advantages, firstly the entire video does not need to be downloaded upfront which makes the whole experience more responsive and if the user effectively fast forwards the content then only data from that part of the video onwards is streamed.

Although this sounds simply the commercial reality is quite different. Typically to stream media you are required to purchase a separate streaming media server, which is often an expensive proposition. For unix users there is the possibility of using the open source Darwin Steaming Media server from apple, but again this means installing a separate software component.

Fortunately there are some alternatives, although these are not full streaming media server they are a decent compromise.

The apache module mod_flvx can be used to achieve pseudo streaming of media. Effectively what happens here is when a user skips to the middle of a video the module will read the contents of the file, truncate the part the user wishes to skip and the send the remaining content.

To use this module the following steps were taken.

1) Install the httpd-devel tools

sudo yum install httpd-devel

1) Download the extension from the apache website

1) Compile the extension

apxs -c -i ./mod_flvx.c

1) Edit your apache configuration file (often called httpd.conf) and add the following lines

LoadModule flvx_module modules/mod_flvx.so
AddHandler flv-stream .flv

1) Restart apache

/etc/init.d/httpd restart

The strange nature of the XMLA Drop Command

Fri, 23 Oct 2009 19:24:09 +0100

During the past few weeks I have spent a great deal of time writing XMLA. I have come across some unexpected issues and behaviours, some which can be explained and some which I currently have no explanation for.

Generally I have to admit that I am fairly impressed with XMLA, although I do find some aspects of it frustrating, perhaps some this is due to the fact that XMLA has been around for so long although some of the design seems illogical.

My first encounter with Analysis Services (SSAS) was with the 2000 version. I recall encountering so many issues... we were running on 32 bit machines and found that it was impossible to process our account dimension, which held around 3 million members. When we first hit this issue we were able to use the 2 gb switch to give a temporary reprieve but ultimately it became apparent that the product did not scale in terms of it’s MOLAP offering. The issue in this case being the fact that incremental dimensional process was really just a marketing term. In reality it pulled all of the records from the relational dimension table and then compared these with the current olap dimension and inserted / updated and deleting the data where necessary. In effect it said “select * from sourceTable”.
Since we realised that the product couldn’t scale the decision was made that we would only hold a fixed amount of data and create stored procedures which would purge older facts and no longer required dimension members.

Things have improved a great deal in the 2005 and 2008 offerings and 64 bit servers and now also much more ubiquitous. On the current project we will be deploying our analysis services cubes on a server with 256Gb of RAM and 32 cores. With this kind of computing power you could hold a vast amount of data in memory, unfortunatley there is now easy way to pin SSAS partitions in memory, indeed Microsoft chose to remove the pin table feature from recent version of SQL Server.

So generally things have improved a great deal, however when you have a very demanding user base and tight SLAs to adhere to there is always the need for compromise. One of the unfortunate issues with using MOLAP cubes is that any structural change to a dimension requires that dimension needs to be fully processed, which has the knock on effect of invalidating (deleting) any cube partitions associated with that dimension.

The fact that all of your historical data is wiped out until you can reprocess really is not ideal. Due to this it is fairly common to build a system which is a compromise. Recent records are held in MOLAP and historical data is held in ROLAP.

One other annoyance is the fact that page level dimension members are treated as separate entities in relation to pivot tables. This is obviously done for performance reasons, however it does mean that many members are displayed which have absolutely no relation to other filtering attributes.

Therefore if you are going to have a mixed architecture of MOLAP and ROLAP then it might be worth splitting these into separate cubes, so that when you display dimension members in the MOLAP pivots you are displaying the more relevant members, yes they still will not be inter-related but the very fact that the dimension is smaller means less combinations are shown.

Given this requirement how can you delete dimension members? Well you could do this act the relational level, and process the dimension but this would also affect your relational ROLAP model and in any case would be a compromise, since the set of dimension members required to support the last month of data will be less than that required to support the historic data.

Within XMLA there is a delete command, so the command below can be used to delete the elements from an Analysis Services Dimension.

<Drop xsi:type="Drop" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ddl2="http://schemas.microsoft.com/analysisservices/2003/engine/2"
xmlns:ddl2_2="http://schemas.microsoft.com/analysisservices/2003/engine/2/2"
xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">
<Object>
    <Database>Warehouse</Database>
    <Cube>Sales</Cube>
    <Dimension>Client</Dimension>
</Object>
<Where>
    <Attribute>
    <AttributeName>Account Number</AttributeName>
    <Keys>
        <Key xsi:type="xsd:integer">5000</Key>
    </Keys>
    </Attribute>
</Where>
<DeleteWithDescendants>true</DeleteWithDescendants>
</Drop>

So what is so weird about this command ? Well firstly the fact that is was necessary to specify a cube. A dimension can be shared across multiple cubes so it seems kind of illogical that it is necessary to specify a dimension. However this is the only way I could get the command to work, without a cube being specified the command refused to execute. Perhaps this is some kind of check to validate if data is held in the cube which could be orphaned by deleting the dimension key, that would seem like a reasonable explanation.

The other thing is that the command actually deleted the element from the underlying dimension table in an Oracle database. This is exactly the kind of behaviour which we were trying to avoid. This is indeed surprising since the dimension is configured to be held as MOLAP and is not a write-back enabled dimension, so why was the data removed from the warehouse ?

At some point in the next week or two I will come back and investigate the subject a little further, perhaps then I will be able to shed some light on the subject.

Installing Perl Packages on CentOS - The magic of YUM

Sun, 4 Oct 2009 21:03:36 +0100

Sometime ago I wrote a perl CGI program to send mail using STMP. In order to create this script I was write required to install some perl packages which proved relatively straight forward to install using CPAN.

However recently I decided to migrate this code to a bare bones CentOS machine.

I ran cpan and then realised I was missing a C compiler. So quit and performed the following operations

yum -y update
yum install gcc
yum install make
yum install unzip

Now I ran cpan again and tried to install the first dependant module in this case Net::SSLeay

Unfortunately I was presented with the following message :

*** Found OpenSSL-0.9.8e installed in /usr
*** Be sure to use the same compiler and options to compile your OpenSSL, perl,
and Net::SSLeay. Mixing and matching compilers is not supported.

Oh no!! I certainly did not want to have to recompile perl, and OpenSSL with the same compiler.

However on this occasion I was saved by the yum package manager. A quick check with the command yum whatprovides 'perl(Net::SSLeay)'
produced the following message:

perl-Net-SSLeay-1.30-4.fc6.x86_64 : Perl extension for using OpenSSL
Matched from:
Other : perl(Net::SSLeay)

What a great result, there is a package already available, no need for CPAN or compiling. So the next step was to issue the command

yum install perl(Net::SSLeay)

So far so good, yum indicated the module was now installed, so the following command was issued to confirm if this was indeed the case and if perl could use the module.

perl -MNet::SSLeay -e 1

No errors were shown with the above command, so all was looking good. Yum resulted in a clean install, which has the added benefit that it can be upgraded along with the rest of the system as and new versions become available.

I have never installed perl modules using a package manager before and have always either used CPAN or compiled them from C. I now feel like I have been living in the dark ages and from this day forward will aways try to use the built in package manager where possible.

This approach also worked like a dream for the following modules

yum install 'perl(IO::Socket::SSL)'

JPalo is a really nice unfortunately there is no default mac download

Wed, 29 Jul 2009 09:43:39 +0100

We really like the JPalo eclipse client. The ad-hoc browsing is excellent, it makes stacking dimensions very simple. The only set back is that there is no version of OSX available in the downloads section. However making this great tool work on the mac is a simple task which can easily be accomplished by any user.

The following step are required to perform this task
• Download the Mac OSX Java EE Edition from here
• Extract the archive from the above file and copy the eclipse folder to the applications section in finder
• Complete the necessary steps to download the eclipse client in either the windows or linux format from the JPalo website (The actual type does not matter, since we simply extract the necessary dependancies from the archive)
• Extract the file downloaded from the JPalo website
• Enter the features directory of the JPalo archive and copy all the the directories which are contained within. Paste these folders into the features folder in the eclipse application which was previously downloaded

• Copy all of the files which begin com.tensegrity from the plugins folder in the archive downloaded from the JPalo website to the corresponding folder in you eclipse directory.

The core files are now in place.

Start eclipse and change the perspective to one of the palo perspectives.

As can be seen you can now perform adhoc queries and drag and drop dimensions to slice and dice the cube at your will.

We have not tried out all of the features offered by the tool and it is possible that some of the features, such as searching for dimension elements are missing since these might depend on code in the Apache Lucene library. If you want to ensure that every feature will work then copy all of the jar files from the plugins folder in the JPalo archive to the the corresponding folder in Eclipse.