How To Obtain ActiveMQ Queue Sizes using Python and Stomp

November 14th, 2012

This post outlines how to use Python and its Stomp client to obtain the queue length of an ActiveMQ queue.

This unit test test cases quickly illustrates how to use ActiveMQ Statistics plugin. The basic use case for doing this is:

  • Enable the ActiveMQ Statistics Plugin on the ActiveMQ instance
    you are interacting with.
  • Send an empty message to ActiveMQ.Statistics.Destination.QueueName with a JMSReplyTo header specifying the queue to return the stats on.
  • Pop the return message off the return Queue to obtain your stats.

To enable the ActiveMQ Statistics Plugin add this to your ActiveMQ configuration.

<broker>
<plugins>
<statisticsBrokerPlugin/>
</plugins>
</broker>

The Unit Test

#!/usr/bin/python

import unittest
import stomp
import time

class QueueListener(object):
"""A queue listener for objects."""

def __init__(self, handler):
self.handler = handler

def on_error(self, headers, message):
print "Received an error fetching message: %s" % message

def on_message(self, headers, message):
try:
self.handler.handleMessage( headers, message )
except Exception as e:
import traceback
print e
print traceback.format_exc()
pass

class testActivemqStats(unittest.TestCase):

def setUp(self):
pass

def getConn(self):
conn = stomp.Connection( host_and_ports= [ ( "127.0.0.1", 61613) ] )
conn.start()
conn.connect()

return conn

def handleMessage(self, headers, msg):
print "Message: %s ; %s" % ( str(headers), str(msg) )

def testSomething(self):
sendConn = self.getConn()
recvConn = self.getConn()

statQueue="/queue/ActiveMQ.Statistics.Destination.%s" % '/uruk/parseQ'
statRecvQueue="/client/stats"

# The ActiveMQ Stats Plugin returns the stats as an ObjectMessage which needs to be transformed
# to JSON or XML to be useful to the Python STOMP client
recvConn.set_listener('MyActiveMQStatListener', QueueListener(self))
recvConn.subscribe(destination=statRecvQueue, ack='auto', transformation='jms-object-json')

# The ActiveMQ Stats Plugin only respondes to messages with a JMSReplyTo header.
print "Sending a request for stats to: %s" % statQueue
sendConn.send( "", destination=statQueue, headers={'reply-to': statRecvQueue, 'JMSReplyTo': statRecvQueue} )

while True:
import time
time.sleep(60)

if __name__ == '__main__':
unittest.main()

The output from the Unit Test should look like this:

{
'expires': '0',
'timestamp': '0',
'destination': '/queue//client/stats',
'priority': '0',
'message-id': 'ID:elric-59664-1352862988577-2:1:0:0:10',
'type': 'Advisory',
'transformation': 'jms-object-json'
}; {
"map": {
"entry": [{
"string": "memoryUsage",
"long": 0
}, {
"string": "dequeueCount",
"long": 0
}, {
"string": "inflightCount",
"long": 0
}, {
"string": "messagesCached",
"long": 0
}, {
"string": "averageEnqueueTime",
"double": 0
}, {
"string": ["destinationName", "queue:\/\/\/uruk\/parseQ"]
}, {
"string": "size",
"long": 0
}, {
"string": "memoryPercentUsage",
"int": 0
}, {
"string": "producerCount",
"long": 0
}, {
"string": "consumerCount",
"long": 0
}, {
"string": "minEnqueueTime",
"double": 0
}, {
"string": "expiredCount",
"long": 0
}, {
"string": "dispatchCount",
"long": 0
}, {
"string": "maxEnqueueTime",
"double": 0
}, {
"string": "enqueueCount",
"long": 0
}, {
"string": "memoryLimit",
"long": 1048576
}]
}
}

Happy Queueing

Oracle PL/SQL Objects

April 5th, 2011

This post gives a high level overview of how one could use Oracle PL/SQL Objects. Much of this post was inspired by Steven Feuerstein’s Chapter 26 of Oracle PL/SQL Programming, 5th Edition. Buy it. It is a good book for a PL/SQL developer.

Let’s jump into it and create some silly Objects to play with.

[sourcecode language=”sql”]
CREATE OR REPLACE TYPE animalType AS OBJECT (
name VARCHAR2(30),
MEMBER FUNCTION makeNoise RETURN VARCHAR2,
MEMBER PROCEDURE printNoise
) NOT INSTANTIABLE NOT FINAL;

CREATE OR REPLACE TYPE BODY animalType
IS
MEMBER FUNCTION makeNoise RETURN VARCHAR2 IS
BEGIN
RETURN ‘‘;
END;

MEMBER PROCEDURE printNoise AS
BEGIN
dbms_output.put_line(name || ‘ says ‘ || self.makeNoise );
END;

END;
[/sourcecode]

Our base type, that can’t be instantiated but that can be extended by other sub types.

Now here are the sub-types that can be instantiated.

Now let’s try the object out. Please note the polymorphic behavior at work.

You should see something like the following in our dbms_output buffer.

sheep says baaahh
lion says roar

The geek in me says “Neat so far!” The skeptic in you says how do we save the object data? “Easy!” says I. Let’s create DAO, Data Access Object, in the form of a package. This example shows the data being stored in an object column.

Brilliant! Let’s save some beasties to the database:

Even better let’s select that data from its table.

Object columns are neat, but Object tables are even better. Let’s re-write the DAO to use an object table instead.

Let’s give the new DAO a spin:

Of course, Oracle gives us all kinds of ways to query the meta data about the objects.

OK. What if you need to pull your objects from an existing legacy table?

Insert some new data into our “legacy” table.

How do the views work. Try these queries to see:

References:

Recovery of a Vista Laptop Hard Drive (using Linux).

February 15th, 2011

Some bad sectors cropped up on my wive’s laptop and her Vista Home Premium x86 would not boot. I, of course, am the resident tech support dude in the house, so it was up to me to restore Vista to a boot-able state.

I took a new drive allocated for the long overdue Windows 7 upgrade and copied a backup of the damaged Vista drive to it. At first, I tried to use SpinRite to scan and repair defects of the original Vista disk, however it took too long. So, I decided to use ddrescue to recover the data on the disk:


DDRLOG=/tmp/ddrescue-20110208.log
# see info ddrescue for more complete notes on its use.
# avoid the bad sectors on the first dump of data
ddrescue -n /dev/sda /dev/sdb $DDRLOG

# rescan the drive around the bad sectors and recover data (attempt 3 reads
# on bad sectors)
ddrescue -d -r3 /dev/sda /dev/sdb $DDRLOG

# save DDRLOG to some place.
ddrescue is part of the excellent “Ubuntu Rescue Remix” distribution. Boot into this Linux distro and then let the magic happen with the above ddrescue commands. For info about this distribution and other recovery tools please see this article.

Note: I had to rescue the disk in its entirety to have Windows Vista boot normally after the data recovery on the new drive. “ddrescue -n /dev/sda2 /dev/sdb2″ did not work for me. I am not sure why. A Vista setup DVD was able to detect /dev/sda2 and found a Vista OS installed on the partition. More over the Vista Setup DVD was able to repair the file system. And it attempted to perform Start Up Recoveries of the Vista partition. However in the end I was not able to make Vista boot properly using the single partition ddrescue copy. [I wasted three calendar days because of this mistake. Don’t repeat my mistake.]

Copying the disk and all its partitions in its entirety in one fell swoop did work for me. So I recommend the first set of ddrescue commands documented in this post when working with Vista.

Why was restoring Vista to a boot-able state so important before the Windows 7 upgrade? In order to install Windows 7 in an “upgrade” mode, Windows Vista must be boot-able and able to run the Windows 7 installer. (Thanks, Microsoft, for making the simple hard.) The Upgrade mode of the installer was preferred by my wife. The Upgrade mode retains all the installed Windows applications and other things added to Windows (such as fonts in my wife’s case).

After the original Vista Home Premium x86 was restored to a bootable state, I attempted an upgrade from Vista to Windows 7. (It was overdue anyway!) We had purchased a Windows 7 Ultimate 64-bit edition DVD, however this edition of Windows can not be applied to a Vista HP (32 bit) in Upgrade mode. Nice, Microsoft! Thanks again for making the simple hard!

So back to the store, I picked up a copy of Windows 7 Home Premium (32 bit). This version of Windows 7 allowed the original Vista Home Premium x86 edition to be upgraded.

After all this, Windows 7 happily boots on the laptop with the restored data. To date of this post, no lost data has been reported by my wife, so I assume the majority of the data on the Windows partition was preserved by ddrescue.

Thank you ddrescue for making the hard easy!

How to ignore Invalid byte sequences in a Postgres Dump File

November 15th, 2010

I needed to restore a client Postgres database recently.  However I discovered the pg_dump v8.3.9 dump utility produced invalid byte sequences in the dump file.

When importing the file using the following v8.4.5 psql command:
- psql -U username dbname -f pg-dbbackup.YYYYMMDD.dump

Each and every invalid byte sequence in the dump throws an error like the following one at a time:

psql:pg-dbbackup.20101115.dump:45711: ERROR:  invalid byte sequence for encoding “UTF8″: 0xc8ed

HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by “client_encoding”.
CONTEXT:  COPY PostRequestField, line 13274

Note, you can identify the line where the byte sequence occurs by finding the line where the ‘COPY “TableName” FROM stdin;’ statement occurs. Add this line number with the 13274 reported above and this should be the line where the byte sequence occurs in the dump file.

Note, if you have one invalid byte sequence, you may have thousands of other invalid byte sequences. One way to deal with the invalid byte sequences in one fell swoop is to use the iconv command.

iconv -c -f UTF8 -t UTF8 pg-dbbackup.YYYYMMDD.dump > pg-dbbackup.YYYYMMDD.dump.txt

pg-dbbackup.YYYYMMDD.dump.txt should import without invalid byte sequences now. Note you might be stripping important information out of your data. But at least the dump re-imports into Postgres database and you can do things with it.

Simple Linux Backup Method

May 13th, 2010

For a simple Linux backup solution, I often use rsync.

You need to identify the file systems that need to be backed up. From the command line:

df -h

As a side note you should create a separate file system for your applicaton. RHEL or Linux does not normally do this for you. The advantage to doing this is, it is much easier to upgrade the operating system when your application files are not stored on the root file system.

Back to the task at hand, let’s say you added an extra disk to your server and have it mounted at /backup. You can run rsync from the command line:

rsync -aPv --one-file-system / /backup/root/

The above command should backup your root file system to /backup/root/. I usually like to add dates to these backup directory so I can restore a file from a point in time. The -P rsync option gives you verbose progress information. This is great when you run it interactively.

You can automate this command by putting it into a shell script.

# create file: /etc/cron.daily/backup.sh

#!/bin/bash

/usr/bin/rsync -av --one-file-system / /backup/root/

You will need to set the execute bit on the newly created backup.sh script

chmod +x /etc/cron.daily/backup.sh

Once you have created /etc/cron.daily/backup.sh it should be run every day by the Linux cron.

The Trouble with Tivos

March 4th, 2010

Tivos and DVRs are great until they stop working. And then you hear all about it from wives and daughters. Oh, how they scream bloody murder when their precious shows and entertainment are not being recorded. It is enough to lose sleep over.

I had a particularly troubling episode occur recently. When we originally signed up for DirectTV, I ordered three DirectTV receivers for the household:
- living room
- basement
- master bedroom

I then received a Samsung SIR-4040R DirectTivo unit from my friend Tom. (Thank you Tom.) I plugged the Tivo into our home video distribution unit and the family can watch DirectTV recorded by Tivo throughout the house. So our need for multiple receivers abated and we had stopped using them. In fact, we had consolidated to:
- one receiver in the living room (for Tivo outages or live events)
- tivo in the basement (redistributed to the rest of the house)

Two receivers sat unused and unplugged in the basement. So after 3 years of paying for something we weren’t using, this past Saturday I de-activated the unused receivers. And something of a Tivo nightmare began the following Sunday. The Tivo stopped recording with the following message “Activate your DVR Service, Message #81. Please call DirectTV”. The Tivo would allow us to watch previously recorded shows and it would let us view live satellite TV. However every attempt to make it record a show resulted in the Message #81.

So I called DirectTV, the helpful reps tried several times to reset the activation of my Tivo’s Access card to no avail. The last DirectTV tech rep told me, “Your Tivo unit is only partially supported, Sir. If it doesn’t respond to these reset commands I am sending, than we have to assume it is in-operable. Let me ship a supported R15 DVR unit to you to replace your broken Samsung.”

I begrudgingly accepted the R15 replacement. I thought my DirectTivo was toast and that the family’s bliss of watching recorded shows on which ever TV they wanted was at an end. In a vain attempt to solve the problem my self, I logged into the www.directv.com support tools and tried to re-activate the Tivo’s access card myself. All the attempts were in vain, however I did notice the DVR Service on my account was still active.

Several days later we received the new R15. I began the install process. As soon as I powered the R15 on, it wanted to be activated. “Please call DirecTV” it asked. So I did. The automated DirecTV system asked: “Do you want to add the DVR service to your account for $7 per month?”. I thought this is strange “DVR Services” are already active on my account from the Tivo. I responded “Yes” anyway and tinkered with the R15. The R15 continued to complain about its “Access Card” being in-active, even though I just walked thru a DirecTV representative activating it for me.

While waiting on hold to talk to another DirecTV representative, I noticed the Tivo was working again. Brilliant I thought! All my troubles are at an end! And they are. The Tivo is recording again and I have an R15 to play with to observe all the well placed complaints about its clumsy interface.

The DirecTV R15 is probably great if you are coming from a plain old DirecTV receiver. The R15 remote has a gazillion buttons that do this, that and the other thing but not what you want to do at the moment. Many of the buttons are context sensitive. In other words the buttons are useless until R15 UI interface is in a certain mode when the buttons can be used. The R15 interface is also busy and hard to navigate, but it looks an awful lot like the interface of a normal DirecTV receiver. The Tivo interface on the other hand is stream lined and simple down to the remote. DirecTV can take my Tivo remote from my cold dead hand.

I am looking forward to a HD Tivo / DirecTV unit that is suppose to hit the market this 2010. Apparently the product will contain Tivo software only. So the box will be built by DirecTV, but a portion of the software will be written by Tivo. So let’s hope it is a good product combination and that DirecTV supports the product far into the future.

To summarize, if your DirectTivo gives you a “Message #81″ and your DirecTV rep. tells you “You need to retire the unit.” Try de-activating your DVR Service for your account and re-activating the DVR Service. It seemed to do the trick for me.

How to make audio work with a Windows XP VMware guest hosted on Fedora 11

July 12th, 2009

Update: December 2010. I have given up using Skype under VMware for quite some time now; for almost a year! The Linux Skype Beta Client 2.1.0.81 software is quite good and supports my “Skype-In” and “Skype-Out” features I previously needed VMware for. So, if you need to use Skype with Linux, I recommend you install the Linux Client it is very stable and full featured now!

My Windows XP instance has enjoyed running on bare metal for the last time in my office. The machine flaked out for no apparent reason and crashed every 10 to 45 minutes. I have a Linux laptop where I do most of my work: e-mail, documents, web browsing and Java development. However recent clients have engaged me to write (gulp) .NET projects. So I have needed a XP workstation for the past several years for SharpDevelop, Visual Studio and the odd Windows tool.  Along with these Windows tools I have fostered an addiction to Skype 3.8.

I use Skype to route calls from my office number to answer on my Desktop. I also use Skype Out to dial US long distance numbers and to conference call multiple parties together on one line to make decisions happen. I should use SIP and Ekiga but I don’t at the moment.

So when my XP workstation melted down, my development and communication tools melted down at the same time.

I was so put off by Microsoft for destroying my productivity that I was determined to put XP into a virtual machine to make restoring it back to health faster. I am no stranger to VMware Workstation and Linux is my friend. So I thought it would be a snap to cobble together a working Windows XP virtual machine with Skype 3.8. I was partially right but it took the better part of two days to make VMware audio work the way I needed.

I installed 64 bit Fedora 11 on a pair of new software raided disks that I installed in my Dell Workstation which previously ran XP SP3. The Fedora 11 install went smooth. Everything worked well. The new Noveau NVIDIA driver drove my dual monitor setup well. The PulseAudio system delivered rock solid audio. VMware installed with relatively little effort after following the instructions found on this site: http://communities.vmware.com/thread/203231

With VMware working, I took my Windows XP virtual machine and resized its partition from 16 GB to 64 GB using clonezilla, gparted and ntfsresize. This all took some time and a lot of patience to execute (By the way why can’t XP check it’s file system reliably and why doesn’t NTFS continuously defrag its file system like ext3?). However there was still one nagging problem, the sound card was not being detected by VMware. Apparently the Linux version of VMware 6.5.2 requires a working /dev/dsp OSS device file to work. The default PulseAudio installation in Fedora 11 doesn’t provide a /dev/dsp device file. Nor did the PulseAudio padsp utility work for me with VMware 6.5.2.
I desperately needed audio support in VMware to run Skype 3.8 in XP. (Yes I know Skype 2.0 runs natively under Linux, but Skype 3.8 supports conference calling with real phone numbers, a 3.8 feature I have become accustomed to.)

Running “padsp vmware” did not produce working sound for me. VMware complained of not being able to detect the sound card.

I found these instructions to make VMware work with PulseAudio under Ubuntu 8.10 as described on this site: http://communities.vmware.com/thread/163605 . However the instructions lead me down a dead end and did not work under Fedora 11.
I experimented a little on my 32bit Fedora 10 and 64 bit Fedora 10 workstations. I discovered I could make audio work under VMware quite easily on Fedora 10. I also discovered that Fedora 10 provides a /dev/dsp device file from its OSS sound adapter. Determined that this was the answer to my 64 bit Fedora 11 workstation audio problem, I researched how to restore the /dev/dsp device file to Fedora 11.

I removed the alsa-plugins-pulseaudio RPM.

  • yum remove alsa-plugins-pulseaudio

Then I made sure the OSS modules were being loaded

  • sudo modprobe snd_mixer_oss
  • sudo modprobe snd_pcm_oss
  • sudo modprobe snd_seq_oss

I added my user account to the “audio” group . I explicitly specified my XP VMware instance to use the /dev/dsp device file with the sound.file parameter and set sound.autodetect = “FALSE”. I rebooted and crossed my fingers.

After more research I discovered this Fedora 11 bug, https://bugzilla.redhat.com/show_bug.cgi?id=501051. The author of comment #17 suggests:

“I needed to uncomment a line in /etc/modprobe.d/dist-oss.conf to load those
modules.. Appears they’re caught up in the “disable OSS” feature of F11.”
Uncommenting the referenced line in dist-oss.conf, ensures the aforementioned oss sound modules are loaded when Fedora boots.
To my surprise VMware was able to detect the sound card and allowed XP access to it. Skype 3.8 is now able to make calls. It appears I am back in business. I wanted to share the steps I took to make audio work with VMware and Fedora 11 in the hope they save someone the time and effort I spent to find my solution.

I also wanted to express my concern that the Fedora project is not good at maintaining backward compatibility. I can see the benefit of PulseAudio. It seems to make sound much more reliable under Linux. But the Fedora Project should make a better effort to maintain backwards compatibility with applications behind the curve such as VMware. From an outsider’s perspective it seems like it would be trivial to create /dev/dsp device files on demand and send the audio to the PulseAudio system. (padsp is suppose to do this but apparently doesn’t work with all applications including VMware) If such a capability existed, it would have been easier to resolve my VMware problems.

Enough ranting. Happy hacking.

The Church channel spirits in Michigan

July 3rd, 2009

[For those interested in some great video of the Church live on the “So Love May Find Us” check out: http://blog.kexp.org/blog/2009/07/08/live-video-the-church-live-at-the-triple-door/ ]
My wife graciously granted me the rare opportunity to make a pilgrimage to see the Church (http://www.thechurchband.com/) at two tour locations. (Thank you Collen. I love you.) I eagerly seized the opportunity and hopped in the auto for two consecutive nights: once to Grand Rapid’s The Intersection and once to Ferndale’s Magic Bag.

The two venues couldn’t have been more different. The Intersection was a fine clean place for a Church concert. However the acoustics of the cavernous concert hall left much to be desired. The spot near the sound mixing station towards the back of the hall provided the best sound at least for my ears.

Adam Franklin & The Bolts of Melody took the stage first. A decent band. I am considering exploring their discography, but this post is not about Bolts despite how good they were.

Things haven’t changed much with the Church. They took the stage in their usual order: Peter Koppes, Steve Kilbey, Marty Wilson-Piper, Craig Wilson(on keyboards) and Tim Powles. The first event of the concert was of course an equipment mal-function. A low murmur (a FM radio signal?) resonated on the amplifier. Steve’s first response to this was “We are channelling spirits tonight.” The audience laughed and the band proceeded to perform despite the mal-function. The murmur was troublesome for many of the quiet moments in their set list. The alleged spirits and the crowd put Steve on edge a few times.

I suffered through the poor acoustics up front for about 5 songs then I thought better of missing the best part of the concert, the lovely melodies crafted by the band. So I moved towards the sound mixing station. It was phenomenal to see the guitar work of Marty and Peter up close. But it was even better to hear the guitar with clean acoustics in the center of the room.

The stand out songs from the play list on this night for me were:

  • Block
  • After Everything
  • Almost with You
  • Pangea
  • Reptile (I loved the intro, Steve’s best impression of a snake.)

All in all, the Grand Rapids concert was a great Church concert. I thought nothing could top it until…

The Magic Bag is an old Movie Theater renovated to be a small concert space / bar. There are tables through out the space and wait staff to deliver beverages to everyone’s thirsty throats. Moving around this venue is a little harder. So I stayed near the front to the right about 15 feet away from Marty.

The acoustics at the Magic Bag were much better than the Intersection and there weren’t any “channelled spirits” at this concert. All the instruments could be heard clear as bells. The boys played their hearts out delivering one of the best live performances I have heard them play in years.
Highlights of the set list for me at the Magic Bag were:

  • Block (again)
  • North South East and West
  • After Everything (again)
  • Deadman’s Hand (again)
  • Pangaea (again)
  • Reptile (again)
  • An Interlude
  • Hotel Womb

A Church concert wouldn’t be complete without a performance of “Under the Milky Way”. Both nights the rendition of this song was solid however it was vastly inferior to the performance of “Reptile” from the same “Star Fish” album. At the risk of disappointing fans, I think the band should consider retiring “Under the Milky Way” for a tour or so. Maybe this absence of play would increase the band’s fondness for the song.

Steve was slightly agitated in both performances by hecklers on his left. However he compensated for his slight temper with amazing vocals. Peter and Marty played remarkable as they always do. Timothy Powell, the band’s tremendous drummer, played astonishing sets. I personally would like to thank Timothy for restoring the band to their former glory. Someday I will be pleasantly surprised to see a live performance of “They Come in to Take Your Place” with Tim on the vocals.

Protected: How to install the Trac WebAdmin Module

July 19th, 2007

This post is password protected. To view it please enter your password below:

Protected: How To Install Trac on Windows

July 9th, 2007

This post is password protected. To view it please enter your password below: