• the ability to easily create rich user interfaces with diagrams, tables and data graphs.
• the ease of use of the graphical programming interface, which provides many pre-build libraries and instrument drivers.
• the very high execution speed of Labview scripts.

However, as I started to build more complex experiments it became increasingly hard to come up with good solutions for interfacing them through Labview. For example, some limitations which quickly started to annoy me were

• the lack of a text-based programming language. Labview uses a graphical programming language called "G", which makes it easy to “write” but hard to copy and read code.
• the dataflow programming model of Labview, which makes it very hard to reuse individual parts of programs and to make use of object-oriented programming techniques. 

So after all, I wasn’t fully happy with using Labview for controlling my experiments since I felt that I would be somehow limited in the extension of my measurement system. Luckily, one day a post-doc working in our group told me that in his former lab a programming language called Python was used to control and interface most of their measurement equipment. Up to that day I only briefly heard of this language and I didn’t have the slightest experience with it. Nevertheless, I became very curious and I wanted to learn more about it. And as it turned out, I found Python to be a great tool for my work: It was flexible, easy to learn and it had many useful libraries that I could use for my data acquisition and processing needs. In fact, as of today Python seems to become more and more the standard scientific programming language, often replacing older and more inconvenient languages such as C/C++ and Fortran. So, being a physicist and a computer geek I was immediately convinced and started to use it on a daily basis. As of today, I have completely rewritten our whole measurement and data acquisition system in Python, with very satisfying results. Unfortunately, during this work I also learned that Python has some shortcomings of its own that can make it complicated to use it in a scientific environment. For me, some of the most troubling (but solvable) issues included

• the Global Interpreter Lock (GIL): The Python interpreter cannot simultaneously execute code in different threads. This makes it quite hard to make full use of a  multi-processor architecture.
• the fact that Python code runs relatively slow: Numerical operations and the manipulation of large amounts of data (e.g. binary strings, lists and numbers), which are very common tasks in a measurement environment, run much slower than e.g. in Labview or C/C++.
• the lack of an easy-to-use (for a scientist, not for a programmer) toolkit for building graphical user interfaces: Although there are several very good toolkits available that make it relatively easy to build a graphical user interface with Python (e.g. PyQt or Tkinter), all of them require some considerable learning effort and a large amount of programming.

Fortunately it turned out that all of these problems can be somehow fixed or circumvented. In the following blog posts I will describe in detail how I resolved these issues and what I think is a good way to build a measurement system with Python.

Please feel free to comment on this article and to share your thoughts and experiences concerning the use of Python in a scientific environment.

import weakref
 
class Subject:
    def __init__(self):
        self._observers = []
        self.isNotifying = False
 
    def attach(self, observer):
        r = weakref.ref(observer)
        if not r in self._observers:
            self._observers.append(r)
 
    def setObservers(self,observers):
      if not observers == None:
        self._observers = observers
      else:
        self._observers = []
 
    def observers(self):
      return self._observers
 
    def detach(self, observer):
        r = weakref.ref(observer)
        try:
            if DEBUG:
              print "Removing observer."
            self._observers.remove(r)
        except ValueError:
            print "Could not remove observer %s" % str(r())
 
    def notify(self,property = None, value = None,modifier = None):
      try:
        #This is to avoid infinite notification loop, e.g. when the notified class calls a function of the subject that triggers another notify and so on...
        if self.isNotifying:
          print "WARNING: notify was called recursively, aborting."
          return False
        self.isNotifying = True
        for observer in self._observers:
            if observer() == None:
              self._observers.remove(observer)
            elif modifier != observer():
              observer().updated(self,property,value)
        self.isNotifying = False
      except:
        print "An error occured when notifying my observers..."
        print sys.exc_info()
        self.ifNotifying = False
        raise
 
#The corresponding observer class. It makes sure that the observing class has an "updated" function.
class Observer:
 
  def updated(self,subject = None,property = None,value = None):
    pass

And here’s how to use it:

class MySubject(Subject):
 
   def __init__(self):
      Subject.__init__(self)
 
   def addNumbers(self,a,b):
      #Do something....
      c = a+b
      self.notify("addNumbersResult",c)
 
class MyObserver(Observer):
 
#...
 
   def updated(self,subject = None,property = None,value = None):
      if property == "addNumbersResult":
         print "Addition complete. Result: %g" % value
 
observer = MyObserver()
subject = MySubject()
subject.attach(observer)
subject.addNumbers(1,2)

This will print ‘Addition complete: 3′.

Several things in the code above are worth noting:

• First, the Subject class stores only a weak reference to the observer. If we would use a normal reference instead, an observer would not be deleted (and thus garbage-collected) without explicitly detaching it from the subject, which in general is not what we want.
• The “notify”  function contains a safeguard that makes sure that it never gets called recursively. This helps to avoid infinite loops which one can produce rather easily by e.g. having an observer which always triggers another “notify” event while being notified itself. This behaviour might not always be exactly what you want, so feel free to remove this feature if you don’t need it.

This simple pattern is both versatile and very easy to use. Nevertheless, it can be dangerous to use it in a GUI application, especially when  multi-threading is involved. This is because most GUI toolkits -such as for example Qt- do not allow you to call any GUI-relevant function from a thread other than the main (GUI-) thread. Ignoring this restriction can yield a lot of unpleasant surprises and will often break your program. Suppose for example that you have a “Subject” class running in a worker thread and that this class notifies some GUI component. Now, if you would call any GUI-related function during the notification process (which gets executed in the worker thread) you will most likely produce some very unpredictable and unpleasant behaviour.

This problem can be circumvented if we introduce an observer class that does not process the “notify” directly upon reception but rather queues it and processes it later from the main (GUI-) thread. The following Python class does exactly this by using a timer function and an event queue. It is designed for use with Qt but can be easily generalized to work with other GUI toolkits as well.

#This is an observer class which is especially tailored to Qt widgets. Instead of processing "updated" calls directly,
#this class queues them and periodically calls a function from the main thread that processes the events.
class ObserverWidget(Observer):
 
  #This function gets called from the notifying (possibly non-GUI) thread.
  #It is NOT safe to call GUI-related functions from this function.
  def updatedThread(self,subject = None,property = None,value = None):
    pass
 
  #This function gets called from the main (GUI) thread.
  #It is safe to call GUI-related functions from this function.
  def updatedGui(self,subject = None,property = None,value = None):
    pass
 
  #We intercept the "update" call from the subject and queue it.
  def updated(self,subject = None,property = None,value = None):
    self.updatedThread(subject,property,value)
    self.updateQueue.append([subject,property,value])
 
  #This function periodically processes the update queue.
  def processUpdateQueue(self):
    while len(self.updateQueue) > 0:
      event = self.updateQueue.pop(0)
      self.updatedGui(*event) 
 
  def clearUpdateQueue(self):
    self.updateQueue = []
 
  def __init__(self,parent = None):
    self.updateQueue =  []
    self.updateQueueTimer = QTimer()
    self.updateQueueTimer.setInterval(100)
    self.connect(self.updateQueueTimer,SIGNAL("timeout()"),self.processUpdateQueue)
    self.updateQueueTimer.start()

To use this pattern you just inherit from “ObserverWidget” and use the functions “updatedThread” (for everything that can be done in the “Subject” thread) and “updatedGui” (for everything that should be done in the main thread) instead of “updated” to process your event notifications. If you use this class, please make sure that the class which you derive from it also derives from QWidget (or a child class of it) and that you call the constructor of this GUI class  BEFORE calling the constructor of the ObserverWidget.

Personally, I often use “reload” to dynamically update classes when doing rapid prototyping. By this technique I can for example reload a single widget of a GUI without having to restart the whole program, saving me tons of precious development time.

To provide the “reload” functionality in an object-oriented fashion, I just wrote a small class called “Reloadable”. This class provides a function “reloadClass” that takes care of the whole reload process: First, it calls “reload” for the module that defined the original class (the name of which is stored in the variable “self.__module__”), fetches a reference to the reloaded class type (by using the class name given in “self.__class__.__name__”) and replaces the “self.__class__” variable by this new type. After this, the class instance will have all its functions and variables replaced by their reloaded version but will nevertheless keep all instance variables that were already defined. Finally, “reloadClass” calls the function “onReload”, which can be redefined in a subclass to implement some specific reloading tasks. Easy, isn’t it? Please also note that all other instances belonging to the reloaded class will stay unaffected by this operations.

Here is the code snippet that defines “Reloadable”:

import sys
class Reloadable():
 
  #Reload the module that defines the class and replace the "__class__" variable with a new instance of the class.
  def reloadClass(self):
    newModule = reload(sys.modules[self.__module__])
    self.__class__ = eval("newModule.%s" % self.__class__.__name__)
    self.onReload()
 
  #Re-implement this function in the derived class to perform specific tasks after the reload (e.g. call "__init__")
  def onReload(self):
    pass

There is also a nice code snippet on code.activestate.com that will automatically reload all instances of a class when the corresponding module is reloaded.

Online Scientific Collaboration: Beyond the tip of the Iceberg

Scientific research is becoming more and more dependent on the use of information technology in order to acquire, process and share research data. The amount of data that is generated within this process increases with an exponential rate, making it very hard and sometimes impossible to manage or even analyze all of it. Therefore, scientists have to rely on a broad range of software tools that help them to be more productive.

Me, being a scientist myself and trying to cut my way through the daily data jungle, I use this blog as a discussion board for some useful tools, techniques and technologies that I use in my everyday work. I will focus mainly on online collaboration tools since I think they are still not as much accepted and used as they should be, but I will also discuss some other techniques that can be used directly in the lab with no social skills whatsoever required

I will only discuss tools which I use and like myself in order to give you some real first-hand experience . To begin with, here are some topics that I’m currently interested in and that I will probably discuss in the upcoming posts:

• To serve and protect: Software Configuration Management tools (especially Subversion) for Scientists.
• Controlling everything: Embeddable scripting languages (especially Lua) for simulation, configuration and automation.
• Snake charmer: Using Python for your everyday scientific needs: Plotting, calculation, measurement, data analysis, …
• Social online tools for scientists: How to use Slideshare, Mendeley& Co to promote your work and to spread your knowledge.
• Knowledge base: Using Wikis and blogs for your research.

I aim at writing one extensive article each week.  Happy reading!