Archive for the ‘Ohne Kategorie’ category

Python for Measurement and Data Acquisition

July 4th, 2010

I’m a PhD student in physics, working on experimental quantum computing with superconducting qubits. So as you might guess, data acquisition and analysis make up a big part of my daily work. To state an example, my current experiment requires me to control 15 different measurement devices such as microwave generators, voltage sources, arbitrary waveform generators, oscilloscopes, spectrum analyzers and fast data acquisition cards. All of this equipment needs to be properly interfaced and in order to perform an actual measurement it is necessary to perform many individual parameter changes and data transfer steps. Furthermore, huge amounts of numerical and binary data often need to be analyzed, aggregated and fed-back as parameters into the measurement system. In order to do all this, the software used in my group when I started my PhD was National Instrument’s “Labview”. Labview is an easy-to-use and quite powerful tool for building measurement and automation applications. In fact, when I first got in touch with it I was quite amazed of how easy it actually was to interface measurement devices and to build simple graphical user interfaces. Some of the features of Labview which I probably liked most were

  • the ability to easily create rich user interfaces with diagrams, tables and data graphs.
  • the ease of use of the graphical programming interface, which provides many pre-build libraries and instrument drivers.
  • the very high execution speed of Labview scripts.

However, as I started to build more complex experiments it became increasingly hard to come up with good solutions for interfacing them through Labview. For example, some limitations which quickly started to annoy me were

  • the lack of a text-based programming language. Labview uses a graphical programming language called "G", which makes it easy to “write” but hard to copy and read code.
  • the dataflow programming model of Labview, which makes it very hard to reuse individual parts of programs and to make use of object-oriented programming techniques. 

So after all, I wasn’t fully happy with using Labview for controlling my experiments since I felt that I would be somehow limited in the extension of my measurement system. Luckily, one day a post-doc working in our group told me that in his former lab a programming language called Python was used to control and interface most of their measurement equipment. Up to that day I only briefly heard of this language and I didn’t have the slightest experience with it. Nevertheless, I became very curious and I wanted to learn more about it. And as it turned out, I found Python to be a great tool for my work: It was flexible, easy to learn and it had many useful libraries that I could use for my data acquisition and processing needs. In fact, as of today Python seems to become more and more the standard scientific programming language, often replacing older and more inconvenient languages such as C/C++ and Fortran. So, being a physicist and a computer geek I was immediately convinced and started to use it on a daily basis. As of today, I have completely rewritten our whole measurement and data acquisition system in Python, with very satisfying results. Unfortunately, during this work I also learned that Python has some shortcomings of its own that can make it complicated to use it in a scientific environment. For me, some of the most troubling (but solvable) issues included

  • the Global Interpreter Lock (GIL): The Python interpreter cannot simultaneously execute code in different threads. This makes it quite hard to make full use of a  multi-processor architecture.
  • the fact that Python code runs relatively slow: Numerical operations and the manipulation of large amounts of data (e.g. binary strings, lists and numbers), which are very common tasks in a measurement environment, run much slower than e.g. in Labview or C/C++.
  • the lack of an easy-to-use (for a scientist, not for a programmer) toolkit for building graphical user interfaces: Although there are several very good toolkits available that make it relatively easy to build a graphical user interface with Python (e.g. PyQt or Tkinter), all of them require some considerable learning effort and a large amount of programming.

Fortunately it turned out that all of these problems can be somehow fixed or circumvented. In the following blog posts I will describe in detail how I resolved these issues and what I think is a good way to build a measurement system with Python.

Please feel free to comment on this article and to share your thoughts and experiences concerning the use of Python in a scientific environment.

No comments »

Posted in Ohne Kategorie

Tags:

QTextEdit with line numbers

June 5th, 2010

The Qt QPlainTextEdit class provides a very nice plaintext editor that can be used e.g. for displaying and editing source code. One thing missing in the class though is the ability to display line numbers in the margin of the editor, which (in my opinion) is a very important feature of any code editor. Luckily, after a bit of searching on the web I found this excellent blog entry which includes a code snippet that extends QTextEdit with the ability to display line numbers. For one of my recent projects I used and slightly modified this code, eliminating among other things the QFrame that was used there and using a QPlainTextEdit instead of a QTextEdit as base class of the editor, which makes the processing of the line numbers slightly faster. Here is my modified version of the code:

#Original code (MIT license): http://john.nachtimwald.com/2009/08/15/qtextedit-with-line-numbers/
from PyQt4.QtGui import * 
from PyQt4.QtCore import *
import sys
 
class LineTextWidget(QPlainTextEdit):
 
    def append(self,string):
        self.appendPlainText(string)
 
    class NumberBar(QWidget): 
 
        def __init__(self, *args):
            QWidget.__init__(self, *args)
            self.edit = None
            # This is used to update the width of the control.
            # It is the highest line that is currently visibile.
            self.highest_line = 0
 
        def setTextEdit(self, edit):
            self.edit = edit
 
        def update(self, *args):
            width = QFontMetrics(self.edit.document().defaultFont()).width(str(self.highest_line)) + 10
            if self.width() != width:
                self.setFixedWidth(width)
                self.edit.setViewportMargins(width,0,0,0)
            QWidget.update(self, *args)
 
        def paintEvent(self, event):
            contents_y = 0
            page_bottom = self.edit.viewport().height()
            font_metrics = QFontMetrics(self.edit.document().defaultFont())
            current_block = self.edit.document().findBlock(self.edit.textCursor().position())
 
            painter = QPainter(self)
 
            # Iterate over all text blocks in the document.
            block = self.edit.firstVisibleBlock()
            viewport_offset = self.edit.contentOffset()
            line_count = block.blockNumber()
            painter.setFont(self.edit.document().defaultFont())
            while block.isValid():
                line_count += 1
 
                # The top left position of the block in the document
                position = self.edit.blockBoundingGeometry(block).topLeft()+viewport_offset
                # Check if the position of the block is out side of the visible
                # area.
                if position.y() > page_bottom:
                    break
 
                # We want the line number for the selected line to be bold.
                bold = False
                if block == current_block:
                    bold = True
                    font = painter.font()
                    font.setBold(True)
                    painter.setFont(font)
 
                # Draw the line number right justified at the y position of the
                # line. 3 is a magic padding number. drawText(x, y, text).
                painter.drawText(self.width() - font_metrics.width(str(line_count)) - 3, round(position.y()) + font_metrics.ascent()+font_metrics.descent()-1, str(line_count))
 
                # Remove the bold style if it was set previously.
                if bold:
                    font = painter.font()
                    font.setBold(False)
                    painter.setFont(font)
 
                block = block.next()
 
            self.highest_line = line_count
            painter.end()
 
            QWidget.paintEvent(self, event)
 
 
    def __init__(self, *args):
        QPlainTextEdit.__init__(self, *args)
 
        self.number_bar = self.NumberBar(self)
        self.number_bar.setTextEdit(self)
 
        self.viewport().installEventFilter(self)
 
    def resizeEvent(self,e):
        self.number_bar.setFixedHeight(self.height())
        QPlainTextEdit.resizeEvent(self,e)
 
    def setDefaultFont(self,font):
      self.document().setDefaultFont(font)
 
    def eventFilter(self, object, event):
        # Update the line numbers for all events on the text edit and the viewport.
        # This is easier than connecting all necessary singals.
        if object is self.viewport():
            self.number_bar.update()
            return False
        return QPlainTextEdit.eventFilter(object, event)
 
#To test the class.
if __name__ == '__main__':
  qApp = QApplication(sys.argv)
  qApp.connect(qApp, SIGNAL('lastWindowClosed()'), qApp,
                    SLOT('quit()'))
 
  MyWindow = QMainWindow()
 
  MyEdit = LineTextWidget()
 
  MyWindow.setCentralWidget(MyEdit)
 
  MyWindow.show()
 
  qApp.exec_()

No comments »

Posted in Ohne Kategorie

Tags:

A thread-safe observer pattern for Python and Qt

April 10th, 2010

Observer patterns are quite useful in many circumstances. As the name implies, such a pattern notifies an “observer” class about some change in another class, often called the “subject”. In Python it is very straightforward to implement such a pattern and the following code snippet -which I use quite often in my everyday programming- shows a possible way to do it:

import weakref
 
class Subject:
    def __init__(self):
        self._observers = []
        self.isNotifying = False
 
    def attach(self, observer):
        r = weakref.ref(observer)
        if not r in self._observers:
            self._observers.append(r)
 
    def setObservers(self,observers):
      if not observers == None:
        self._observers = observers
      else:
        self._observers = []
 
    def observers(self):
      return self._observers
 
    def detach(self, observer):
        r = weakref.ref(observer)
        try:
            if DEBUG:
              print "Removing observer."
            self._observers.remove(r)
        except ValueError:
            print "Could not remove observer %s" % str(r())
 
    def notify(self,property = None, value = None,modifier = None):
      try:
        #This is to avoid infinite notification loop, e.g. when the notified class calls a function of the subject that triggers another notify and so on...
        if self.isNotifying:
          print "WARNING: notify was called recursively, aborting."
          return False
        self.isNotifying = True
        for observer in self._observers:
            if observer() == None:
              self._observers.remove(observer)
            elif modifier != observer():
              observer().updated(self,property,value)
        self.isNotifying = False
      except:
        print "An error occured when notifying my observers..."
        print sys.exc_info()
        self.ifNotifying = False
        raise
 
#The corresponding observer class. It makes sure that the observing class has an "updated" function.
class Observer:
 
  def updated(self,subject = None,property = None,value = None):
    pass

And here’s how to use it:

class MySubject(Subject):
 
   def __init__(self):
      Subject.__init__(self)
 
   def addNumbers(self,a,b):
      #Do something....
      c = a+b
      self.notify("addNumbersResult",c)
 
class MyObserver(Observer):
 
#...
 
   def updated(self,subject = None,property = None,value = None):
      if property == "addNumbersResult":
         print "Addition complete. Result: %g" % value
 
observer = MyObserver()
subject = MySubject()
subject.attach(observer)
subject.addNumbers(1,2)

This will print ‘Addition complete: 3′.

Several things in the code above are worth noting:

  • First, the Subject class stores only a weak reference to the observer. If we would use a normal reference instead, an observer would not be deleted (and thus garbage-collected) without explicitly detaching it from the subject, which in general is not what we want.
  • The “notify”  function contains a safeguard that makes sure that it never gets called recursively. This helps to avoid infinite loops which one can produce rather easily by e.g. having an observer which always triggers another “notify” event while being notified itself. This behaviour might not always be exactly what you want, so feel free to remove this feature if you don’t need it.

This simple pattern is both versatile and very easy to use. Nevertheless, it can be dangerous to use it in a GUI application, especially when  multi-threading is involved. This is because most GUI toolkits -such as for example Qt- do not allow you to call any GUI-relevant function from a thread other than the main (GUI-) thread. Ignoring this restriction can yield a lot of unpleasant surprises and will often break your program. Suppose for example that you have a “Subject” class running in a worker thread and that this class notifies some GUI component. Now, if you would call any GUI-related function during the notification process (which gets executed in the worker thread) you will most likely produce some very unpredictable and unpleasant behaviour.

This problem can be circumvented if we introduce an observer class that does not process the “notify” directly upon reception but rather queues it and processes it later from the main (GUI-) thread. The following Python class does exactly this by using a timer function and an event queue. It is designed for use with Qt but can be easily generalized to work with other GUI toolkits as well.

#This is an observer class which is especially tailored to Qt widgets. Instead of processing "updated" calls directly,
#this class queues them and periodically calls a function from the main thread that processes the events.
class ObserverWidget(Observer):
 
  #This function gets called from the notifying (possibly non-GUI) thread.
  #It is NOT safe to call GUI-related functions from this function.
  def updatedThread(self,subject = None,property = None,value = None):
    pass
 
  #This function gets called from the main (GUI) thread.
  #It is safe to call GUI-related functions from this function.
  def updatedGui(self,subject = None,property = None,value = None):
    pass
 
  #We intercept the "update" call from the subject and queue it.
  def updated(self,subject = None,property = None,value = None):
    self.updatedThread(subject,property,value)
    self.updateQueue.append([subject,property,value])
 
  #This function periodically processes the update queue.
  def processUpdateQueue(self):
    while len(self.updateQueue) > 0:
      event = self.updateQueue.pop(0)
      self.updatedGui(*event) 
 
  def clearUpdateQueue(self):
    self.updateQueue = []
 
  def __init__(self,parent = None):
    self.updateQueue =  []
    self.updateQueueTimer = QTimer()
    self.updateQueueTimer.setInterval(100)
    self.connect(self.updateQueueTimer,SIGNAL("timeout()"),self.processUpdateQueue)
    self.updateQueueTimer.start()

To use this pattern you just inherit from “ObserverWidget” and use the functions “updatedThread” (for everything that can be done in the “Subject” thread) and “updatedGui” (for everything that should be done in the main thread) instead of “updated” to process your event notifications. If you use this class, please make sure that the class which you derive from it also derives from QWidget (or a child class of it) and that you call the constructor of this GUI class  BEFORE calling the constructor of the ObserverWidget.

No comments »

Posted in Ohne Kategorie

Python: Reloading a class definition at runtime

March 14th, 2010

Python comes with a very useful function called “reload”, which allows to dynamically reload the code of a given module at runtime. This technique is quite useful if one wants for example to reload some part of a larger application without having to close and restart the whole system.

Personally, I often use “reload” to dynamically update classes when doing rapid prototyping. By this technique I can for example reload a single widget of a GUI without having to restart the whole program, saving me tons of precious development time.

To provide the “reload” functionality in an object-oriented fashion, I just wrote a small class called “Reloadable”. This class provides a function “reloadClass” that takes care of the whole reload process: First, it calls “reload” for the module that defined the original class (the name of which is stored in the variable “self.__module__”), fetches a reference to the reloaded class type (by using the class name given in “self.__class__.__name__”) and replaces the “self.__class__” variable by this new type. After this, the class instance will have all its functions and variables replaced by their reloaded version but will nevertheless keep all instance variables that were already defined. Finally, “reloadClass” calls the function “onReload”, which can be redefined in a subclass to implement some specific reloading tasks. Easy, isn’t it? Please also note that all other instances belonging to the reloaded class will stay unaffected by this operations.

Here is the code snippet that defines “Reloadable”:

import sys
class Reloadable():
 
  #Reload the module that defines the class and replace the "__class__" variable with a new instance of the class.
  def reloadClass(self):
    newModule = reload(sys.modules[self.__module__])
    self.__class__ = eval("newModule.%s" % self.__class__.__name__)
    self.onReload()
 
  #Re-implement this function in the derived class to perform specific tasks after the reload (e.g. call "__init__")
  def onReload(self):
    pass

There is also a nice code snippet on code.activestate.com that will automatically reload all instances of a class when the corresponding module is reloaded.

No comments »

Posted in Ohne Kategorie

Tags:

Presentation: Python for Scientists

March 9th, 2010

This is a presentation I gave at our weekly group seminar. It discusses the use of the programming language Python in a scientific environment, putting emphasis on data acquisition, processing and presentation. Enjoy!

Python for Scientists

View more presentations from japh44.

No comments »

Posted in Ohne Kategorie