The newick tree

The Newick tree format is a way of representing a graph trees with edge lengths using parentheses and commas.

A newick tree example:

(((Espresso:2,(Milk Foam:2,Espresso Macchiato:5,((Steamed Milk:2,Cappucino:2,(Whipped Cream:1,Chocolate Syrup:1,Cafe Mocha:3):5):5,Flat White:2):5):5):1,Coffee arabica:0.1,(Columbian:1.5,((Medium Roast:1,Viennese Roast:3,American Roast:5,Instant Coffee:9):2,Heavy Roast:0.1,French Roast:0.2,European Roast:1):5,Brazilian:0.1):1):1,Americano:10,Water:1);

A graphical representation for the newick tree above (using the http://www.jsphylosvg.com/ library):

The Newick format is commonly used for store phylogenetic trees.

The problem

A phylogenetic tree can be highly branched and dense and even using proper visualization software can be difficult to analyse it. Additionally, as a tree are produced by a chain of different software with data from the laboratory, the label for each leaf/node can be something not meaningful for a human reader.

For this particular problem, an example of a node label could be SXS_3014_Albula_vulpes_id_30.

There was a spreadsheet with more meaningful information where a node label could be used as a primary key. Example for the node above:

The problem consists in using the tree and the spreadsheet to produce a new tree with the same structure, where each node have a more meaningful label.

The approach

The new tree can be mounted by substituting each label of the initial tree with the respective information from the spreadsheet. A script can be used to automate this process.

The solution

After converting the spreadsheet to a CSV file that could be more easily handled by a CSV Python library the problem is reduced to a file handling and string substitution. Fortunately, due the simplicity of the Newick format and its limited vocabulary, a tree parser is not necessary.

Source-code at Github.

Difficulties found

The spreadsheet was originally in a Microsoft Office Excel 2007 (.xlsx) and the conversion to CSV provided by Excel was not good and there was no configuration option available. Finally, the conversion provided by LibreOffice Productivity Suite was more configurable and was easier to read by the CSV library.

In the script, the DictReader class showed in the the long-term much more reliable and tolerant to changes in the spreadsheet as long the names of the columns remain the same.

P.S. due to the nature of the original sources for the tree and spreadsheet I don’t have the authorization for public publishing their complete and original content. The artificial data displayed here is merely illustrative.

DNA inspired sculpture by Charles Jencks. Creative Commons photo by Maria Keays.

What is GenBank?

The GenBank sequence database is a widely used collection of nucleotide sequences and their protein translations. A GenBank sequence record file typically has a .gbk or .gb extension and is filled with plain text characters. A example of GenBank file can be found here.

Filename problem

Although there are several metadata are available inside a GenBank record the name of the file are not always in accordance with the content of the file. This is potentially a source of confusion to organize files and requires an additional effort to rename the files according to their content.

Approach using Biopython

The Biopython project is a mature open source international collaboration of volunteer developers, providing Python libraries for a wide range of bioinformatics problems. Among other tools, Biopython includes modules for reading and writing different sequence file formats including the GenBank’s record files.

Despite the fact that is possible to write a parser for GenBank’ files it would represent a redundant effort to develop and maintain such tool. Biopython can be delegated to perform parsing and focus the programming on renaming mechanism.

Biopython installation on Linux (Ubuntu 11.10) or Apple OS X (Lion)

For both Ubuntu 11.10 and OS X Lion, a modern version of Python already comes out of the box.

For Linux you just need to install the Biopython package. One method to install Biopython in a APT ready distribution as Ubuntu 11.10 (Oneiric Ocelot) is:

# apt-get install python-biopython

For an Apple OS X (Lion) you can install Biopython using easy_install, a popular package manager for the Python. Easy_install is bundled with Setuptools, a set of tools for Python.

To install the Setuptools download the .egg file for your python version (probably setuptools-0.6c11-py2.7.egg) and execute it as a Shell Script:

sudo sh setuptools-0.6c11-py2.7.egg

After this you already have easy_install in place and you can use it to install the Biopython library:

sudo easy_install -f http://biopython.org/DIST/ biopython

For both operational systems you can test if you already have Biopython installed using the Python iterative terminal:

$ python
Python 2.7.2+ (default, Oct 4 2011, 20:03:08)
[GCC 4.6.1] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import Bio
>>> Bio.__version__
’1.57′
>>>

Automatic rename example through scripting

Below the Python source-code for a simple use of using Biopython to rename a Genbank file to it’s description after removing commas and spaces.

Using the the previous example of GenBank file, suppose you have a file called sequence.gb. To rename this file to the GenBank description metadata inside it you can use the script.

python gbkrename.py sequence.gb

And after this it will be called Hippopotamus_amphibius_mitochondrial_DNA_complete_genome.gbk.

Improvements

There is plenty of room for improvement as:

• Better command line parsing with optparse and parameterization of all possible configuration.
• A graphical interface
• Handle special cases such multiple sequences in a single GenBank file.

Surprisingly python doesn’t have a shortcut for flatten a list (more generally a list of lists of lists of…).

I made a simple implementation that doesn’t use recursion and tries to be written clearly.

I get a element from a “bad” list (a list that can have another lists). If the element is not a list we store in our flat list. If the element is still a list we deal with him later. The flat list always have elements that are not a list.
To preserve the original order we reverse the elements at the end.

Quantos zeros tem entre um e mil?

É mais fácil responder perguntas desse tipo escrevendo pequenos programas usando o suporte a programação funcional e compreensão de lista que algumas linguagens como Python oferecem.

Para contar os zeros de um número, transformamos ele em uma string e contamos quantas substrings ’0′ ele contém. Por exemplo o 800:

str(800).count('0')
# 2

Para gerar uma lista ordenada com os elementos do intervalo entre um e mil, inclusive os valores um e mil:

xrange(1,1001)
# [1, 2, ... , 999, 1000]

Pegamos esse intervalo  e geramos uma outra lista onde cada elemento é a contagem dos zeros do número do intervalo.

[str(x).count('0') for x in xrange(1,1001)]
# [0, 0, ... , 0, 3]

Por exemplo, 1 não tem nenhum zero. Dois também não. 999 também não. 1000 tem três.

Somamos todos os elementos da lista temos o número de algarismos zero entre um e mil.

sum([str(x).count('0') for x in xrange(1,1001)])

E a resposta é 192.

O mesmo poderia ser obtido contando quantos zeros há na representação de string da lista do intervalo.

str(range(1,1001)).count('0')

Mas essa abordagem apesar de menor é menos geral se você quiser modifica-la para contagens mais complexas.

A diferença do range pro xrange é que o range constrói a lista real do intervalo real em memória e o xrange uma representação da lista do intervalo. Em geral mas não sempre, a performasse do xrange é melhor.

De toda forma, em ambos os casos, o resultado é o mesmo.

I was looking for a date and time representation useful for registering stock quotes in a simple plain file.

I found that the standard ISO 8601 is just the answer for this, it’s called “Data elements and interchange formats — Information interchange — Representation of dates and times”. Here is a example:

2010-01-20 22:14:38

There’s this good article from Markus Kuhn, “A summary of the international standard date and time notation”. This notation allow us to using simple lexicographical order the events.

Some examples of how to do this in Python (thanks for the Jochen Voss article “Date and Time Representation in Python”) The first for displaying the current date and time:

from time import strftime
print strftime("%Y-%m-%d %H:%M:%S")
# 2010-01-20 22:34:22

Another possibility is using strftime from datetime object.

from datetime import datetime
now = datetime.datetime.now()
print now.strftime("%Y-%m-%d %H:%M:%S")
# 2010-01-20 22:12:31

Is that. Using this notation in the begging of each line is easy to sort them in any language or using the unix sort.

Eu não achei onde coletar os dados diários de rentabilidade dos fundos de investimento do Banco do Brasil em formato bem estruturado.

Num mundo ideal as coisas seriam assim, você faria uma requisição numa url como esta:

http://bb.com.br/apps/rentabilidade?fundo=Siderurgia&saida=xml

E ele cuspiria um XML com as informações da rentabilidade diária desse fundo, isso se eu não especificasse através de outro parâmetro qual a data ou intervalo de datas desejado ou outro tipo de dados para saída como YAML ou JSON. Mas por enquanto não temos isso, nem unicórnios, então temos de fazer as coisas do jeito mais difícil, que é puxando os dados feitos para humanos e escrevendo um programa pra extrair à força os dados que desejamos e quem sabe usar eles para algum uso relacionado a mineração de dados.

A primeira abordagem que eu tentei foi a de criar um desses pequenos parsers XML que eu já mostrei como fazer antes, mas o código fonte desse documento se mostrou muito incompatível com o XML que o parser estava disposto a trabalhar. A solução alternativa foi tratar o documento linha a linha.

import urllib
 
# abrimos o documento referenciado pela url
url = 'http://www21.bb.com.br/portalbb/rentabilidade/index.jsp?tipo=01'
documento = urllib.urlopen(url)
 
# fundo de investimento que me interessa
fundo = 'small caps'
 
# estados
INICIO = 0
ACHOU_FUNDO = 1
FIM = 2
 
# estado inicial
estado = INICIO
 
# vamos analisar linha a linha do fluxo do documento
for linha in documento:
	# simplificamos, tudo pra minusculas
	linha = linha.lower()
 
	# no inicio, procura uma linha que tenha o fundo
	if estado == INICIO and linha.find(fundo) != -1:
		estado = ACHOU_FUNDO
 
	# depois, procuramos o proximo inicio de tabela html.
	# dessa linha, pegamos o que vem depois do primeiro >
	# e entao o que vem antes do primeiro <
	# e trocamos a virgula por ponto.
	elif estado == ACHOU_FUNDO and linha.find('>')[1].split('<')[0].replace(',','.')
		estado = FIM

E para usar:

$ python rendimento_small_caps.py
0.881

Geralmente estamos mais interessados em saber o valor da cota daquele fundo, daí podemos calcular o rendimento total sabendo a cota que compramos a ação inicialmente. Nesse caso o dado está na 11º coluna.

import urllib
 
# abrimos o documento referenciado pela url
url = 'http://www21.bb.com.br/portalbb/rentabilidade/index.jsp?tipo=01'
documento = urllib.urlopen(url)
 
# fundo de investimento que me interessa
fundo = 'small caps'
 
# estados
INICIO = 0
ACHOU_FUNDO = 1
FIM = 2
 
# estado inicial
estado = INICIO
coluna = 0
 
# vamos analisar linha a linha do fluxo do documento
for linha in documento:
	# simplificamos, tudo pra minusculas
	linha = linha.lower()
 
	# no inicio, procura uma linha que tenha o fundo
	if estado == INICIO and linha.find(fundo) != -1:
		estado = ACHOU_FUNDO
 
	# para cada coluna, conta a coluna, mas nao faz nada
	elif estado == ACHOU_FUNDO and linha.find('<'):
		coluna += 1
 
	# quando chegar na coluna onze, retira o conteudo entre os sinais &gt; e &lt;
	# e troca virgula por ponto, transforma em float e joga na tela
	if estado==ACHOU_FUNDO and coluna == 11:
		print float(linha.split('>')[1].split('<')[0].replace(',','.'))
		estado = FIM

$ python cota_small_caps.py
6.156906634

Essa é uma abordagem que eu não gosto nem recomendo porque ela é muito frágil e está extremamente acoplada a formatação de dados para humanos. Esta formatação está interessada no saída gráfica que o usuário vai obter e não em facilitar a extração (não humana) desses dados. Isso torna a solução muito frágil:

• Se mudarem os nomes internos dos elementos, a solução pode falhar.
• Se mudarem a formatação da tabela, a solução pode falhar.
• Se mudarem a disposição interna dos elementos html, a solução pode falhar.
• Se mudarem a url do documento, a solução vai falhar.
• Se o documento não puder mais ser tratado linha a linha, a solução vai falhar feio.

É provável que quando você estiver lendo isso ela nem funcione mais do jeito que está descrita aqui.

Por outro lado, a solução funciona e nesse caso é o que me interessa. Quando ela quebrar, se ainda for do meu interesse eu posso rapidamente conserta-la e os dados já coletados no passado continuam válidos.

Isso somado  a uma programa como o Cron pode se tornar uma ferramenta realmente poderosa.

Here is a useful tip on Python XML decoding.

I was extending xml.sax.ContentHandler class in a example to decode maps for a Pygame application when my connection went down and I noticed that the program stop working raising a exception regarded a call to urlib (a module for retrieve resources by url). I noticed that the module was getting the remote DTD schema to validate the XML.

<!DOCTYPE map SYSTEM "http://mapeditor.org/dtd/1.0/map.dtd">

This is not a requirement for my applications and it’s a huge performance overhead when works (almost 1 second for each map loaded) and when the applications is running in a environment without Internet it just waits for almost a minute and then fail with the remain decoding. A dirty workaround is open the XML file and get rid of the line containing the DTD reference.

But the correct way to programming XML decoding when we are not concerned on validate a XML schema is just the xml.parsers.expat. Instead of using a interface you just have to set some callback functions with the behaviors we want. This is a example from the documentation:

import xml.parsers.expat
 
# 3 handler functions
def start_element(name, attrs):
    print 'Start element:', name, attrs
def end_element(name):
    print 'End element:', name
def char_data(data):
    print 'Character data:', repr(data)
 
p = xml.parsers.expat.ParserCreate()
 
p.StartElementHandler = start_element
p.EndElementHandler = end_element
p.CharacterDataHandler = char_data
 
p.Parse("""<?xml version="1.0"?>
<parent id="top"><child1 name="paul">Text goes here</child1>
<child2 name="fred">More text</child2>
</parent>""", 1)

The output:

Start element: parent {'id': 'top'}
Start element: child1 {'name': 'paul'}
Character data: 'Text goes here'
End element: child1
Character data: '\n'
Start element: child2 {'name': 'fred'}
Character data: 'More text'
End element: child2
Character data: '\n'
End element: parent

I’m using the Tiled Map Editor for a while, I even wrote that tutorial about it. It’s a general purpose tile map editor, written in Java but now migrating to C++ with Qt, that can be easily used with my set of free pixelart tiles.

A map done with Tiled is stored in a file with TMX extension. It’s just a XML file, easy to understand.

As I’m creating a map loader for my owns purposes, the procedure I’m doing here works we need some simplifications. I’m handling orthogonal maps only. I’m not supporting tile properties as well. I also don’t want to handle base64 and zlib encoding in this version, so in the Tiled editor, go at the menu Edit → Preferences and in the Saving tab unmark the options “Use binary encoding” and “Compress Layer Data (gzip)”, like this:

When saving a map it will produce a TMX file like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE map SYSTEM "http://mapeditor.org/dtd/1.0/map.dtd">
<map version="1.0" orientation="orthogonal" width="10" height="10" tilewidth="32" tileheight="32">
 <properties>
  <property name="Author" value="Silveira Neto"/>
  <property name="Year" value="2009"/>
 </properties>
 <tileset name="mytiles" firstgid="1" tilewidth="32" tileheight="32">
  <image source="free_tileset_version_10.png"/>
 </tileset>
 <layer name="grass" width="10" height="10">
  <data>
   <tile gid="261"/>
   <tile gid="260"/>
    ...
   <tile gid="160"/>
   <tile gid="0"/>
  </data>
 </layer>
</map>

For processing it on Python I’m using the event oriented SAX approach for XML. So I create a ContentHandler that handles events the start and end of XML elements. In the first element, map, I know enough to create a Pygame surface with the correct size. I’m also storing the map properties so I can use it later for add some logics or effects on the map. After that we create a instance of the Tileset class from where we will get the each tile by an gid number. Each layer has it’s a bunch of gids in the correct order. So it’s enough information to mount and draw a map.

# Author: Silveira Neto
# License: GPLv3
import sys, pygame
from pygame.locals import *
from pygame import Rect
from xml import sax
 
class Tileset:
    def __init__(self, file, tile_width, tile_height):
        image = pygame.image.load(file).convert_alpha()
        if not image:
            print "Error creating new Tileset: file %s not found" % file
        self.tile_width = tile_width
        self.tile_height = tile_height
        self.tiles = []
        for line in xrange(image.get_height()/self.tile_height):
            for column in xrange(image.get_width()/self.tile_width):
                pos = Rect(
                        column*self.tile_width,
                        line*self.tile_height,
                        self.tile_width,
                        self.tile_height )
                self.tiles.append(image.subsurface(pos))
 
    def get_tile(self, gid):
        return self.tiles[gid]
 
class TMXHandler(sax.ContentHandler):
    def __init__(self):
        self.width = 0
        self.height = 0
        self.tile_width = 0
        self.tile_height = 0
        self.columns = 0
        self.lines  = 0
        self.properties = {}
        self.image = None
        self.tileset = None
 
    def startElement(self, name, attrs):
        # get most general map informations and create a surface
        if name == 'map':
            self.columns = int(attrs.get('width', None))
            self.lines  = int(attrs.get('height', None))
            self.tile_width = int(attrs.get('tilewidth', None))
            self.tile_height = int(attrs.get('tileheight', None))
            self.width = self.columns * self.tile_width
            self.height = self.lines * self.tile_height
            self.image = pygame.Surface([self.width, self.height]).convert()
        # create a tileset
        elif name=="image":
            source = attrs.get('source', None)
            self.tileset = Tileset(source, self.tile_width, self.tile_height)
        # store additional properties.
        elif name == 'property':
            self.properties[attrs.get('name', None)] = attrs.get('value', None)
        # starting counting
        elif name == 'layer':
            self.line = 0
            self.column = 0
        # get information of each tile and put on the surface using the tileset
        elif name == 'tile':
            gid = int(attrs.get('gid', None)) - 1
            if gid <0: gid = 0
            tile = self.tileset.get_tile(gid)
            pos = (self.column*self.tile_width, self.line*self.tile_height)
            self.image.blit(tile, pos)
 
            self.column += 1
            if(self.column>=self.columns):
                self.column = 0
                self.line += 1
 
    # just for debugging
    def endDocument(self):
        print self.width, self.height, self.tile_width, self.tile_height
        print self.properties
        print self.image
 
def main():
    if(len(sys.argv)!=2):
        print 'Usage:\n\t{0} filename'.format(sys.argv[0])
        sys.exit(2)
    pygame.init()
    screen = pygame.display.set_mode((800, 480))
    parser = sax.make_parser()
    tmxhandler = TMXHandler()
    parser.setContentHandler(tmxhandler)
    parser.parse(sys.argv[1])
    while 1:
        for event in pygame.event.get():
            if event.type == QUIT:
                return
            elif event.type == KEYDOWN and event.key == K_ESCAPE:
                return
        screen.fill((255,255,255))
        screen.blit(tmxhandler.image, (0,0))
        pygame.display.flip()
        pygame.time.delay(1000/60)
 
if __name__ == "__main__": main()

Here is the result for opening a four layers map file:

That’s it. You can get this code and adapt for your game because next versions will be a lot more coupled for my own purposes and not so general.

Download:maploader.tar.bz2 It’s the Netbeans 6.7 (Python EA 2) project file but that can be opened or used with another IDE or without one. Also contains the village.tmx map and the tileset.

Here is a Pygame Sprite animation using the approach presented by Joe Wreschnig and Nicolas Crovatti. It’s not yet exactly what I need but is very suitable.

import pygame, random
from pygame.locals import *
 
class Char(pygame.sprite.Sprite):
	x,y = (100,0)
	def __init__(self, img, frames=1, modes=1, w=32, h=32, fps=3):
		pygame.sprite.Sprite.__init__(self)
		original_width, original_height = img.get_size()
		self._w = w
		self._h = h
		self._framelist = []
		for i in xrange(int(original_width/w)):
			self._framelist.append(img.subsurface((i*w,0,w,h)))
		self.image = self._framelist[0]		
		self._start = pygame.time.get_ticks()
		self._delay = 1000 / fps
		self._last_update = 0
		self._frame = 0
		self.update(pygame.time.get_ticks(), 100, 100)	
 
	def set_pos(self, x, y):
		self.x = x
		self.y = y
 
	def get_pos(self):
		return (self.x,self.y)
 
	def update(self, t, width, height):
		# postion
		self.y+=1
		if(self.y>width):
			self.x = random.randint(0,height-self._w)
			self.y = -self._h
 
		# animation
		if t - self._last_update > self._delay:
			self._frame += 1
			if self._frame >= len(self._framelist):
				self._frame = 0
			self.image = self._framelist[self._frame]
			self._last_update = t
 
SCREEN_W, SCREEN_H = (320, 320)
 
def main():
	pygame.init()
	screen = pygame.display.set_mode((SCREEN_W, SCREEN_H))
	background = pygame.image.load("field.png")
	img_orc = pygame.image.load("orc.png")
	orc = Char(img_orc, 4, 1, 32, 48)
	while pygame.event.poll().type != KEYDOWN:
		screen.blit(background, (0,0))
		screen.blit(orc.image,  orc.get_pos())
		orc.update(pygame.time.get_ticks(), SCREEN_W, SCREEN_H)
		pygame.display.update()
		pygame.time.delay(10)
 
if __name__ == '__main__': main()

Here is it working:

Uptade: I put this source and images at the OpenPixel project in Github

Here’s a simple key handle in Pygame wheres you move a circle using keyboard.

import pygame
from pygame.locals import *
 
def main():
	x,y = (100,100)
	pygame.init()
	screen = pygame.display.set_mode((400, 400))
	while 1:
		pygame.time.delay(1000/60)
      # exit handle
		for event in pygame.event.get():
			if event.type == QUIT:
				return
			elif event.type == KEYDOWN and event.key == K_ESCAPE:
				return
 
      # keys handle 
		key=pygame.key.get_pressed()
		if key[K_LEFT]:
			x-=1
		if key[K_RIGHT]:
			x+=1
		if key[K_UP]:
			y-=1
		if key[K_DOWN]:
			y+=1
 
		# fill background and draw a white circle
		screen.fill((255,255,255))
		pygame.draw.circle(screen, (0,0,0), [x,y], 30)
		pygame.display.flip()
 
if __name__ == '__main__': main()

Here’s a video of it working:

Function pygame.key.get_pressed Returns a sequence of boolean values representing the state of every key on the keyboard. It’s very useful because usually on others game platforms I have to create it by myself.

This approach allow me to handle more than one key at time. For example, left and up keys can be pressed and each one is handled separately creating a diagonal movement.