Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Thursday, September 1, 2011

Modern Portfolio Theory - A Python Implementation


I was surprised last week to find there was no accessible Python implementation of the calculation of the Efficient Frontier (as defined by Markowitz in his presentation of Modern Portfolio Theory ~1957).* Since the problem seemed simple to solve with the tools at hand, I set out to "right the wrong" and develop an open implementation (available under an open, BSD license via github) that meets my needs.

In the process of building the basic metrics, I was able to experiment with NumPy's datetime support -- as mentioned in my earlier post. The API's are still a bit in flux, but I thought I would write up a full workflow using the tools as they are. I would greatly appreciate feedback/contributions/suggestions.  So let's dive in ...

Thursday, August 25, 2011

NumPy's New datetime Support and a Financial Database Roundtrip

Last month when I was in Austin, TX at the 10th Annual (!) SciPy Conference I asked around about the status of native datetime support in NumPy. Based on the response of my good friends at Enthought, I've been living under a rock for a while -- it's been finished for some time now.

So, a month later, I've wrangled some time to take a look. I also happened to have a particular problem I wanted to solve. Here's a teaser preview of the output of my Modern Portfolio Theory implementation:



I'll get into the details of that implementation in a later post. For now, let's tackle some data handling.

The immediate task I undertook was to refactor a simple asset price data fetcher (from Yahoo Finance, in this particular case) that I use. I always end up hammering the Yahoo Finance server to pull down price data when I'm doing some ad-hoc analysis, and I thought I should build a simple function to stash that data away locally in a sqlite database file. I know there are other approaches for fetching and manipulating this data out there -- notably pandas aggregation functions, paired with the matplotlib finance functions. I've looked to those tools for guidance, but wanted to have a bit more functional approach, fewer (or at least different) dependencies, and SQL as an interface to the data. I'll say up front that all the code is available under a BSD license via github.

First, I cleaned up the old data fetcher and made it more functional (in the programming sense -- rather than object oriented). Really, I just wanted to be able to do this:



I prefer my api for this over the matplotlib api mainly because I don't want to import datetime, followed by all the startdate = datetime.datetime(2000,1,1) nonsense, just to get a date (pun intended). Anyway, once I've got my data, I want to stuff it into a sqlite db file and be able to (losslessly) pull it back out into a numpy array. This seems fairly simple because pysqlite offers conversion support for python datetime objects. Unfortunately, I wanted dates stored as floats of seconds since the epoch, so I had to do a bit more work. This resulted in the code:



These two functions provide for accurate round-tripping despite the nuances of daylight savings time treatment by numpy and sqlite.

Another important point in the use of the converter function is how one determines that the type is a "datetime." Sqlite uses datetime as a designator for a built-in type -- I'm just overriding that when I register the converter function. In order to have the proper type recognized, we need to specify the detect_types argument when we instantiate our connection to the database for INSERT-ing or SELECT-ing. Also, notice how I specify the type using the as keyword in my SELECT statement:



Now, I have a nice interface for this kind of interaction with my data:



I'm still looking into potential problems with how queries might perform where I've indexed a float column for the date. Also, any suggestions about how to better handle the DST issue would be most welcome. It's pretty hackish now and I'd like to avoid wading into the morass that is Python's congenital complex about round-tripping of timestamps and how to properly do timezones.

Next step: Port some metrics code over to use the numpy datetimes rather than my manual handling of dates. I'll be sharing this code as well as an explanation in a later post.

UPDATE: I've added a handy convenience function, price_utils.symbol_exists to check what data is in the sqlite database for a particular symbol.



It returns a tuple of the form (num_recs, first_date, last_date). Again, the code is available via github. Enjoy!

Wednesday, February 17, 2010

Python Jobs Trend

As Matt Asay pointed out, there seems to be a real uptick in jobs in the Python space. Here's the chart:



You can play with the chart here.

While the raw number of jobs is still much lower, the rates of change are encouraging for open languages.

Tuesday, September 29, 2009

Test Driven Python Development Experiment: P&L Statement - Step 1

I began trading equities in my personal portfolio earlier this Spring using an account with Interactive Brokers (whom I'd recommend for their nice tools, good access to products, and published APIs). One of the issues I've had with them, however, is the lack of clarity in a daily P&L statement. Without bogging down in too many details, it boils down to the fact that I'd like to see a few things that are not provided out-of-the box by Interactive Brokers:

  • Realized gain rolled up for each day.

  • An intraday, calculated P&L in the event that I wish to use it as part of an automated trading system.

  • A choice between FIFO, LIFO and what I call WIFO (worst-in-first-out, a more accurate and better-rhyming name for HIFO) handling of trades.

Developing these capabilities seems like an excellent exercise for me to experiment with Test Driven Development (TDD) and share my experiences with others, so consider this a test for TDD using a below-average developer with simple needs. The current state of the code is available at github.

Position Object


The first thing needed is a data structure to hold information about each position. The data populating this data structure will come from a text log of transactions, or, potentially, a database with transaction records. Interactive Brokers provides several formats of text output. I chose to use a text file provided in a TradeLog format. Looking at a few lines of a typical log gives an idea of the type and format of data provided:

ACCOUNT_INFORMATION
ACT_INF|UXXXXXXX|Test Company|Advisor Client|1111 Main Street, San Antonio TX 78201 United States

OPTION_TRANSACTIONS
OPT_TRD|305665068|APVGG|AAPL JUL09 135 C|ISE|BUYTOOPEN|O|20090512|10:22:55|USD|23.00|100.00|5.55|12765.00|-16.10|1.00
OPT_TRD|305668016|APVGG|AAPL JUL09 135 C|NASDAQOM|SELLTOCLOSE|C|20090512|10:24:59|USD|-23.00|100.00|5.61|-12903.00|-10.35|1.00
OPT_TRD|305681552|APVGG|AAPL JUL09 135 C|ISE|BUYTOOPEN|O|20090512|10:36:59|USD|46.00|100.00|5.30|24380.00|-32.20|1.00
OPT_TRD|305721409|APVGG|AAPL JUL09 135 C|ISE|BUYTOOPEN|O|20090512|11:25:29|USD|47.00|100.00|5.15|24205.00|-32.90|1.00
OPT_TRD|305767844|APVGG|AAPL JUL09 135 C|ISE|BUYTOOPEN|O|20090512|12:30:38|USD|23.00|100.00|4.70|10810.00|-16.10|1.00
OPT_TRD|306301447|APVGG|AAPL JUL09 135 C|NASDAQOM|BUYTOOPEN|O|20090513|14:19:58|USD|30.00|100.00|3.19|9570.00|-13.50|1.00

The optimal way to put this information into a data structure in python would probably be a to use a numpy structured array. However, since performance is not an issue (at this time), and I'd like to use this problem as a TDD test for myself, I'll forgo the numpy approach and use more of a brute-force, object-oriented method.

If I were to codify some requirements for my data structure by writing a test, it might look something like this:


import position

def position_test():
""" Dead simple test to see if I can store data in my position object
and get it back out
"""
p = position.Position(symbol="AAPL", qty=1000, price=185.25,
multiplier=1., fee=7.0, total_amt=185250.,
trans_date=1053605468.54)

assert p.price==185.25
assert p.description==""
assert p.display_date=="05/22/2003"
assert p.display_time=="08:11:08"

OK, so there are several things implicit in this test (don't worry, I'll break all those asserts into different tests before going forward). One obvious point is that I seem to require some fairly sophisticated handling of datetimes and default values. Other than that, we have a big pile of keyword arguments and a bunch of attributes in a fairly brain-dead object. The conventional approach is to build a giant __init__ with a bunch of lines like this nonsense:

if not description is None:
self.description = description
else:
self.description = 0.0

Frankly, we deserve better than to waste our lives on silly boilerplate code for initialization, validation and defaults. This is where Traits comes to the rescue (there's a nice tutorial here). I'll not go into all the details here, but you'll see how how traits allows us to have much cleaner code as I progress.

As a very simple first test, let's take our first assert and make a stand-alone test:

def position_attribute_test():
""" Dead simple test to see if I can store data in my position object
and get it back out
"""
p = position.Position(symbol="AAPL", qty=1000, price=185.25,
multiplier=1., fee=7.0, total_amt=185250.,
trans_date=1053605468.54)

assert p.price==185.25

We can solve it with the following code, which includes all the fields (attributes/traits) I think I'll need:

from enthought.traits.api import (HasTraits, Enum, Float, Int,
Str)

class Position(HasTraits):
""" Simple object to act as a data structure for a position

While all attributes (traits) are optional, classes that contain or
collect instances of the Position class will probably require the following:
symbol, trans_date, qty, price, total_amt

"""


side = Enum("BUYTOOPEN", ["SELLTOCLOS", "BUYTOOPEN", "SELLTOOPEN", "BUYTOCLOSE"])
symbol = Str
id = Int
description = Str
trans_date = Float
qty = Float
price = Float
multiplier = Float(1.0)
fee = Float
exchange_rate = Float(1.0)
currency = Str("USD")
total_amt = Float
filled = Str
exchange = Str

I now can throw my test code into a directory called "test" adjacent to this module (which I named position.py) and I use the excellent nosetest tool to harvest and run any tests I've got. It passes! ... but that was actually not much of a challenge, so let's look at another test function:

def position_initialization_test():
""" Test to see if I can handle fields for which I provide no data.
"""
p = position.Position(symbol="AAPL", qty=1000, price=185.25,
multiplier=1., fee=7.0, total_amt=185250.,
trans_date=1053605468.54)


assert p.description==""

... this is similarly a soft pitch over the middle of the plate. Traits has already solved this for us. The test passes with no change to the code.

How about something a little more challenging:

def position_dates_test():
""" Test to see if I'm handling dates correctly
"""
p = position.Position(symbol="AAPL", qty=1000, price=185.25,
multiplier=1., fee=7.0, total_amt=185250.,
trans_date=1053605468.54)

assert p.date_display=="05/22/2003"
assert p.time_display=="08:11:08"

So now I have to actually write some code. I won't go into all the painful issues with Python date handling, I'll just provide a link to the date utility code that I wrote to help smooth things over.

I do want to step back and write a test that defines a need for date handling that is currently a bit broken (at least in Python 2.5) -- converting a python datetime to a timestamp and a timestamp to a python datetime:

from date_util import dt_from_timestamp, dt_to_timestamp


def date_util_test():
""" Simple test of correctly transforming a timestamp to a python datetime,
and back to a timestamp
"""

# test a not-very-random sequence of times
for ts in range(0, 1250000000, 321321):
# simply see if we can round-trip the timestamp and get the same result
dt = dt_from_timestamp(ts)
ts2 = int(dt_to_timestamp(dt))
assert ts2 == ts


Picking back up with my position_dates_test ... the instructive part of what I need to do here is in using a Property Trait as a good human interface for my trans_date. The trans_date value is actually a float value indicating seconds (and fractional seconds) since the epoch (which happens to fall in the same year that we put the first man on the moon, the summer of love, Woodstock and, notably, the year of my birth -- 1969!). While the current tally of seconds since 1969 may be at the front of my mind, it's not a particularly good way to represent dates for most people. In fact, most GUIs that handle datetime entries separate the date and time. This makes sense -- we're used to interacting with the two with separate pieces of hardware (a calendar and a clock). The date and time properties are defined after the other traits in my class like so:

date_display = Property(Regex(value='11/17/1969',
regex='\d\d[/]\d\d[/]\d\d\d\d'),
depends_on='trans_date')
time_display = Property(Regex(value='12:01:01',
regex='\d\d[:]\d\d[:]\d\d'),
depends_on='trans_date')

###################################
# Property methods
def _get_date_display(self):
return dt_from_timestamp(self.trans_date, tz=Eastern).strftime("%m/%d/%Y")

def _set_date_display(self, val):
tm = self._get_time_display()
trans_date = datetime.strptime(val+tm, "%m/%d/%Y%H:%M:%S" )
trans_date = trans_date.replace(tzinfo=Eastern)
self.trans_date = dt_to_timestamp(trans_date)
return

def _get_time_display(self):
t = dt_from_timestamp(self.trans_date, tz=Eastern).strftime("%H:%M:%S")
return t

def _set_time_display(self, val):
trans_time = datetime.strptime(self._get_date_display()+val, "%m/%d/%Y%H:%M:%S")
trans_time = trans_time.replace(tzinfo=Eastern)
self.trans_date = dt_to_timestamp(trans_time)
return

The getter and setter methods behave just as you would expect, and the whole thing provides a meaningful interface to the timestamp stored in my trans_date trait. There are some imports that I don't show here for brevity, but it's fairly clean code considering what it does. The best part of all -- my test passes!

Finally, I want to try another test, because I know I'll need to sort my Position objects within a collection. Here's a simple test function:

def position_sort_test():
""" Test to see if I can collect and sort these properly.
The objective is to have the objects sort by the trans_date
trait.
"""


p0 = position.Position(id=102, symbol="AAPL", qty=1000, price=185.25,
multiplier=1., fee=7.0, total_amt=185250.,
trans_date=1045623459.68)

p1 = position.Position(id=103, symbol="AAPL", qty=-1000, price=186.25,
multiplier=1., fee=7.0, total_amt=-186250.,
trans_date=1053605468.54)

p2 = position.Position(id=101, symbol="AAPL", qty=500, price=184.00,
multiplier=1., fee=7.0, total_amt=62000.,
trans_date=1021236990.02)

plist = [p0, p1, p2]

plist.sort()

assert plist[0]==p2
assert plist[1]==p0
assert plist[2]==p1

This just collects a few positions into a list, calls the .sort method on the list and checks to see if it accurately sorts by the trans_date. The method I add to my class to accomplish this is actually quite simple. I define a __cmp__ method which correctly compares objects of this class and I'm good to go:

# support reasonable sorting based on trans_date
def __cmp__(self, other):
if self.trans_date < other.trans_date:
return -1
elif self.trans_date > other.trans_date:
return 1
else: return 0

This pretty much wraps up the functionality I want (so far) in my position class. This is evidenced by my nosetests results:

Oh, and one more thing -- thanks to the miracle of traits I can call the edit_traits method on any position object and get a nice form with all the fields. I've added a view definition to my position class to pare this down a bit:

traits_view = View(Item('symbol', label="Symb"),
Item('date_display'),
Item('time_display'),
Item('qty'),
buttons=['OK', 'Cancel'], resizable=True)

Now if p is a Position object then calling p.edit_traits() pops up the following dialog:

Again, all the code for this is available via github.

Given this simple exercise in TDD, my initial impression is that it's helpful for ensuring good test coverage for features, but I really need to go back and write corner-case tests and tests which will pick up minor regressions in the state of my code better. Also, I found that considering how to best design the code was often a separate exercise from designing tests -- so I could really sense I was switching mental contexts to try to drive the development with well thought-out tests. I'm willing to concede that this is unique to this developer. Overall, I think the jury is still out. I get the feeling I don't have enough experience with TDD to do it well, but hope springs eternal.

I'll handle proper queueing and adding/removing positions in a portfolio in the next post in this series. Until then, please raise any questions or corrections in the comments below.

Saturday, August 8, 2009

A Very Simple GUI Using Traits

I ran across this post today about some magic with simple syntax for GUI building that Richard Jones is playing around with. It occurred to me to give a simple example of traits usage to do the same thing.

So, thanks to traits, our class definition looks quite clean, and we can provide some manifest typing as well. I show a screen shot of my ipython session, which was launched with ipython -wthread:


So executing line [5] pops up the GUI shown. One nice thing about traits is the built-in MVC architecture, which allows me to change the value in the choices class attribute and it informs the listening label , which is updated automatically. Notice that the value f.choices inspected at the command line is updated as well:


The main drawback I see between this approach and Richard's is the size of the tool chain. The TraitsGUI piece requires wxPython (or QT--it works with either), and some dependencies.

Saturday, August 1, 2009

Infographic in Python using Chaco

This week I ran across a blog post about this New York Times infographic, which explains one of the measures of the "business cycle" based on industrial production (the data comes from the OECD originally). [Update: I also saw the post over at Juice Analytics in which they implemented this in excel.] Never one to pass up an opportunity to re-invent the wheel, I thought it would be a good exercise to implement this in Python using the excellent Chaco plotting toolkit which comes included in some python distributions. So, here's the beginning of that effort, after a few hours digging around the docs and hacking together a GUI:

This is a good start. I've posted the code for this at github.

To really flesh it out, you'd need to add in the Composite Leading Indicator data and make some of the elements update based on the selected range. It would also be cool to dynamically switch out the data for various countries, or view them concurrently. Any takers?

Information Density

I think what makes such a simple interface so compelling is that you are able to see the relationship between three pieces of data. Cross-plots are a great mechanism for visualizing relationships between two data sets, but they're made even more useful when you can highlight a range in a common index (e.g. time, in the case of time-series data, or depth, in the case of depth indexed data in the geophysics arena.) Even with a lot of information presented, the display is very clean--even sparse.

State and State-Transition

This particular graphic also reveals the "state" of the business cycle by partitioning the graph into quadrants. This data set has a very straightforward state inherent in it's construction, but one might imagine more sophisticated calculations of state decorating time series data such as this. I'm beginning to investigate the application of this to stock price streams and some derived state that can be displayed in ways that can be "replayed" and analyzed. Whether real "information" can be teased out of the data will remain to be seen, but I'll try to leverage the visual cortex to gain intuition about the data.

Any comments/suggestions about the approach are welcome.

Wednesday, July 29, 2009

Hello World (Map)

I'm resurrecting my blogging effort outside of my former company--in the hopes of sharing useful tips I discover in my new software pursuits.

One particular area of distraction for me has been in GIS and mapping. I'm using the excellent mapnik toolkit to produce mapping visualizations. Mapnik sits on a fairly heavy stack of dependencies including:
  • GDAL/OGR - libraries and utilities for dealing with a multitude of raster and shape (vector) files. These have very good python bindings
  • PostgreSQL/PostGIS - nice, but not required
  • PROJ4 - Cartographic Projections library
  • BOOST - The forever-to-compile C++ libraries that mapnik uses for wrapping C++ codes.
Getting all of this running on OSX is a huge pain in the neck--especially if I want to use my preferred Python Distribution. Much of this pain is alleviated with Framework installs of many of the dependencies from the excellent kyngchaos site (many thanks!). I'm still not convinced I have all my libraries linked properly (who needs ICU_LIBS anyway?!).

My original mapping goal was fairly simple: Develop a composite map of roads, terrain and some custom features for a nice large-format printout for my son's room. Technically, this meant a few things:
  • converting srtm elevation data [1] to nice contour shape files (gdal_contour).
  • generating a hillshade raster image from srtm data
  • generating a color relief from srtm data
  • properly composing the custom features (considering Cartographica for this, but it's a bit expensive for a hobby project and the product seems at bit...nascent. Suggestions?).
  • beautifully rendering everything using a nice scanline approach.
It's important to note that the data and the tools I've used (with the exception of Cartographica) are all freely available (some tools do have copy-left provisions, and the data are not to be 'sold' in many cases). There is a massive amount of data for my region of interest, and a half day of effort gave me something like this:


Well...there was still much work to be done. The above image used the downloadable contour sets from the two Texas counties that this image spans, and the renderer I was using (cartographica) doesn't do very good text-layout "collision detection," nor does it have very fine-grained control over things like max angles and spacing for labels. So, putting together a simple mapnik script, using my own generated contours, applying hill shading, and tweaking the color relief to make it more wife-friendly yields something like this:


I'm getting close. I need to re-append the roads data and add all my custom features (labels, points of interest, etc.) I'm pleased with the progress so far.

This exercise has led me to think about potentially different approaches. While mapnik is extremely nice, they do use boost which, unfortunately, leads to a more lengthy and difficult build/install. (I'll try to document the OS X steps that I used when I set it up on a new machine--I had to deviate from the "official" installation instructions in a couple of places.) Why didn't they use SWIG? Is the mapnik C++ layer heavily templated? Also, I'd like to see the rendering layer use my favorite stack of 2D rendering tools -- Kiva and Enable. This would allow an intermediate layer which would provide more general vector output routines. I'll try to get a handle on the level of effort for this in the coming weeks, as time allows.

----
[1] The srtm data set is one of the cooler products I've come across--a shuttle mission to map the world and provide free access to global elevation data at 10m or 30 resolution (gladly sponsored by this American taxpayer!).