Friday, February 8, 2013

...


gnuplot> plot 'tmp' using 3 with boxes linecolor rgb "#000333" fill solid, 'tmp' using 2 with boxes linecolor rgb "#336699", 'tmp' using 1 with points lt 1 linecolor rgb "#99ccff", 'ts.dat' with lines lw 3 linecolor rgb "white"




gnuplot> plot 'tmp' using 3 with boxes fill solid lt 3, 'tmp' using 2 with boxes fill solid lt 2, 'tmp' using 1 with boxes fill solid lt 1, 'ts.dat' with lines lw 3 linecolor rgb "white"

Monday, January 14, 2013

How far from Omonia / Athens?

OpenTripPlanner have released OpenTripPlanner Analyst to analyse city transportation network GTFS data. The tool provides a visualisation of public transport travel times from a specific location. The following map shows how far you can travel from the centre of Athens by bus and metro in 30,60,90,120 minutes. Plan to include this as an overlay on zee.gr at some point.


Friday, May 18, 2012

real time stocks with gnuplot

Just for fun -- snatch real time price quotes from NASDAQ, plot the results, create audible alert on new trend or if low/high of day is breached: Here, I'm watching $VELT



$ cat >velt.sh

#!/bin/bash
echo "NASDAQ"
echo "======"
lynx --source http://www.nasdaq.com/symbol/velt/real-time |egrep "_[A-Z].*span" |sed 's/.*id="//g; s/\">/ /g; s/\" style.* / /g; s/<\/span>.*$//g' |egrep -v "_NO|_Market" |sed 's/_TradeTimeStamp /Time Stamp\t=\t/; s/_LastSale /Last\t\t=\t/; s/_NetChange /Change\t\t=\t/; s/_PctChange /%\t\t=\t/; s/_Volume /Volume\t\t=\t/; s/_PreviousClose /PreC\t\t=\t/; s/_TodaysHigh /HOD\t\t=\t/; s/_TodaysLow /LOD\t\t=\t/' >velt.tmp
cat velt.tmp
last="`tail -1 velt.log |awk '{print $3}'`"
echo `cat velt.tmp |grep "Time" |awk '{print $4 " " $5}'` `cat velt.tmp |egrep "Last" |awk '{print $3}'` >>velt.log


echo "(`cat velt.log |grep "." |tail -20 |awk '{print $3"+"}' |tr -d '\n' |sed 's/\+$//'`)/20" |bc -l >ma20.tmp
echo "(`cat velt.log |grep "." |tail -50 |awk '{print $3"+"}' |tr -d '\n' |sed 's/\+$//'`)/50" |bc -l >ma50.tmp

ma20=`cat ma20.tmp`
ma20=`printf "%.2f" $ma20`
ma50=`cat ma50.tmp`
ma50=`printf "%.2f" $ma50`

hod=`cat velt.tmp |grep HOD |awk '{print $3}'`
lod=`cat velt.tmp |grep LOD |awk '{print $3}'`
result=$(awk -vn1="$last" -vn2="$hod" 'BEGIN{print (n1>n2)?1:0 }')

if [ $result -eq 1 ]; then
        echo "exceeded high of day" |festival --tts 2>/dev/null &
else
        result=$(awk -vn1="$last" -vn2="$lod" 'BEGIN{print (n1>n2)?1:0 }')
        if [ $result -eq 0 ]; then
               echo "low of day breached" |festival --tts 2>/dev/null &
        fi
fi


if [ $ma20 != $ma50 ];then
result=$(awk -vn1="$ma20" -vn2="$ma50" 'BEGIN{print (n1>n2)?1:0 }')

if [ $result -eq 1 ]; then
        if [ -e velt.trend ] ; then
                trend=`cat velt.trend`
                if [ $trend -eq 0 ] ; then
                        echo "positive trend. " $ma20 " over " $ma50 |festival --tts 2>/dev/null &
                        echo "1" >velt.trend
                fi
        else
                echo "positive trend. " $ma20 " over " $ma50 |festival --tts 2>/dev/null &
                echo "1" >velt.trend
        fi 
else 
        if [ -e velt.trend ] ; then
                trend=`cat velt.trend`
                if [ $trend -eq 1 ] ; then
                        echo "negative trend. " $ma20 " under " $ma50 |festival --tts 2>/dev/null &
                        echo "0" >velt.trend
                fi
        else
                echo "negative trend. " $ma20 " under " $ma50 |festival --tts 2>/dev/null &
                echo "0" >velt.trend
        fi 
fi
fi

Fire this up and run every 5 seconds:
$ watch -n5 ./velt.sh

Watch with gnuplot:
$ echo "pause 1; replot; reread;" >loop_forever.gnu
$ gnuplot -e set label ''; set key off; set obj 1 rectangle behind from screen 0,0 to screen 1,1; set obj 1 fillstyle solid 1.0 fillcolor rgbcolor 'black'; plot 'velt.log' using 3 with lines, 'velt.log' using 3 with lines smooth bezier lw 3;load 'loop_forever.gnu' -p

Monday, January 16, 2012

Athens 24 Hour time lapsed transit video

Using the data from zee.gr/bus/ I've made this 24 hour time-lapsed visualization of the public transit system in Athens from Saturday 4am to Sunday 4am. The blue dots represent the metro vehicles, bus = red, trolley = yellow, tram = orange and proastiakos = green.


I made it by taking a screenshot every 5 seconds in order to check the performance of a new feature I am working on for my zee.gr/bus app and it turned out to look pretty interesting, so I used some encoding tools to create a video:

sudo ../../boost.sh ;xset b off ;xset s off ;xset -dpms ;for((i=0;;i++)) ;do import -window root $i.png ;sleep 5 ;done
for((i=9;i<=15149;i++)) ;do convert $i.png -resize 84% -crop 1280x720+6+75 $i.png ;done
for((i=9,j=1;i<=15149;i++,j++)) ;do cp -v $i.png img`printf %05d $j`.png ;done
ffmpeg -r 55 -i img%05d.png -i song.mp3 -acodec copy -s hd720 -vcodec libx264 -bf 0 -crf 16 -threads 0 -b 5000 athens.mp4

Check it out on youtube here in high-definition :)

Saturday, January 7, 2012

Athens public transport planner

I've finally created a Beta version of my Athens public transport trip planner, schedule and routing app.


The app supports the following main features:
  • Browse the Athens public transport system by route - A service list under the "Network" tab is automatically populated depending on the intersection of routes in the bounding box of the map view. (Shown above) - Services are sized according to trip frequency.
  • Estimated live departure times from each stop in Athens (shown in the popup window above) - This is useful if you want to minimise the time waiting at bus stops.
  • Browse the stops of each route including the first and last departures depending on the current day (Metro line 2 shown above).
  • Quickly plan a route by right clicking on a source and destination position on the map (shown above). Alternative routes become highlighted on-mouse-over. More advanced planning features are also available, such as the ability to specify departure/arrival times and exclude specific modes of transport.
I started the app in 2009 when I moved to Athens (I've recently left). I found it difficult to make any sense of the public transport system. With the exception of the Metro, the rest of the transport means seemed a mystery to me. The only available information at the time were lists of stops and streets from the OASA website. Furthermore, there were even inconsistencies between time table data in Greek and English. The only semi-useful information seemed to be some outdated pdf route maps probably created some time around the Athens olympics.

At the time, I thought it would be a nice idea to create a trip planning app using data which I planned to scrape and create myself. This seemed pretty straight forward at the time and I estimated I could complete the task by devoting a few weekends to it over a period of 4-6 months. However, given that I'm posting this now (January 2012) its pretty obvious that I underestimated the complexity of the job. I ended up working on this app for about 3 years, spread over the odd weekend here and there whenever I had nothing to do, which is quite rare if you live in Athens :)

I began by focusing on harvesting the data I would need for the app. This involved designing a database schema to hold routes, time tables and stop information. I populated this schema by defining the Metro, ISAP and Proastiakos stops and routes by hand - creating line-strings and stop locations. I then populated the Proastiakos time table data by hand (literally) by inputting time table data I found in the stations. The metro and ISAP data was a little more interesting, since the only available data was frequency based. So given frequencies, first and last departures, I created some throw away code to calculate complete schedules for all stations by simulating the minimal number of vehicles needed to maintain the published frequencies given the length of each route.

The next step was daunting, and having spent quite some time to generate just the rail data, I knew that generating data for the entire Athens bus network would be the most time consuming part. Nonetheless, I continued with the hope that I could find a nice way to automate and speed up the data generation process. I started by snatching and parsing route descriptions from OASA which described each route by a textual ordered list of stops and corresponding list of streets. Given this raw data, the next task was to somehow generate a set of line-strings and stop locations for every single route.


To do this, I created a kind of bus route creation tool (shown above) using google maps api (at the time v2) directions and geocoding services. The tool worked by geocoding the list of roads for each route, and then with the help of google's directions service, extract the path for the set of geocoded roads. Then, for each automatically generated path I estimated the bus stop locations by evenly distributing the number of stops for each route over each path. The next phase involved checking each estimated route and manually redefining stops and line-strings where the automated generation had failed.


Eventually I ended up with the prototype shown above - this simply allowed viewing of individual routes and stops. At this point I dropped the project for some time to work on other things. When I returned I was happy to learn that OASA, via the Greek government open geo-data initiative had released time table data in GTFS format. This really helped me, since I had not yet started on creating time table data for buses as I had done earlier with the train network. This, in my opinion, is a great move since it allows developers like me to concentrate more on the creation of an app rather than harvesting and structuring scrapped data. This video I re-posted some time ago on this blog pretty much sums up the benefits.

I also believe the initiative to publish official feeds makes more sense than spending millions(?) on projects such as Attica Routing Portal and the more recent Opti-Trans project, since once these contracted projects are complete, they are likely to lay rotting due to the fact that funds are required to maintain them and eventually just serve as relics based on depreciated routing data.

Although a good move, the GTFS feed itself is not at the moment in good shape. I've been told that OASA are working to fix it, but in the mean time if you are interested in using this data there are a few things to note: (1) The feed is invalid - E.g., it does not conform to the GTFS spec. (2) Quite a few of the stops are in the wrong order. (3) The schedule data for intermediate stops is very wrong for some routes, showing buses traveling greater than the speed of light!

Thursday, January 5, 2012

Genetic Algorithm for Image Evolution

Some time ago I came across this, this and this - an interesting idea to reproduce an image given a minimal set of polygons, utilising evolutionary search. The original idea used a hill climbing strategy to randomly mutate a collection of polygons, keeping a mutation only if the change yielded an improvement, defined by the sum of pixel by pixel differences between the original image and the collection of polygons in the new image.

I was curious if the method could be improved by using a genetic algorithm (using a population of candidate solutions instead of just 1) and ended up with this . Using this image as input, the objective is to evolve a new image constrained by a maximum number of polygons. Here is the result:


The algorithm initialises a population of candidate images where each image contains 1 randomly coloured and positioned polygon. At each evolutionary step, each individual (candidate image) in the population is evaluated according to a fitness function which determines the closeness of the candidate image from the original and then assigned a fitness score. Then, individuals from the population are selected using fitness proportionate selection such that individuals with greater fitness (closer proximity to original image) have a greater chance of mating. Selected individuals then produce offspring using a genetic crossover technique and are then subject to mutation. Crossover involved copying polygons from each parent to form new offspring, while mutation involved random changes to polygon structure and colour as well as the possibility to add or remove a polygon from the newly created offspring.

The main difficulty with using a GA for this problem is the choice of crossover function. I tried a number of schemes and found them to be destructive - Ultimately mutation led the search for image improvements, however this simply results in random search. The second difficulty is the quality of the fitness function: I tried the sum of differences between the original image and the candidate image over each RGB value, histogram comparison and a combination of both. Ultimately, I could not really improve on random search so am thinking of new ways to encode the problem so that the search space is better defined.

Friday, October 14, 2011

Athens: The City Before

A neighboring balcony from a newly constructed modern apartment building partially demolishes a neoclassical building in Athens. 

Based on a sequence of frames extracted from a 1980s film "Here is Athens…..the city before" (Nikos Grammatikopoulos), documenting the destruction of neoclassical architecture in Athens.


Friday, May 6, 2011

Taxi calculation reloaded

Since September last year my taxi calculator app has been used by 44 thousand people looking to estimate the cost of their taxi journey in Athens. After a lot of feedback, last weekend I decided to re-implement it and introduce some new features.


The first version of the app had many limitations. It was only intended to work in Athens, it did not include "Tariff 2" in any calculations (double tariff for part of trip outside of city limits), it did not include the new flat rate 35-50 euro fare to and from Athens airport and Athens centre (specifically inside the Daktylios Athinon), and most importantly, did not take into account the fare incurred when the taxi is stationary (in traffic or lights).

Finally, it turned out that the google directions API I was using to calculate the possible taxi route did not work for the center/daktylios ring of Athens. This is because the centre of Athens operates a traffic restriction policy based on the day of week and parity of number plate, and due to its complication, google or teleatlas have marked the area as pedestrian only. As a consequence, google directions do not work at all for Athens centre and are sub optimal for the Athens greater area because of the diversion from the pedestrian only zone.

The new version now has the following features:
  • Works for most of Greece, including Greek islands (fares differ on islands due to different VAT/FPA)
  • Tariff 2 fare now included for trips originating from Athens or Thessaloniki if the trip exceeds city limits
  • Includes fixed tariff for trips to and from Athens centre and Athens airport
  • 24 hour Forecast of traffic conditions in Athens or Thessaloniki and impact on taxi fare
  • New routing/directions service based on OpenStreetMap data
  • Toll calculation for all of Greece
  • JSON API for taxi cost calculation, toll calculation and dynamic traveling times (contact me if you would like to use it in your own app)
The original app was 100% client side (Javascript) - this is great for scalability, since the client does all of the work, however for the new version I wanted to pull most of the calculation server-side - specifically because the new features require a number of geometry calculations and quite a lot of geospatial data.

In my day to day work I primarily use Java and for my own projects I tend to dump the bloat of xml and overweight frameworks for something more productive, powerful and less suffocating - Specifically a functional language - I've been playing around with Lisp, Erlang quite a lot (at a novice level) and starting to appreciate Javascript as a very powerful language and considered using nodeJs for the application re-write. However, I came across Play Framework which is a full stack web app framework for Java inspired by Ruby on rails. A little skeptical at first, I thought I'd give it a spin, and quickly found it to be a productive and high quality framework.

Eventually I ended up with this stack:
The MapQuest open directions API is a very good open routing service exposed via JSON/XML and makes use of data from the open street map project. The documentation and support is top-quality enabling me to make use of this web service in my app in next to no time.

The most significant addition to the taxi app is the inclusion of a traffic model for Athens and Thessaloniki. As a separate side project I have been working on a forecasting/simulation system for traffic flow which will eventually make use of an implementation I started a few years ago of "NEAT" (Neuro Evolution of Augmenting Topologies) as a forecasting technique.

For now, the model is very crude and is based on the assumption that the deviation from the average optimal driving speed (average driving speed with no-traffic) is very strongly correlated with trip duration. To model this in the taxi app, I use a matrix of weights which represent the deviation from optimal driving speed for an averaged 24 hour period in Athens.

During the hours between midnight and ~5am, the traveling conditions are "optimal", meaning average driving speed will be >= the average of speed limits inside the city - this has a weight of "1" (no deviation). During the first peak/rush hour, the weight is 1.55, or according to my assumption, total (moving) traveling time will be 1.55 times more (on average during weekdays) and total (stationary) time/traffic lights/gridlock with also be 1.55 time more (likely).

The graph above shows my weights which have been derived from actual observations. The 5 labels (A to E) are very interesting and say quite a lot about life in Athens (or indeed many cities). It is obvious that in (A) congestion is much higher from ~7am, peaking exactly at 9am (B). Between 8 and 9, the situation rapidly deteriorates - People are going to work, and evidently the majority of the population want to be there at 9. The sudden peak is due to the size of the city - it is small, you can go quite far in 1 hour and most people will leave for work during this period. Label (C) shows that during weekdays most people will be at work and some perhaps heading out for lunch. Label (C) also shows a peak during saturday, which is probably people who are working on saturdays (having again arrived at 9 - smaller peak below B) and are leaving earlier at around 1pm. Label (C) also marks an interesting point - At around ~2pm the traffic suddenly starts to rise again up until it's peak at ~5pm. This long-tailed rise shows that unlike the sharp 9am peak, where most people want to be at work for 9, the time for people leaving work is obviously quite variable, but peaks around 5pm and long-tails off to settle down at around 8pm. Perhaps I'm in the wrong job because most people i know (working as programmers in Athens) have 0 chance of leaving work before 8. The final label (E) shows a peak of Saturday traffic at around 9pm - tsiporisation :)

Monday, January 3, 2011

Monday, November 8, 2010

Athens transit weighted graph I

Experiment, code->paint: Bash, MySQL Geospatial, Java, Linestring weighted edge clustering, Oil on canvas.
Central Athens bus network, edges weighted by OASA frequency.



Sunday, October 10, 2010

Visualisation of Athens Public Transport

"Unfortunately, no one can be told what The Matrix is. You have to see it for yourself."

Using the new styled map functionality of Google maps API v3, I've created these visualisations of the bus network in Athens.


The network is made up of 569 bus routes covering a combined distance of 6500 Km - enough to stretch to central China... Each route is shown here in red - where bright red indicates a higher concentration of buses.



Central Athens - This detail demonstrates how well the centre of Athens is connected. It puzzles me why anyone would need to use a car (or taxi ;)) I'm in favor of banning all cars in the central zone...


Connections to the port from central Athens

Port of Piraeus

Agia Paraskevi

Monday, September 20, 2010

Athens Taxi Cost Estimation

Tired of being overcharged for taxi trips in Athens? At the weekend I made this simple tool to help estimate the cost of traveling through Athens by taxi: www.zee.gr/taxi/ The costs have been calculated according to the most up to date laws (2010) and include a full itemisation of charges you can expect from your taxi trip. The charges also include the 11% VAT introduced for Athens taxi services in July.

To use the tool, simply right-click on the map to specify your starting and destination address or use the input fields in the left hand side menu. The screenshot below shows a trip from the port to the airport including additional port, toll-road (Attikh Odos) and airport fees.

The tool was made using Google's Beta Maps API v3, Google geocoding and directions services and some very ugly hacked up css/jquery/javascript. Improvements coming soon :)

Open Transport Data

A Case for Open Data in Transit from Streetfilms on Vimeo.

Saturday, May 1, 2010

Eastern Telegraph Co Ltd


This interesting looking building was once the hub of telecommunications for Greece, established c,1854 (I think). It is the site where the first submarine cables were laid, in 1858, connecting Syros to mainland Greece via Pieria. Later, cables were laid connecting Syros and Chios, subsequently providing the first telecommunications to Constantinople and Alexandria.

According to this reference Greece under King George, "There are 8,958 kilometres of wire, and 186 telegraph- offices. The most important service is that of the Eastern Telegraph Company, with its head-quarters at Syros." The building later housed the Merchant Navy School.

The interior of the building is fascinating. Strange mechanical devices lay dormant amidst the call of squabbling pigeons.

The structure was severely damaged during the Second World War, although restored in 1947 to house the TTT (Panhellenic Telephony Telegraphy & Postal Service).

Lazaretta



Next to the Eastern Telegraph Company building lay the ruins of "Limokathartirio"; a large complex, constructed in 1839 by the Bavarian architect & painter, Wilhelm von Weiler. Weiler was also the architect of the Old Military Hospital in Athens, constructed earlier in 1834. The Lazaretta/Limokathartirio building in Syros is considered to be Weiler's masterpiece. Initially, the complex was used as Quarantine until the late 19th century.






The main part of the building consists of 32 rooms (all equipped with a kitchen, wash area and lavatory), in which mainly Eastern travelers would be required to stay for a minimum of 7 days "clearing period" before entering Ermoupoli.



Until the late 19th century, this clearing period was implemented as a measure to protect the local population of Syros from major diseases of the time, such as the deadly cholera pandemics. From 1908, the complex was used as an asylum for the insane. And later still, up until 1961, used as a prison holding political prisoners en-route to be expelled to deserted islands.



The building has been divided up according to function (Quarantine, Asylum, Prison). It is possible to identify the purpose of each area by looking at the colour of the walls. Saffron is the quarantine area, while Red is the Asylum area and Grey, the prison.



It is difficult to find any specific historical events (although there must be many) related to the occupants of the building. Although, through a little searching, it is apparent that a Greek priest, intellectual & revolutionary (Theophilos Kairis) died in the building (assuming the building was also a prison at the time) on the 10th January, 1853.



In modern Greek history, Theophilos Kairis (b. 1784) is an important figure. On the 10th May, 1821 he declared the War of Independence by raising the Greek flag on the island of Andros (See wikipedia article). He also imported revolutionary thinking, such as western science & liberalism in to Greece and is notably credited with introducing the first telescope to Greece.

Despite this, his "radical" ideas did not impress the Greek Orthodox Church. Consequently, the never ending obstacle of rational thought that is - The Church -, declared him as a heretic & sent him to Syros for trial. Theophilos Kairis, along with his collaborators were tried on the 21st December 1852 in Ermopoli, where he was sentenced to 2 years imprisonment. Being 68 years old and frail, he perished after only 10 days of imprisonment in the Lazaretta building.

He was buried without any ceremony in the yard of the building; Later, priest's, along with followers of the church, desecrated his grave by exhuming his body, cutting open his belly and stuffing it with Lime in protest of his "heresy".

Off topic note: according to this reference, the desecration happened as above - I find this a little hard to believe. It is known that Theophilos Kairis died of "natural causes" - after only 10 days in prison. He died in a time of plague & disease, he was buried without ceremony, and further still, in a building which was also used as quarantine. If these facts are taken into consideration, along with the strange use of Lime, I believe: Lime (Quick lime / powder - Calcium Oxide), in times of plague, has historically been used to cover corpses. This is because calcium oxide/lime speeds up the decomposition process, thereby minimising the risk of disease to the living population. Lime, in it's raw form, was also used to mask the smell of death. Given this, it's more probable that since he perished of "natural causes", it may have been suspected that he died of sickness, and had to be buried quickly - hense the lack of ceremony.



It's a fascinating structure, with a strange, and as is the case with Theophilos Kairis; a dark history. Unfortunately, many of the red brick arches have become the victim of developers (one of whom is known locally as "The Dragon") who have apparently removed many bricks by hammering away at the building under the cover of night in order to build fireplaces in modern day homes.

In 1976, a developer removed over 50 bricks from von Weiler's masterpiece to be used in reinforced concrete for a patio for a holiday villa in the nearby village of Finika. Subsequently, the magnificent arches at the front of the building collapsed. Soon after this, in 1978, the Greek state recognised the building as an official monument; although since then, it has been left to ruin. More unfortunate are the plans to use the site as a casino :( -- I believe this to be a shame, given the site's notable architecture and its importance in modern Greek history.


Industrial Ruins

Ruins of the Ladopoulou Tannery

Ropes

May day

Friday, January 15, 2010

Map of interesting architcture in Athens, Greece


Using google maps API and by geocoding data from EIE I've constructed a map of interesting architecture in Athens. Buildings can be filtered according to year of construction by using a slidebar (see screenshot above) + the map has a basic search function.



Using a map of Athens published in 1890 from Meyers Konversationslexikon, following the design of modern Athens by architects Stamatis Kleanthis, Eduard Schaubert & Leo von Klenze , I aligned the streets from the 1890 survey with the streets as they exist today. The 1890 survey is astonishingly accurate - It's fascinating how this level of precision cartography was achieved without modern day tools such as GPS. Subsequently, the original survey can be imposed on to the modern map with very minor corrections.



Above is a detail of Kolonaki. According to the 1890 map, the modern day Hilton hotel was built on top of a Barracks & everything south of Vasilissis Sofias Avenue was undeveloped.. You can also see how the development of Kolonaki has spread up into Lykavittos :/


By comparing the Acropolis with the survey from 1890, it's possible to see how much things have changed... Lot's of interesting things to see here :)

















Back to the architecture map - I've noticed this building a few times whilst coming back from Athens in a taxi. According to EIE, it was built in 1925. I especially like this one - it stands in huge contrast to the hastily built modern slabs of concrete around it. Here it is on the map - 46 Septemviou & Marni Steet

Link