Adon Metcalfe's Journal


Previous Entry Share Next Entry

works update =D

I guess now is time for an update (warning technical stuff ahead)

I've been working on lots of python code for the last ages - work and labyrinth related mainly. We do lots of server management stuff at work (department of environment & conservation, fire management services), and because I'm lazy/like bash/frustrated with existing config management tools that require centralised deployment (chef, puppet, etc) and provisioning tools that are install only (imaging with clonezilla, debian preseeding, redhat cobbler, suse studio) which are all very cool but fall over when either pulling in local changes easily, copying an existing servers setup (though clonezilla is pretty awesome at that), and most painfully backing up non config based info (databases, imagery etc). So I wrote a tool to do it myself in like 2 weeks at the end of last year, then deployed 40 servers with it, and after doing refinements every couple of weekends its kinda useable (I've still got a bit more work to do on getting some documentation sorted before I release it, which will probably happen July sometime).

hgbackup: stick it on a server with networking, use it to manage launching/messing with anything you can call from bash/anything python can call on a system, tell it to backup directories locally using mercurial or flag them to be backed up remotely, any system with it installed can backup any other system over ssh (uses rsync + local versioning depending on remotes config)

I wrote it from the perspective that I want to vim .history, copy/paste commands into a script("sudo apt-get install supertux => H.cmd("sudo apt-get install supertux", method="subprocess"), the subprocess method allows user interaction, default returns output of command for processing in python (uses commands.getoutput)), but have it easily redeployable elsewhere. The backup commands are just directories or method:data i.e. for postgres you go "pgdump:dbname".

SO that was kinda cool and its been working quite well (maintain backups for two sites, 20+ vms each at eachother on 5 minute intervals, and it manages pretty well), though needs more documentation

That brings me to cool stuff were doing for labyrinth data services which is a private business I run with Patrick Coleman and Scott Percival. We wrote a fancy frontend for KVM in Django, that lets our clients manage their vps power controls and account information (billing, passwords etc) as well as provide a vnc tunnel to their vps so they can administer it remotely. At the moment were doing lots of migrations of customers from our old hardware to newer hardware, and part of the plan is to automate provisioning netbooted servers + config deployment so we don't have to copy template/setup vps manually like we do now, and customers can just pick they want a new *linux distro of choice* server. The app will boot a vm of the network imaging server running clonezilla, which launches hgbackup on first boot, retrieves a config generated by django and deploys custom packages/settings for that user as well as standard packages to make a shiny new VPS (and automatically bill customer of course =P)

We also spent a bit of time on magical network shaping, and abolished quotas because their annoying. OpenBSD has an incredible firewall stack called pf (Packet Filter) that lets us do stuff like shape on WAIX/non WAIX traffic based on routes we receive over BGP (we have our own /22 of ipv4 addresses and a whopping /48 of ipv6 addresses (thats the existing internet ^2.5, i.e. 1 trillion (10^12) squared! cmon everyone get on ipv6 already!) and do elegant linkshare allocation (we use HFSC) to guarantee minimum rates, burst rates, and sharing unused bandwidth. Most ISP's operate at 40:1, we run at 20:1 contention by default, which we find means our users can generally get their full burst rate, while only paying for 1:20th of it (bandwidth in perth is expensive as =()

So those are the personal projects I've been hammering away at for the last 6 months or so, while at work for the last 2 years or so I've been working on a massive project for resource management and data visualisation. An aside, Adrian who I work with has spent the last year implementing lots of OpenBSD routers built on the excellent yawarra embedded boxes that we also use at labyrinth. Anyway we used to have this commercial app called datagate made by datalink systems in america, but it cost a lot, and per user licenses were $1000, so as we have like 2000 people in the department that can use resource information the plan was to write our own resource tracking tool to display resources across the state. The majority of the application (firesource) was developed in Django with clientside javascript being written in dojo. After a pretty good run of about a year we are now redeveloping the app to be a generic vector and raster visualisation tool (it already aggregates data from bureau of met, nafi, firewatch (landgate) etc) and after a bit of bouncing around I settled on JQuery for the clientside js, google closure for compiling and javascript templating, and django as always for the backend. To serve imagery we use mapserver for raster data, and geoserver for vector data pointed at a postgis database. Django is backed with a postgis/postgres database which holds about 2million tracking points for the last 2 years of operation, and we query that entire dataset for the latest points for each device (we have around 300) in 4 seconds almost every minute, with the capability to process a maximum of 10000 incoming spatial records a minute.

I've spent a lot of time working on the versioning/auditing for the new application to ensure that the system is able to do indexed queries across all versions in the dataset (like fullhistory or django-reversion but better =P), being able to display any snapshot in the history of the application (so users can see what vehicles were at fire X that had a rough boundary by time Y etc), and this work is leading into map production, plan is the application will produce a printable map at 300dpi at up to ISO A0 for operational staff to create maps quickly in the field on any machine with a web browser, including live vehicle position information. The Audit stuff is quite generic and I feel I should split it into a separate reusable django app at some point, but at this point theres nooo tiiime (theres a whole bunch of stuff going on with enterprise architecture design happening at work also and somehow I got dragged onto the committee for that also =O)

Well that was interesting, but yeah any questions on bits and pieces please comment, otherwise this is just a braindump for google to index so I can find it all when I get AMNESIA (futurama awesome).

For those whose eyes glazed over - ASCII NINJA TURTLE =)

           ,;;;;;;, `\. `\         .,c$$$$$$$$$$$$$ec,.
      ,;;!!!!!!!!!!!>; `. ,;!>> .e$$$$$$$$"".  "?$$$$$$$e.
 <:<!!!!!!!!'` ..,,,.`` ,!!!' ;,(?""""""";!!''<; `?$$$$$$PF ,;,
  `'!!!!;;;;;;;;<!'''`  !!! ;,`'``''!!!;!!!!`..`!;  ,,,  .<!''`).
     ```'''''``         `!  `!!!!><;;;!!!!! J$$b,`!>;!!:!!`,d?b`!>
                          `'-;,(<!!!!!!!!!> $F   )...:!.  d"  3 !>
                              ```````''<!!!- "=-='     .  `--=",!>
                         .ze$$$$$$$$$er  .,cd$$$$$$$$$$$$$$$$bc.'
                     z$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$c .
                    $$$$$$$$$$$$$$ dbc `""?$$$$$$$$$$$$$$$$$$$$$$?$$$$$$$c
                    ?$$$$$$$$$$$$$$$$$$c.      """"????????"""" c$$$$$$$$P
         .,,.        "$$$$$$$$$$$$$$$$$$$$c.   ._              J$$$$$$$$$
 .,,cc$$$$$$$$$bec,.  `?$$$$$$$$$$$$$$$$$$$$$c.```%%%%,%%%,   c$$$$$$$$P"
$$$$$$$$$$$$$$$$$$$$$$c  ""?$$$$$$$$$$$$$$$$$$$$$bc,,.`` .,,c$$$$$$$P"",cb
$$$$$$$$$$$$$$$$$$$$$$$b bc,.""??$$$$$$$$$$$$$$FF""?????"",J$$$$$P" ,zd$$$
$$$$$$$$$$$$$$$$$$$$$$$$ ?$???%   `""??$$$$$$$$$$$$bcucd$$$P"""  ==$$$$$$$
$$$$$$$$$$$$$$$$$$$$$$$P" ,;;;<!!!!!>;;,. `""""??????""  ,;;;;;;;;;, `"?$$
$$$$$$$$$$$$$$$$$$$P"",;!!!!!!!!!!!!!!!!!!!!!!!;;;;;;!!!!!!!!!!!!!!!!!;  "
$$$$$$$$$$$$$$$" ;!!!!!'`.z$$$$$$$$$$$$$ec,. ```'''''''``` .,,ccecec,`'!!!
$$$$$$$$$$$$$" ;!!!!' .c$$$$$$$$$$$$$$$$$$$$$$$c  :: .c$$$$$$$$$$$$$$$. <!
$$$$$$$$$$$" ;!!!!' .d$$$$$$$$$$$$$$$$$$$$$$$$$$b ' z$$$$$$$$$$$$$$$$$$c <
$$$$$$$$$F  <!!!'.c$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$b  $$$$$$$$$$$$$$$$$$$$r
$$$$$$$P" <!!!' c$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$, "$$$$$$$$$$$$$$$$$$$$
$$$$$P" ;!!!' z$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$  $$$$$$$$$$$$$$$$$$$$

next time I should chuck up some screenshots

Awesome. pf is pretty cool, although the constant rate of change to the syntax gets a bit annoying.

haha yeah, we plan on templating up our configuration which will make it easier to adjust going forwards =)

you should ask patrick about the massive debate we had over python3's deprecating of the print verb =P

yeah, that was about my thought process too >.>

Wow, all that stuff sounds incredibly awesome.

hydrogen generator for car


2011-02-18 11:23 am (UTC)

Nice website, I think I recognize this design from somewhere, is it a template? It suits your website anyway.

immergas assistenza


2011-02-21 09:23 am (UTC)

maria capitanu
berlin parfum
[url=http://procug7161.myip.hu/1vkr.html] Air canada floor plan for basketball [/url] [url=http://liama9806.myip.hu/kvlj.html] Muscle weakness lipitor [/url] [url=http://enex1697.myip.hu/7pot.html] Super bowl xlii final score photo [/url] [url=http://liama9806.myip.hu/lugx.html] Ibuprofen pain releif [/url] [url=http://dorssmar8959.myip.hu/0smu.html] Digital playground pass [/url] sri dinesh
404 vodafone
[url=http://crerar2287.myip.hu/vlwj.html] Ibuprofen 200 mg [/url] [url=http://artio8426.myip.hu/owlo.html] Ibuprofen salt titration [/url] [url=http://houghgnac3923.myip.hu/mhns.html] Abilify pediatric use [/url] [url=http://procug7161.myip.hu/1ozu.html] Machine to play stick cricket on [/url] [url=http://artio8426.myip.hu/mdqt.html] Sexual side effects accutane [/url]

Beneficial info and excellent design you got here! I want to thank you for sharing your ideas and putting the time into the stuff you publish! Great work!

I don’t bookmark sites but i will bookmark this! LOL!