My Little World

Well posting is a lot faster when I select --no-preview for nanoblogger. No error messages either. So I made a shell script called make-entry to run this command for me every time I want to add a blog entry.

Every time I add an entry with nanoblogger I get an error message at the end of the posting saying something about preview finding an error in the index.html xml code. So I'm posting this post with the --no-preview option selected. Let's see how it goes.

Today I've been thinking about my collection of shell accounts again. I started adding a writing directory into each of them. And a few weeks ago I started exploring the use of Perl to automate running processes on all of these accounts simultaneously. I should write a Perl script that backs up my writing stuff into each node sort of like what Dropbox does.

So I've been thinking about how to do this. I guess I could run a cron job that checks the contents and timestamps of each node's writing directory and synchronizes them every few minutes or so. I think I got a simple SCP Perl script working and was successfully able to transfer files to a few of my test nodes. Now I need to figure out how to link all these little Perl scripts together. Maybe I should write a nice and tidy module to do all these functions, and save all my node information to an encrypted text file.

So yeah. Ideas. I also want to work out a better system for running asynchronous jobs on my node cluster. Perl is going to be my go-to tool. I've explored some of the stuff other's have made with Bash, Java, and C but they are all too hard to use and setup. And they don't do exactly what I want them to do either. They are good for running the same job on each node, with just the data split among nodes. What I want to do is run different jobs on each node and then collect results with the main node I'm interfacing with.

Like say for example I want to find the best model that describes a dataset. I want to try various models on that same dataset. So I write a script that assigns a range of models to try and report back on how well that model fit. So I write a script that splits the models across various nodes and they all return a report when they are done. I combine all the reports and analyze the results from the main node. Sounds simple enough, but now I need to consider the various differences I have between the nodes. Some nodes only allow 200 MB of storage space. Some nodes don't have the latest version of Perl installed. Some have Java and others do not. So if I wanted to use Weka I would be limited to only a few nodes.

So then the real work becomes developing Perl scripts that implement all of the various models I want to try out. The amount of work I need to do to get started in machine learning becomes pretty huge. Someone needs to write a pure Perl machine learning library that is small and easy to install on machines with limited resources.

That's all for now. I'll post on update when I'm done daydreaming. I think I'll try designing a file backup server first.