learning from manjaa

01 May 2011 chennai

A little background

I was trying to make a static-site generator cum blogging engine called patang for myself. It was inspired by jekyll (which runs this site), and was supposed to be a light-weight variation of jekyll, written from scratch. It isn’t yet finished. patang source

My urge for getting blogging engine specific to my needs was strong, and my developer optimism was high. Hence I came up with another product idea that would fit my blogging needs well, and would be cool. Presenting hereby … manjaa

  • It is a web application. This is important so that I can write my blogs in a browser instead of a text editor. I keep changing my computers every six months, hence so.

  • It should run a static site generator on the backend. Jekyll right now. So my posts should be static pages generated by jekyll, and I can add liquid templates whenever I want to change the look and feel.

  • It should be able to push the posts to github, so that blog is hosted on github pages. On my gihub page url - zerothabhishek.github.com. This one was most important.

  • It can be used by anyone by singing up. This means having separate jekyll folders and git repos for all the users.

So what I needed to implement was mainly a simple blogging application, a jekyll backend and github api. Blogging application in rails is a matter of minutes, integrating with jekyll should take a few hours at max. The only real challenge was github api. I had seen some implementations of oAuth before, never done one myself. I can’t say how I imagined it could all be done in two days.

And after spending more than two weeks on it, I had to abandon it. Here are some learnings.

Learnings

  • Background jobs are important in rails projects. Any action that may take long time or that can be done later in the background should be done in background. There are way too many tools to accomplish the task, and most of them solve the problem differently. Decision should be made based on whether you want jobs stored in database, if you want rails environment loaded in context of the jobs, whether priority is needed, if threads are allowed (jruby)

  • Testing background jobs is hard. The manual testing especially - you need to switch between terminals or open log files. Automated integration testing (which I didn’t do here) may help.

  • Beanstalkd and stalker are a simple solution if the problem is simple.

  • Gihub Api documentation exists, but support/examples are limited. As per the docs, I could create repo, but in actual practice I could not. And no one replied to support requests.

  • OAuth2 requires a callback URL to be specified. And that can’t be localhost. As a hack, I set up a small sinatra app that receives the callback and redirects the browser to some URL on localhost. Perhaps something better than this could be done.

  • OAuth2 access_token expires. In case of github, it is unpredictable, but works for a few hours. When it expires, we need the user to get us access through the browser. And the only way to check if it has expired is to catch a certain exception, flag the expiry and present it to the user whenever he logs-on/refreshes.

  • Deployment is always a struggle. capistrano documentation is bulky and not easy, and recipes are not that easy to write. A better approach is to start deployment after first commit only. Or perhaps use this gem

  • Some tools might just fail because on linux they have a different version, when installed via apt-get. For example, version mismatch in Beanstalkd client for ubuntu(older) and mac.

  • bundler should be run vendored on installing so it installs all the gems within the project directly only, and not disturb the system gems. See bundler website (well documented)

  • Here are some very useful sysadmin commands

      ps -el | grep beanstalkd		## shows ps info if the program called beanstalkd is running
      ls -l /proc/21345/exe			## shows the executable file that the process id 21345 is running
      ls -l /proc/21345/fd			## shows the files the process 21345 has opened eg log files etc.
      nohup  rake:somedaemon 			## nohup runs the program as daemon - it doesn't die on closing the terminal
    
  • It is a good idea to have any script that needs the app objects as rake task. And anything else in script/*. And to run daemon by having the rake task or script run via the nohup. A better idea is to use a tools like god or monit

  • There were some wrong assumptions regarding the behavior of ssh keys. A single public key can’t perhaps be used for multiple accounts. Need more info on that.

  • A project with too many moving parts - independent components integrated together is hard to implement and test. A project with third party dependencies (api) is risky

Edited with ACE

blog comments powered by Disqus