Strehler: state of the art

I know that i'm not a frequent writer so a lot of interesting things about me doesn't appear on this place because i'm lazy. So I decided to write just a post to summing up how Strehler is going on after the 1.0.0 release.

Well, a lot of things changed and got better since that day. The first, fundamental step, with 1.1.0 was landing on CPAN with a real test suite and some supporting script. Now you can install Strehler using just cpan -i Strehler and getting an up and running CMS with few commands. In many ways the 1.1.0 release of Strehler is the real first release, because I know that the long, boring and messy installation procedure of 1.0.0 is something NOONE would use, out of me, obviously, but I'm the father of Strehler and my judge is irrilevant.

Strehler 1.1.0 was my first package released on CPAN so I learned a lot releasing it. I learned about PAUSE, about distzilla and about how many thing to your code after you upload it on the perl repository. I did a lot of things wrong because mistakes are the way nature teach you things, so in the week after 1.1.0 release I released FIVE bugfix packeges bringing rapidly Strehler to 1.1.5 version. Well, it's a good thing, high numbers make your software truly professional.

With a quite stable version of my software on the public repository, I started work about little features not yet implemented, like function to order admin list and searchbox. All these things went in the 1.1.6 version and i'm quite proud of it. Yes, a lot of work could be done on the search box because it is implemented in the easiest and plainest way I found, with no indexing or optimizations. It's good because it's accessible just from the backend of your site, but it's probably not a good implementation if you want it also on the frontend.  Probably in the future I'll do something, pluggable if I can, about it.

I'll probably release soon a 1.1.7 version. I wanted to wait after some implementations, but I found a big bug in the strehler script that make impossible to create a strehler database on your custom site, making all the installation process useless. I fixed it and I want to share it with the world as soon as possible.

After that, a lot of adventures are waiting for us!

The spring of HTML::FormFu

HTML::FormFu arrived on his first major release (1.00) this christmas. It's quite an event considering that previews release was on 05 Oct 2012.

I'm very happy for that because now form rendering is better and cleaner. Problem is that this new rendering had some incompatibility with my Strehler forms (javascript and CSS) so, when I upgraded, thing went a little messed up.

The most interesting thing (the only worth to be written here) is about my little CategoryUnique validator.

Strehler backend manage a category tree that you can use to organize contents. For example:

foo
foo/bar
foo/ber
yetanothercategory
newcategory
newcategory/newson

When you create a new category you have to give its name and its parent. The CategoryUnique validator check if the category already exists under that parent.

In the example up there you can create bar under newcategory, you can't under foo, because there's already one category named that way.

So the validator can't just check about the value of the category field, but it has also  to consider the second parameter of the form, to check if the category has the same name under the same parent.

With HTML::FormFu 0.09010 it was easy:

my $self = shift;
my $parent = $self->form->param_value('partent');

Easy, yes, but apparently forbidden by documentation.

If the value is invalid or the form was not submitted, it returns undef.

In a validator a value cannot be already valid :-)

But it worked because there was a bug, a bug fixed in the December 2012.

We could exploit it just because no new release of HTML::FormFu come after that fix... until December 2013.

What can we do now, with the shiny fixed 1.00 version of the software? The only way I find to do the same thing is to use the undocumented HTML::FormFu::FakeQuery. The query method of the form, infact, doesn't return an hash as documentation still say, but an instance of this object.

Luckly, using it is easy enough:

my $self = shift;

my $query = $self->form->query();
my $parent = $query->param('parent');


without passing through the param method of the form, that HTLM::FormFu documentation itself doesn't trust.

The clock king

I wrote about timezones telling how avoid problem using them and, in particular, using timestamp fields. As I said in the end of the article the solution has a big flow: it depends on MySQL configuration.

This is not good because best practice is to make your code as system-indepentent as possible. It's good if you can install your sotware without checking MySQL timezone. Because MySQL timezone could be configured by someone who is not you, because you could never know the real MySQL timezone and because YOU'LL NOT CHECK MySQL timezone at installation time. You'll forget it. Don't try to fool me, we both know it will go this way.

So? Is there a better solution? Obviously yes.

Problem is that timestamp field is a field that carry in it timezone information calculating it using its envionment. This property bind it to MySQL timezone and gives us headaches. So, solution is throw timestamp field away and use datetime instead.

Datetime field has no timezone information (basically, it's not converted to UNIX Timestamp when saved on the DB table) so it's just a magic string with a day and an hour. It's less powerful so you can have better control on it. The only thing you have to keep in mind is to ensure that, every time your app write a datetime on the database the timezone it uses is always the same. UTC is usually a good choose.

So, when you insert a new line and you want to trace the moment of insert you no more relay on the MySQL auto-update, but you assign:

timestamp => DateTime->now()

DateTime->now() is already with UTC timezone.

Now you can display your timestamp using any timezone you like. The value you'll receive from database will be UTC so conversion is easy.
Just:

my $ts = $dbix_row->timestamp;
$ts->set_time_zone('UTC') #Remember? Value arrive from  database with floating timezone
$ts->set_time_zone('TIMEZONE_YOU_LIKE') #Starting from UTC the datetime will be converted

Habemus Strehler!

Well, i worked a lot about it behind the scene, so i'm now a little excited to talk about Strehler, my most complex Perl work.

Link to github first: https://github.com/cym0n/strehler

As the README says, Strehler is a light-weight, nerdy, smart CMS in perl based on Perl Dancer2 framework. The story of his creation is very simple: I had to create a website (well, more than one) and after I decided to use Dancer2 I also decided to write the backend interface for it once for all, developing a tool easy to install on any type of webapp you need and with a lot of customizations available out of the box.

Strehler implements just the standard CRUD needed for a standard site. It can create articles, upload images, organize things  with tags and categories. I said more about it in the github wiki.

In my opinion the software reached enough stability to deserve version 1.0.0 tag. My hope now is for people to download it, try it and tell me where I'm wrong thinking this. At this point of development Strehler needs to be seen by fresh eyes, this is the reason i'm announcing that with so much emphasis.

You can try Strehler, you can use it, you can just find something you don't like and tell me. If you like collaborative work or if you think that Strehler can be what you need with some modifications, just work on it. Code is open and the git repository is there to be used. If you need some advice about how to improve the tool, try the TODO page. It's there for me as for everyone ho like to develope using Perl.

Strehler is ready! Cheers!

Entering the time zone

It' a lot of time I don't write here, i'm working on something that will make me write a lot (I hope) but i'm keeping silence until it is finished. However, today I had to understand timezones more that how much I did before so I want to throw all the knowledge I acquired in this blog, to make it remeber it instead of me.

Problem of the day is common, but not simple. You have a database with a timestamp field, you want to print it on the screen. It seems easy, a timestamp is just a date and an hour, what could go wrong? Easy, the hour you print IS NOT the hour you expected.

We are a lot of people who lives in a lot of places. Sun can't shine on all these places at the same time. But everyone wants to wake up in the morning and go to bed in the evening so: timezones are born.

We assume that you have a MySQL database, that you will read the timestamp column using DBIx::Class and then that a Dancer app will make it visibile on the screen. Every step has its timezone issue.

Timezone Steps

MySQL

MySQL knows its timezone and every time it gives you a timestamp the timestamp is considered in that timezone.

MySQL timezone is the default_time_zone option in my.cnf. If no timezone is provided, the server will use the one of the machine where it it hosted.

DBIx::Class

DBIx::Class has a lot of tools to manage timestamps. Using InflateColumn::DateTime is nearly a standard. Problem is DBIx::Class can't ask MySQL what is its timezone so you have to configure it when you write the model. Your timestamp field will be:

  "timestamp",

  {

      data_type => "timestamp",

      datetime_undef_if_invalid => 1,

      default_value => \"current_timestamp",

      timezone => "Europe/Rome",

      is_nullable => 0,

  }

Dancer

Dancer is not really involved in this question. Every time you want to print a timestamp you'll go through this code. But this could be used when Dancer has to give you an output on the screen. Here you can decide once more what is the timezone to use. 

Few lines:

$ts->set_time_zone("Europe/Rome");

It's a trap!

What could go wrong? 

If timezone is not configured in the DBIx::Class you will have, as output, the hour in the timezone of MySQL, but with a "floating" timezone. This means that conversion will be... well... erratic.

If timezone configured in DBIx::Class is different from the timezone in the MySQL, MySQL timezone will be ignored although the time you receive is made from it. DBIx::Class timezone will be attached to timestamp instead. No automatic conversion will be triggered and you'll have to work just with wrong data.

How to deal with this

Problem I see with this situation is that MySQL is part of the environment while the rest is part of your code. You write the code, but the environment can change. I think that a good idea is always configure the timezone when you output the data ("Dancer" level) so you have a good point to change it if it's needed, without touching the structure below. Than be always sure of your MySQL configuration when you write DBIx::Classes.

Ballroom Blitz (2)

Silence in the library

When you write a perl program you don't need just the perl interpeter, but also a lot of libraries that someone else wrote. It's easy to fetch them while you're in your developement environment, but when you go to production you have to manage a lot of issues about this problem.

First of all, there's the scenario where you don't have root access for the system where you are so libraries installation appear to be an impossible task. 

A worse scenario is the one where you have root access and you install the libraries directly in the system.

The worst case scenario is the one where you're installing things in the system, the system die and someone behind you start hitting you with a club. No, the problem is not the one with the club, the problem is you. The club is the solution.

Never do installations at system level. System level is just for... the system, the libraries installed that are the libraries need by your OS, not the libraries you need for your site. In many years of developement and deployment  people found many ways to do things better.

The article I linked last time, for this task, uses perlbrew, one of the best practices for that. Perlbrew manages not only libraries, but also perl version, so it can also be used to ensure that the bizarre incompatibilty between 5.16.0 and the cutting edge 5.18.0 is not an issue because you will configure on your machine just the version that works.

But timtowtdi so in this article I'll speak about probably the oldest practice about managing libraries, the oldest but still useful, clean and sharp: local::lib.

local::lib is a library that you have to install at system level:

(as root) cpan -i local::lib

 after that you'll have the power of playing with perl libraries just moving directories. Remember the directory where we decided to place our site, /var/www/com/example/my/?

Create under this directory a directory env and then a file named perl_sandbox with these lines:

libdir=/var/www/com/example/my/env
export PS1='\[\e[1;35m\]\u@\h:\w\$\[\e[0m\] '
eval $(perl -I$libdir -Mlocal::lib=$libdir)

Now source the file:

source perl_sandbox

This way your shell will become pink (the export PS1 line) and all the operations you'll do about perl libraries will have effect just on the env directory. However, with the perl_sandbox active, every perl command you'll use will see the libraries in the env directory.

Just a last tip: remember to install cpanm

cpan -i App::cpanminus

and use that for installations, with --notest option. Life is too short to let cpan to check all the repository every time you need an environment.

Up to the plack

Plack is a server used to run PSGI applications. PSGI is a interface that we can use to communicate with the webserver that will bring our site on the World Wide Web, so Plack is something we really need to make things work.

Talking about this issue, working with Dancer makes thing really easy because the standard bin/app.pl script that run your webapp can be used as is as a PSGI startup file, as the Dancer2 Deployment page says.

bin/app.pl script must run under a perl web server supporting PSGI. My favourite for this is starman (yes, I like David Bowie, but it's not just for that).

Copy&Paste from Dancer2 Deployment page how to run plackup and make your site works, however, is not enough for something that will need maintenance in the future.

The last piece of the puzzle is Server::Starter, a perl module that provide you the start_server script that will change you plackup server in a daemon (Argh! WItchcraft!), making easy for you stop/start procedures and monitoring.

After installing that module and after creating a /tmp directory under our site directory, you can try THIS copy&paste:

WORKINGDIR=/site/directory
PROJECT=/where/bin/directory/is

source perl_sandbox #as seen in the previous paragraph 

          cd $PROJECT

start_server \
--pid-file=$WORKINGDIR/tmp/starman.pid \
--status-file=$WORKINGDIR/tmp/starman.status \
-- \
plackup -E ec2 -s Starman --workers=2 -l $WORKINGDIR/tmp/plack.sock -a bin/app.pl & 

as your start_server script.

And this as last command:

start_server \
--pid-file=$WORKINGDIR/tmp/starman.pid \
--status-file=$WORKINGDIR/tmp/starman.status \
--restart

For restart.

For the timtowtdi section, take a look at Daemon::Control for an alternative implementation of the daemon. For a PSGI webserver general purpouse, one of the most used technologies of these days is uWSGI.

Gentlemen, start your NGINX

Last step is the easiest one. NGINX is just a proxy from the internet to your plackup structure. NGINX and Plackup will communicate using the plack socket that we configured in the start script: /my/site/dir/tmp/plack.sock. Obviously we're talking about an environment where Plack and uWSGI are on the same machine, otherwise you'll have to sail for the TCP/IP, as everything on the network.
Let's see nginx configuration:

upstream siteplack {
  server unix:/your/site/directory/tmp/plack.sock;
}

server {
  server_name www.yoursite.com;

  access_log /your/site/directory/logs/www.site.it_access.log;
  error_log /your/site/directory/logs/www.site.it_error.log;

  root /your/site/directory/project/src/public;

  location / {
    try_files $uri $uri @proxy;
    access_log off;
    expires max;
  }

  location @proxy {
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://siteplack;
  }
}

Few things to say. The public directory of the Dancer2 project should be server directly, because DancerApp has nothing to do with static things like css and javascripts. All the others things are just to copy and paste.

Restarting NGINX, praying all your gods, if all you did was did well your site will rise.
If not probably it will be my fault writing something wrong in here, but I'll be too far from you to be blamed. Just solve your problems and tell me how to fix the article. Comments are there for this reason.

Ballroom blitz (1)

I read this interesting article about Dancer2 deployment. I have a lot to learn about Dancer2 deployment so I thought seriously about apply this practice to my sites. However, working on it, I didn't feel so confortable with the techniques applied there, probably because of my habist more than other reasons.
So, at the end, I came to a different solution and I want to describe it here, to clarify it in my head and to demostrate, once more, that timtowtdi.
I will skip the part about the provider subscription, I want to focus just on the technical issues and I don't want to talk about a specific environment. Let us think just about you having a server, anywhere you want, and the Power of Root is in your hands.

Who's the man?

When you're young and stupid you can think that root is enough, because root is powerful, shiny, charming and has a silver armour with gold drawings on it. Uncle Ben died to teach you that with great power comes great responsibility and you're still so stupid to use root and just root, also for pron watching!

Then you grow up and you know that there're three type of users:

• root: the boss. Period. You don't annoy the boss with stupid deployment issues.
• human users: they have to log in on your machine and do dirty things. They need a little, walled garden (some guards on the wall, too)
• application users: they're on the machine just to work. They're strictly linked to the processes they run.

So, you need to start your app deployment and you have to choose a user that will do it. For this issue, after years chasing not-so-writable directories and evoking the sacred chmod sevensevenseven, I chose the easy way: do all using the nginx application user (nginx or www-data, usually). It's not perfect, it's not the cleanest way, it's not the most secure, but it's tidy enough and give you a lot of space to manage things.

From now on, consider yourself logged as nginx. It's home directory will be /var/www.

Where's the man?

Just a small paragraph about directories. If you don't have a strict discipline your system will become a mess. It's its destiny, it's nature. Entropy is part of every job in the universe.

My favourite idea about how to organize different sites on the same machine is to use reverse domain as subdirectories.

So, mylittlepony.evilgenius.com will go under /var/www/com/evilgenius/mylittlepony and fluffy.bunny.org will be /var/www/org/bunny/fluffy.

This way it will be easy to find the site you're looking for and directory names will never be too long. The policy for domain names is also one of the strictest of the computer science, so you'll never have to manage UTF-256 characters or things like that.

Beam me up Scotty!

This article will speak also about the obvious. 

The easiest way to put the code of your webapp on your server is through some versioning system like git. You just have to pick up an account on github or bitbucket and then, after finishing the development, push all you did on a repository. Then, on your webserver, you just have to clone the project and then maintaining that pulling every update you push.

There's a part of me that don't like working this way. The cleanest and most clever way to deploy a software is packing it someway and then installing it on the server. Packing and installing must be done through a clean, sharp and smart suite of scripts tailored on the software you are managing.

This is the old school procedure and as a lot of old school procedure it's still good, but I have to admit that git is easier and perfect for lazy people.

But! Remember! :

• Think about if you need a private repository. It could be that someone doesn't want that the code of the site is exposed to everyone with a google bar. I know, I know, open is better, open is good, open i what tender kittens wants, but sometimes it's not what YOU want.
• gitignore is the most important file of your repository. Design it thinking that nothing about your server environment should be in the git repository. For two simple reasons. One: if you have on git something related to an environment it could change the environment configuration on deploy just because on your developement server something is different and making a git pull a disaster. Two: you don't want your production database password written on a public github repository... and probably you don't want it also on a private one, if you don't know all the contributors to your project.
• Take a branch as the production branch and never touch it before deploy time. When you have to develope a feature branch from production and develope there. On deploy, merge your new branch with production and push. You have to trust in production branch. Every time you go on your server and do "git pull origin production" you have to know exactly what will happen. Nothing, in the most cases, the deploy of something you planned, on the happy release date.
  Again, if you don't trust all your contributors, there're many ways to prohibit people to push on production. Enforce them. Your contributors are not always malicious, but they're still users. They do users' things. Users things are often stupid things. Stop them.

This project about Dancer2 deployment will be quite long. I'm putting in it a lot of random things from my experience. So I stop it now and I put a number on the title. We'll be back, i hope soon.