How to become a Strehler developer

Strehler is a strange package to develope. It's easy to install from repository and use, but while you are just writing it you have to arrange many tricks to be confortable with the work. For example, Strehler needs a Dancer2 site to run, something the respository doesn't contain.

In this little tutorial I'll try to explain how to setup an environment to become a Strehler developer, because Strehler needs you, as any of the packages published on CPAN. It's a very basic tutorial, useful for beginners too.

I hope it will help.

The repository

You probably already have a sandbox created in one of the many ways available for perl development. I'm still a fan of the good'ol local::lib but it's good to use also, for example, perlbrew. Your sandbox has to be active during Strehler development and you have to put in it Dancer2

cpanm --notest Dancer2

But NOT Strehler. After installing Dancer2 most of the library useful to work with Strehler will be in place, but probably you'll need something more, libraries specific to Strehler work. Just install them in your sandbox the same way using cpanm when running the code point out their absence..

You'll surely need:

• Dancer2::Plugin::DBIC
• Dancer2::Plugin::Ajax
• HTML::FormFu
• Authen::Passphrase::BlowfishCrypt
• IO::Prompt
• DateTime::Format::SQLite

All this packages are in the requirements of Strehler package and would come automatically installing Strehler for CPAN. Installing them manually is the cleanest way to obtain them, but you can try some trick, like installing Strehler and then deleting it from repository, keeping all its dependencies.

The code

Just pull down all the source from the git repository in a directory you like

git pull git@github.com:cym0n/strehler.git 

Downloading the repository you'll have a directory named $GITURL/src/lib and one named $GITURL/src/script. These two directories will come useful soon.

The environment

Now that you have Strehler code it's easy to modify it and manage changes using git. Less obvious how to use this code and see it working. We just need to customize a little the environment. Open a file named strehler-devel and put in it

alias perl="perl -I$GITURL/src/lib"

alias strehler="perl -I$GITURL/src/lib $GITURL/src/script/strehler"

When the file is ok source it.

source strehler-devel

Testing site

All Strehler power is now available, coming directly from the git repository on your machine. Now you can easly create a testing site with Strehler onboard using strehler script itself.

strehler demo 

Just run it as

perl bin/app.pl 

to see Strehler and (eventually) your modifications in action.

Manipulating databases

Database management is something very difficult to do, because databases are strange things created by strange people and database are also many, each of them doing things the way they think better. There'ra no two databases that thinks the same way.

When I wrote strehler initdb command I was quite impressed by the deploy function of DBIx::Class::Schema that worked like magic , creating databases as I want, But creating a database from zero is an easy task, changing an existing one, possibly without DROP TABLE commands that trash data is more interesting.

There're packages that do exactly this, like DBIx::Class::Schema::Versioned, but they have a static management. You build your database, you give a version to your schema. You change your database, you give another version and something calculate the diff. It's good, but I was looking for something more dynamic.

Why doesn't a dynamic way to change database exist? Because, obviously, changing a database dynamically is a bad idea in many cases, you should have control on it, but I wasn't thinking about a CMS that change databases through a web interface, I just wanted to do it as a script, to embed it in my strehler script.
And I want to cver just the easy case where I add brand new tables, no modifications to the existing ones.

My first option was starting from schema. I took my existing DBIx::Class::Schema, I added my tables to it with register method and then I tried a new deploy, turning off the DROP TABLE option. Result was good but... DBIC gave me back a bunch of error about existing tables that it cannot create again, so the output of the operation was really dirty.

Problem is that noone considered the concept of deploying just a table and, for me, deploying a schema containing only new tables wasn't enough, because this way all the foreign keys to old tables wouldn't be created.

Solution I found is dirty but... it works and, for now, I didn't find a way to break it. Starting point is the same of the precedent case: take DBIx::Class::Schema and add new tables to it registering DBIx::Classes. Then there's some dirty work you have to do by yourself.

A step back: deploy function in DBIx::Class::Schema starts from your schema, create queries from it considering the target database and then execute them. The tool that extract queries from yout classes is SQL::Translator and you can use it also without DBIx::Schema::Class deploy function.

You can run SQL::Translator on your original schema, obtaining a list of queries to create it. Then you can run it on you database modified using register function. Considering that you're just adding tables you'll have a new array with same queries from before and a couple of new ones.

Deploying new tables only is about calculating the diff between these two arrays and run just it on your DBIC connector... with little tricks to avoid running queries in wrong order.

If you want to see an implementation of this little hack you can see it in Strehler::Element::Role::Maintainer.

EX power

So this weekend Strehler received a major release and two minor bugfix releases and it's now at 1.4.2.

When I completed Strehler backend I was so excited about how easy was to install it and start writing contents for a site. Then, I started working on a real frontend and I found doing that a very boring work, with a lot of copy-and-paste and clumsy writing.

Creating Strehler I was obsessed by the opportunity to make it highly configurable because in my life I always stop loving packages like it when I found what they decided by themself, without asking my opinion. Most of Strehler has a parameter that control it, side effect of this is that you have to write a lot to make the whole thing having the right place.

High level refactoring of this exists. What about taking all the boring code I wrote for fulfill simple tasks and "hide" it behind Dancer2 keyword? The EX plugin was born.

The EX plugin is a collection of keywords for standard cases. In particular they can be used to serve a content called using its slug, to display a list of contents of a category or just to manage a page with contents that you write once and then you throw away (like homepage boxes).

EX plugin keywords can still be a little complex to understand, because they need obviously nested hashes to work well, but at least you have just to understand the meaning of their parameters and not perl code to use them and documentation can help you doing this.

It's also a point of my todo page so i'm really proud of it!

I'm less proud about the two minor releases. The first was done because I wantend more options and configurable things in EX plugin plus a smarter way to write code. The second was done because I did the first too fast :-)

A matter of time

Strehler 1.3.0 is released. This means that all the problems with Dancer2 0.15x are solved. More than that, requirement about Dancer2 moved up from Dancer2 0.11 to Dancer 0.15002.

In the previous episodes we saw that Dancer2 App, from version 0.15, try to find their configuration file in a directory near the one they are installed, ignoring where is the script that uses them. This was very bad for Strehler becase Strehler apps like Strehler::Admin are installed using CPAN so they are in the local repository, a place where no configuration file can be stored.

Only available solution (for now) is overriding thi behaviour setting environment variable DANCER_CONFDIR to the path where our config.yml actually is. Can we do it inside a script so that we can use this solution in any context, like CPAN tests? The answer is yes, if you know the right things about perl.

First of all, any perl script can access environment variables through the %ENV hash,  a read/write variable always available. So, first step should be to write in your code:

$ENV{DANCER_CONFDIR} = /the/directory/I/want

Now you can think that, including my app, all will work, so you're thinking about something like this:

$ENV{DANCER_CONFDIR} = /the/directory/I/want
use Strehler::Admin

Don't run too fast, this code WILL NOT work.

There's a lot of interesting concepts about perl interpreter behind this problem. Concepts, I hope, one day you'll be curious about. For now it's enough to say that "use" directive will run before DANCER_CONFDIR assignment even though you write it in the end of your file.

 Modules must be loaded before any other piece of code because consistency of any piece of code can be verified only with all the module loaded. Perl interpreter does it without asking nothing, ignoring that you necessary need to do something before.

Can you change this behaviour? Yes, two ways to do it:

• BEGIN directive will run all the code inside its block before any other code, including use.

BEGIN

{

   $ENV{DANCER_CONFDIR} = /the/directory/I/want

}

use Strehler::Admin;

• require directive instead of use load the module only when someone ask for it,with no precedence on anything.

$ENV{DANCER_CONFDIR} = /the/directory/I/want

require Strehler::Admin;

You can see an example of this here.

Here are described many ways to configure DANCER_CONFDIR.

Why don't I want to consider this as the definitive solution for the problem? Because I think it's still a dirty way to do things because force you to use an environment variable and disable the original App behaviour that allow each of them to have a separated config file, something that could come useful in some situations.

Plack::Test and Dancer Session

PSGI is very easy to understand: a black box, something goes in, something goes out. What goes in: a request made of a path and headers. What goes out: a response with a content and more headers.

Plack::Test follows this structure. What you have to test is the black box, you inject the request in it and analyze the response you receive back.

Important thing to understand is that Plack::Test is something very different from an emulation of a browser like LWP::UserAgent. Most important difference is, obviously, in session management, because that's something that should survive between calls and in Plack::test every call is a test by itself, with no memory of what happens around it.

What do you have to do  if you want to test something like a login using Plack::Test? Well, the solution is very easy because the concept of session is AN ILLUSION throughout all the web. It's not just Plack::Test, the HTTP protocol itself is session-less. The "session" you see is just a set of tricks that clients and servers do under the hood.

As we said: a request go, a response returns, nothing more, so the session is something that must be hidden there. In particular, session is controlled throught headers, in particular headers that control browser cookies.

There're a lot of documentation about how cookies and response headers have to interact and probably studying it could be very interesting, but HTTP::Cookies module contains all the knowledge you need so learn to use it will be enough.

Let's do a login and get the response from it:

$r = $cb->( POST '/admin/login', [ user => 'admin', password => 'admin' ] );

Response object $r contain many headers, one of that ask the client to write in a cookie the identifier of dancer session. We can use HTTP::Cookies to extract this information.

my $jar = HTTP::Cookies->new;
$jar->extract_cookies($r);

Now we want that, in the next call, the server find us logged in.

When a browser does a request to a site, it passes in the request all the informations related to the cookies it has. This way it can say to the server what is its dances session identifier. Using that, the server can retrieve any information about the caller, like its status as logged user.

Let's cook a call for admin homepage.

my $req = HTTP::Request->new(GET => $site . '/admin');

WARNING: Remember to ALWAYS create request with complete URL, starting from http, otherwise HTTP::Cookies method add_cookie_header will not work!

Now we want that this call carry information about the cookies of the previous call, where logged in user session was configured. We just have to get information out from the cookie jar.

$jar->add_cookie_header($req);

Now this req will been seen by the server as the logged user from the previous call did it.

Well, if you configured in a clever way the server too.

Remember? There's no memory between requests client-side AS server-side. Client memory is in the cookies and HTTP::Cookies helps us to manage it. Where is server memory?

Standard session management in Dancer leave session information in RAM. On a development server this is enough, but a development server is something that stay alive between different calls so you can consider information stored in RAM persistent. Plack::Test instantiate the black box fresh new at every call so information about sessions is always trashed at the end of it.

Harddisk is more reliable if you want to store persisten information so switch session management to YAML when doing this kind of test.

Here is a little example of what we said.

The darkest hour

I've been away from this place for long time, I know, and I also know that also Strehler has been left with no updates for many months. There're technical questions about all this silence and I want to explain them here.

This summer Dancer2 reached release 0.150000, quite an improvement! A lot of things changed with this release of the framework, most of them related to the concept of App.

Major architectural changes are something I know I have to deal with, talking about Strehler's developement. Dancer2 code is not stable, it's usually considered alpha and I know that chasing its releases is one of the most difficult but important tasks for me as mantainer of a module based on it.

Problem is 0.150000 Dancer2 was too much. I described the issue introduced by this release here, Briefly, an App installed in your local repository gives a lot of trouble when it's time to read configuration files, because App uses its own path to find site resources, ignoring the path of the script that manage site startup.

For now, in my opinion, there's no way to solve the question fixing Strehler. Problem is inside Dancer2 architecture so I hope that the Dancer2 development team will do something. Waiting for it, this is what I can say if you want to use Strehler:

• Strehler latest release is perfectly working with any Dancer2 0.14x release. So, if you don't update Dancer2 packages too much you have nothig to fear.
• You can make Strehler works with Dancer2 0.15. You just have to ignore all the tests and run your site setting the environmental variable DANCER_CONFDIR as the directory where your config file is. It's quite simple and I know I could consider it the definitive fallback, but it's not clean and still I don't know how to use it to run the test suit, so I take this as a temporary workaround.

Well, I don't have much more to say, this is the situation at today. I hope it will change soon and I think it will, but patience, for a bit, has to be your new best friend.

Last announcement: Strehler has now a site. Click here for it. Important articles from this blog will be reported there, as a place to aggregate every communication about the development.

Strehler 1.2.1: the hive

Strehler made the jump from 1.1.x to 1.2.x. Obviously, the first attempt had a dirty bug related to old perl versions, so the package we cleebrate now is the 1.2.1.

Strehler 1.2.1 is about Strehler frontend. No real front-end will be ever developed in the Strehler project, Strehler will never become a Wordpress or a Joomla! (but a CMS like them could be built on top of it). Otherwise, 1.1.x system is too limited, because it gives you no way to access contents you created. Yes, you can write a site that do that, but there's nothing out-of-the-box.

Most trendy way to serve contents in our days is through APIs, preferably APIs that some clever JS library like AngularJS can manage. So I added to the projct a package, Strehler::API, that, using the same logic of the admin views, generate paths to return Strehler contents in JSON(p) format.

Here below, the rules I found on internet about APIs that I used developing Strehler's (just a reminder for myself):

• Return format is JSON. It's the most friendly for javascript and open systems. Other formats will probably follow in further development.
• Along with JSON you can ask data in JSONp format in any situation, just adding a callback parameter. This is important when working with cross-domain platforms and... it's so easy to do that not implementing it would be just cruel.
• JSON response content-type is 'application/json', JSONp response content-type is 'application/javascript' (don't laugh, there're many questions on Stackoverflow about this).
• API path contains a reference to API version, so is /api/v1/... I can't imagine, now, a reason API version can change but it's always better to expect the unexpected.
• Error codes are returned with a JSON content with the error message. In my opinion is right that error "pages" are JSON too, because whatever is calling your API, you can assume it's not a human being using a browser. Giving him back a kitten with a sorry face on 404 will be probably not easly understood by IT. (probably this feature should be improved a little, considering 1.2.1 version of the package)
• Singular and no ending slash => an element, Plural and ending slash => a list. "Ending slash" rule is not written anywhere, it's just how I like things. I think it works well.