How to become a Strehler developer

Strehler is a strange package to develope. It's easy to install from repository and use, but while you are just writing it you have to arrange many tricks to be confortable with the work. For example, Strehler needs a Dancer2 site to run, something the respository doesn't contain.

In this little tutorial I'll try to explain how to setup an environment to become a Strehler developer, because Strehler needs you, as any of the packages published on CPAN. It's a very basic tutorial, useful for beginners too.

I hope it will help.

The repository

You probably already have a sandbox created in one of the many ways available for perl development. I'm still a fan of the good'ol local::lib but it's good to use also, for example, perlbrew. Your sandbox has to be active during Strehler development and you have to put in it Dancer2

cpanm --notest Dancer2

But NOT Strehler. After installing Dancer2 most of the library useful to work with Strehler will be in place, but probably you'll need something more, libraries specific to Strehler work. Just install them in your sandbox the same way using cpanm when running the code point out their absence..

You'll surely need:

• Dancer2::Plugin::DBIC
• Dancer2::Plugin::Ajax
• HTML::FormFu
• Authen::Passphrase::BlowfishCrypt
• IO::Prompt
• DateTime::Format::SQLite

All this packages are in the requirements of Strehler package and would come automatically installing Strehler for CPAN. Installing them manually is the cleanest way to obtain them, but you can try some trick, like installing Strehler and then deleting it from repository, keeping all its dependencies.

The code

Just pull down all the source from the git repository in a directory you like

git pull git@github.com:cym0n/strehler.git 

Downloading the repository you'll have a directory named $GITURL/src/lib and one named $GITURL/src/script. These two directories will come useful soon.

The environment

Now that you have Strehler code it's easy to modify it and manage changes using git. Less obvious how to use this code and see it working. We just need to customize a little the environment. Open a file named strehler-devel and put in it

alias perl="perl -I$GITURL/src/lib"

alias strehler="perl -I$GITURL/src/lib $GITURL/src/script/strehler"

When the file is ok source it.

source strehler-devel

Testing site

All Strehler power is now available, coming directly from the git repository on your machine. Now you can easly create a testing site with Strehler onboard using strehler script itself.

strehler demo 

Just run it as

perl bin/app.pl 

to see Strehler and (eventually) your modifications in action.

Manipulating databases

Database management is something very difficult to do, because databases are strange things created by strange people and database are also many, each of them doing things the way they think better. There'ra no two databases that thinks the same way.

When I wrote strehler initdb command I was quite impressed by the deploy function of DBIx::Class::Schema that worked like magic , creating databases as I want, But creating a database from zero is an easy task, changing an existing one, possibly without DROP TABLE commands that trash data is more interesting.

There're packages that do exactly this, like DBIx::Class::Schema::Versioned, but they have a static management. You build your database, you give a version to your schema. You change your database, you give another version and something calculate the diff. It's good, but I was looking for something more dynamic.

Why doesn't a dynamic way to change database exist? Because, obviously, changing a database dynamically is a bad idea in many cases, you should have control on it, but I wasn't thinking about a CMS that change databases through a web interface, I just wanted to do it as a script, to embed it in my strehler script.
And I want to cver just the easy case where I add brand new tables, no modifications to the existing ones.

My first option was starting from schema. I took my existing DBIx::Class::Schema, I added my tables to it with register method and then I tried a new deploy, turning off the DROP TABLE option. Result was good but... DBIC gave me back a bunch of error about existing tables that it cannot create again, so the output of the operation was really dirty.

Problem is that noone considered the concept of deploying just a table and, for me, deploying a schema containing only new tables wasn't enough, because this way all the foreign keys to old tables wouldn't be created.

Solution I found is dirty but... it works and, for now, I didn't find a way to break it. Starting point is the same of the precedent case: take DBIx::Class::Schema and add new tables to it registering DBIx::Classes. Then there's some dirty work you have to do by yourself.

A step back: deploy function in DBIx::Class::Schema starts from your schema, create queries from it considering the target database and then execute them. The tool that extract queries from yout classes is SQL::Translator and you can use it also without DBIx::Schema::Class deploy function.

You can run SQL::Translator on your original schema, obtaining a list of queries to create it. Then you can run it on you database modified using register function. Considering that you're just adding tables you'll have a new array with same queries from before and a couple of new ones.

Deploying new tables only is about calculating the diff between these two arrays and run just it on your DBIC connector... with little tricks to avoid running queries in wrong order.

If you want to see an implementation of this little hack you can see it in Strehler::Element::Role::Maintainer.

EX power

So this weekend Strehler received a major release and two minor bugfix releases and it's now at 1.4.2.

When I completed Strehler backend I was so excited about how easy was to install it and start writing contents for a site. Then, I started working on a real frontend and I found doing that a very boring work, with a lot of copy-and-paste and clumsy writing.

Creating Strehler I was obsessed by the opportunity to make it highly configurable because in my life I always stop loving packages like it when I found what they decided by themself, without asking my opinion. Most of Strehler has a parameter that control it, side effect of this is that you have to write a lot to make the whole thing having the right place.

High level refactoring of this exists. What about taking all the boring code I wrote for fulfill simple tasks and "hide" it behind Dancer2 keyword? The EX plugin was born.

The EX plugin is a collection of keywords for standard cases. In particular they can be used to serve a content called using its slug, to display a list of contents of a category or just to manage a page with contents that you write once and then you throw away (like homepage boxes).

EX plugin keywords can still be a little complex to understand, because they need obviously nested hashes to work well, but at least you have just to understand the meaning of their parameters and not perl code to use them and documentation can help you doing this.

It's also a point of my todo page so i'm really proud of it!

I'm less proud about the two minor releases. The first was done because I wantend more options and configurable things in EX plugin plus a smarter way to write code. The second was done because I did the first too fast :-)

The darkest hour

I've been away from this place for long time, I know, and I also know that also Strehler has been left with no updates for many months. There're technical questions about all this silence and I want to explain them here.

This summer Dancer2 reached release 0.150000, quite an improvement! A lot of things changed with this release of the framework, most of them related to the concept of App.

Major architectural changes are something I know I have to deal with, talking about Strehler's developement. Dancer2 code is not stable, it's usually considered alpha and I know that chasing its releases is one of the most difficult but important tasks for me as mantainer of a module based on it.

Problem is 0.150000 Dancer2 was too much. I described the issue introduced by this release here, Briefly, an App installed in your local repository gives a lot of trouble when it's time to read configuration files, because App uses its own path to find site resources, ignoring the path of the script that manage site startup.

For now, in my opinion, there's no way to solve the question fixing Strehler. Problem is inside Dancer2 architecture so I hope that the Dancer2 development team will do something. Waiting for it, this is what I can say if you want to use Strehler:

• Strehler latest release is perfectly working with any Dancer2 0.14x release. So, if you don't update Dancer2 packages too much you have nothig to fear.
• You can make Strehler works with Dancer2 0.15. You just have to ignore all the tests and run your site setting the environmental variable DANCER_CONFDIR as the directory where your config file is. It's quite simple and I know I could consider it the definitive fallback, but it's not clean and still I don't know how to use it to run the test suit, so I take this as a temporary workaround.

Well, I don't have much more to say, this is the situation at today. I hope it will change soon and I think it will, but patience, for a bit, has to be your new best friend.

Last announcement: Strehler has now a site. Click here for it. Important articles from this blog will be reported there, as a place to aggregate every communication about the development.

Strehler 1.2.1: the hive

Strehler made the jump from 1.1.x to 1.2.x. Obviously, the first attempt had a dirty bug related to old perl versions, so the package we cleebrate now is the 1.2.1.

Strehler 1.2.1 is about Strehler frontend. No real front-end will be ever developed in the Strehler project, Strehler will never become a Wordpress or a Joomla! (but a CMS like them could be built on top of it). Otherwise, 1.1.x system is too limited, because it gives you no way to access contents you created. Yes, you can write a site that do that, but there's nothing out-of-the-box.

Most trendy way to serve contents in our days is through APIs, preferably APIs that some clever JS library like AngularJS can manage. So I added to the projct a package, Strehler::API, that, using the same logic of the admin views, generate paths to return Strehler contents in JSON(p) format.

Here below, the rules I found on internet about APIs that I used developing Strehler's (just a reminder for myself):

• Return format is JSON. It's the most friendly for javascript and open systems. Other formats will probably follow in further development.
• Along with JSON you can ask data in JSONp format in any situation, just adding a callback parameter. This is important when working with cross-domain platforms and... it's so easy to do that not implementing it would be just cruel.
• JSON response content-type is 'application/json', JSONp response content-type is 'application/javascript' (don't laugh, there're many questions on Stackoverflow about this).
• API path contains a reference to API version, so is /api/v1/... I can't imagine, now, a reason API version can change but it's always better to expect the unexpected.
• Error codes are returned with a JSON content with the error message. In my opinion is right that error "pages" are JSON too, because whatever is calling your API, you can assume it's not a human being using a browser. Giving him back a kitten with a sorry face on 404 will be probably not easly understood by IT. (probably this feature should be improved a little, considering 1.2.1 version of the package)
• Singular and no ending slash => an element, Plural and ending slash => a list. "Ending slash" rule is not written anywhere, it's just how I like things. I think it works well.

1.1.10 is da boss + something about AJAX

It's impossible to be sure about something like this, but I think I can consider Strehler 1.1.10 version as the first actual mature version of the product.

When I released on CPAN 1.1.0 I didn't consider it mature because I knew that a lot of features were implemented in the wrong way and many tools were hard to be used by a developer. There were too much cut and past, too much hocus pocus, too complex documentation to read.

Now, in my opinion, a developer can implement his site in a very quick way, understanding what he's doing and with the right level of trust in what I did and in what I will do (for example: update my code).

So, this time i want to make a toast bigger than the ones about previous releases.

Closing the last little bugs about Strehler I found something very interesting about  Dancer2::Plugin::AJAX that I want to share with you.

Dancer2 AJAX plugin is a little piece of code that wrap few interesting features you like to have where you serve an AJAX request.
More than this: it's really easy to use, you should never understimate that.

The strange thing about the plugin is that answer content type default is XML... In my opinion this is a little weird because when I think about AJAX I usually think about Javascript interaction where JSON is easier to use.

You can just configure the plugin to give you back a different content type, but when we modify global configuration I don't completely understand, I become a little paranoid. So my decision for my routes was to force the right content-type directly in the function.

Strehler 1.1.9 released

Released but not perfect. There're little problems about documentation and I probably found a little error in strehler script, layout function. Well, as usual...

I'm not sure I can find a way to test Strehler script but I should, at least to check if it gives an error in standard situations. The script, in my opinion, is a very sensitive part of the package. Strehler script go to war on the open ground of user's file system, it can find strange things and fight insidious enemies, it deserves attention. For example, I'm not sure how it works under Windows...

Another point in my TO-DO list is find a way to see how CPAN page will be rendered, after package upload, just to avoid little glitches in documentation as I'm experimenting now.

Well, stop with the bad things, a Strehler release is always something to celebrate, 1.1.9 bring order in FormFu elements and configuration and we love it for that!

HTML::FormFu Beastmaster

I'm not a great fan of HTML::FormFu package.
Let me explain: HTML..FormFu does a lot of dirty job and give you an easy way to write down forms with a lot of business logic, but when you try to open is source code to do some customization it's quite a labyrinth with too much trolls and few indications about the right road.
In my opinion the module could be ten times more powerful giving an easy way to hack elements and modify them for new purposes.

Writing down Strehler interface I obviously had to manage a lot of forms. I wrote many of them following Strehler business logic to offer easy ways for javascript to work with them, this logic is something built on top on HTML::FormFu elements.

The category selector, probably the most complex widget I had to use, in forms configuration files is something like this:

    - type: Select
      id: category_selector
      name: category
      label: Category
      constraints:
        type: Required
        message: 'Category needed'
    - type: Select
      name: subcategory
      label: Sub-category
      id: subcat

Problem is that Strehler should give other developers the opportunity to implement their own entities and forms managing them. These forms could contain my category selector and I can't ask people to copy and paste all this code in their YML files. It's really clumsy and what if I change this structure? I should call them one by one and ask to change their files.

TIdy implementation wants that, when you need a category selector, you just add in your form configuration:

- type: "+Strehler::FormFu::Element::Category"

An ad hoc HTML::FormFu element.

There's not a real path to implement something like that, best way I found is to hack a HTML::FormFu::Element::Block nailing in it a fixed configuration with the elements of my category selector. The result is here.

You can do a lot of thngs exploiting the after BUILD of the HTML::FormFu::Element sub-classes, but you'll be always limited to the possibilities given by the class and its methods and, for example, you'll not have any chance to add custom configuration entries. So this pattern can be useful when you just want to make a package of some existing configurations, but nothing more.

Strehler::FormFu::Element::EntitySelect is something a bit more smart, as you can see here, but it's still an hack on Select I did after giving up on creating a FormFu::Element completely by myself.

Working on that, I drew a class diagram of HTML::FormFu (still 0.9 version), doing a bit of reverse engeneering on it. Here it is, as a gift from me.

Refactoring configuration

I'm going through an interesting refactoring that I'm doing using roles (my first time) and I'd like to summing up what I'm doing in this blog.

I think that an important feature of Strehler is the possibility to manage custom entities, custom "database tables" that can be manipulated through Strehler backend using all the features Strehler makes available.
New entities can be easly (I hope) added as said in this tutorial, but to give enough freedom to developer, as you can see, there's a lot of flags available in Strehler configuration to change the behaviour of the entity itself.

The actual problem is that not only new custom entities need values for those flags, but also standard entities: Article, Image, User and Log and they can't just use all the default values. Log, for example, is not creatable or editable, it's interface has to be read-only while Article, Image and User allow a complete CRUD flow.
Big question is: where configurations for them are?

Solution until Strehler 1.1.8 is horrible. I've and horrible method (see it here) that fetch informations from config.yml and assign hardcoded values for standard entities.

This solution has, in my opinion, two problems:

• It's in the wrong place. Six months from now I'll never think that Article configuration is in a curious Strehler::Helper packages package.
• Makes config.yml the only way to configure entities. Flags are A LOT and many configurations can make this file a real mess.

This is the reason I'm introducing a role consumed by Strehler::Element where every parameter is a method returing, with no input parameters, the value of the flag itself. Standard implementation of every method is just a little piece of code reading from config.yml, as in Strehler::Helper.

Why is this solution better? Because is... more object oriented and allows me (and Strehler developers) to keep informations about entities in the entity class, where they should be. 

So, for example, Article parameters doesn't need no more an external method in a strange place. They're just in Strehler::Element::Article, an override of the standard methods where fetching from config.yml is replaced by returning just the hardcoded value... hardcoded in the right place.

This way, a Strehler developer creating a custom entity can now choose between write down all the configurations in the config.yml or just leave in config.yml minimal configuration needed and add all the parameters in the custom class.

I think that both methods should be used: config.yml when a flag can change through time or in different environments, class methods when a parameter is... "endemic" for the designed entity.

I talked about methods and not attributes because in many cases I need to know class configuration, without an instanced object. Is the same logic of the metaclass_data method.

I hope to release this new configuration soon!

Messing a lot with files

So we have a 1.1.8 just to bugfix strehler statics. 
Problem using it to update (not create) files on a package update was that file were created all read-only, as they are saved in the CPAN package, and when new ones tried to overwrite them they obtain a permission denied.
Only solution I found is to delete the public/strehler directory from the target project and then copy again it with all the files in the last version. This is not an operation easy to do using perl libraries, I found them a little complicated to use. They don't actually map the shell commands (rm, cp, mv) but follow their logic, so using them for what I needed was a bit difficult.
For example there's not a straight way to copy a directory from a place to another as in:

cp -r $DIR $DESTINATION

If you just use  dircopy from File::Copy::Recursive as:

dircopy($DIR, $DESTINATION)

then all files in $DIR will be trasferred under $DESTINATION (files, NOT the directory).
To obtain what I said you have to... well, copy and paste the magic string:

$File::Copy::Recursive::CPRFComp = 1;

The magic and not-exactly-human-readable string.

I already released a new version because update problem wasn't the only problem of the statics command... Well, the whole command implementation was a mess, with wrong messages and other problems, so i wasn't confortable leaving it in CPAN for a long time. It's just a little number increase...

All the releases come with a bug

So I released Strehler 1.1.7. Now the filter is improved and you can select a category to obtain all the elements in it and in its children.

The real reason I did it, however, is that 1.1.6 has a terrible bug that break the strehler initdb command making impossible a complete Strehler installation as described in the documentation. Strehler installation process is something that MUST go well plain and fast or I judge the entire package useless, because I created it to make a CMS deployment easy.

But... now I found a little problem on strehler statics when it comes to use it to update your statics after an update of the package. Less vital than the solved bug on initdb, but still annoying, probably e new release will be done soon.
At the moment, if you update Strehler package from a version to another (for example from 1.1.6 to 1.1.7) to make all works delete the public/strehler directory in your project and then run strehler statics again. Al the resources will be regenerated in the right version.

Also the Dancer2 0.14000 release gave me some work. As described in the release notes, Runner-Server was untangled. I like the explanation about why this was done, my problem is that all my tests were written using Test::TCP relying on the Dancer2 Standalone Server for the server part. With Dancer2 0.140000 al this part went broken.
So I managed to change this line of code:

Dancer2->runner->server->port($port);

with this if clause

          if( $Dancer2::VERSION < 0.14 )
         {        Dancer2->runner->server->port($port);    }
         else
         {        Dancer2->runner->{'port'} = $port; }

just to keep retrocompatibility.

Strehler: state of the art

I know that i'm not a frequent writer so a lot of interesting things about me doesn't appear on this place because i'm lazy. So I decided to write just a post to summing up how Strehler is going on after the 1.0.0 release.

Well, a lot of things changed and got better since that day. The first, fundamental step, with 1.1.0 was landing on CPAN with a real test suite and some supporting script. Now you can install Strehler using just cpan -i Strehler and getting an up and running CMS with few commands. In many ways the 1.1.0 release of Strehler is the real first release, because I know that the long, boring and messy installation procedure of 1.0.0 is something NOONE would use, out of me, obviously, but I'm the father of Strehler and my judge is irrilevant.

Strehler 1.1.0 was my first package released on CPAN so I learned a lot releasing it. I learned about PAUSE, about distzilla and about how many thing to your code after you upload it on the perl repository. I did a lot of things wrong because mistakes are the way nature teach you things, so in the week after 1.1.0 release I released FIVE bugfix packeges bringing rapidly Strehler to 1.1.5 version. Well, it's a good thing, high numbers make your software truly professional.

With a quite stable version of my software on the public repository, I started work about little features not yet implemented, like function to order admin list and searchbox. All these things went in the 1.1.6 version and i'm quite proud of it. Yes, a lot of work could be done on the search box because it is implemented in the easiest and plainest way I found, with no indexing or optimizations. It's good because it's accessible just from the backend of your site, but it's probably not a good implementation if you want it also on the frontend.  Probably in the future I'll do something, pluggable if I can, about it.

I'll probably release soon a 1.1.7 version. I wanted to wait after some implementations, but I found a big bug in the strehler script that make impossible to create a strehler database on your custom site, making all the installation process useless. I fixed it and I want to share it with the world as soon as possible.

After that, a lot of adventures are waiting for us!

The spring of HTML::FormFu

HTML::FormFu arrived on his first major release (1.00) this christmas. It's quite an event considering that previews release was on 05 Oct 2012.

I'm very happy for that because now form rendering is better and cleaner. Problem is that this new rendering had some incompatibility with my Strehler forms (javascript and CSS) so, when I upgraded, thing went a little messed up.

The most interesting thing (the only worth to be written here) is about my little CategoryUnique validator.

Strehler backend manage a category tree that you can use to organize contents. For example:

foo
foo/bar
foo/ber
yetanothercategory
newcategory
newcategory/newson

When you create a new category you have to give its name and its parent. The CategoryUnique validator check if the category already exists under that parent.

In the example up there you can create bar under newcategory, you can't under foo, because there's already one category named that way.

So the validator can't just check about the value of the category field, but it has also  to consider the second parameter of the form, to check if the category has the same name under the same parent.

With HTML::FormFu 0.09010 it was easy:

my $self = shift;
my $parent = $self->form->param_value('partent');

Easy, yes, but apparently forbidden by documentation.

If the value is invalid or the form was not submitted, it returns undef.

In a validator a value cannot be already valid :-)

But it worked because there was a bug, a bug fixed in the December 2012.

We could exploit it just because no new release of HTML::FormFu come after that fix... until December 2013.

What can we do now, with the shiny fixed 1.00 version of the software? The only way I find to do the same thing is to use the undocumented HTML::FormFu::FakeQuery. The query method of the form, infact, doesn't return an hash as documentation still say, but an instance of this object.

Luckly, using it is easy enough:

my $self = shift;

my $query = $self->form->query();
my $parent = $query->param('parent');


without passing through the param method of the form, that HTLM::FormFu documentation itself doesn't trust.

Habemus Strehler!

Well, i worked a lot about it behind the scene, so i'm now a little excited to talk about Strehler, my most complex Perl work.

Link to github first: https://github.com/cym0n/strehler

As the README says, Strehler is a light-weight, nerdy, smart CMS in perl based on Perl Dancer2 framework. The story of his creation is very simple: I had to create a website (well, more than one) and after I decided to use Dancer2 I also decided to write the backend interface for it once for all, developing a tool easy to install on any type of webapp you need and with a lot of customizations available out of the box.

Strehler implements just the standard CRUD needed for a standard site. It can create articles, upload images, organize things  with tags and categories. I said more about it in the github wiki.

In my opinion the software reached enough stability to deserve version 1.0.0 tag. My hope now is for people to download it, try it and tell me where I'm wrong thinking this. At this point of development Strehler needs to be seen by fresh eyes, this is the reason i'm announcing that with so much emphasis.

You can try Strehler, you can use it, you can just find something you don't like and tell me. If you like collaborative work or if you think that Strehler can be what you need with some modifications, just work on it. Code is open and the git repository is there to be used. If you need some advice about how to improve the tool, try the TODO page. It's there for me as for everyone ho like to develope using Perl.

Strehler is ready! Cheers!