About Elvis Ciotti

Founder and Software Engineer at Software Engineering Solutions (UK), currently working as a Senior PHP Developer for the UK Ministry of Justice.

SymfonyCon, Berlin 2016, notes and thoughts

Berlin

Berlin

I wrote an article for the Italian PHP conference last year, so I decided to repeat that and write up some notes I took at the Symfony Conference in Berlin 2106. Talks and notes at this link.

Conference format

I personally prefer conferences around a wide subject, in order to list to more variety of talks, spreading across the whole area of the technology. This one was a 2 days conference around a single framework, so I was expecting to get a bit bored, and I was right. Lots of talks around minor functionalities on the framework that I can easily read online, or things I already know. Some talks were fortunately more general therefore more of my interest. And that’s because I want a talk to inspire me, giving me clues, tips, but not repeating what the online documentation says.

Talks

Some talks around the Symfony 3.3 new features, SensioCloud (a kind of heroku for Symfony that smells a bit commercial and coupled with Symfony at first glance), PHP 7 improvements I missed (static variables persisted in memory among requests), PHP types (things I’ve already heard many times, but good to hear them again, with updates on latest PHP updates).

One of the talk I like the most was about when to abstract, where lots of useful concepts were mentioned. Concepts that I knew already, but I always found it difficult to explain to other younger developers or the business.

  • Predicting patterns should be done very carefully, we can’t really know how the business logic is evolving, and a premature abstraction leads to difficulty to change the product. The risk is facing an over-engineered or over-architected product at the time you have to make changes. In this talk is suggested to develop first with duplication, abstract later. Completely agreed;
  • Refactor  is not only and improvement, but the best way to let the business patterns and logic emerge from your code, in case you don’t fully know it. I knew that, but I never though of explaining to the business that way. I understand “refactor” sounds scary to the business, like the builders in your house say “we need a paid (by you) day to refactor the wall we are building” when you clearly saw the were half way through after 2 days.
  • A code rewrite, instead, means losing some of the domain rules. In the talk it was mentioned 40%, but I think it depends, it was much less in my experience, also considering that some functionalities are not needed anymore, so it’s good to lose unknown and useless functionalities, and re-implemented the updated version if the users and business require those;
  • API should be optimised for stability, projects (what you build with a framework for example) for change, products (e.g. Symfony /Wordpress) for stable core. Agreed, again.

Other talks I followed worth linking:

Other tools

I spoke to guys in the sponsor stands. Interesting to see blackfire.io in action, that I’ll definitely try out next time I need to optimise an app. Interesting to see heroku to basically deploy and handle the “devops” part of an application (create server instances, install packages, manage servers, watch logs) entirely from the command line without a single SSH command inside the box. I wasn’t particularly lured by the whole SensioCloud idea, as I never felt the need for something like that, also not sure I want to use a platform created by the framework creator, that I’m not sure I can (easily) use with other frameworks.

How to make changes on your own repositories imported with Composer

composerIn a previous article, I talked about reusing and exporting code. This article is more practical, and it’s about implementing that with PHP.

In the PHP world, composer is the de-factor tool to put together, solve dependencies and autoload external libraries (the tons you can find on github) or your own repos (that means everything non-application-specific that you can export and it’s not already decently implemented by anyone else).

Export your libraries into a public github repo 

Say you locate some code that is non-application-specific and you want to export it into another public repository. To do that, create a new Github repo, using this skeleton for example, and add there your libraries, tests and composer.json with autoload information. Commit and push.

Use your new library on your main project

On your main project, add the require for your repo, and specify source as a preferred install. See the example below.
composer repo

You can now run

composer update your-github-username/your-repo

And have your repository autoloaded into your main one.

Make changes to your repo

You can make changes your your imported repository with your IDE directly inside the vendor/your-github-username/your-repo directory.

If you want to commit a change, bash into the vendor directory, commit and push the change.

Before you commit changes (that use the modified version of your imported repo) on your main project, remember to run

composer update

so that the composer.lock file will now point at the new version.

Point to a specific commit hash

If you want to point your main repo at a specific commit of your imported repo, get the commit hash (git log), then in your composer file use

dev-master#adbd92dcf17b4ca5ecf5de61964a01daf9f92ebd

instead of

dev-master

If you want to make changes to an existing github repository that is not yours

If you need to use your fork instead an official repo, please refer to another article on this blog about using your own forks with composer. Making changes will be the same as described above. If you need to pull changes from the original repository, just add the remote origin (git remote add upstream <originalRepoUrl)). Refer to git remote operations for that.

 

PHPDay, the Italian PHP conference 2016, notes

veronaThis year I decided to take a couple of days off work and attend the 2016’s Italian PHP conference.

I was curious to see what the Italians audience thought of talks from international speakers, where the business is normally different. In Italy, according to my freelance experience in 2002-2008 and confirmed by the conference’s attendees, there is a majority of very small businesses requiring small/medium CMSes, often serving tourism needs (e.g. booking platforms), with a small budget, sometimes requiring to maintain old platforms and obliging developers to share their time doing multiple roles: devops, backend and frontend, sometimes design too, SEO and marketing) and/or working for lots of clients with different (sometimes legacy) platforms.

What I observed is that the backend world is more or less the same as the previous years. Basics are still the same. I even re-heard some recommendations read in the old 1994’s OOP bible book .

But there were some interesting points and tools. I would group the talks this way:

PHP 7

php 7In summary much more performances by just upgrading PHP to the version 7, a very few backwards incompatibilities, a few language improvements. Interesting to see some stats from the PHP’s creator himself: WordPress and composer seem to be hugely faster now, and use much less memory (I don’t remember the numbers for each but at least 2x). Tips about smem (a tool to better measure memory consumption excluding shared memory), settings tuning (realpath_cache_sizecommand_buffer_too_small, DocumentRoot in tmpfs), considerations about multiprocessors and NUMA.

I attended a talk from badoo – the widest dating/social network site, so quite a lot of servers – that switched to PHP 7 and implemented the needed upgrades to all the used extensions. A viable solution for a big company, whereas a small one would probably not afford that and would be obliged to wait for the stable repositories and extensions upgrades before switching. Pinba (MySQL storage engine to accumulate PHP profiling info sent over UDP, similarly to a local NewRelic setup) was used for some measuring. Runkit was used to modify constants, user-defined functions and classes in 60k tests, and – since it was not supported in PHP 7 – they ended up developing their own mock framework  and distributing it for free on GitHub (well, thanks !).

Docker

A bit too “DevOps” for a PHP conference, but since it replicates the platform architecture locally, and simplifies the deployment, I guess it’s becoming a MUST. At MOJ, fortunately, we already use it, thanks to your dedicated DevOps team. Nothing new to learn for me, apart from the pipeline jenkins plugin suggested in this talk, that I might play with when I have time, instead of simply using job triggering.

Event sourcing

That basically means storing DB changes and be able to query on those changes. I’ve already implemented something similar in the past using doctrine listeners, and IMO a great approach when the data to save is connected to entity operations. I didn’t like how the argument was covered but good to hear and become curious to learn more during the talk, ending up reading the Martin Fowler’s article about event sourcing, play with Prooph framework for it, along with doctrine and mongo adapters. The Command Query Responsibility Segregation (in short, different models for updating and displaying) pattern was also mentioned, but IMO not necessarily connected to event sourcing as I heard.

Doing something already existing, but in PHP

Interesting talk about the fann extension, for Artificial neural networks, and an application example of machine learning in this talk, where the “intelligence” was recognising PHP code from human language in code comments, by initially defining what a code normally contains (“$”, “->” and “;” symbols), then launching it on many inputs (=code comments), and using an iterative approach to improve the results.

Another talk about Raspberry PI and  PHP libs (alternative here) to pilot it. Not something developers normally do for their clients, but good to hear something refreshing and different. Raspberry PI’s OS is a Debian distribution, so a web server with PHP could be installed on it, and pilot a huge variety of sensors. Good to know. I might use it to recognise pigeons on my balcony and pilot a plastic bullet BB Gun to shoot at them !

 

Others

  • Packing and distribution: Lots of useful tips from this talk, thanks to which I found a useful skeleton for new projects, refreshed the semantin version concepts, a tool to select a licence, conventions and other stuff;
  • Middleware, ways of glueing software. I’ll create a specific post for this, ZF3 and other frameworks like slim have a support for the idea;
  • PPI framework: to load and bootstrap multiple frameworks. I normally include libs with composer, so never had the need to use this framework, but I might play with it in the future;
  • API recommendations: in this talk, some recommendations, most of which I have already heard, but good to brush up. Among the most useful ones: revoking permissions will break/alter the client’s behaviour, so better to start with restricted permissions, and open them little by little (increasing permission will not break the consumers’ code). Good suggestion about not being 100% “perfect”, better to have a “RESTlike” working API than a 100%-compliant over-engineered and difficult to understand one. Loved the definition of pragmatism as cutting corners safely and be realistic. Interesting the Maslow pyramid from usability (base) to creativity (top);
  • PHPSpec for TDD. I skipped those talks. I already heard of it more than once in the past. I already do BDD with Behat, and TDD with PHPUnit that always proved to be a great combination of tools to guarantee application stability and maintainable code thanks to safe refactors. I haven’t found PHPSpec useful so far. IDEs also help a lot with code generation, so I don’t need more TDD/OOP tools. I personally prefer to spend my time on other aspects of software development (both frontend and DevOps) and business.

What I didn’t hear

No talks about unit testing, maybe because there is no much more to say ?

No talk about functional testing, one of the most underrated thing to discuss IMHO. The sSoftware has to respond to business’ needs, be reliable and bug-free. We should never forget and stop improving on this side. I hear developers talking and focusing too much on speed and performances, without even knowing the optimise-measure iterative process. Also I hear tools and framework adoptions based on personal preferences or by just trusting what’s new and sold “better” without objectively comparing the alternatives.

Summary

  • London’s environment, developers, community (and – as a consequence – clients) are always at the top in terms of framework and tools choices, so apparently no new technologies/approaches to learn from Italy;
  • No big news in terms of new frameworks and way of developing. One the reason I don’t spend too much time learning new frontend frameworks. The JS community seems to jump from a framework to another too often, a sign that things need more time to become mature before being worth spending lots of time on them. The only JS stable adoption seems to be JQuery, that pragmatically solves most of the problems elegantly (when JS is only an enrichment layer on the top of the application and not uniquely used as a front-end renderer).
  • Code distribution on GitHub and composer.json is definitely an emerging habit among developers, always good to share and stop re-inventing the wheel. Very few people in other professions think so broadly;
  • PHP 7 is a huge improvement from the past by 2x or more, for free, without coding (unless fixing a very few backwards compatibilities). That means fewer costs to host PHP apps, happier clients, happier users, happier developers. Never heard a so big improvement for other open source technologies. Not sure JAVA or .NET or even Python or Ruby communities we’ll hear one day that the new compiler/interpreter version is 2-5 times faster. Probably because they were already optimised from the start, you might say, at which I would add: if PHP made a long way without being optimised, it must have been able to listen to devs and business needs more than all the others;
  • PHP is somehow a language proving the  “Premature optimisation is the root of all evil” and lean startup rules. It started simple as a very simple scripting language, so developers started coding solutions quickly, and businesses liked it for its low costs and quick response to the market needs. Frameworks and tools were built and with time, both languages and framework grew and improved, more people and business moved to it gradually, and further improvements were added. Now the stack of tools available for PHP developers has nothing to envy from Java and .NET. Also, I noticed businesses preferring open source to closed platforms. The former have proved to be less risky, for example by avoiding the vendor lock-in problem. If I had the opportunity to work with PHP for a gov.uk service, it’s also thanks to this winning approach.

Stand on the shoulders of giants, and export your code

stand shoulder giantAmong the many software principles to adopt, like KISS eYAGNI, another important one is DRY (Don’t repeat yourself), IMO connected to the Standing on the shoulder of giants principle. In other words: Great people already faced your problems, solved them elegantly and also published their solution. So why not using highly skilled people’s solutions instead of developing yours ?
Unless you do things for learning, it’s more efficient to re-use things and avoid reinventing the wheel. Is, therefore, important to learn how to contribute to export and distribute some code to the public. Below, a list of some libraries I’ve exported from some projects of mine because I thought I’d use them in the future, or somebody else would need them.

Currency converter

A few years ago I developed a component to convert GBP into EUR for a client who needed an auction website and price filtering in different currencies. No PHP library was available at the time (not sure there still is one), so I’ve developed this unit-tested parser able to read values from the European central bank feed. I’ve used it a couple of time since then, so it was a good idea exporting it. If I need to convert currencies, it’ll take minutes to add this to the project. Note: the library doesn’t download and cache the feed on purpose. I prefer to have repositories doing one thing only. Downloading and caching is a responsibility of the caller, and it could be doctrine caching and a simple PHP file_get_content or guzzle call, depending on what you already use in your project.

Mockery stub chain

Mockery (PHP stubs framework for testing) allows to mock chains of called methods, but no arguments can be specified. If your arguments are strings or integers, you can use this simple extension and write the same assumptions in a simpler way.

Twig text extension

I couldn’t find a simple twig extension to limit some text by the number of words (without truncating them), and – in case they are truncated – add a suffix like “continue …”. Also, there was nothing to remove accents (useful to Slugify), so I created this unit-tested one, where I’ll add more general purpose Twig filters in the future.

Array collection

A simple extension of doctrine array collections that already work great, but a.t.m. misses functionalities like sorting, create a new one from a merge of multiple arrays/collections, get not empty values.
Very simple and worth exporting. It’s unit tested, so I next time I need to work with a ArrayCollection, I’ll add this repo to the composer file and save time. Besides, any improvements will get saved into the repository itself, and not into a private project that might be abandoned, and therefore not lost.

WooCommerce layered nav sort

Simple WordPress Woocommerce plugin to order woocommerce layered navs filters by order descending. Took minutes to implement.

Better google tasks

Google Tasks web client, displaying all the lists on one page, functionalities to drag and drop tasks from one list to another. Written with AngularJs and Silex. I’ve abandoned this project due to some priorities, but some parts of it could be useful for similar projects. Sorry, no demo available

Text spinner

There are free language dictionaries on the web, but no open source library to parse them.
I’ve created one, that can be used for auto text spinning or suggesting synonyms.

How to create DEB packages for ubuntu/debian

linuxTo create a DEB package, you just need to create a directory with the files, respecting the same structure starting from “/”, plus a couple of files with the configuration.

The command dpkg –build will then create the DEB files that you can install with dpkg -i mysqldb2html.deb.

For detailed, instructions, read this article on ubuntuforums.org.

For an example of application of mine, just packaging a single script, click here

ZF2 subforms usage and validation

zf2Differently from Zend Framework 1, I couldn’t find any good documentation about the usage of subforms in Zend Framework 2.
After a bit of investigation, I understood how to use them.
When the subforms extend the `Fieldset` ZF2 class, they can be added to the form with a name, and that name (e.g. subform1 and subform2) will be used with getData() and setData() as a key of the subform data array.
The validation can stay in each related subform, the isValid() ZF2 Form will recursively validate all the subforms.
Example:
A `MainForm` contains two subforms (named `subform1` and `subform2`) of the same class `SubForm`.
Once filled in, getData() will return the following. Equally, the same array structure will fill in the related fields.
    $data = array(
      ‘formElement’ => ‘…’,
      ‘subform1’ => array(
         ‘first’ => ‘…’,
         ‘second’ => ‘…’,
      ),
      ‘subform2’ => array(
         ‘first’ => ‘…’,
         ‘second’ => ‘…’,
      )
    );
Here the code:

How to create mobile apps with PhoneGap

Making a “hello word” application for my android mobile phone was in my Todo-list for ages, and last night (the flu blocked me home) I spent a couple of courses challenging myself to develop a mobile application starting from scratch (including downloading and setting up the needed tools).

I decided to use PhoneGap, at which I had a look last spring and it soon looked great: using CSS and JavaScript to develop cross-platform (Android, iPhone, windows mobile).

Tools

I timeboxed my hack-night and I kept things super simple, opting for the PhoneGap desktop app to run the server (the mobile phone connects to it, downloads and execute the code in real-time, in beta version and sometimes not refreshing as expected) and for the  online compiling tools (free up to one application, a bit slow but they do the job).

Issues

The only problem I had was the layout. The PhoneGap CSS layout didn’t suit my needs and I started with a new one, ending up spending most of the time with font size and media queries to render the application properly on different devices (the app executed via the desktop app rendered differently from the compiled APK one on my Nexus 4).

Coding !

This was my “hello world” application so I kept it simple, without any complicated phone API interaction, just a calculator for runners (insert distance and time, calculate velocity).

running tools android appApart from understanding where to load the events (I had two different ways to do that, and none of them worked as expected, I think due to a non-back-compatible old event system), the JavaScript code worked like a charm on my local browser as on the mobile phone.

Download & repo

If you want to give at try, download the APK. Source code on Github. It asks for lots of permissions, but don’t worry, I just haven’t had the time to take out the one I didn’t need from the default settings.

The app compiled in a 480Kbsize APK file, that is good considering the app wasn’t written in the native Java language, and I haven’t even spent a second trying to strip out the unnecessary stuff (nobody cares about size, do you ? ;-))

 

Nice, uh? take a coffee and impement your own !

 

 

 

 

Quick intro to Postgres for developers

postgresIn this article, a quick guide to the most commonly used Postgres commands

INSTALL

sudo apt-get install postgresql postgresql-contrib

ADMIN OPERATIONS

Create superuser from the commad line

sudo -u postgres createuser –superuser digideps

Create database from command line

sudo -u postgres createdb newDatabaseName

Login via command line (postgres is the default root user)

sudo -u postgres psql [dbname]

DEVELOPER OPERATIONS / QUERIES

Describe table1

d+ table1;

Change password of the postgres user

password postgres

Grant permission on db1 to user1

grant all privileges on database db1 to user1

List databases

l

List users

du

List tables in a db

dt

IMPORT/EXPORT

Export db snapshot

sudo -u postgres pg_dump dbname —clean > dump.sql

Import db from snapshot

sudo -u postgres psql dbname < dump.sql

The simplest Observer design pattern example

A few days ago I got involved in a chat about the usage of Object-oriented design patterns. I brought up the problematic of their usage when not needed, that ends up in adapting the code to the pattern and altering it, risking to increase its complexity. I mentioned an example of usage of the Observer design pattern, that I’ve seen used even where one object A (Subject) just needed to keep an array of reference of objects B (Observers), but B didn’t actually need to refer back to A.

From Wikipedia: The observer pattern is a software design pattern in which an object, called the subject, maintains a list of its dependents, called observers, and notifies them automatically of any state changes, usually by calling one of their methods.

In this post, I’ll expose a case where the usage of that pattern is very simple to understand: a Newsletter system, with senders (Subject) and subscribers (Observers).

In this example, we have two users, John and Mike who both subscribe to the NewsletterSender (Subject). When the email is sent, both are notified by NewsletterSender::notifySubscribers() that iterates the users and call their update() passing its reference. Via this reference, the users will then be able to call the getter to read the email. So, John and Mike will both receive the reference to the NewsletterSender, John will print out the email text, but Mike only decides to say that there is an email (different implementation).

See the comment near each method with the classes and method names of the corresponding PHP interfaces SplObserver and SplSubject

script to modify DB values filtering by column and rows

One of my clients recently needed to implement an application-level encryption for some existing applications.

Unfortunately, the design pattern used for the database layer obliged me to write an independent script to browse the database and encrypt the data for each user using a user-specific two-way encryption key.

So, I ended up writing a simple utility class that applies a user-defined function (modifies) to the defined columns and records (via a filter query) of each database table.

It can be used for any relational database server supported by PDO (MySQL, SQL Server, MySQL, Oracle). I’ve only used it with MySQL but it should work smoothly with all the others.

Repository:
Db modifier https://github.com/elvisciotti/DbModifier

An example of usage is in the README, hope it’s clear enough.

You can basically use this to mass editing Db records with a function that is NOT easily implementable with a simple MySQL Query, e.g. encrypt the data with a custom PHP function.