Automation frameworks

Automation frameworks

When considering a move to cloud computing, businesses should strongly consider adopting automation as a key development practice before taking the plunge.

What most businesses don't realise when moving away from their existing data centres is that moving to the cloud does not mean they can adopt a fire-and-forget attitude to deploying software. In fact, deploying software to the cloud can actually mean more work in the long run if they retain the old practices of deploying software to the data centre.

Automation can transform the software development process by significantly reducing the amount of effort required by the development and operations teams in deploying and supporting software in the cloud. Deploying software can be an arduous and error prone process and often requires extensive understanding of the underlying platform. Teams can spend weeks or months getting software deployed and production ready.

Wouldn't it be nice if businesses could automate away all this development effort ? Wouldn't it be even better if they could pick and choose bits of infrastructure such as databases, caches, APIs and microservices that fit their needs and deploy this into a modular framework that took care of deploying all this into the cloud without spending weeks of effort ?

Enter the automation framework.

https://docs.google.com/presentation/d/17Im0mSbi6nLw6jaoIZ2xWjONRl8C2LpxUCQI7TifkoA

Why bother with Cloud Computing ?

Cloud vs Data centre

So what is the difference really between running your IT infrastructure in your data centre and running it on a cloud platform such as AWS ? After all "The Cloud is just someone else's computer"!

Most data centres now are using virtualisation, which has been around for a long time in computing terms. The first release of the Xen hypervisor was back in 2003, which along with VMWare brought virtualisation to the masses.

By now most people understand the benefits that virtualisation beings to IT infrastructure - cost, efficiency, faster provisioning, application isolation, Disaster recovery. The list is huge.

So what does the cloud actually offer over and above a data centre employing virtualisation ?

A quick summary is in order here:

 - Huge cost savings due to economies of scale and shared tenancy hardware.
 - Even bigger savings when reserving hardware.
 - Business can shift responsibility to cloud vendor for managing IT infrastrcture, over and above a vanilla data centre.
- Best practice security at every level of your infrastructure.
 - Almost instantaneous provisioning means new applications can get to market as fast as your developers can build them.
- Advanced services can provide easy to use Big Data, Machine Learning, Caching, CDN, SQL and Non SQL Databases, Data Warehousing and many more services. Eliminating the need to manage these services in-house.
- Highly available, Highly scalable solutions with minimal effort.
- Multi region DR solutions with a bit of effort.
- Ability to up-scale at the push of a button when expecting high demand (and even when not).
- Powerful APIs to allow complete automation and Infrastructure-As-Code so you can write configuration to literally codify your entire Cloud infrastructure and spin this up anywhere in the world in a few minutes!
- Powerful tools to allow migration of applications, databases, filesystems and document stores from your data centre to the Cloud.
- Fast dedicated lines to your data centre to allow seamless connectivity and hybrid solutions.


And this is just the start - new services are being introduced all the time. For instance AWS and Google are now offering Serverless computing which allows applications to be written with literally no servers to deal with. You upload your code, pay for the number of seconds runtime and MB of memory you use and that is it!

So with all these benefits, why wouldn't you want your business running it's IT in the cloud ? Well there's a few reasons :

- Poor understanding of what Cloud computing actually is.
- Locked into lengthy contracts with existing vendors.
- Too hard to move monolithic applications and databases.
- Lack of expertise.

Let me cover off that last one a bit. Moving to the Cloud is not a silver bullet to fix all your IT woes. It is hard and requires special training and deep understanding at every level of the business to effectively leverage the benefits. If you don't believe me, take a look at some of the AWS whitepapers...

https://aws.amazon.com/whitepapers/

There are hundreds of them, covering every topic from Security, Architecture, DevOps and Cost efficiencies, and these just scratch the service of the platform. Is it huge, and growing all the time.

To really get the most (or anything at all) out of the Cloud you need to have experienced people to make it work. The good news is that there is a lot of very good training material available online, so anybody with the time and commitment can learn to use Cloud platforms effectively.

All the other issues I mentioned above can be resolved with the right people and the right training. Systems can be migrated over with the right knowhow and AWS in particular offers a wide array of Hybrid cloud solutions to help businesses leverage the Cloud and their existing data centres.

Cloud computing really is a revolution in the way we do computing, and extends far past simply where we put our applications and databases. Simply "lifting and shifting" your existing systems to the Cloud will offer many great benefits, but the real power is in architecting your solutions for the Cloud from the ground up.

AWS in particular enables building hugely powerful solutions that with the right design principles can really perform in the Internet age, which can withstand a barrage of spiky traffic, deal with DDOS and other attacks, resist entire data centre and network outages and can be automated to deploy all the way from the source code to production without the usual drawn out release process that many companies endure.

Javascript: language of the future (and the present)

The past

There was a time, not so long ago, when Javascript was derided as a poor man's programming language. It was slow, it was buggy, and it couldn't really do much other than update a few elements on a web page, or intercept a button click. Worse, a lack of cross browser compatibility meant that writing code in Javascript was an extremely time consuming prospect - while that didn't stop people using it, it certainly slowed widespread adoption - and developers opted for easier, but proprietary solutions to enable interactivity in web pages - Flash, Java and Silverlight all became popular for web pages... OK, well not Java.

In many cases in the Enterprise, web pages were shunned in favour of client side solutions such as Java's Swing and the vastly superior .Net Winforms and WPF (Windows Presentation Foundation). While these frameworks offered rich client side UI solutions, they entailed all the usual problems of managing client side software. Deployments become a challenge, and homogeneous client machines are mandated due to the challenge of making the system work on various Operating Systems. Normally this regresses everybody towards working on Windows.

Gradually we started moving more code to the Web again, and with the release of jQuery, and the death of IE6, developers were able to develop web-pages that could work on any browser, and on any Operating System. This vastly simplified the process of application development in the Enterprise, because now software could be deployed to a server and the UI served to the users via HTTP and rendered in a browser. Standards and Open Source code could be used, meaning less overhead for developing a new application and fewer proprietary solutions (at least for the UI).

The Present

Moving to present day, there has been an explosion in popularity of Javascript frameworks. There is more open source Javascript code than ever before, new frameworks that leverage the power of the language and huge strides in performance of browser Javascript engines (V8 and SpiderMonkey to name two) - and the release of HTML5 and CSS3 - to provide rich, interactive web-pages and code reuse on a scale never seen before in web development.

Javascript frameworks such as Angular.JS and React (there are others), combined with CSS frameworks such as Bootstrap and Foundation enable developers to very quickly deliver professional looking web sites, loaded with features and updated in real time, to replace aging client side applications and tired old table based pages with little or no interactivity and limited browser support.

These frameworks also have another excellent benefit, moving more processing to the browser. Today's fast clients; Desktops, tablets, mobiles now come with multiple cores, and Gigabytes of RAM as standard - moving code to these devices lightens the load on the server and enables excellent scalability. The rise of the SPA (Single Page Application) has enabled the server to stop rendering each page and serving it to the browser, and instead serve up static resources and let the browser take care of the whole UI.

I shouldn't overlook the impact of CSS frameworks too here, these have enabled the rise of responsive web design (RWD) - a page that once written will reorder and scale itself to enable it to display on any device, be it Desktop, tablet, mobile or even a TV.

Back to Javascript - what else can it do? How about run on the server? Yes - with Node.JS it is possible, not only that, but it is widely reported to be even more scalable than Java .^{[citation needed]} - how is that possible? Javascript has a single threaded runtime after all - ie no Multithreading. Well it is basically down to it's totally non-blocking asynchronous design - this means instead of swapping threads all the time to service different requests, it simply offloads code into collections and executes it out-of-order, serving each request as soon as it is ready to be executed. This vastly lowers the overhead of managing threads, all worker processes; fetching http requests, waiting on file handles - this is shunted into the fast C++ code running the Javascript engine. The non blocking nature of Javascript means that execution is never paused while waiting for something to happen, IO / Network request, you name it.

The upshot now is that we have two very powerful Javascript environments, one on the client (browser) and one on the server. Both can be implemented with the same standardised syntax, code can be reused between both, Engine improvements benefit both, we can use the same build tools on both (which are also written in Javascript). See where i'm going with this ?

The future

Javascript has the opportunity to become the first ubiquitous development language that can literally be written for any environment. Furthermore, the client / server expertise divide is gone - developers can be proficient in both without having to learn a whole new language. Yes there are barriers, different frameworks to learn - but these barriers are significantly lower than they used to be.

Another great advantage to Javascript is that because it is interpreted, not compiled, we can transpile (Babel) the syntax to our hearts content. Want to write in the newer ES6 spec ? Go for it, want to write your entire application in CoffeeScript ? Why not. OK, so we can do this too in other languages, but in Javascript it is easy!

So we have a programming language that can be written in (almost) any syntax, executed on either the client or server, run on any device. Where the actual language spec is standardised, and as such decoupled from the engine that manages it's runtime. Where security and performance improvements are rolled out globally, and competing organisations battle to build the fastest, most secure engine. Then we have the nature of the language being functional and not OO (believe me, it's a good thing), and that it can be either dynamically or statically typed (TypeScript).

Is there any other language out there that has anything like the potential of Javascript ? Not that I know of.

So what's wrong with it ?

Javascript still has it's detractors, they will tell you that it is hard to program, full of nasty quirks and nonsensical syntax - to be fair - they are right, and wrong. Dropping in to Javascript without learning it's idiosyncrasies is bound to result in pain, there is an awful lot of cruft in the language, with more being added all the time. The trick is to isolate what works, and disregard the rest. I heartily recommend reading Douglas Crockford's Javascript: The Good Parts, for an idea of how the language works and how to get the most from it. Also, use a linter on your projects to stop the bad code.


Next time - I will talk about some of the killer features of Javascript and how to use them properly. I will also talk about one of the major problems facing Javascript today - classical OO developers* - and why they present a problem to the language, and how by trying to appease them, the language actually gets more confusing rather than clearer.

* before the outrage starts, I should point out that I am an classical OO developer. I have used Java for 8 years prior to my foray into Javascript.

My philosophy on code development in the enterprise using Java and Javascript

About me

I am an agile software developer specialising in enterprise software development. I have over 9 years experience in developing software solutions - long enough to be opinionated on the subject.

My philosophy

My aim is to build enterprise software in a way that simplifies and encourages code reuse and where core domain services remain unencumbered with client side, infrastructure or proprietary code. This is important because a vast majority of enterprise code will have a long shelf-life which is typical of software in large organisations, this domain centric code will have been developed at high cost to the business and should be shielded where possible from the technical and infrastructural changes in the environment, yet at the same time should not hinder upgrades to the infrastructure and software platforms due to high costs in changing it – achieving re-usability is vital to developing software for enterprises.

Considering the entire SDLC, I believe that Agile software development offers the most reliable return on investment for large organisations – the reason for this is the iterative nature of development can get software into production much faster than traditional methodologies, this software typically has a sub-set of the total project specification, but the organisation can start benefiting from it sooner – and the project team can get feedback sooner which leads to a healthy software delivery cycle.

For these reasons amongst others, I am a strong advocate of agile software development; I apply a blend of core practices such as SCRUM, Feature Driven Development, Continuous Integration, and Test Driven Development (TDD) and I tend to adopt a flexible approach to implementing agile practices in the workplace. I prefer to develop my code using TDD and have found it to produce exceptional results and high quality code, not to mention a huge time-saving lifeline should any subtle defects arise – it also enables a very powerful methodology for breaking down complex requirements into simple repeatable steps which I find indispensable.

I adopt a delivery focused approach to development, which essentially means writing 'good enough software' – favouring simplicity and re-usability over clever, complex designs. Where complexity is unavoidable it needs to be encapsulated, documented and easily testable – or else risk falling into the bucket of 'untouchable code'.

Traditional Java

For most of my career I have developed in Java, though I started originally with .Net and made the jump when the opportunity arose.

I have found Java to be reasonably effective in delivering requirements for the enterprise; it is fast, the code is easy to understand and there is a lot of active development going on and excellent frameworks being developed.

One of the best frameworks for Java is undoubtedly Spring. Spring has enabled a move away from big heavyweight JEE containers, and all the overhead that they entail, into smaller, more component oriented applications, but retaining all the best features of the JEE containers such as transactions, ORM support and messaging.

One of Java's biggest drawbacks - which uncoincidentally has long been recognised as one of it's biggest strengths - is backwards compatibility. In many ways Java has become a victim of it's own success; having been eagerly adopted by the enterprise over the last 15 years, so much legacy code now exists built on the Java platform - and particularly by organisations that traditionally have not been willing, or able to keep up with the accelerating pace of technology - that backwards compatibility has become the defacto goal for all new revisions of the Java platform.

The problem with this is that Java has been unable to embrace new features at a rate that most developers would like - and when those features have arrived they have often been encumbered by legacy APIs that cannot change for one reason or another, because to do so would slow down adoption of the new versions of Java by organisations using code written for legacy Java platforms.

To name two fundamental APIs in Java that have long been a cause of developer pain; dates and files. These have long been overdue a rewrite in Java as both are messy APIs with many subtle bugs and an unwieldy requirement to use Calendar and Stream classes respectively to get the job done. These APIs have been overhauled in Java 8, but it has taken a long time to arrive and it will be even longer until Java 8 is the standard Java platform in the enterprise.

Other areas where Java has suffered has been with it's unhealthy fixation with Xml for configuration of, well everything - the number of systems written with page after page of unreadable Xml configuration, Xml for Spring, Web server, logging, build management, it all serves to make any solution more complex than necessary and because it can't be compiled you discover your mistakes at runtime instead of compile time as with code. To rub salt into the wound you can't debug Xml either, so spending lots of time looking for errors in your config is standard practice.

Xml is meant to be an open standard machine readable language that humans can also understand - not the primary method for humans to configure machines. Luckily many frameworks (such as Spring) are moving away from Xml and towards Java based configuration. Indeed the Servlet 3 spec (JSR-315) allows for Xml-less configuration of a Servlet container - excellent !

Java generics is another feature that adds some essential flexibility to the Java language, but also an awful lot of complexity and more trip wires for the new or uninitiated developer - type erasure is the real issue with Java's generics, leading to lots of bugs and head scratching when trying to resolve types at runtime. Often this leads to lots of messy code and type casting where type information has been lost and the programming language forces the developer through these hoops.

Java has also been slow to adopt functional features, and non blocking APIs. It also has a difficult to use threading model based on shared memory and syncronisation - leading to more head scratching and pain for the unwary developer.

Another area in which Java has built up extra layers of complexity over time is it's build system - it's default compiler Javac is fine for compiling code into byte code, but it offers no support for any other build automation and as a result a number of build tools have arisen to take over this fundamental requirement. Once this would have been Ant, but now Maven has become the standard. Maven is an excellent build tool, but it is very complicated and requires an expert to use it effectively - anyone can run a build with it, but troubleshooting dependency related issues can be a burden on any developer. Again like much of Java, Maven comes with a ton of Xml as standard.

Having said all the above, there is still a lot to like about Java, and it's adoption in the enterprise shows little sign of drying up. It's portability allows organisations to move between operating systems with relative ease which is a great thing - long one of Java's best features - however there is another technology which takes portability to a new level, and that is JavaScript.

 

Moving application development to Javascript

JavaScript is currently undergoing a resurgence of popularity – Once viewed as a two-bit language for hackers, it is fast becoming the most popular programming language in the world. Huge strides in JavaScript engine performance, culminating in Google's V8 JavaScript engine (which powers Chrome), coupled with ever improving browser compatibility and standardisation has made JavaScript a real option for developing entire software solutions in a single language.

One of JavaScript's most useful features, and the reason that I see it has a bright future in the enterprise is it's portability. Similarly to Java, JavaScript can be run anywhere - however it has an ace up its sleeve compared to Java and that is its standardisation - it has taken a very long time for this to come about and browser vendors are principally at fault for this - but it's here now, and it's a game changer.

With JavaScript, you can now write code that can be deployed anywhere, run in the browser, run on any PC, any mobile device, any TV with a web browser, and any server. It really is "write once, run anywhere"; and because the ECMAScript (The formal, and terrible name for JavaScript) specification is an open standard, it means we don't have one huge tech giant running the whole show, as we do for Java (Oracle) or .Net (Microsoft).

Often JavaScript gets a bad rap due to the painful hours spent manipulating the DOM with it; however as Douglas Crockford has stated before, this is really the fault of the early browser developers (Netscape and Microsoft) principally for the horrible DOM interface used to manipulate web pages. JavaScript often gets the blame for this, unfairly in my opinion. Luckily since the invention of JQuery, and the standardisation of web browsers, much of the DOM pain has gone away.

JavaScript is a language that does not need compiling (though it can be minified for deployment), it can run on any operating system, it can run equally well on most browsers and power dynamic web sites, but in addition to this it now also runs on the back end servers just as well, as a result of the development on Node.JS. We now have a programming language that can run everywhere, and can power full end-to-end solutions, and is fully standards compliant.

JavaScript has other big advantages and those are it's fully functional nature, non blocking APIs, simple single threaded event loop and it's excellent closure implementation. Put together it is really very easy to write high performance code, without the complexity of managing a lot of threads.

Two particular frameworks I have focused on in JavaScript, provide a very powerful and feature-rich platform for developing a client-server based solution in the enterprise :

Angular.JS sets the standard for front-end development, and powerful templating and data binding functionality building on the successes of JQuery before it, coupled with responsive design means that developers can now write front-end applications for the web that are every bit as rich and powerful as native applications, but with the cross-platform support that the web offers.

Node.JS on the back end offers a powerful single-threaded, non-blocking, high throughput JavaScript platform where heavy frameworks are shunned in favour of a modular design. High performance non-blocking code is written in C++ and exposed via JavaScript bindings - the application developer need never look at a single line of C++ code however. The Node.JS engine uses Google's V8 JavaScript engine.

Put together, these two technologies offer a unique opportunity to develop software for the "full stack" - in fact to do so is easier than developing separate solutions for the front and back end because the entire solution can be written in one language.

There are a lot of very good frameworks to assist in writing great applications, such as Jasmine (unit testing), Protactor (Angular E2E using selenium), Express.JS (for building web services) to name but a few.

I think in general that Java still enjoys the best tooling and enterprise level support, but JavaScript is fast catching up.

In summary I believe that JavaScript could eventually be the dominant programming language in the enterprise, there is a lot of innovation going on, and it is a very expressive, multi-use language with a lot to like about. In addition, the tooling support is getting better by the day, and it is not uncommon to see native JavaScript support in IDEs now.