Author: Paul Grigoruta

One of the best reasons to use Magento are the community extensions. Free or paid, they augment Magento’s core features and save a lot of development time. Magento always supported this effort by offering a curated extension list. In the old days, this was called Magento Connect. It was simply an extension directory, Magento did not intermediate support or purchasing the extension. It still had value though as it did include a review list which of course, was more relevant than the 5-star reviews on the vendor’s site.

A short history of the marketplace

Magento Connect had a lot of down sides though. The approval time was very long. All the focus was on the marketing. You had to have pretty images and descriptions, but you could get away with very low quality, or even broken code. What was the worst though is that Magento heavily promoted the idea that you can go to the marketplace, get an extension, upload it via SFTP to your server and use it. Magento was not (and it isn’t to date), that kind of system. This resulted in a large number of bad reviews from non-technical merchants (“This extension breaks your site”, “I am not able to upload, PLS HELP!”, “Not good! Do not USE!”). Magento’s approach frustrated the open source community. It’s one thing to charge for the extension and provide support to merchants that cannot afford a developer. A totally different thing to provide free software, but having to deal with installing the extension/fix incompatible merchant stores. This resulted in a large number of open source developers simply pulling off their extensions from Magento Connect and keeping them on Github only, where the audience is more technical. I found myself searching for an extension on Github first and on connect second, which defeated the whole purpose of the effort to have an extension directory.

Fast forward to Magento 2, the approach was changed completely and Magento Connect was replaced by Magento Marketplace. There were major improvements right from the start:

Magento intermediates the purchasing (and charges a fee to the extension vendor). Now I can at least address Magento for a refund instead of obscure vendors.

You can no longer post a review unless you actually purchased the extension.

What did not improve from the start was the extension quality. Actually, initially the quality was worse than on the old Magento Connect. Probably Magento needed to have a big number of extensions to offer, so they accepted pretty much anything (as long as you had pretty pictures and descriptions!). Vendors tried to release their Magento 1 extensions for Magento 2 asap, ignoring all coding standards and architectural approaches.

Luckily, Magento worked hard and improved this. Here is how the current process looks like:

First, there are now two parallel, completely separate tracks – Marketing and Technical.

Marketing

The marketing flow is about the pretty pictures and description. Magento actually made this part really useful…

you need to add screenshots. Really important as they are the easiest way to understand quickly if the extension does what you need it todo.

the description has stricter guidelines so that it’s focused on the feature set, not on the sales pitch.

you have to submit a installation and user manual.

you have to clearly specify Magento version compatibility and pricing, including whether the extension relies on another third party subscriptions.

Technical

This is very important for a developer/agency. Personally, I try my best to write good quality code in my projects. Then the merchant buys an extension from the official marketplace and I am puzzled at how low quality it is. Or at least, this is how it used to work. Now there is a technical review. Mainly, it has three steps:

an automated code sniffer. It catches quite a few things. It even caught a few things in my code even I consider myself “seasoned”. While it’s still just an automated checker, you cannot do blatant mistakes anymore.

a Varnish test. Basically, check that the extension does not break caching. I had to ask for refunds on extensions in the past due to their architecture simply disabling caching and relying on it.

a manual QA test. While I am not sure what happens there, I like to think that a real person actually checks the basic flows of the extension and maybe looks over the code too.

I am sure the above works. First, there is no way to bypass the review process. If the automated code tester finds issues, you have to fix them. Then, I can simply feel how the quality has increased. It’s becoming the exception that I buy an extension and have a bad experience. Actually, I am currently only using the marketplace to buy extensions as I trust Magento’s reviews. At least for me, the Magento-owned marketplace concept finally worked.

Why is it better

Besides the above, there are a few not-so-obvious improvements that really helped:

the manual review team seems a bit more trained. I did not get my extensions rejected for silly reasons in a while.

the review process is faster. No more waiting for months.

the documentation is better on how the submission should look like, at least on the marketing side.

What’s still missing

While the extension markeplace is better, it’s still a long way from great imho. Here is what I’d like to see in the future:

A Magento-owned demo site so I can check the extension before buying. The vendors now take care of the demos, but not all of them do it properly.

A git repo for the extension. Being able to see the commit history helps me a lot.

Magento should own security issues. Sadly, vendors do a poor job at communicating a security issue. I’d like to be able to submit a form to Magento when I have one, Magento should work with vendor to correct, then all extension owners should be notified. This is left at the discretion of the vendor now. Most of them simply release a new version but forgot about the patch, or even about mentioning the upgrade fixes a critical security issue.

As an extension vendor, I’d love to see subscription-based pricing.

Conclusion

In the last year, I started to trust the marketplace as an authoritative entity in the extension world. While there are a few things to improve, Magento is definitely moving in the right direction. I expect that by then end of 2019, we will have an even better marketplace.

One of the most common tasks I have to deal with in Magento are integrations. Almost any system I worked with needs to push order or product data to an ERP or marketing software, or get stock/product info/tracking numbers from an external source. Every integration is unique in a few ways…

the actual data mapping. We either pass fields as-is or combine/transform them.

However, any good integration has a lot in common:

a log must exist so we can refer back to when and how each piece of information was synchronized

errors must be logged, with an alert system so we are aware of any failures

the system should be able to queue data in case the integration is down. We cannot lose info especially when dealing with financial records.

the system must be able to retry failed records

the actual field mapping must be easy to change, ideally without changing code

I have been looking for good solution to build integrations for a while. On the lower end, there is the do it yourself custom shell script. Easy to build, but usually missing key elements, like retries or flexible data mapping. On the higher end, we have full ETL solutions. They tend to be expensive and add new software components to the mix.

Almost accidentally I stumbled upon Xtento’s Import/Export suite – https://www.xtento.com/magento-2-extensions.html . They flew under my radar as I had bad experiences in the past with such extensions and for a while, concluded that the best import/export is the one you built on top of Magento’s default.

Let’s go over the steps involved to export orders from Magento via Xtento. First, one starts with defining destinations. A destination is the place where your orders will go. You have a few options:

on the local server in a specified folder

on a remote server via a SFTP or FTP connection

to an email

to a HTTP server (i.e. REST)

to a web service. Use this for XML-RPC or SOAP integrations

via a custom class, where you can define the communication with your exotic system

So the above options should cover any possible target. One nice thing is that you can have multiple destinations, so you could place the orders on a SFTP but also mail a copy of the files to you for later review.

After defining the destinations, the next step is to define the export profile. Most options are obvious so I will go only over the important ones:

you can choose the entity to export, i.e. orders. Usually each exportable entity corresponds to a specific extension that you need to buy from Xtento.

you can define filters. For example, you can decide that you only want to export “processing” orders, keeping the “pending” ones in queue until they are reviewed

You can define the export mapping as an XSLT template. This was the feature that I was sold on. XSLT templates allow you to be as expressive as you need to. You can use all the standard fields, apply transformations, use all the fields in the order/related tables (your custom ones too). All this with a nice autocomplete interface and a dead-simple way to test the template with an existing order. Once you get the hang of it, you almost never need to refer to the docs/examples, it’s that easy. If you do need help though, https://support.xtento.com/wiki/Magento_2_Extensions:Magento_Order_Export_Module#XSL_Template has you covered.

You can define when your export runs. Do you want to export orders on schedule? How often? Do you export all orders in the last N hours or only what’s new? If the export process is time consuming, you have a CLI command to run it outside the main Magento cron.

You have a flexible manual export option in case you need to replay any of the missed exports, or simply test the process.

Everything comes with logging. You have filterable grids where you can see the exported entries and errors. You also have the option to locally store the export files for later review.

If you need to import entities, Xtento has you covered too, The process is very similar in that you still have sources where you can pull from, profiles you can define, same logging capabilities, a way to map the data. In addition to imports, you have an actions section where you can define what happens when an entry is imported. For example, when you import a tracking number, you can have Xtento ship and bill the order automatically.

I should mention that currently Xtento does not offer a product import solution. You can import stocks, but not product data. I’d love to see that on their offer sometimes.

What I really like about the extensions is that they are developer-friendly. Almost everything in their system has a fallback to a custom class. You have a very exotic export destination? You can define a class to implement the communication logic. Need to map your data in a way the XSLT template does not support? You can define a class and a method just for that. Finally, having logs for all operations make it easy to identify random issues. It scales ok too, I have been exporting importing 100k records per day with no performance issues.

Here are a few usecases where I have used Xtento successfully, usually without writing any line of code:

product exports to marketplaces/product aggregators. I still prefer specific extensions for big marketplaces, like Amazon or Google Shopping, but will use Xtento for the ones that have poor extensions or none at all.

pushing order data to ERPs and getting back tracking info, billing and shipping automatically. That’s a huge one, before Xtento I used to spend a lot of time on these type of implementations.

pushing order and product data to marketing software, like mailing list managers.

importing stocks from an external source (usually ERP).

Xtento might be lacking all the bells and whistles of an ETL solution, but in an ecosystem where not everyone has a fortune to spend on an integration, their extension suite is more than decent to get things done.

Magento 2 has never been ultra-fast. Even with careful customisation and not a very large catalog, it still takes somewhere near 2s to render the average category view page. This slowness is usually made better by the full page cache, but navigation and filtering patterns are too different to hope that in a browsing session, a user will hit only cached pages.

Now, 1s-2s server-side generation time is not too bad. But more and more merchants expect an almost-instant browsing experience. Magento is hard at work with the PWA implementation, which should deliver a very smooth experience, but that’s still in progress. And when it’s done, time will be needed for the extension vendors to catch up. In my opinion, all the approaches for Magento PWA, either official or community based, call for a big implementation budget.

And here comes Algolia. Stop now and go here, check the load speed as you filter through the left-hand navigation. Almost instant.

Algolia is a search service. They integrate with Magento via an open source module. Basically, it will replace all your search and category pages with an empty catalog page that at load time, will do a client-side call to Algolia, get a json-formatted list of products, then use html templates that are under your control to render the results. So the initial page load time includes Magento bootstrapping (which should be under 500ms), plus a few hundred milliseconds for the Algolia call. Even better, the navigation is ajax-based, so any further filtering is almost instant. Of course, they also have very relevant (and customisable) search results.

That’s a huge selling point for me. While I still have to deal with performance issues on product detail pages, cart and checkout, using Algolia makes catalog navigation ultra-fast. And it takes a lot of load from the server, since we don’t process the catalog there.

All this sounds great, but there are a few reasons for which you may not be able to use it.

It is a paid service. Their pricing scheme is good though, I am almost positive the increase in conversion rate will pay off. Not to mention the countless hours you’d spend in trying to optimize the catalog page, if you get to that point.

It is a SaaS. You are fully dependent on them. When I was working on a project, my free trial ended and the site instantly became unusable. That got me thinking that an Algolia outage will break my site. But they look well funded and have a great uptime.

If you have an existing project, it’s way harder to move to Algolia. Since they replace all templates on the catalog pages, any skinning/custom features will be gone. Of course you can re-do them using their templates, but it will mean a lot more than just installing the Algolia module. Don’t expect any community extension that touches the category page to work out of the box either.

I also had to customise Algolia a bit and luckily, there are options. Roughly, the module works by creating a product indexer that pushes data to Algolia. Via the admin, you get to decide what attributes are searchable, filterable etc. All changes to products will be pushed at indexing time. When rendering the results, you have access to the json and can render them as you wish using html and a proprietary markup language.

Overall, it’s very easy to change CSS styling and add more attributes or custom logic, but you should be aware that at render time, you don’t have access to the product collection, so you cannot call Magento for any logic. Or well, I guess you could, but it defeats the purpose of using Algolia. The only approach should be to send any data you need at index time (even precalculate some attributes if needed), then simply display them in the templates. Algolia has you covered for default more complex features, like different pricing per customer group, but I can see how things could get complicated if you have custom logic in displaying data based on the current customer’s session. Still doable though.

All in all, Algolia is in my top 3 recommendations for a new Magento 2 project. It’s also in the top 3 recommendations for websites that have performance issues. I do hope that Magento will provide a similar experience based on elasticsearch once PWA is finalised, but till then Algolia is one of the best integrations you can have for your store.

Six-seven years ago when I started with Magento, the import feature was lacking. While it improved, it still has quite a few issues even today. Mostly every import task I worked on was delayed due to unexpected core bugs. Extensions that promise to help with this come with their own set of issues. There is hope with the https://github.com/magento-engcom/import-export-improvements project, but till that’s done we have to get by.

First, the imports do not have a programmatic interface you can hook into, i.e. pass an array with data and get it imported. Luckly, Firegento fixed this with https://github.com/firegento/FireGento_FastSimpleImport2 . The module is just a simple wrapper over the standard import, so not many chances to break. Of course there is the standard Magento 2 product API, but it’s pretty useless for anything else than a few products due to slowness.

The second thing I ran into is that tier price imports are a special entity, i.e. you first import the product data in a format, then import the tier pricing in a totally different one. This is not what a merchant would expect and sadly, not even the Firegento extension has a programmatic interface for tiers.

Then another gotcha is that importing empty values does nothing by default. So if you set a product color, then decide to unset it by importing an empty value, it will still keep the same value as before. Similarly, if you import category associations and want to remove an older association, you are out of luck, the old link remains. This is almost a “feature” in Magento – https://github.com/magento/magento2/issues/7930 . The proposed fix is to use the “Replace” behaviour of imports, but that creates another issue – with “Replace”, product data is wiped out and recreated. Not something you want todo if you care about stats/old orders since all the product IDs will change.

Another nuisance is that all custom attributes need to be crammed in a single csv field like:

For most of my clients, additional attributes is where the bulk of the data lives, so having them formatted as above is clunky.

For the good parts, the import is pretty fast, it can import images correctly and once you get the format right, most of the stuff works.

The conclusion? If you’re working on a import task, pad your estimates substantially, you will need the time to go through a various set of quirks you never thought of. By the way, all of the above refers strictly to the simple products, other product types might come with their own set of problems.

Ten years ago, PHP was definitely the better choice for the web. Things seem to have changed though. As a new developer, you are faced with quite a few options. First, there is Javascript that can now be used both in the frontend and backend. Then, there is Python which has seen a huge momentum. Ruby has always had his steady share. And there are many other languages coming out everyday. The junior programmer is left wondering what language should they learn as a web developer.

First, the question of what programming language to learn/use is a lot less relevant than a junior developer would think. The hard things in programming are organising your code, naming your variables, getting into the OOP/functional mindset, not repeating yourself and creating simple solutions. None of these are programming-language dependent. Once you get a hold of those, learning a new programming language only means getting over the nuisance of getting used to the syntax.

So how does one choose a programming language for a specific task? In some cases, the choice has already been made and you have to continue using the same language as you are not starting the project, but continuing it. In others, the client simply prefers a specific language because they worked with it in the past. But let’s assume total freedom of choice.

Programming languages by default offer very small building blocks, like if, for, while constructors. Then they offer libraries with small, contained objects and methods to work with things like files and databases. While usually they are very far away from what you have to build, the building blocks are small enough so with enough work, you can combine them to achieve any kind of result. Actually programming languages can be splitted into two different categories:

general languages, like PHP, Python, Javascript. They are not trying to solve a specific problem, but give you the tools to solve any problem. With enough work, that is.

domain-specific languages. While you cannot do anything you want with them, they can help you solve a problem in a specific domain much faster. For example, SQL helps you create databases, then query for data in a very efficient way. Mathcad helps solving math problems. HTML is a declarative language that helps you format web pages. CSS gives you a good way to style them.

You usually find yourself needing to use at least one general programming language, along with a number of domain specific ones. As a web developer, you will probably have to use HTML, CSS, SQL along with your general language programming of choice.

When it comes to general programming languages, you care mostly about what exists out there than can match the needs of what you want to build. I tend to categorize “what’s out there” in three distinct areas:

libraries. Code that allows you to use new methods and objects, without imposing a structure. For example, a statistics library will give you pre-built methods to calculate averages, means, deviations etc. You don’t need any specific code style to use them. Just search the documentation for what functions/objects are defined and call them in your code.

frameworks. They are more about helping you have a sane structure for your code. For example, a web framework will probably force you to use MVC, organise your files in a specific way, be mindful about security etc. They are invaluable for the beginner as you don’t have to go through the huge exercise of finding the best way to organise your code. They are great for seniors too, as they provide a very well documented way of working which helps board new team members faster, or even hire people with experience on the same framework, if popular enough.

full solutions. Think WordPress. You install it, then you have a full, working blog. You can use it as-is, or start making changes to suit your needs.

Given the above, the smart choice is to use the programming language that has most pre-built components for your needs. For example, it’s almost a no-brainer that if you are doing machine learning, Python or R are the better choice. That’s not really related to programming languages, but to the fact that Python has the most diverse, production grade libraries related to data analysis and machine learning.

Luckily enough, most languages do have frameworks. Once you’re done with learning the basic syntax, spend some time in learning the most popular/suitable framework for your needs. See how it feels.

This brings me to the last feature, full solutions, and slowly to the point why PHP still matters. Most problems are not that unique in the programming world. For example, I have been making a living in the last 10 years by mostly building e-commerce stores. While each store is unique, there is a lot that is common to all of them. So then it only feels natural to ask the question whether you can start from a “base” e-commerce store and build from there. And of course you can. And here is where PHP shines.

Mostly because PHP is almost as old as the modern web, there are a lot of full production grade solutions built with it. For example, most of the popular open source e-commerce solutions (Magento, Open Cart, Presta), are built in PHP. Then there is the hugely popular WordPress on which you can build a blog or any presentation/content site really. If WordPress looks like too restrictive to manage your content site, then there is always Drupal. Now since e-commerce and content sites cover maybe 80% of the sites that are out there, that’s a huge part. And of course, there are solutions for forums, mail clients, database admins and many others, but they don’t seem like the kind of things one would build nowadays.

Given that with PHP, you have pre-built solutions, it only makes sense to use them when the requirements match. There are a few arguments against it that I will try to defer….

“I can build a blog from scratch in X hours/days”. Yeah, “a blog”. But you definitely cannot build WordPress, with all the flexibility it offers.

Popular software like WordPress or Magento have a huge marketplace aroud them. You can buy a jaw-dropping theme for WordPress to have a great design for a few dollars. You can buy integrations with your preffered payment gateway for Magento at a small fraction of the cost to build it. Try that with your custom solution.

You can add more people to the team by simply searching for people with experience on the platform, very low learning curve.

“But I organise my code better, make it faster, more readable”. Most of the time this is simply not true. But even if it is, an effort a few orders of magnitude higher rarely makes sense business-wise.

“But Amazon, Facebook, Google, [Insert big brand here] do not use a pre-built solution”. This is totally correct. There is a point when maybe it makes sense to move to a custom solution. But that should be done when you are able to have a team of top-level developers and a huge budget. Trying to prototype a solution with not enough time and a team of juniors will end up in a disaster. Also keep in mind that some of the solutions used by the big companies are custom just because they started that way and never made the switch.

Overall, I feel comfortable with PHP as I can use software like Magento and WordPress to kick-start my efforts. It’s a no-brainer for me to use those when requirements match. What about custom solutions? I still prefer PHP. As I was stating in the beginning, good software is about organising the code correctly. I can do this in PHP as well as I could do it in other languages. I can also use a decent framework (preferring Laravel) as well as any other. But I am faster with PHP and Laravel as I have used them for years, so I am lazy enough to not switch to something else just do implement the same thing differently. The only valid reason for which I use another language is when that language offers a great deal of libraries to help with the problem at hand, libraries that are missing/are inferior in PHP. And in that case, I only need to get over the syntax errors and language quirks. Most programming languages are similar anyway.

Today I had to generate native XLS files from Magento 1. The https://github.com/PHPOffice/PhpSpreadsheet is the best library for the job, but it’s modern in that it uses composer and PSR. Since I really wanted to use this specific package, here is a quick way to include it in the now dated Magento 1 codebase:

Then, still in lib/PhpSpreadsheet, let composer download the needed files:

composer install

To actually use the library, we will have to tell Magento how to include the directory in the path and also require the generated composer autoloader. In your helper/model/controller, add a init method to do just that:

When discussing with a prospect, out of respect for their time, you should find out quickly whether you can really work together. After a few long negotiations with clients, I came up with a list of three questions I ask upfront to make sure the partnership is a good fit.

What is your estimated monthly budget for development?

As a freelancer, you have multiple clients. I have found out that by myself, I can handle about 5 of them. Each client will have urgent needs, Black Friday is in the same day for everyone, each needs meetings for time to time. Because of this reason, aim for clients that budget at least 20% of your desired monthly income. Less than that is too much noise and you won’t be able to serve them well.

Each client pays you out of their revenue. If they don’t make money, they won’t be able to pay you long term. No matter how well you do your job, they will not afford you at some point. Basically, a client will take a loss of profit only if:

They are a startup and hope that their product will sell a lot when ready

They have other sources of revenue (i.e. an online shop that’s backed up by a brick and mortar store)

They are heavily investing hoping to increase the business

They have no idea what they are doing

Evaluate well. While making some extra money is ok, it is better to have partnerships with profitable businesses.

Do you work hourly or fixed bid?

Personally I found out that working hourly is the best for my clients. If they agree on this, I simply provide my hourly rate upfront. With new clients however, you sometimes have to earn trust so you might have to go fixed bid for a while. To make sure that your expectations are aligned, give some examples from the past, i.e. “I have built this payment method integration for X”. The client will quickly realise whether you are a match for them.

Isn’t this to blunt?

No. As I stated in the beginning, you should make sure the basic terms are in order as soon as possible out of respect for the client’s time. There are some clients (especially in the lower tier level), that will wonder why do you ask these questions. Personally, I give them the same reasons I have outlined here.

What other challenges are there?

Even with the above questions answered and agreed upon, there are still challenges that might appear after signing the contract…

They might pay late or not at all

They always have urgent needs

In case you signed up for a fixed bid, they might ask for a lot of quotes and meetings, ending up by implementing very little, making most of your time unbillable

You simply cannot communicate well with them

I was not able to find the right questions for the above, so my approach is to start with a “probation” period, in which both me and the client can part ways without much heads-up. With the probation set up-front (and possibly a lower rate if it happens), you will never have an unhappy client even if you stop the collaboration. Usually, a broken partnership is reciprocal, they will want to get out of it as much as you do.