Genetic Programming

by Irina for IBA Group
Posted on August 28, 2014

IBA Group
Pavel Charnysh

Artificial intelligence, intelligent self-learning machines, systems that can advise on how to do work better, and robotics – all of this has always been like magic to me. When I studied at the university, I was carried away by these topics. It is even more fascinating to use knowledge from one field for another and thus solve the tasks that seemed unsolvable.

What do you think about the interaction of artificial intelligence and the theory of evolution, one of the most interesting open issues in biology? As I worked with algorithms and not hardware, I kept wondering, how we can teach a computer to be intelligent. I did research on the topic within an internal project at IBA.  This article gives an overview of Genetic Programming and my speculations on how to use it in software development.

As Wiki says, ‘Genetic Programming (GP) is an evolutionary algorithm-based methodology inspired by biological evolution to find computer programs that perform a user-defined task. Essentially GP is a set of instructions and a fitness function to measure how well a computer has performed a task. It is a specialization of genetic algorithms (GA) where each individual is a computer program. It is also a machine learning technique used to optimize a population of computer programs according to a fitness landscape determined by a program’s ability to perform a given computational task’.

http://en.wikipedia.org/wiki/Genetic_programming

In my view, GP is a way to find a solution using atomic user-defined blocks.

The following are the main principles of GP:

1. Initially, we are given past performance data to build the most suitable program for reproducing the same principles of data structure in the future. We assume that the initial data were produced by a kind of a black box. We have historical input and output data to reproduce this system with the same rules and principles for future use.

2. All programs are members of a population. It means that we get not the only solution, but a set of solutions and can choose the most suitable one.

3. Changes in a population are made using an iterative method. At each iteration, the programs that are most fit for crossover and replenishment of the population are selected.

4. A fitness function is used to determine, if a program is fit for the purpose. It is a user-defined metric that numerically presents the ability of a program to solve the defined task (to fit the mapping of the input and output parameters for a given data set).

5. The fittest individuals of a population are selected to develop the population just the way evolution selects species.

6. Changes that can be implemented in the surviving members are similar to biological evolution. A member can be mutated or crossed with another member of the population.

7. A stop condition is defined for a fitness function, when one can stop GP and pick up a solution.

To grow up a population, a programmer must define primitive blocks which will form an individual. These are terminals, including constants and variables, and primitive expressions such as  +, -, *, /, cos, sin, if-else, foreach, and other predicates. Any program can be presented as a tree built up of these blocks. This way, any individual of a population can be presented.


In this case, mutation and crossover are represented as following.


We just take a randomly selected node from one individual and use it to replace a randomly selected subtree of another individual. That is a crossover. We can also take a randomly selected node of an individual and replace it with a randomly generated subtree. That is mutation.

Genetic Programming is used for neural network learning and numeric computing, as well to approximate complex functions. I was researching GP to imitate activity of a definite person. An employee while doing his or her job can make both optimal and non-optimal decisions, which makes human thinking different from digital technologies. When you are asked to point to the south, you won’t be able to do it without a mistake, and this ‘white noise’ is our individual quality. Sometimes, we do not need an accurate answer to solve a problem. After gathering the data, we can imitate this employee’s behavior for new tasks using a computer program.

Genetic programming is also useful when creating an artificial player for a game with different difficulty levels. The behavioral algorithms of an artificial intelligent player are typically ideal and therefore a human cannot beat it, if it didn’t play at give-away. Consequently, we need to make the artificial intelligence a little bit human, allowing it to make mistakes from time to time. To this end, the GP algorithms are in place because they teach the artificial intelligence human behavior, including the ability to make mistakes.

Isn’t that remarkable? I think it is real magic and those who create smart computers are magicians. Who knows, maybe it’s a way put a soul into a computer.

Is jargon preventing an acceptance of Big Data?

by Irina for IBA Group
Posted on August 18, 2014

IBA Group
Mark Hillary

The Smart Data Collective blog recently published a view that there is too much jargon circulating in the industry related to Big Data. In fact, as I mentioned in my last blog, Big Data is itself a term that is often misunderstood and needs more clarity.

The blog is interesting because the author takes a good example of an over-used business term, ‘digital’, and explores what we mean when we read and use this term. Many of the definitions from the dictionary have nothing at all to do with the definition of digital business you might expect – modern, hi-tech, and connected.

In fact, many more terms are taken from the dictionary and bent and shaped into something new by technology companies. Innovate, disrupt, and thought leadership are all terms that mean something different if you are not working for a technology company, but how can we improve the use of Big Data as a term?

The advantage we have is that Big Data is a genuine and meaningful area of data science. It’s not just jargon created for use by MBA students as they discuss their plans for ‘wealth-generation’.

Big Data needs to be understood by the general public and by the company leaders that have never really felt that they had to understand technology before. But almost everyone has now used Facebook, or contacted a customer service centre, so it is becoming easier to connect the theory of how Big Data can be used to the ways in which people see it every day.

http://smartdatacollective.com/tracey-wallace/222436/big-data-jargon-we-all-need-reign-right-now

Understanding Big Data

by Irina for IBA Group
Posted on August 11, 2014

IBA Group
Mark Hillary

Big Data is a subject we have explored often on this blog because it’s an area where IBA has extensive experience and knowledge, but it is often difficult to explain. How big does a database need to be before it can be considered ‘Big Data’ and why do we need this separate terminology to refer to manipulating and analysing this data when relational databases have been in use for decades?

One example that goes a long way to answering these questions is the way that customer service is changing – especially for retailers. Products used to have a phone number and email address that customers could use to reach the manufacturer or retailer – usually to complain after a purchase.

Now, customers use online forums, review sites, Twitter, Facebook as well as the more traditional and direct channels such as online chat, emails, or a voice call. Customers are not always directly contacting a brand when they comment on a product, yet they usually expect a response.

Exploring this mass of information is a classic Big Data example. The retailers want to connect together all these communication channels into an ‘omnichannel’, yet this is impossible when they are considered to be regular databases.

If a customer emails a complaint, then tweets about the problem because their email is not answered and then finally calls because neither tweet nor email has been answered then the ideal situation for a retailer is that the agent on the phone knows about the earlier email and tweet.

But to make this work is not easy. The company has no control over Facebook or Twitter – it’s not internal data. And how can comments on a tweet be connected to a customer on the telephone?

All this is feasible, if you have enough information from various sources and you can analyse it quickly enough. Every company that interacts with their customers is now exploring this problem so maybe Big Data is about to hit the headlines again.

Messaging and messengers

by Irina for IBA Group
Posted on August 4, 2014

IBA Gomel
Ihar Kalesnik

It is common knowledge today that communication is king. The word ‘communication’ refers to face-to-face, online, and telecommunications, video conferencing, and other communication methods, including texting or messaging.

Mobility is another aspect of our everyday life. Mobile devices embrace a growing number of life areas, making our life easier and less tied to specific places, be it home, office or something else.

Accordingly, a growing number of mobile applications have shown up, the old ones are increasingly replaced with new applications. However, mobile applications often fail to meet our expectations. It happens that the features that were accessible earlier and made the mobile applications so convenient are no longer available. Some are too hard to set up or tune. Others on the contrary are installed easily but have too many functions, most of which are useless. Finally, we begin looking for something new again.

This is also true about message exchange software. Depending on personal habits and preferences, everyone has his or her own choice of favorite messengers.

Taking into account our personal experience and preferences, IBA developed a mobile messaging system that functions on Android. IBM Lotus Sametime served as a starting point and a basis for the application.

IBM® Sametime® products integrate real-time social communications into business environment, providing a unified user experience through instant messaging, online meetings, voice, video and data. With just one click, you are immediately connected to the person behind the information, which helps you meet the ongoing demands of everyday business.

In the IBA messenger, we implemented a customary and useful set of functions, making the application easy to customize and utilize. As an option, we added connection with the address book of a mobile device and a possibility to make a mobile phone call to any contact in the address book.

IBA Messenger screenshot1 IBA messenger screenshot 2 IBA messenger screenshot 3 IBA messenger screenshot 4

After the application was published on PlayMarket, many people have been using it successfully. Numerous positive references testify to its usability and popularity.

At present, messaging systems have different approaches to interface, the used protocols, and methods of interaction. Producers and developers keep working to modify the existing applications and create new ones. Using these applications, it is possible to exchange text, images, audio and video files, conduct voice and video communication, and use many other functions.

IBA is also planning to expand the messenger’s functionality. Currently, the IBA team is working on the next release of the application to include new functions based on the user feedbacks from PlayMarket. The release will present such useful features as file transfer, extended status support, and automatic reconnection. We intend to launch the new release on PlayMarket in September, 2014.

In the near future, we are going to develop an iOS version of the application.

I invite you to try the messenger in use. You are also welcome to leave comments on how we can improve it :)

https://play.google.com/store/apps/details?id=com.iba.gomel.andy.chat&hl=en

Big Data and Cloud in decision-making

by Irina for IBA Group
Posted on July 25, 2014

IBA Group
Aleš Hojka, CEO of IBA CZ
Vitězslav Košina, Business Consultant at IBA CZ

It is a reality today that organizations have to deal with a multitude of unstructured documents and other data. These data have true value, if they are properly and timely processed and extracted, and also are supplied with really useful links.

Almost any business has in some way implemented a data management system (DMS), a content management system (CMS) or a business intelligence (BI) solution. Unfortunately, a new system often provides a very low added value, especially when a large amount of data is involved. Getting reasonable output from unstructured data is problematic. Why is it so?

DMS, CMS, and BI systems can be really effective, if they meet two essential requirements. The first is a sufficiently flexible environment and quick access to information complemented by strong information security. A really flexible environment shoul be able to respond to resource and computing requests in a very short, practically in real time. This can be easily achieved by using a cloud-based solution. As these requests are dynamic and cannot be easily predicted at the time when the system is designed, cloud can be very instrumental. In addition, keeping resources for a «rainy day» is not an effective allocation of resources.

Therefore, cloud is a prerequisite for dealing with big data, but it is not the only one. In some cases, data and documents are not available anytime and anywhere without limitations, though we have an environment designed for large amounts of data. This is true.

It should be noted that effective decision-making involves an increasingly growing amount of information. As the information should be available on mobile phones and tablets, an information management system cannot transfer huge amounts of data and should have short response time. If we meet these two basic requirements, then we are poised for truly efficient extraction and processing of large amounts of data, which can be further aggregated and analyzed to make a grounded decision, deal, and etc.

For many companies, using Software as a Service (SaaS0 is a problem, because they have to do with sensitive commercial information or client data. From the perspective of a cloud-based solution, a dedicated cloud is needed as apposed to shared capacities.

Many organizations outsource only the infrastructure, while we recommend to outsource the entire solution. With a cloud solution, the issue of security is solved to the extent that is normal for corporate clients. In this case, only the needed capacity is commissioned and there is no need to reserve resources for the team that takes care of the infrastructure and applications. The organization can thus focus on business and management objectives, as well as provide added value to the clients.

With mobile devices, security is a very thorny issue because attacks on their security are more likely than on the computers that are located in the company’s premises, where they are protected physically. The good news is that these devices are so powerful that allow for implementing the strongest encryption and other elements of modern security.

Taking into account the links between documents and their associated metadata, as well as other data sources, cloud computing is the best solution one can choose. It is however possible, if needed, to move from SaaS to the model, in which the solution is managed by the customer and yet not loses the flexibility that cloud offers us in terms of resources. Using cloud is much cheaper than building one‘s own infrastructure. Moreover, it enables organizations to concentrate on their requirements and business needs instead of specific software or infrastructure.

Internet of Things

by Irina for IBA Group
Posted on July 18, 2014

IBA Group
Mark Hillary

I have written recently about Business Analytics (BA). What is BA? How does it affect your IT strategy and your business in general? I have also observed that there is a relationship between BA and Big Data (BD) – they are related concepts.

To clear up any confusion, I would say that BA is related to taking a set of data, performing a modelling operation, and using the model to predict some kind of future state – what-if calculations. BD is more of a continuous analysis of very large-scale business information.

But the business concept that is driving forward the important of both Business Analytics and Big Data is really The Internet of Things (IoT).

http://www.forbes.com/sites/mikekavis/2014/06/26/the-internet-of-things-will-radically-change-your-big-data-strategy/

Even for a fairly short blog post, this is already starting to fill up with three-letter-acronyms so let’s define what the IoT really is. More and more devices are capable of communication using the existing Internet infrastructure. It used to be computers that we would connect to the Internet, then laptops, then smart phones. Now it is tablets, ebooks, televisions, and every corporate electronic system you can think of – from security systems to electricity meters to photocopiers.

This revolution in making almost every device connect to the Internet is the starting point for the IoT. The classic consumer example is usually the connected fridge that can recommend a dinner based on what is inside, though a more useful example might be your car diagnosing a problem and communicating with the service centre without your own interaction.

In 1999, about 250mb of data per person was created each year. By 2012 ten times this amount of data per person was being generated. Data creation is increasing and the speed of increase is accelerating. Every day people are generating data with their smart phones without doing anything – just by switching it on, connecting to the Internet and allowing applications to work in the background.

This change in both the consumer and corporate environment is driving both the need for continuous Big Data analysis and also the ability to predict what may happen next based on Business Analytic tools.

Forbes: A short history of Big Data

Frontend development tools: HTML 5

by Irina for IBA Group
Posted on July 11, 2014

IBA Group
Alexei Zagorsky

Until recently, HTML was used exclusively to code web pages. At the early stages of the mass use of the internet, it was simple and clear that HTML is a markup language, CSS defines the styles of the webpage elements, and JavaScript is used to create specific effects that cannot be achieved with HTML and CSS.

This is of course an idealistic picture. The reality was that the browsers that existed at that time displayed the pages in an unpredictable way and met no standards. On the other hand, the standards did not catch up with the pace of life.

The situation gained momentum after the internet attracted the interest of large-scale businesses. It became quickly clear that with the internet one can dramatically increase sales. No additional software is needed! Just run the internet.

It is a browser that became the client part of a client-server application in the time when the word frontend was not yet used ‘right and left’. The word website became too generic in a way and the time came for web applications to step to the foreground.

The problems with HTML standards were not yet solved but the situation progressed quickly. Technologies improve dramatically, if they are in high demand. Even HTML and JavaScript allowed developers to create masterpieces when in capable hands. Web development was implementing a combination of features provided by different specifications, such as HTML 4.01 and XHTML 1.1, as well as well as widely-used methods, including different ‘workarounds’ to prevent browser errors.

HTML 5 simplified and, which is more important put in good order many things that were available before. It became a unified markup language that combined the capabilities of HTML and XHTML. In addition, it preserved compatibility with the earlier versions.

The main features of HTML 5 are the following:

• Tools for semantic markup of a document. Using relevant tags, one can easily identify the title, content, navigation, and etc. A similar approach can be applied to structure the content

• Multimedia and graphic support at the level of HTML tags. It radically simplifies implementation of audio and video content on a web page. As for the drawing capabilities provided by the canvas tag, they take one’s breath away

• New elements and attributes, and exclusion of the features that are better handled by CSS

• Improvement in web form handling, including new field types, additional attributes, and validation ‘on the fly’

• New JavaScript APIs for complicated web applications, including drag and drop, document editing, 2D drawing, and management of browsing history.

These are just basic innovations. For all HTML 5 features, visit the W3C website at http://www.w3.org/TR/2011/WD-html5-diff-20110405/
The web pages built on HTML 5 look simpler in terms of the code, are quicker to run and friendlier to search robots. Today, HTML 5 is indispensable for a professional website. Thanks to the new capabilities of HTML 5, web applications are actually similar to classic desktop applications in terms of frontend. One more important features of HTML 5 is its compatibility not only with PCs and notebooks but also with mobile devices, such as smartphones and tablets, as well as with modern TVs that have internet access. Therefore, it is possible to develop a unified interface that looks the same on different devices.

The HTML 5 standard is being further modified. The current version is available at http://www.w3.org/TR/html5/. However, one cannot be sure that the existing browsers fully embrace these specifications. Trust but prove :) . We should also bear in mind that the standards developed by W3C are not mandatory and their implementation depends on browser producers. These are the captains of the IT industry like Google and Microsoft.

See also Frontend versus backend. Where to begin?

How is Business Analytics used in the real world?

by Irina for IBA Group
Posted on July 2, 2014

IBA Group
Mark Hillary

As I have mentioned in the last couple of blogs Business Analytics (BA) is about using data and models to make better decisions. This can help governments to improve policymaking and companies to develop better strategies. The aim is to use information to improve the day-to-day performance of the organisation.

But what are the real business decisions being taken with BA tools and how does the data analysis process improve the decisions?

    - In the airline business, employees and aircraft need to be in the right place at the right time to maximise efficiency. This is a complex planning procedure when things are going right, but when a flight is delayed or a key staff member is ill or a plane requires unscheduled maintenance then the knock-on effect can be huge. Analyzing the possible options is a great example of how BA can influence real decisions.
    - Hotels want guests in their rooms as often as possible. Empty rooms in a hotel mean lost profit and a hotel that quickly sells out a particular night may have been able to charge more for the rooms on that evening. Analyzing past customer behaviour, the competition, and other influencing factors such as big sport or music events in the neighbourhood is another classic example of how data can feed into a specific business decision – how much to charge for a room.
    - Banks and finance companies need to make quick decisions about whether to lend money to a particular customer based on limited information – such as the salary. However there are many other factors that could determine whether the individual is good for the loan or not and these can all be quickly factored together to provide a decision.
    - Supermarkets always seem to have full shelves these days. Can you remember Saturday afternoons where many big supermarkets would run out of important products? With the entire history of each product and how it sells in each store it is now far easier for supermarket managers to ensure they are ordering the correct stock levels.

These are just four entirely different industries, but in each one it is clear that Business Analytic tools are changing how managers make decisions in those companies.

Better decision-making leads to increased efficiency, a better use of existing resources, and the opportunity to perform better – to earn more by delivering a better service. For this reason, managers in all industries should be thinking about what Business Analytics can do for them.

Selecting the right business analytics partner

by Irina for IBA Group
Posted on June 16, 2014

IBA Group
Mark Hillary

IDC is one of the leading global industry analysts, so it’s always interesting to see their own guidance on choosing a supplier. Their recent report ‘IDC MarketScape Excerpt: Worldwide Business Analytics Consulting and Systems Integration Services 2014 Vendor Assessment’ focused exclusively on the questions you need to ask when searching for a Business Analytics partner.

1. Pay attention to domain knowledge. Over the years, some service providers have built deep industry expertise across certain business needs.

2. Create a culture of analytics. It is not enough for you to have access to the right data, you need to create the processes that can make use of this data across your entire organization.

3. Don’t neglect the basics. Companies often find they have data issues once the migration has commenced, which will then delay the entire migration project. Stop and avoid all these roadblocks by taking the data cleansing stage seriously so the migration can run smoothly.

4. Align the strength of your supplier with project success. If you find a good match then ensure that your partner gets stronger as the project succeeds.

Of course, point 5 from IDC was to utilize their own research when selecting a supplier. Most of these points are what any manager with a good experience of outsourcing would be planning anyway, but it cannot be stressed enough that good planning for the migration and creating a culture of data analytics are essential for success.

For more information on IDC research reports, visit their website at http://www.idc.com

Business Analytics and Big Data

by Irina for IBA Group
Posted on June 10, 2014

IBA Group
Mark Hillary

Business Analytics is a topic that is often confused with Big Data. While the analysis of Big Data is related to the continuous analysis of business information through an analytical process, they are related concepts rather than exactly the same thing.

There are various kinds of analytics to start with:

Descriptive Analytics: how to gain insight from historical data, creating reports and scorecards that give a better vision of some existing data.

Predictive analytics: modeling through the use of predictive models and machine learning – allowing the system to learn what might happen next based on the data that is being studied usually in real-time.

Prescriptive analytics: taking a large data-set and attempting to create decisions, choosing possible paths, simulating what might happen if certain decisions are taken.

Decisive analytics: this supports human decision-making with very visual analytic information that helps the user.

So the field of Business Analytics is more related to the process of taking data and either modeling outcomes or predicting what may happen next, rather than just attempting to spot trends in a large data set.

Business Analytics is really a tool that can support executives to make better decisions by supporting their decisions with data, rather than just estimates or guesses. By using actual data from the business and modeling potential outcomes based on decisions that could be taken, the data can help to support the direction a business leader should take.

This type of process has existed for a long time, but it has been the creation of vast pools of business data – the move towards a Big Data environment – that has really stimulated the need for improved Business Analytics. The increased amount of data has provided more information that can be analysed and yet has also made it more difficult to reach a conclusion on the right decisions – without better analysis.