Blog: Knowledge Engineering & Systems: Key to Intelligence Amplification (IA)
When I started my career in data & analytics 15 years ago, a typical industry challenge was collecting highly dispersed, fragmented and somewhat low quality information and structuring it for insights in a cost-effective way. More recently though, the industry need has flipped, and the new challenge is condensing large swaths of dense and high volume information into actionable insights (of course, still in a cost-effective way). So, while the volume, velocity and variety of the data has changed dramatically, and the cost function has moved from searching for information to processing of information, one aspect has remained the same in many ways — the greater purpose of information: amplify human intelligence!
In the early 80s when artificial intelligence and machine learning was gaining initial traction, Knowledge Systems emerged as a viable way of augmenting human intelligence. A knowledge based system is a system that uses artificial intelligence techniques in problem-solving processes to support human decision-making, learning, and action.
Some of the initial systems developed focused on medical diagnosis or expert systems for satellites etc. Over the last four decades, as we have moved intensely towards knowledge based economy globally, the development of knowledge systems has also expanded. This offered immense benefits to enterprises and knowledge workers. The book ‘Knowledge Engineering and Management — The CommonKADS Methodology’ published by MIT Press provides a good framework of these benefits:
However, despite the benefits of knowledge systems, advancement in artificial intelligence and enterprise endeavors to develop them, many use cases of enterprise knowledge systems in the recent times have not been successful. That begs the question, why?
Some key observations emerged based on my research and current market dynamics:
1. Hyper marketing and hype cycles are distorting context of machine capability: The advertised promise of artificial intelligence / machine learning (as the end of human labor) is untrue, at least for the foreseeable future. However, the era for Intelligence Amplification (IA) is certainly going on and is unfortunately, not widely discussed. IA refers to leveraging power of machines and algorithms to augment human intelligence. In my view, AI based systems (aka knowledge systems) can now generate 30–60% productivity & quality in many use cases, however, undermined due to over promise. Simply changing the narrative from AI to IA and focus on knowledge automation can therefore, have a significant impact on purpose, methods and expectations of knowledge systems. There is good literature on this, but highlighting just one: https://pdfs.semanticscholar.org/eb82/c04d987c1db532e0fa5eaf07d82839c5e24c.pdf
2. Too much focus on inference methods rather than knowledge base: A research paper by Stanford in 1980 (https://apps.dtic.mil/dtic/tr/fulltext/u2/a092574.pdf), explained that an knowledge system’s performance is primarily a consequence of its knowledge base, and only secondarily a consequence of inference method applied. Therefore, expert systems should be knowledge rich even if they methods poor! While in 2019, we don’t need to compromise on inference methods given the advancements, a disproportionate time is spent on inference methods rather than developing the knowledge base. This, many a times, neglected component of modeling knowledge, is knowledge engineering! Knowledge engineering enables systematic & scientific extraction, integration and representing of complex knowledge structures within a computer system.
Additional reading material: http://mkusuma.staff.gunadarma.ac.id/Downloads/files/29404/KBS-Review.pdf
3. Knowledge systems need to factor the evolving nature of knowledge: One of the key reasons companies such as Amazon and Uber became highly successful at what they do was not due to their business idea only. A significant contributor was the high velocity feedback loop from users that allows them to fix/enhance their supply chains, user experience, products, etc. at a rapid pace. In contrast, enterprise systems are not at all nimble and despite great technology appear very monolithic. They do not allow Wiki type features to experts/users to continually ingest new knowledge understanding into the system and therefore, lose relevance over time.
Consequently, enterprises still leverage less than 15% of known information in their day to day processes and this reflects poorly on productivity, decision making and performance.
A key way to overcome many of the issues surrounding knowledge systems, apart from market conditions and narrative, is effective knowledge engineering. Knowledge engineering is at the core of a successful knowledge system and will play the same role in the fourth industrial revolution as mechanical, electrical and chemical engineering played in the industrial revolution that transformed manual labor.
Reading the book on CommonKADs methodology and other publications, some key themes on knowledge engineering:
- Recognize knowledge engineering as its own discipline: In several instances, knowledge engineering is a shadow role or highly heuristics based. This is not appropriate, as knowledge engineers have specific roles for knowledge acquisition, organization and representation.
Therefore, while the data engineer focuses on building the data pipes, the data scientist focuses on inference methods, knowledge engineer should focus on modeling structural use cases and detailing concepts of expert’s knowledge. The work of the knowledge engineer is a key input for the AI and software developers.
2. Know the problem to solve & leverage templates for knowledge models: Knowledge engineering methods adapt to use cases of knowledge and can model for specific requirements and in many cases produce reusable formats. The book describes a typical knowledge template hierarchy and each template deploys relatively unique approach for system development:
2. Balance efforts between knowledge engineering and inference methods: Developing knowledge rich systems should be the primary goal of knowledge based systems. There are several methods for acquisition and representation of knowledge depending on the nature of knowledge (illustration below):
Transforming this conceptual knowledge into programming language has progressed significantly in the last three decades with advancement in ontology engineering and knowledge graphs, however, should be applied diligently on projects.
3. Knowledge engineering sits at the cross section between business experts and technical experts: Much too often, the systems engineering teams and business experts are very disconnected from the eventual purpose and that is the primary reason why knowledge systems fail. Ideally, knowledge engineers should sit in the middle and facilitate transition of knowledge from the expert brain to the product.
Knowledge systems represent one of the most promising areas within artificial intelligence. Successful companies such as Amazon apply knowledge engineering extensively for product development (e.g. training Alexa). However, based on my research and experience, there is a greater need to recognize and deploy the concepts of knowledge engineering to effectively supplement human intelligence.