Blog: Extreme visualization and real-time simulation capabilities are the keys to the commercialization…
On May 15th, at the Artificial Intelligence Summit held by SenseTime Technology, the 51VR founder and CEO, Yi Lee, was invited to attend and gave a speech on “When VR Meets AI ”. In the speech, Yi Lee focused on the importance of the digital twin world for AI training, and the need to truly realize the commercialization of digital twinning. The two core competencies are essential, namely the ultimate visualization capability and real-time simulation capability.
Yi Lee emphasized that digital twin not only help AI training become more efficient but also provide a visual, simulation and intelligent decision-making infrastructure for smart cities after deep integration with AI.
In the speech, Yi Lee not only talked about the future trend but also talked about how 51VR can realize the industry step by step and shared many cases.
The following is the speech from the founder and CEO of 51VR, Yi Lee, at the SenseTime Artificial Intelligence Summit:
Recently, the concept of digital twin has become a hot issue, stocks across the board have turned into a bull market, Baidu index also was broken ten thousand. For your information, the Baidu Index last year was basically close to zero. It can be seen how crazy everyone is in pursuit of popular concepts.
Digital twin is the key to achieve Industry 4.0. It is a very important application scenario, which is used to train AI. The AI algorithm training needs in a specific scenario are achieved by reducing the real world by 1: 1. Such a reduction, not just a visual 1: 1 reduction, also includes real-world physics rules and operations, to strengthen the superposition of AI training environment.
In the process of continuously training AI, the environment built by digital twin is constantly learning itself. It continuously integrates real data into the digital breeding platform through self-evolution, and finally completes the AI of the whole system. A digitally-generated environment based on AI for continuous self-growth will plan and predict in larger scenarios, enabling more features that are currently unimaginable.
As early as three years ago, we began to put forward a concept — earth clone, and to achieve this goal step by step, every year to hold the Earth clone conference. This year will be the third Earth Clone Conference. There is also a word especially popular this year — the mirror world, the meaning of which is exactly the same as earth clone we have been using. It contains a particularly important concept — digital twin.
But based on our long-standing efforts in this field, I would like to say that digital hygiene is not a concept of empty space. This technology is really to be realized. It is true that the future of the world will be cloned, and the mirror world will be good. The premise must be Based on the realization of two core capabilities: extreme visualization and real-time simulation. Without the support of these two core competencies, digital students are difficult to scale to all walks of life.
The ultimate visualization is a basic ability to restore the real world, and it is also an ability of a wide range of applications in the current industry. But only the visual restoration is not enough for the virtual scene to take on more functions, and it is not enough to be a powerful complement to the real world to complete the impossible creation in the real world. Therefore, real-time simulation will be another core capability that is indispensable for the digital generation.
Today, I am honored to be invited to participate in the SenseTime Artificial Intelligence Summit. I would also like to take this opportunity to share how 51VR can generate two key capabilities based on digital twin: extreme visualization and real-time simulation capabilities. How such capabilities can help AI and help the city become smarter.
51VR and SenseTime first met due to the introduction of Star VC. Both two sides have very high matching in each product line and technical ability, and they highly appreciated each other, which opens the close cooperation between 51VR and SenseTime.
At the end of 2017, the SenseTime Technology participated in the B round of financing 51VR.
Since then, the two sides have cooperated frequently in smart cities and automated driving. SenseTime provides computer vision support for 51VR help transform massive unstructured video data in commercial scenarios into structured data. 51VR provides realistic scene support for SenseTime unmanned algorithm training and combines simulation capabilities to provide excellent data derivation and data display support for algorithm training, helping its AI algorithm evolve. At the same time, 51VR ‘s 51City OS visual operating system in smart city products cooperates with SenseTime smart city overall solution to serve customers.
SenseTime Technology has always advocated “ AI can empower all industries”, meanwhile what 51VR has been doing is “ accelerate AI and eliminate difficulties”.
Below, I mainly talk about how 51VR empowers AI with ultimate visualization and real-time simulation capabilities.
Extreme visualization is the first step. I begin with a detailed share of 51VR accumulation in the 3D scene to build on.
All of our data comes from real-world acquisitions, which allows for a complete comparison with the real world 1: 1. Data collection includes more than ten types of data sources such as drone photos, radar car scans, satellite maps, GIS data, BIM data, and more than 1.3 million digital assets deposited within four years of our company.
These data are quickly realized by 3D modeling through self-developed machine learning and automation tools, and finally, realize the reductions from different precision type scenes of L1-L5 according to actual needs.
Currently, our system can automatedly generate a reduction of L1. The L1 precision can distinguish different buildings from the shape but does not reflect the specific difference of appearance color.
L2 precision is based on L1, adding materials, lighting, and so on. The following Seattle project built by 51VR belongs to this precision category.
L3 precision is our main category implementing in smart cities. Based on the L3 precision city model, we can realize the visual operation of the park or city scale by imposing various types of data, including water and electric heating, police system, government system, and roads.
The L4 precision is even higher with an accurate to centimeter level. AI training needs to be done at this level of precision.
As for the L5 precision, it can be accurate to millimeter level, but the cost is relatively high. At present, one of the world’s top Internet giants is the 51VR customer at this level of precision, used to train its AI robot.
After talking about the first core competency, let me introduce the accumulation of 51VR in real-time simulation capabilities.
Automated driving simulation, letting AI algorithms process more precise and faster
More than two years ago, we began to consider the layout of the 51VR in simulation.
We have seen that in China’s entire automotive industry, no domestically produced original software has been widely accepted by the industry, all using European and American software. In China’s automobile industry funds are used for the exchange of technology.
When the automated driving L3 and L4 began to thrive, we began to think, is there any chance to make a Chinese original simulation software in this new trend?
In 2017, we investigated the needs of many OEMs and algorithm companies, and comprehensively judged our own advantages, and finally decided to enter the field of automated driving simulation testing.
The end of December 2018, we officially released China’s first fully modular automated driving simulation test platform, 51Sim-One, the platform can do the whole life cycle simulation, complete the need of hardware and software providers, algorithm companies, sensor-making companies, OEMs and test agencies.
At present, the 51Sim-One automated driving simulation test platform can be used without input code, and the main vehicle, various sensors, weather, traffic flow, and case scene can be directly configured through the operation interface, thereby making automated driving cars trained automatically. If there is a bug in the algorithm during the training process, you can restore the scene of the accident through the playback case to see where the problem is.
Currently, we have invested more than 100 experts and engineers throughout the automated driving simulation to focus on this matter.
Based on the development of automated driving simulation, there are three main cores of 51VR, namely data structuring, dynamic reduction and expansion, and sensor simulation.
The first is to structure the road information. The world that the machines see is completely different from the world that the human eye does. Structured treatment is actually dealing with the real world as information that the machine can understand.
In terms of dynamic reduction, in order to train the automated driving AI algorithm, there must be enough accident cases. Currently, we use the self-developed case reduction and extension tools to meet the training and testing needs of exponentially increasing AI algorithms.
In terms of sensor simulation, such as camera simulation, we have close cooperation with SenseTime. SenseTime tests the semantic segmentations which were derived from our system with real-time data and uses these on their discriminative machine learning system. We continue to improve the realism of the simulation scene and the reliability of the camera simulation according to the evaluation mechanism of machine learning.
At present, we have achieved good results in the field of automated driving simulation. Not only have we been recognized by domestic car manufacturers and automated driving algorithm companies, but also have good cooperation with German clients.
Real-time traffic flow simulation, check for traffic congestion
The structuring of road and traffic data can be used not only for automated driving simulation but also for real-time traffic flow simulation and real-time data structuring, which play a great role in the entire transportation system.
Based on the traffic real-time data flow of the roadside camera, we work with partners to extract and smooth the vehicle moving trajectory to achieve the reduction of the traffic flow in the virtual environment, and transform the unstructured video data into structured data that can be tracked and analyzed.
Real-time forecasting of traffic and scientific decision-making provides a model basis, thus achieving early warning, real-time response, decision analysis, and planning optimization.
At the same time, we can extract typical standard and dangerous case cases from these structured traffic flow data, and use the case generalization function for the test and iterative evolution of the automated driving algorithm.
In the 51VR simulation training platform, as the training data is continuously precipitated, the data can be fed back to the algorithm.
Real-time simulation capabilities will be an extremely important capability in the future, and it will transcend big data and have a far-reaching impact on the real world.
For example, is it reasonable to predict whether the current morning and evening traffic peak in Beijing are reasonable and need to be adjusted?
For this problem, the Traffic Management Bureau cannot give one hundred measures, one by one. But in the simulation system, you can try a thousand, or even 10,000 times, whether to change a certain road or optimize traffic lights, etc., let the system find the optimal solution, making the decision more scientific.
For another example, when a city reaches 200,000 automated driving vehicles on the road, what will happen to the transportation system? How does it influence the flow of people and where to set the location of important infrastructure? These questions will be answered in the simulation system.
Urban intelligence, the road to the future from manipulation to simulation
For example, in this system, the monitoring system is no longer a single screen of dozens of splits, but what you see is what you get. Since the entire park has been completely reduced by 3D, you can directly view the overall flow chart of the park from a top-down perspective, and you can directly view and retrieve detailed monitoring images from any location.
Since the monitoring system and the security system have been connected to the underlying data, for some peculiar situations, the nearest security guard can be arranged to view.
In addition to monitoring and security, this platform can superimpose all data, real-time analysis, real-time mobilization, and real-time control.
There are several levels to build the system.
The first is the access layer.
The access layer needs to interface with various types of hardware and software interfaces, just like various hardware interfaces in Windows systems. You can build an API after all kinds of hardware, such as sensors, are plugged in.
The second layer, the data layer.
This layer is mainly used to organize the data generated by the connected software and hardware. The data here includes both structured data and unstructured data.
The third layer, the interface layer.
This layer mainly solves the problem of data isolation. Through the 51City OS system, all data is opened and displayed based on an interface. Anyone, with just ten minutes of training, can get started directly, and the operation is very intuitive. This is a meaning with the UI interface in Windows.
The fourth layer, the application layer.
This layer mainly solves the problem of manipulation. All the connected IoT devices have been linked, and the administrator can complete all the operations on one screen, just as easy and agile as clicking the application directly in the Windows system.
Going to the next level is the simulation layer.
In the future, as the system continues to improve, we will be able to access the simulation capabilities. At that time, the system can truly serve more simulation requirements beyond big data, providing optimal solutions for campus and city level.
Since its launch in December last year, 51City OS has been in more than 10 cities in 5 months. Every customer sees such an operating system and gives us useful feedback: the labor cost is greatly reduced, the learning cost of the operator is greatly reduced, the management becomes more direct, and the efficiency is greatly improved.
Whether it is real-time simulation applied to driverless training, traffic flow prediction, or city-level system simulation from control to simulation based on software and hardware integration, it is constantly being integrated into more and more data, allowing manipulation, training, and simulation have become more realistic and accurate. At the same time, such an environment that integrates extreme visualization and real-time simulation is also constantly self-learning, and it becomes more intelligent through data feeding and becomes self-growth, thus completing system-level AI and forming A powerful digital world. As a result, this digital twin world will also achieve large-scale predictions in larger scenarios, achieving more unimaginable creations in the current real world.
Of course, back to the present, only the ultimate visual and real-time simulation capabilities are excellent, and the scale of digital twinning is commercialized.