19 September 2021

Tesla Dojo and Hydranet and AI and Deep Learning with New Super Computer Dojo and D1 Chip

 

Tesla Dojo and Hydranet and AI and Deep Learning

Tesla Has Done Something No Other Automaker Has: Assumed The Mantle Of Moore’s Law

Steve Jurvetson shared on Twitter that Tesla now holds the mantle of Moore’s law in the same manner NVIDIA took leadership from Intel a decade ago. He noted that the substrates have shifted several times, but humanity’s capacity to compute has compounded for 122 years. He shared a log scale with details.

https://www.flickr.com/photos/jurvetson/51391518506/

The link Jurvetson shared included a detailed article explaining how Tesla holds the mantel of Moore’s Law. Tesla’s introduced its D1 chip for the DOJO Supercomputer and he said:


“This should not be a surprise, as Intel ceded leadership to NVIDIA a decade ago, and further handoffs were inevitable. The computational frontier has shifted across many technology substrates over the past 120 years, most recently from the CPU to the GPU to ASICs optimized for neural networks (the majority of new compute cycles).”


“Of all of the depictions of Moore’s Law, this is the one I find to be most useful, as it captures what customers actually value — computation per $ spent (note: on a log scale, so a straight line is an exponential; each y-axis tick is 100x).”

“Humanity’s capacity to compute has compounded for as long as we can measure it, exogenous to the economy, and starting long before Intel co-founder Gordon Moore noticed a refraction of the longer-term trend in the belly of the fledgling semiconductor industry in 1965.”

Project Dojo: Check out Tesla Bot AI chip! (full presentation)


“In the modern era of accelerating change, it is hard to find even five-year trends with any predictive value, let alone trends that span the centuries. I would go further and assert that this is the most important graph ever conceived (my earlier blog post on its origins and importance).”

“Why the transition within the integrated circuit era? Intel lost to NVIDIA for neural networks because the fine-grained parallel compute architecture of a GPU maps better to the needs of deep learning. There is a poetic beauty to the computational similarity of a processor optimized for graphics processing and the computational needs of a sensory cortex, as commonly seen in neural networks today. A custom chip (like the Tesla D1 ASIC) optimized for neural networks extends that trend to its inevitable future in the digital domain. Further advances are possible in analog in-memory compute, an even closer biomimicry of the human cortex. The best business planning assumption is that Moore’s Law, as depicted here, will continue for the next 20 years as it has for the past 120.”

In the detailed description of the chart, Jurvetson pointed out that in the perception of Moore’s Law, computer chips are compounding in their complexity at near-constant per unit cost. He explained that this is one of many abstractions of the law. Moore’s Law is both a prediction and an abstraction this abstraction is related to the compounding of transistor density in two dimensions. He explained that others related to speed or computational power.

He also added:

“What Moore observed in the belly of the early IC industry was a derivative metric, a refracted signal, from a longer-term trend, a trend that begs various philosophical questions and predicts mind-bending futures.”

“Ray Kurzweil’s abstraction of Moore’s Law shows computational power on a logarithmic scale, and finds a double exponential curve that holds over 120 years! A straight line would represent a geometrically compounding curve of progress.”



He explained that, through five paradigm shifts, the computation power that $1,000 buys has doubled every two years. And it has been doubling every year for the past 30 years. In this graph, he explained that each dot represented a frontier of the computational price performance of the day. He gave these examples: one machine used in the 1890 Census, one cracked the Nazi Enigma cipher in WW2, and one predicted Eisenhower’s win in the 1956 presidential election.

He also pointed out that each dot represented a human drama and that before Moore’s first paper in 1965, none of them realized that they were on a predictive curve. The dots represent an attempt to build the best computer with the tools of the day, he explained. And with those creations, we use them to make better design software and manufacturing control algorithms.

“Notice that the pace of innovation is exogenous to the economy. The Great Depression and the World Wars and various recessions do not introduce a meaningful change in the long-term trajectory of Moore’s Law. Certainly, the adoption rates, revenue, profits, and economic fates of the computer companies behind the various dots on the graph may go through wild oscillations, but the long-term trend emerges nevertheless.”

Tesla now holds the mantle of Moore’s Law, with the D1 chip introduced last night for the DOJO supercomputer (video, news summary).

Tesla’s BREAKTHROUGH DOJO Supercomputer Hardware Explained

This should not be a surprise, as Intel ceded leadership to NVIDIA a decade ago, and further handoffs were inevitable. The computational frontier has shifted across many technology substrates over the past 120 years, most recently from the CPU to the GPU to ASICs optimized for neural networks (the majority of new compute cycles). The ASIC approach is being pursued by scores of new companies and Google TPUs now added to the chart by popular request (see note below for methodology), as well as the Mythic analog M.2

By taking on the mantle of Moore’s Law, Tesla is achieving something that no other automaker has achieved. I used the term “automaker” since Tesla is often referred to as such by the media, friends, family, and those who don’t really follow the company closely. Tesla started out as an automaker and that’s what people remember most about it: “a car for rich people,” one of my close friends told me. (She was shocked when I told her how much a Model 3 cost. She thought it was over $100K for the base model.)

Jurvetson’s post is very technical, but it reflects the truth: Tesla has done something unique for the auto industry. Tesla has progressed an industry that was outdated and challenged the legacy OEMs to evolve. This was is a hard thing for them to do, as there hasn’t been any new revolutionary technology introduced to this industry since Henry Ford moved humanity from the horse and buggy to automobiles.

Sure, over the years, designs of vehicles changed along with pricing, specs, and other details, but until Tesla, none of these changes affected the industry largely as a whole. None of these changes made the industry so uncomfortable that they laughed at the idea before lated getting scared of being left behind. The only company to have done this is Tesla, and now new companies are trying to be the next Tesla or create competing cars — and do whatever they can to keep up with Tesla’s lead.

Teaching a Car to Drive Itself by Imitation and Imagination (Google I/O'19)

For the auto industry, Tesla represents a jump in evolution, and not many people understand this. I think most automakers have figured this out, though. Ford and VW especially.

Of all of the depictions of Moore’s Law, this is the one I find to be most useful, as it captures what customers actually value — computation per $ spent (note: on a log scale, so a straight line is an exponential; each y-axis tick is 100x).

Humanity’s capacity to compute has compounded for as long as we can measure it, exogenous to the economy, and starting long before Intel co-founder Gordon Moore noticed a refraction of the longer-term trend in the belly of the fledgling semiconductor industry in 1965.

Why the transition within the integrated circuit era? Intel lost to NVIDIA for neural networks because the fine-grained parallel compute architecture of a GPU maps better to the needs of deep learning. There is a poetic beauty to the computational similarity of a processor optimized for graphics processing and the computational needs of a sensory cortex, as commonly seen in neural networks today. A custom chip (like the Tesla D1 ASIC) optimized for neural networks extends that trend to its inevitable future in the digital domain. Further advances are possible in analog in-memory compute, an even closer biomimicry of the human cortex. The best business planning assumption is that Moore’s Law, as depicted here, will continue for the next 20 years as it has for the past 120.

For those unfamiliar with this chart, here is a more detailed description:

Moore's Law is both a prediction and an abstraction

Moore’s Law is commonly reported as a doubling of transistor density every 18 months. But this is not something the co-founder of Intel, Gordon Moore, has ever said. It is a nice blending of his two predictions; in 1965, he predicted an annual doubling of transistor counts in the most cost effective chip and revised it in 1975 to every 24 months. With a little hand waving, most reports attribute 18 months to Moore’s Law, but there is quite a bit of variability. The popular perception of Moore’s Law is that computer chips are compounding in their complexity at near constant per unit cost. This is one of the many abstractions of Moore’s Law, and it relates to the compounding of transistor density in two dimensions. Others relate to speed (the signals have less distance to travel) and computational power (speed x density).

Unless you work for a chip company and focus on fab-yield optimization, you do not care about transistor counts. Integrated circuit customers do not buy transistors. Consumers of technology purchase computational speed and data storage density. When recast in these terms, Moore’s Law is no longer a transistor-centric metric, and this abstraction allows for longer-term analysis.

Tesla’s MIND BLOWING Dojo AI Chip (changes everything)

What Moore observed in the belly of the early IC industry was a derivative metric, a refracted signal, from a longer-term trend, a trend that begs various philosophical questions and predicts mind-bending futures.

Ray Kurzweil’s abstraction of Moore’s Law shows computational power on a logarithmic scale, and finds a double exponential curve that holds over 120 years! A straight line would represent a geometrically compounding curve of progress. 

Through five paradigm shifts – such as electro-mechanical calculators and vacuum tube computers – the computational power that $1000 buys has doubled every two years. For the past 35 years, it has been doubling every year. 

Each dot is the frontier of computational price performance of the day. One machine was used in the 1890 Census; one cracked the Nazi Enigma cipher in World War II; one predicted Eisenhower’s win in the 1956 Presidential election. Many of them can be seen in the Computer History Museum. 

Each dot represents a human drama. Prior to Moore’s first paper in 1965, none of them even knew they were on a predictive curve. Each dot represents an attempt to build the best computer with the tools of the day. Of course, we use these computers to make better design software and manufacturing control algorithms. And so the progress continues.

Notice that the pace of innovation is exogenous to the economy. The Great Depression and the World Wars and various recessions do not introduce a meaningful change in the long-term trajectory of Moore’s Law. Certainly, the adoption rates, revenue, profits and economic fates of the computer companies behind the various dots on the graph may go though wild oscillations, but the long-term trend emerges nevertheless.

Any one technology, such as the CMOS transistor, follows an elongated S-shaped curve of slow progress during initial development, upward progress during a rapid adoption phase, and then slower growth from market saturation over time. But a more generalized capability, such as computation, storage, or bandwidth, tends to follow a pure exponential – bridging across a variety of technologies and their cascade of S-curves.

In the modern era of accelerating change in the tech industry, it is hard to find even five-year trends with any predictive value, let alone trends that span the centuries. I would go further and assert that this is the most important graph ever conceived.

Why is this the most important graph in human history?

A large and growing set of industries depends on continued exponential cost declines in computational power and storage density. Moore’s Law drives electronics, communications and computers and has become a primary driver in drug discovery, biotech and bioinformatics, medical imaging and diagnostics. As Moore’s Law crosses critical thresholds, a formerly lab science of trial and error experimentation becomes a simulation science, and the pace of progress accelerates dramatically, creating opportunities for new entrants in new industries. Boeing used to rely on the wind tunnels to test novel aircraft design performance. Ever since CFD modeling became powerful enough, design moves to the rapid pace of iterative simulations, and the nearby wind tunnels of NASA Ames lie fallow. The engineer can iterate at a rapid rate while simply sitting at their desk.

Tesla unveils "Dojo" Computer Chip | Tesla AI Day 

Every industry on our planet is going to become an information business. Consider agriculture. If you ask a farmer in 20 years’ time about how they compete, it will depend on how they use information, from satellite imagery driving robotic field optimization to the code in their seeds. It will have nothing to do with workmanship or labor. That will eventually percolate through every industry as IT innervates the economy.

Non-linear shifts in the marketplace are also essential for entrepreneurship and meaningful change. Technology’s exponential pace of progress has been the primary juggernaut of perpetual market disruption, spawning wave after wave of opportunities for new companies. Without disruption, entrepreneurs would not exist.

Moore’s Law is not just exogenous to the economy; it is why we have economic growth and an accelerating pace of progress. At Future Ventures, we see that in the growing diversity and global impact of the entrepreneurial ideas that we see each year. The industries impacted by the current wave of tech entrepreneurs are more diverse, and an order of magnitude larger than those of the 90’s — from automobiles and aerospace to energy and chemicals.

At the cutting edge of computational capture is biology; we are actively reengineering the information systems of biology and creating synthetic microbes whose DNA is manufactured from bare computer code and an organic chemistry printer. But what to build? So far, we largely copy large tracts of code from nature. But the question spans across all the complex systems that we might wish to build, from cities to designer microbes, to computer intelligence.

Reengineering engineering

As these systems transcend human comprehension, we will shift from traditional engineering to evolutionary algorithms and iterative learning algorithms like deep learning and machine learning. As we design for evolvability, the locus of learning shifts from the artifacts themselves to the process that created them. There is no mathematical shortcut for the decomposition of a neural network or genetic program, no way to "reverse evolve" with the ease that we can reverse engineer the artifacts of purposeful design. The beauty of compounding iterative algorithms (evolution, fractals, organic growth, art) derives from their irreducibility. And it empowers us to design complex systems that exceed human understanding.

Tesla AI Day

Why does progress perpetually accelerate?

All new technologies are combinations of technologies that already exist. Innovation does not occur in a vacuum; it is a combination of ideas from before. In any academic field, the advances today are built on a large edifice of history. . This is why major innovations tend to be 'ripe' and tend to be discovered at the nearly the same time by multiple people. The compounding of ideas is the foundation of progress, something that was not so evident to the casual observer before the age of science. Science tuned the process parameters for innovation, and became the best method for a culture to learn.

From this conceptual base, come the origin of economic growth and accelerating technological change, as the combinatorial explosion of possible idea pairings grows exponentially as new ideas come into the mix (on the order of 2^n of possible groupings per Reed’s Law). It explains the innovative power of urbanization and networked globalization. And it explains why interdisciplinary ideas are so powerfully disruptive; it is like the differential immunity of epidemiology, whereby islands of cognitive isolation (e.g., academic disciplines) are vulnerable to disruptive memes hopping across, much like South America was to smallpox from Cort├ęs and the Conquistadors. If disruption is what you seek, cognitive island-hopping is good place to start, mining the interstices between academic disciplines.

Predicting cut-ins (Andrej Karpathy)

It is the combinatorial explosion of possible innovation-pairings that creates economic growth, and it’s about to go into overdrive. In recent years, we have begun to see the global innovation effects of a new factor: the internet. People can exchange ideas like never before Long ago, people were not communicating across continents; ideas were partitioned, and so the success of nations and regions pivoted on their own innovations. Richard Dawkins states that in biology it is genes which really matter, and we as people are just vessels for the conveyance of genes. It’s the same with ideas or “memes”. We are the vessels that hold and communicate ideas, and now that pool of ideas percolates on a global basis more rapidly than ever before.

In the next 6 years, three billion minds will come online for the first time to join this global conversation (via inexpensive smart phones in the developing world). This rapid influx of three billion people to the global economy is unprecedented in human history, and so to, will the pace of idea-pairings and progress.

We live in interesting times, at the cusp of the frontiers of the unknown and breathtaking advances. But, it should always feel that way, engendering a perpetual sense of future shock.

The D1 is the second semiconductor designed internally by Tesla, following the in-car supercomputer released in 2019. According to Tesla Official, each D1 packs 362 teraflops (TFLOPs) of processing power, meaning it can perform 362 trillion floating-point operations per second.

Is the ‘D1’ AI chip speeding Tesla towards full autonomy?

The company has designed a super powerful and efficient chip for self-driving, but can be used for many other things

Tesla on its AI day, unveiled a custom chip for training artificial intelligence networks in data centers

The D1 chip is part of Tesla’s Dojo supercomputer system, uses a 7-nm manufacturing process, with 362 teraflops of processing power

The chips can help train models to recognize items from camera feeds inside Tesla vehicles

Will the just-announced Tesla Bot make future working optional for humans - or obsolete?

Elon Musk says Tesla robot will make physical work a ‘choice’

Back at the Tesla 2019 Autonomy Day, CEO Elon Musk unveiled its first custom artificial intelligence (AI) chip, which promised to propel the company toward its goal of full autonomy. The automaker then started producing cars with its custom AI within the same year. This year, as the world grapples with a chip shortage conundrum, the company presented its in-house D1 chip — the processor that will power its Dojo supercomputer.

Tesla's Dojo Supercomputer, Full Analysis (Part 1/2)

Tesla's Dojo Supercomputer, Full Analysis (Part 2/2)


The D1 is the second semiconductor designed internally by Tesla, following the in-car supercomputer released in 2019. According to Tesla Official, each D1 packs 362 teraflops (TFLOPs) of processing power, meaning it can perform 362 trillion floating-point operations per second. 

Tesla combines 25 chips into a training tile and links 120 training tiles together across several server cabinets. In simple terms, each training tile clocks in at 9 petaflops, meaning Dojo will boast over 1 exaflop of computing power. In other words, Dojo can easily be the most powerful AI training machine in the world.

The company believes that AI has limitless possibilities and the system is getting smarter than an average human. Tesla announced that to speed up the AI software workloads, its D1 Dojo custom application-specific integrated circuit (ASIC) for AI training will be of great use, the software that the company presented during this year’s AI Day that was held last week.

Although many companies including tech giants like Amazing, Baidu, Intel and NVIDIA are building ASICs for AI workloads, not everyone has the right formula or satisfies each workload perfectly. Experts reckon it is the reason why Tesla opted to develop its own ASIC for AI training purposes.

Tesla and its foray into AI

The system which is called the D1 resembles a part of the Dojo supercomputer used to train AI models inside Tesla’s headquarters. It is fair to note that the chip is a product of Taiwan’s TSMC’s manufacturing efforts and is produced using the 7nm semiconductor node. The chip reportedly is packed with over 50 billion transistors and boasts a huge die size of 645mm^2.

Now, with the introduction of an exascale supercomputer which management says will be operational next year, Tesla has reinforced that advantage. Since AI training requires two things: massive amounts of data, and a powerful supercomputer that can use that data to train deep neural nets, Tesla has the added advantage. With over one million autopilot-enabled EVs on the road, Tesla already has a vast dataset edge over other automakers. 

All this work comes two years after Tesla began producing vehicles containing AI chips it built in-house. Those chips help the car’s onboard software make decisions very quickly in response to what’s happening on the road. This time, Musk noted that its latest supercomputer tech can be used for many other things and that Tesla is willing to open up other automakers and tech companies who are interested. 


At first it seems improbable — how could it be that Tesla, who has never designed a chip before — would design the best chip in the world? But that is objectively what has occurred. Not best by a small margin, best by a huge margin. It’s in the cars right now,” Musk said. With that, his newest big prediction is that Tesla will have self-driving cars on the road next year — without humans inside — operating in a so-called robo-taxi fleet. 


Tesla introduced the Tesla D1, a new chip designed specifically for artificial intelligence that is capable of delivering a power of 362 TFLOPs in BF16 / CFP8. This was announced at Tesla’s recent AI Day event.

The Tesla D1 adds a total of 354 training nodes that form a network of functional units, which are interconnected to create a massive chip. Each functional unit comes with a quad-core, 64-bit ISA CPU that uses a specialized, custom design for transpositions, compilations, broadcasts, and link traversal. This CPU adopts a superscalar implementation (4-wide scalar and 2-wide vector pipelines).

This new Tesla silicon is manufactured in 7nm process, has a total of 50,000 million transistors, and occupies an area of ​​645 mm square, which means that it is smaller than the GA100 GPU, used in the NVIDIA A100 accelerator, which is 826 mm square in size.

Each functional unit has 1.25 MB SRAM and 512 GB/sec bandwidth in any direction on the unit network. The CPUs are joined in multichip configurations of 25 D1 units, which Tesla calls "Dojo Interface Processors" (DIPs).



Tesla claims its Dojo chip will process computer vision data four times faster than existing systems, enabling the company to bring its self-driving system to full autonomy, but the two most difficult technological feats have not been accomplished by Tesla yet, this is the tile to tile interconnect and software. Each tile has more external bandwidth than the highest end networking switches. To achieve this, Tesla developed custom interconnects. Tesla says the first Dojo cluster will be running by next year.

The same technology that undergirds Tesla’s cars will drive the forthcoming Tesla Bot, which is intended to perform mundane tasks like grocery shopping or assembly-line work. Its design spec calls for 45-pound carrying capacity, “human-level hands,” and a top speed of 5 miles per hour (so humans can outrun it).

IBM’s Telum Processor is the latest silicon wafer chip and a competitor to the Tesla D1. IBM’s first commercial processor, the Telum contains on-chip acceleration and allows clients to use deep-learning interference at scale. IBM claims that the on-chip acceleration empowers the system to conduct inference at a great speed.

IBM’s Telum is integral in fraud detection during the early periods of transaction processing while Tesla’s Dojo is mainly essential for computer vision for self-driving cars using cameras. While Telum is a silicon wafer, Dojo has gone against industry standards: the chips are designed to connect without any glue.

The most powerful supercomputer in the world, Fugaku, lives at the RIKEN Center for Computational Science in Japan. At its tested limit it is capable of 442,010 TFLOPs per second, and theoretically it could perform up to 537,212 TFLOPs per second. Dojo, Tesla said, could end up being capable of breaking the exaflop barrier, something that no supercomputing company, university or government has been capable of doing.

Tesla unveils "Dojo" Computer Chip | Tesla AI Day

Dojo is made up of a mere 10 cabinets and is thus also the smallest supercomputer in the world when it comes to size. Fugaku on the other hand is made up of 256 cabinets. If Tesla was to add 54 cabinets to Dojo V1 for a total of 64 cabinets, Dojo would surpass Fugaku in computer performance.

All along, Tesla seemed positioned to gain an edge in artificial intelligence. Sure, Elon Musk’s Neuralink — along with SpaceX and The Boring Company — are separately held companies from Tesla, but certainly seepage among the companies occurs. So, at the Tesla AI event last month, when the company announced it would be designing its own silicon chips, more than ever it seemed Tesla had an advantage.

The AI event culminated with a dancing human posing as a humanoid robot, previewing the Tesla Bot the company intends to build. But the more immediate and important reveal was the custom AI chip “D1,” which would be used for training the machine-learning algorithm behind Tesla’s Autopilot self-driving system. Tesla has a keen focus on this technology, with a single giant neural network known as a “transformer” receiving input from 8 cameras at once.

“We are effectively building a synthetic animal from the ground up,” Tesla’s AI chief, Andrej Karpathy, said during the August, 2021 event. “The car can be thought of as an animal. It moves around autonomously, senses the environment, and acts autonomously.”

CleanTechnica‘s Johnna Crider, who attended the AI event, shared that, “At the very beginning of the event, Tesla CEO Musk said that Tesla is much more than an electric car company, and that it has ‘deep AI activity in hardware on the inference level and on the training level.’” She concluded that, “by unveiling the Dojo supercomputer plans and getting into the details of how it is solving computer vision problems, Tesla showed the world another side to its identity.”

Tesla’s Foray into Silicon Chips

Tesla is the latest nontraditional chipmaker, as described in a recent Wired analysis. Intel Corporation is the world’s largest semiconductor chip maker, based on its 2020 sales. It is the inventor of the x86 series of microprocessors found in most personal computers today. Yet, as AI gains prominence and silicon chips become essential ingredients in technology-integrated manufacturing, many others, including Google, Amazon, and Microsoft, are now designing their own chips.

Tesla FSD chip explained! Tesla vs Nvidia vs Intel chips

For Tesla, the key to silicon chip success will be deriving optimal performance out of the computer system used to train the company’s neural network. “If it takes a couple of days for a model to train versus a couple of hours,” CEO Elon Musk said at the AI event, “it’s a big deal.”

Initially, Tesla relied on Nvidia hardware for its silicon chips. That changed in 2019, when Tesla turned in-house to design chips that interpret sensor input in its cars. However, manufacturing the chips needed to train AI algorithms — moving the creative process from vision to execution — is quite a sophisticated, costly, and demanding endeavor.

The D1 chip, part of Tesla’s Dojo supercomputer system, uses a 7-nanometer manufacturing process, with 362 teraflops of processing power, said Ganesh Venkataramanan, senior director of Autopilot hardware. Tesla places 25 of these chips onto a single “training tile,” and 120 of these tiles come together across several server cabinets, amounting to over an exaflop of power. “We are assembling our first cabinets pretty soon,” Venkataramanan disclosed.

CleanTechnica‘s Chanan Bos deconstructed the D1 chip intricately in a series of articles (in case you missed them) and related that, under its specifications, the D1 chip boasts that it has 50 billion transistors. When it comes to processors, that absolutely beats the current record held by AMD’s Epyc Rome chip of 39.54 billion transistors.


Tesla says on its website that the company believes “that an approach based on advanced AI for vision and planning, supported by efficient use of inference hardware, is the only way to achieve a general solution for full self-driving and beyond.” To do so, the company will:

Build silicon chips that power the full self-driving software from the ground up, taking every small architectural and micro-architectural improvement into account while pushing hard to squeeze maximum silicon performance-per-watt;

Perform floor-planning, timing, and power analyses on the design;

Write robust, randomized tests and scoreboards to verify functionality and performance;

Implement compilers and drivers to program and communicate with the chip, with a strong focus on performance optimization and power savings; and,

Validate the silicon chip and bring it to mass production.

“We should have Dojo operational next year,” CEO Elon Musk affirmed.

Keynote - Andrej Karpathy, Tesla


The Tesla Neural Network & Data Training

Tesla’s approach to full self-driving is grounded in its neural network. Most companies that are developing self-driving technology look to lidar, which is an acronym for “Light Detection and Ranging.” It’s a remote sensing method that uses light in the form of a pulsed laser to measure ranges — i.e., variable distances — to the Earth. These light pulses are combined with other data recorded by the airborne system to generate precise, 3-dimensional information about the shape of the Earth and its surface characteristics.

PyTorch at Tesla - Andrej Karpathy, Tesla

Tesla, however, rejected lidar, partially due to its expensive cost and the amount of technology required per vehicle. Instead, it interprets scenes by using the neural network algorithm to dissect input from its cameras and radar. Chris Gerdes, director of the Center for Automotive Research at Stanford, says this approach is “computationally formidable. The algorithm has to reconstruct a map of its surroundings from the camera feeds rather than relying on sensors that can capture that picture directly.”

Tesla explains on its website the protocols it has embraced to develop its neural networks:

Apply cutting-edge research to train deep neural networks on problems ranging from perception to control;

Per-camera networks analyze raw images to perform semantic segmentation, object detection, and monocular depth estimation;

Birds-eye-view networks take video from all cameras to output the road layout, static infrastructure, and 3D objects directly in the top-down view;

Networks learn from the most complicated and diverse scenarios in the world, iteratively sourced from a fleet of nearly 1M vehicles in real time; and,

A full build of Autopilot neural networks involves 48 networks that take 70,000 GPU hours to train, and, together, output 1,000 distinct tensors (predictions) at each timestep.

Training Teslas via Videofeeds

Tesla gathers more training data than other car companies. Each of the more than 1 million Teslas on the road sends back to the company the videofeeds from its 8 cameras. Hardware 3 onboard computer processes more than 40s the data compared to Tesla’s previous generation system. The company employs 1,000 people who label those images — noting cars, trucks, traffic signs, lane markings, and other features — to help train the large transformer.


At the August event, Tesla also said it can automatically select which images to prioritize in labeling to make the process more efficient. This is one of the many pieces that sets Tesla apart from its competitors.

Conclusion

Tesla has an advantage over Waymo (and other competitors) in three key areas thanks to its fleet of roughly 500,000 vehicles:

  • Computer vision
  • Prediction
  • Path planning/driving policy

Concerns about collecting the right data, paying people to label it, or paying for bandwidth and storage don’t obviate these advantages. These concerns are addressed by designing good triggers, using data that doesn’t need human labelling, and using abstracted representations (replays) instead of raw video.

The majority view among business analysts, journalists, and the general public appears to be that Waymo is far in the lead with autonomous driving, and Tesla isn’t close. This view doesn’t make sense when you look at the first principles of neural networks.

Wafer-Scale Hardware for ML and Beyond

What’s more, AlphaStar is a proof of concept of large-scale imitation learning for complex tasks. If you are skeptical that Tesla’s approach is the right one, or that path planning/driving policy is a tractable problem, you have to explain why imitation learning worked for StarCraft but won’t work for driving.

I predict that – barring a radical move by Waymo to increase the size of its fleet – in the next 1-3 years, the view that Waymo is far in the lead and Tesla is far behind will be widely abandoned. People have been focusing too much on demos that don’t inform us about system robustness, deeply limited disengagement metrics, and Google/Waymo’s access to top machine learning engineers and researchers. They have been focusing too little on training data, particularly for rare objects and behaviours where Waymo doesn’t have enough data to do machine learning well, or at all.

Wafer-scale AI for science and HPC (Cerebras)

Simulation isn’t an advantage for Waymo because Tesla (like all autonomous vehicle companies) also uses simulation. More importantly, a simulation can’t generate rare objects and rare behaviours that the simulation’s creators can’t anticipate or don’t know how to model accurately.

Pure reinforcement learning didn’t work for AlphaStar because the action space of StarCraft is too large for random exploration to hit upon good strategies. So, DeepMind had to bootstrap with imitation learning. This shows a weakness in the supposition that, as with AlphaGo Zero, pure simulated experience will solve any problem. Especially when it comes to a problem like driving where anticipating the behaviour of humans is a key component. Anticipating human behaviour requires empirical information about the real world.

Compiler Construction for Hardware Acceleration: Challenges and Opportunities

Observers of the autonomous vehicles space may be underestimating Tesla’s ability to attract top machine learning talent. A survey of tech workers found that Tesla is the 2nd most sought-after company in the Bay Area, one rank behind Google. It also found Tesla is the 4th most sought-after company globally, two ranks behind Google at 2nd place. (Shopify is in 3rd place globally, and SpaceX is in 1st.) It also bears noting that fundamental advances in machine learning are often shared openly by academia, OpenAI, and corporate labs at Google, Facebook, and DeepMind. The difference between what Tesla can do and what Waymo can do may not be that big.

2020 LLVM in HPC Workshop: Keynote: MLIR: an Agile Infrastructure for Building a Compiler Ecosystem

The big difference between the two companies is data. As Tesla’s fleet grows to 1 million vehicles, its monthly mileage will be about 1 billion miles, 1000x more than Waymo’s monthly rate of about 1 million miles. What that 1000x difference implies for Tesla is superior detection for rare objects, superior prediction for rare behaviours, and superior path planning/driving policy for rare situations. The self-driving challenge is more about handling the 0.001% of miles that contain rare edge cases than the 99.999% of miles that are unremarkable. So, it stands to reason that the company that can collect a large number of training examples from this 0.001% of miles will do better than the companies that can’t.

More Information:

https://www.datacenterdynamics.com/en/news/tesla-detail-pre-dojo-supercomputer-could-be-up-to-80-petaflops/

https://www.allaboutcircuits.com/news/a-circuit-level-assessment-teslas-proposed-supercomputer-dojo/

https://heartbeat.fritz.ai/computer-vision-at-tesla-cd5e88074376

https://towardsdatascience.com/teslas-deep-learning-at-scale-7eed85b235d3

https://www.autopilotreview.com/teslas-andrej-karpathy-details-autopilot-inner-workings/

https://phucnsp.github.io/blog/self-taught/2020/04/30/tesla-nn-in-production.html

https://asiliconvalleyinsider.com/2020/03/08/waymo-chauffeurnet-versus-telsa-hydranet/

https://www.infoworld.com/article/3597904/why-enterprises-are-turning-from-tensorflow-to-pytorch.html

https://cleantechnica.com/2021/08/22/teslas-dojo-supercomputer-breaks-all-established-industry-standards-cleantechnica-deep-dive-part-3/

https://semianalysis.com/the-tesla-dojo-chip-is-impressive-but-there-are-some-major-technical-issues/

https://www.tweaktown.com/news/81229/teslas-insane-new-dojo-d1-ai-chip-full-transcript-of-its-unveiling/index.html

https://www.inverse.com/innovation/tesla-full-self-driving-release-date-ai-day

https://videocardz.com/newz/tesla-d1-chip-features-50-billion-transistors-scales-up-to-1-1-exaflops-with-exapod

https://cleantechnica.com/2021/09/15/what-advantage-will-tesla-gain-by-making-its-own-silicon-chips/


Share:

0 reacties:

Post a Comment