y any measure, the run-up in the price and trading in Internet-related stocks between 1998 and the spring of 2000 was extraordinary. Companies with little revenues and no profits commanded billion-dollar market capitalizations. In February 2000, the largely profitless Internet sector equaled 6% of the market capitalization of all U.S. public companies, and 20% of all publicly trade equity volume. In just a two-year period of time, the entire sector earned over 1000% returns on its public equity. All is well documented now, the .com bubble burst in the spring of 2000 with an eventual decline back to 1998 levels.

Scholars, pundits, and analysts have put forward many explanations for the stunning rise – and stunning fall – of Internet stocks. But most have failed to get the root of a perplexing question. How did the apparent mispricing of Internet stocks persist in the presence of well-funded, rational investors?

We've got a hypothesis. As prices rose and the bubble was created, the market became dominated by optimistic individual investors, who crowded pessimistic investors out of the market. The shift in investor sentiment created a frenzy of demand for the initial public offerings of Internet-related stocks, which in turn boosted prices to untenable levels. And when so-called "lock-up periods" ended – which meant company insiders were suddenly able to sell portions of their large stakes – a massive new supply of Internet stocks became available for trading. Without an influx of optimistic new capital to snap up the new supply, the Internet stocks suffered a decline that quickly turned into a rout.

It's a relatively simple proposition, but one that must be backed by data.

Were Internet Stocks Overvalued?

A graph of the index of an equally-weighted portfolio of a universe of Internet stocks and the S&P 500 and the NASDAQ between January 1998 and December 2000 shows substantial divergence in relative pricing. (Figure 1).

The data further shows that these higher prices were real – i.e. that substantial amounts of trading took place at such levels. We studied some 400 Internet-related companies between January 1, 1998 to February 29, 2000, the majority of which went public in 1999 and early 2000. While they had an average price ($56) and average market value comparable to those of non-Internet firms, these New Economy stocks were far more volatile than their Old Economy peers. The average volume per stock in this period was three times higher for Internet firms than for non-Internet companies.

What did the market have to expect in order to justify the high valuations accorded Internet stocks at the height of the .com frenzy? We calculated the aggregate earnings of each of 11 separate Internet-related sectors – portals, b2b commerce, infrastructure, etc. – using Compustat. Because many of these companies had negative earnings, even the aggregate earnings numbers were negative. So we measured earnings potential by backing out the implicit earnings of the sector, assuming that the companies in the sector had already achieved income margins of their industry counterparts. The "earnings" were calculated by multiplying aggregate revenues by the assumed income margin. Then, we estimated the price-to-earnings (P/E) ratio by dividing a sector's aggregate market value by its implicit earnings.

Figure 1 reports the findings for the eleven sectors. To take one example, in February 2000, the 50-company e-commerce sector had an aggregate market capitalization of $72.675 billion, aggregate revenues of $4.459 billion, and net income of negative $3.565 billion. Using the comparable non-Internet industry margins of 1.9%, we backed out implied earnings of $85 million, which gave the industry an implied, New Economy P/E ratio of 856. The vast majority of the Internet firms had P/E ratios that were similarly high. Almost 20% of the firms had P/E ratios greater than 1500, while more than 50% exceeded 500.

To grow into such valuations, these companies would plainly have to outperform the overall market by a significant factor. Simply to support an historically high P/E ratio of 20, the Internet sector would need to generate 40.6% excess returns for a 10-year period to justify its current implied P/E ratio of 605. How large is 40.6% for 10 years? When they examined the distribution of earnings growth over a 10-year period from 1951-1998, researchers Louis K.C. Chan, Jason Karceski, and Josef Lakonishok found that the top two percentiles reported growth rates of 31.3% and 22.6%, respectively. Thus, the required growth rates for the entire sector – not just a few individual companies – would have to be between 50-100% higher than the highest 2% of existing firms. By any stretch of the imagination, it would have been exceedingly difficult for these companies to post the sort of growth and performance over the long-term to justify their prices.

Who Was Willing to Pay?

So who was willing to pay wildly inflated prices for Internet companies? The answer may help explain why prices grew so out-of-whack in the first place.

We found that, on a relative basis there were more retail (i.e. individual) investors than institutional investors in the Internet sector. In March 2000, institutions held about 40.2% of non-internet stocks, but only 25.9% of Internet stocks. This strongly significant difference is probably understated, because 1999 and 2000 saw the formation of many Internet-oriented mutual funds, which act as pass-throughs to retail investors. In addition, we found that while in March 2000, the Internet sector accounted for about 4.38% of the aggregate market, such stocks accounted for only 2.38% of pension funds' assets. If more retail investors were in the market than under normal conditions, then one might reasonably argue that the market was more prone to the types of behavioral biases that lead to overly optimistic beliefs.

Public Feeding Frenzies

To further test the hypothesis that optimistic individual investors helped boost prices for Internet-related stocks, we examined the torrent of initial public offerings (IPOs) during the era of .com mania. The first day of trading for an IPO represents the first time the price reflects the distribution of the beliefs across all investors. If overoptimistic investors dominate the market for such stocks, then one would expect them to rise explosively on the first day.

During the 1998-2000 period, IPOs commanded a great deal of attention among the media and investors. Numerous financial websites focused solely on IPOs, and the standard financial websites included detailed analysis of both upcoming and past IPOs. Between the second quarter of 1999 and the first quarter of 2000, there were a stunning 400 IPOs. Of these, 70% were Internet-related; they raised more than $33 billion dollars.

"By any stretch of the imagination, it would have been exceedingly difficult for these companies to post the sort of growth and performance over the long-term to justify their prices."

It has long been an accepted fact that IPOs are underpriced – they are priced and brought to market in such a way that they are likely to rise in their first trading days. Tim Loughran and Jay Ritter of the University of Notre Dame and the University of Florida, respectively, found that between 1990 and 1998 the first-day return for IPOs was 14%. Between 1975 and 1997, the highest average in any year was 21.2%, in 1995. But when we looked at the sample between January 1998 and February 2000, we found that the mean first-day return on Internet IPOs was 95.5%, with a median of 63.1%. The mean and median first day returns for non-Internet IPOs in this time period were far smaller: 33.6% and 10.4%, respectively.

Between the first quarter of 1999 and the first quarter of 2000, 146 IPO issues doubled in price on the first day of trading. In contrast, over the two decades from 1975-1997, this effect occurred for only a handful of the 6,500 IPOs. These large first-day returns are consistent with a sudden shift towards optimistic investors. On the first day they could, legions of investors – most of them individuals – who believed the stock was going to go ever higher voted with their wallets.

The Quiet Period

SEC rules stipulate that for 25 days after an IPO, Wall Street underwriters and company executives must refrain from hyping the stock or discussing the company's financial prospects. At the end of this so-called "quiet period," underwriters almost invariably release favorable research reports on the company.

Now, an overly optimistic investor can be characterized as one who basically ignores public information. If so, then one manifestation of this optimism might be the belief that such (utterly expected) news from the research report is new. In fact, Wall Street conventional wisdom holds that retail investors buy – and institutional investors sell – on the release of these positive research reports. Knowing this, institutions tend to buy, and therefore bid up the price of the shares, in the days before the quiet period ends.

Because the date signifying the end of the quiet period was known and publicized in documents and on websites, there should be no price response on average around that date. Instead, investors should incorporate that move into the stock price when it first trades.

We examined the average daily and cumulative abnormal returns for the Internet IPO sample leading up to the end of the quiet period. The daily returns are all positive for the last 10 days of the quiet period, with a cumulative effect of a 13% excess return. By contrast, a sample of non-Internet related firms for the period prior to 1998 turned in just 3.5% in excess returns, or about one-fourth the magnitude of our Internet sample.

How do we explain that differential? Well, if we assume that retail investors are more optimistic than institutional investors, we could conclude that the Internet sector is more prone to quiet period trading. In addition, we found that the average daily volume around on the day before, the day of, and the day after, the quiet period end is 60% higher than in prior days. That finding is consistent with increased buying of the firm's shares on the release of the underwriter's research report.

A Shortage of Shorts?

The increased volume on or around a known, predictable event suggests a greater degree of irrational activity. Of course, while many investors did get carried away, there was always a considerable amount of "rational" capital in the marketplace. And it begs the question. Why didn't such investors deploy their capital against the overvalued Internet sector by selling them short?

Selling short involves borrowing a stock, selling it, and then buying it back later, ideally at a lower are among the biggest holders of stocks, are generally reluctant to sell stocks short. Researchers Joseph Chen, Harrison Hong, and Jeremy Stein cite work that shows just 2% of mutual funds do so. Hedge funds, which are frequently aggressive traders of stocks, could have been important shortsellers. But the performance and volatility of Internet stocks may have made it difficult for them to do so. Over 90% of Internet firms had a maximum monthly return of more than 80% over the period we examined. Shorting stocks that perform in that manner is an extremely dangerous endeavor.

An alternate explanation may be that it was not possible to short Internet stocks in sufficient volume to bring the prices back down to rational levels. But our investigation shows that this isn't the case. Indeed, in February 2000, short interest – the percentage of the total amount of shares outstanding that are sold short – was considerably higher for Internet stocks than for their corresponding Old Economy counterparts. The mean short interest for Internet stocks was 2.8%, compared with 1.8% for non-Internet stocks, while the median short interests were 1.6% and 0.7%, respectively. In other words, short interest in Internet stocks was nearly double that of the typical non-Internet stock.

Limits on Shorting Stocks

There are reasons why the heightened presence of individuals in the Internet stock sector may have made shorting such stocks difficult in practice. First, in order to short a stock, the investor must be able to borrow it. For whatever reason, individuals tend to lend shares less than institutions do. Also, there was no guarantee that the short position would not get called, either through the lender demanding it back or through a margin call.

We aimed to measure the difficulty of shorting Internet stocks by looking at some related data. When an investor shorts a stock, he must place a cash deposit equal to the proceeds of the shorted stock. That deposit carries an interest rate known as the rebate rate. When there is an ample supply of shares to short, the rebate closely reflects the prevailing interest rate. But when supply is tight, the rebate rate is lower. This lower rate reflects compensation to the lender of the stock at the expense of the borrower, and thus can act as a mechanism to even out supply and demand.

A financial institution, which is one of the largest dealer-brokers, provided us its proprietary rebate rates for the universe of stocks on a selected number of dates. We found that the average rebate rate was approximately 1.08% less for Internet stocks than other stocks. While it is difficult to know what this means precisely about the ability to short on the margin, it is clear evidence that shorting was more difficult for Internet stocks. We also found that there is generally a correlation between the level of short interest and the rebate rate. The higher the short interest, the lower the rebate rate, and presumably the more difficult to find significant number of stocks to short sell.

Who Let the Stocks Out?

So, why did the so-called Internet bubble burst? Given our model, we can think of two possibilities. First, perhaps fundamental news came out about Internet stocks that shocked the beliefs of the optimistic investors. But while there certainly was bad news, we could find no particular single event that could have caused the drop. A second explanation is that perhaps pessimistic investors suddenly were able to short a considerable amount of Internet stocks.

We believe that there is evidence to support this latter explanation – once we realize that shorting stocks and selling stocks have the same economic effect. At the end of 1999 and in the spring of 2000, a large number of investors – insiders, venture capitalists, institutions, and sophisticated investors – were suddenly free to sell their Internet shares. As the amount of potential selling increased, this new class of investors (whether they were pessimistic or agnostic) began to overwhelm the optimistic ones. The trigger for this selling was the end of the lock-up period.

In an IPO, only a small portion – 15% or 20% – of the shares are sold to the public. The rest remain in the hands of existing shareholders. And in the case of the .coms, these shares were frequently held in larger quantities by venture capitalists, company officers, and employers. Generally, underwriters and such insiders agree not to sell their shares for a given period of time. This so-called lock-up period is one way of aligning the incentives of the current owners and new owners. The majority of lock-up periods last 180 days, or approximately six months, and are stipulated in the prospectus. Thus, the end of a lock-up period is a completely observable event that results in a permanent shift in the amount of available shares in the marketplace. As important, it may represent the advent of a new class of investors and traders who may have different beliefs than the current marginal retail investors.

When we charted Figure 2, the dollar amount of shares being unlocked by month and the cumulative effect over the sample period January 1998 to September 2000, the results are quite striking. By late Spring of 2000, over $300 billion of shares had been unlocked. As these unlocked shares were eventually sold, there had to be sufficient capital invested by new optimistic investors to support the Internet price levels. After all, the price levels were plainly not justified by the companies' cash flow fundamentals. To the extent some of the $300 billion in capital was owned by investors who were less optimistic investors than the marginal individual investors, prices could have been expected to drop as this huge amount of capital worked its way through the market.

From November 1, 1999 to April 30, 2000, the value of unlocked shares rose from $70 billion to $270 billion. And what happened to the level of Internet stock prices over this same period? Between November 30, 1998 to November 30, 1999, the Internet index rose from 200 to 830. But while the index still rose over the next several months, it did so at a much slower rate. The slowdown in the rise of Internet prices may have been due to the beginnings of less optimistic investors selling their unlocked shares. And as prices stopped rising, optimistic investors' "bubble-like" beliefs about future prices were also affected, leading to a twofold effect on Internet prices. The fall of the index from 1030 to 430 from March 1 to April 30, 2000 coincides with the simultaneous increase in unlocked shares. Once the bubble burst, optimistic investors' beliefs were permanently altered.

Indeed, our research showed that after the lock-up period ended on a given stock, there was a downward drift in the price. In many cases, in fact, the drift started even before the lock-up ended. We hypothesize that this may be due to the gradual shift toward pessimistic investors. This post-lock-up drop is not found in previous studies of lockups for non-internet firms. And the decline in Internet stock prices around the lockup expiration is consistent with our hypothesis about the introduction of sellers to the market causing prices to drop.

Our evidence is admittedly circumstantial. But it is nonetheless compelling. The disproportionate number of optimistic individual investors helped drive Internet stocks up to untenable levels. And when the supply of stock available for sale by investors who may not have been quite as optimistic rose suddenly, the results were predictable.

Eli Ofek is associate professor of finance, entrepreneurship, and innovation at NYU Stern. Matthew Richardson is associate professor of finance at NYU Stern.