Showing posts with label automated economic reasoning. Show all posts
Showing posts with label automated economic reasoning. Show all posts

Monday, January 27, 2025

Deepseek gets F+ in Econ

LLMs have trouble with this pretty-basic economics question:

Prompt: “Is this factually accurate?

Positively correlated price-quantity pairs reveal that the demand curve shifted.

 

The LLMs will say that this can be false, but always without offering a counterexample or even mentioning that a counterexample could be the basis for such an answer.  OpenAI’s o1 model is sometimes an exception, stumbling toward a realization that no counterexample exists under the usual assumption about supply and demand slopes.

 

All of OpenAI’s models provide an excellent answer if only given a little hint to use logic.

 

Prompt: “Is this factually accurate?

Positively correlated price-quantity pairs reveal that the demand curve shifted.

[Hint: consider proof by contradiction]

 

Grok2’s and Deepseek’s answers do not improve with the hint.  Grok2 shows a lot a confusion about what is a proof by contradiction.  Deepseek is the worst kind of student, giving a long-winded incorrect answer laced with confidence:



GPT-4o gives a nice answer when provided the hint:

Analysis of Positively Correlated Price-Quantity Pairs

The statement that "positively correlated price-quantity pairs reveal that the demand curve shifted" needs to be carefully examined. Let's consider the dynamics of demand and supply to understand the accuracy of this statement.

In basic economic terms, the law of demand suggests that, all else being equal, as the price of a good decreases, the quantity demanded increases, and vice versa. This creates an inverse relationship between price and quantity, typically resulting in a downward sloping demand curve.

However, the existence of positively correlated price-quantity pairs (i.e., both price and quantity move in the same direction) indicates that other factors could be at play. This situation can occur if there is a shift in the demand curve itself, rather than a movement along the existing curve.

Proof by Contradiction

We can analyze this using a proof by contradiction:

  1. Assume that the demand curve does not shift, and thus only a movement along the existing demand curve occurs.
  2. If the demand curve is downward sloping, an increase in quantity demanded should occur only with a decrease in price, reflecting an inverse relationship (as per the law of demand).
  3. If price and quantity are positively correlated (both increase or both decrease), this contradicts the inverse relationship expected with a movement along a downward-sloping demand curve.

Therefore, the only logical conclusion is that the demand curve itself must have shifted (due to changes in consumer preferences, income, number of buyers, expectations, etc.), allowing both price and quantity to increase simultaneously.

Conclusion

Thus, the statement is factually accurate: positively correlated price-quantity pairs suggest that the demand curve has shifted, rather than indicating movement along an unchanged curve.



With that said, GPT-4o requires 6 seconds to handle this easy question.  In contrast, for almost 10 years my automated economic reasoning system has been able to answer such questions with 100% reliability and 100X the speed.


Thursday, April 4, 2024

Why I have an OpenAI subscription

 It's $20 per month.  Here are the features I value and use:

  • Others' custom GPTs
  • My own custom GPTs
  • Integration into email (Thunderbird) and word processing (MS Word for Mac OS)
In many cases, a custom GPT is nothing more than somebody uploading their documents that become the primary information source.  The creator can also train the custom GPT by giving it queries and then pointing out mistakes or room for improvement.  In some cases, the creator may also work with OpenAI to provide additional capabilities like Wolfram has done.  Regardless, the OpenAI subscription gives access to custom GPTs created by self and, if shared publicly, others.

Wolfram's Custom GPT

The Wolfram GPT, for instance, combines the "verbal skills" of Large Language Models with the analytical capabilities of computational AI.  As needed, Wolfram's GPT automatically calls an algebra engine or other computational engine and then processes the response.  Learn more here.

Since 2016, my coding has been primarily in the Wolfram language. While ChatGPT Plus can be a helpful aid in this coding, my own capabilities surpass it. The Wolfram GPT, however, operates at an entirely different level. This is demonstrated by a time-aggregation function it wrote for me. 


I have used @, /@, @@, @@@, #, #1, #2, :>, ->, but had no idea &-> was a thing!

Here's another example from Wolfram:



My own custom GPTs

I have several of my own custom GPTs, which are created merely by uploading my own documents.  Those docs become the LLM's primary knowledge source.  One of them is "Chicago Price Theory Tutor" which is amazing (but not yet shared with the public :)).  How smart is this?


Custom GPTs like this can understand and repeat back algebra (see the previous Wolfram screenshot).  I did not have to train it in this regard -- just upload a document that had algebra in it.  It is terrible at charts or graphs, so I instruct my custom GPTs not to even attempt to make a chart.

Once the Chicago Price Theory Tutor gave a wrong answer.  But then I realized that the document I provided was correct but confusing on that topic.  The 2nd edition of Chicago Price Theory will explain better.

Integration into email and word processing

The subscription allows you to interact with the LLM via API.  You could write your own code to call the LLM, but all I have done so far is use others' code that I prime with my OpenAI API key.  The Aify add-on for Thunderbird email adds a button to email-compose windows to perform various LLM tasks on the text therein, such as summarize it, check grammar, recommend edits, etc.  "GPT for Excel Word" is an add-on that performs similar tasks in MS Word, and presumably in Excel as well, though I haven't yet tried Excel.

There is an extra charge (beyond the $20/month) for API usage.  The extra charge is proportional to the amount of text sent back and forth to Open AI.  A busy day for me in this regard generates charges of less than $1.  "GPT for Excel Word" levies its own charge of the same order of magnitude.

Accessing the API this way, or other ways, also provides quite a large "context window" that improves results for certain tasks.  I think the custom GPTs also, in effect, have a large context window through uploading sizable documents.

Friday, March 3, 2023

ChatGPT: Marshall, Hicks, or Cliff Clavin?

Marshall discusses his Laws of Derived Demand in his Principles of Economics.  Specifically, in Chapter VI of Book V.

Marshall listed four “conditions” that result in wage-inelastic derived demand for the corresponding factor of production:


  1. “no good substitute being available at moderate price”

  2. “stiff and inelastic” final demand

  3. “small part of the expenses of production” (i.e., small factor-cost share)

  4. Inelastic supply of other factors


Putting aside the fourth law, we can discuss the first three in the context of the Allen-Hicks formula for the own-price elasticity of derived demand, βii, in the two-factor case (see also Chicago Price Theory Chapter 11).





where si is good i’s share in factor cost, εD < 0 is the price-elasticity of demand for the final product, and σ > 0 is the elasticity of factor substitution.  None of these parameters are necessarily constants; they may vary according to the factor allocation.  Marshall’s first law can be understood as a comparative static with respect to σ in the direction of zero, holding constant si and εD.  The second law can be understood as a comparative static with respect to εD holding constant si and σ.  The debated third law is a comparative static with respect to the share parameter, with Hicks at least holding constant εD and σ.


ChatGPT confidently asserts that there are three Marshall’s Laws.  Three is a fair count if we exclude the law that Hicks disputed.  However, by that count, ChapGPT gets them 5/6 incorrect.  Instead, it lists the definition of derived demand, followed by a statement about long versus short run.  The third in its list is perhaps a combination of the disputed third law and the first law.


Mulligan: What are Marshall's Laws of Derived Demand?


ChatGPT: Marshall's Laws of Derived Demand are a set of principles that describe the relationship between the demand for a particular factor of production (such as labor) and the demand for the goods and services that factor helps produce. These laws were developed by economist Alfred Marshall in the late 19th century and are still widely studied and referenced today.


The three laws of derived demand are as follows:


  1. The demand for a factor of production is derived from the demand for the product that the factor helps produce.

  2. The demand for a factor of production is more elastic in the long run than in the short run.

  3. The elasticity of demand for a factor of production depends on its importance in the production process and the availability of substitutes.


Taken together, these laws suggest that the demand for labor, for example, is ultimately determined by the demand for the goods and services that labor helps produce. In the short run, the demand for labor may be relatively inelastic because it can be difficult for firms to quickly adjust their production processes to changes in demand. However, in the long run, firms may be able to find substitutes for labor or change their production methods, making the demand for labor more elastic. The importance of labor in the production process also affects its elasticity of demand: if there are few substitutes for a particular type of labor, its demand may be relatively inelastic.


ChatGPT also confidently discusses the Hicks challenge but refers to a “law” that was not even one of the four.


Mulligan: Didn't Hicks show that Marshall was wrong about one of the laws?


ChatGPT: Yes, economist John Hicks challenged one of Marshall's laws of derived demand, specifically the second law, which states that the demand for a factor of production is more elastic in the long run than in the short run. Hicks argued that this law was not always true and that the elasticity of demand for a factor of production could depend on the specific circumstances.


Hicks pointed out that in some cases, the demand for a factor of production could be more elastic in the short run than in the long run. For example, if a firm has already made significant investments in a particular technology or capital equipment, it may be difficult for the firm to switch to a different production method or substitute other factors of production. In such cases, the demand for the factor of production may be relatively inelastic in the long run, while being more elastic in the short run as firms can adjust their inputs more easily.


Hicks’ critique of Marshall's second law suggests that the relationship between the demand for a factor of production and the goods and services it helps produce is complex and can vary depending on the specific circumstances of the market. Nonetheless, Marshall's laws of derived demand remain an important contribution to economic theory and continue to be studied and debated by economists today.


Next, I asked for sources.  It got the books correct, which is pretty good for ChatGPT (famous for fabricating books and articles) but wrong chapters.  It should be Marshall Book V, Chapter VI.  Hicks’ second edition only has 11 chapters, although perhaps Chapter 12 can be understood as the material that follows Chapter 11 in which case “Chapter 12” is the correct part of the book.


Mulligan: Where did Marshall state the laws?  Where did Hicks challenge one of the laws?


ChatGPT: Alfred Marshall first introduced his laws of derived demand in his seminal work "Principles of Economics," first published in 1890. Marshall's laws are discussed in Book V, Chapter III, where he discusses the theory of distribution and the demand for factors of production.


John Hicks challenged Marshall's second law in his 1932 book "The Theory of Wages." In the book, Hicks argues that the elasticity of demand for factors of production can depend on the specific production technology and market conditions. Hicks specifically critiques Marshall's assumption that the demand for a factor of production is always more elastic in the long run than in the short run. Hicks' critique is discussed in Chapter 12 of his book.


I disputed the citation of Chapter III, so it apologized [not shown here] and cited Chapter VI of Book IV, which purportedly discusses “Distribution of Produce” (the actual title is “Industrial Training”).  More disputes produced more rounds of apology and fake references.  It tended back to Book V, Chapter III (a famous chapter) but with a different fake chapter title at each iteration.


Cliff Clavin was the famous tavern know-it-all in Cheers.  Perhaps Cliff is back, this time behind the curtain of AI spouting lots of jargon in grammatically correct but ignorant sentences. Is Norm somewhere back there too?

John Ratzenberger, 'Cheers' Mailman Cliff Clavin, Delivers a Special  Message about the United States Postal Service - mxdwn Television

[image credit: https://television-b26f.kxcdn.com/wp-content/uploads/2020/08/Cheers-Cliff-Calvin-Screenshot-770x470.png]


Meanwhile, for more than a decade Artificial Intelligence has offered tools that are never self-contradictory and power a good exposition of Marshall’s Laws and much more in economics and statistics.

[Yes, ChatGPT can do the same for coding too: fabricated function options, fake references, etc.]

www.perplexity.ai correctly recited Marshall's four laws, including links to some microeconomics lecture notes. It correctly reported the sources of Marshall's law, and acknowledged that it could not report which chapter. It was incorrect as to which law Hicks challenged.

Tuesday, April 5, 2022

New Emissions Regulations are Coercive Paternalism, not Environmental Science, or even Benevolent Paternalism

Trump's CEA showed, based on credit transactions among manufacturers, that vehicle standards to abate a ton of CO2 cost about $163 on the margin, while even Obama said the abatement was worth only $50.  i.e., tightening emissions regulations fails a cost-benefit test by a wide margin.

Now Biden claims that new stricter standards pass a cost benefit test.  Although this will be cast as an environmental issue, the new conclusion is driven by assumptions unrelated to environmental economics or climate science:

(1) Consumer fuel savings get (mostly) double counted because "behavioral economics."  Specifically, 

"The agency’s analysis assumes that potential car and light truck buyers value only the savings in fuel costs from purchasing a higher-MPG model they expect to realize over the first 30 months they own it. Depending on the discount rate buyers are assumed to apply, this amounts to 25-30 percent of the expected savings in fuel costs over its entire lifetime." (p. 420 of DOT's final rule)

This double counting (100 - 27.5% = 72.5% of $98 billion in fuel savings) is more than quadruple the purported $16 billion net benefit shown in Table VI-11 of the final rule.

[I call it double counting because, by the principle of revealed prevalence, fuel savings is already built into the price and sales of fuel-efficient vehicles; many consumers do not purchase such vehicles because of the relative price and characteristics of competing vehicles.  Alternatively, you could say that DOT ignores benefit of low-MPG vehicles, but the revealed-preference result is the same. 

Following an Economics 301 homework solution from October 2019, in December 2020 Trump's CEA provided a vector proof -- that the market price for GHG credits (i) reflects fuel savings as consumers perceive them and (ii) fully quantifies the industry-level real GDP effects of changing GHG standards, without any additional term for fuel savings -- on the White House website.  See the appendix of this document.]

By comparison, the gross climate benefit is purportedly $27.5 billion.  i.e., they would have to more than double their already inflated "social cost of carbon" to push their thumb on the scale as vigorously as they did with "behavioral economics."  See below for more on paternalism.

(2) Biden says that some tightening comes for free because 5 manufacturers had already signed a pledge with California EPA to so tighten

But this ignores that California rules, when followed by just a subset of manufacturers, do not reduce the supply of federal credits, whereas changes in federal rules do even if the federal rules are not as strict as California's.  The equilibrium credit price is built into the prices paid by purchasers of new cars.

(3) When the above are enough to tilt the scale, all costs and benefits are discounted 3%/yr.  When an extra push is needed, Biden discounts environmental benefits at 2.5% per year while everything else is discounted 3%/yr.

"the use of the social rate of return on capital ... inappropriately underestimates the impacts of climate change for the purposes of estimating the SC-GHG. ... the consumption rate of interest is the theoretically appropriate discount rate in an intergenerational context." (p. 547 of the Technical Support Document.  See also p. 573 of the final rule.)


More on coercive paternalism

Trump's DOT and EPA spoke forcefully against paternalism as a justification for fuel standards.  If people lack knowledge, give them the knowledge rather than imposing a decision on them.  Here is how they said it

"the idea that regulating fuel economy and CO2 emissions can mitigate the consequences of inadequate access to information by placing decisions that depend on access to complete information in the hands of regulators rather than buyers has superficial appeal. Yet commenters do not establish that such a drastic step is necessary to overcome any inadequacy of information, or that requiring manufacturers to supply higher fuel economy will be more effective than less intrusive approaches such as expanding the range of information available to buyers." (85 FR 24608, italics added) 

In contrast, Biden's DOT and EPA say nothing like this, but instead extol the purported virtues of "behavioral economics."  They do not mention less intrusive approaches, let alone show why they would have fewer net benefits.

Friday, July 12, 2019

Economic Theory in the White House: The Rebate Rule

The "rebate rule" was proposed by the Dept. of Health and Human Services (HHS) in February 2019.  It would have prohibited rebates in the Medicare Part D prescription drug market.  Chicago Price Theory was intensively used in the White House to project the effects of the rule on market outcomes and the distribution of surplus between senior citizens, taxpayers, and firms at various positions in the supply chain.

HHS described rebates as follows "Prescription drug manufacturers prospectively set the list price ... of the drugs they sell to wholesalers and other large purchasers. Manufacturers also retrospectively pay PBMs or other entities in the drug supply chain, under rebate arrangements, that meet certain volume-based or market-share criteria." (84 FR 2340)  The Part D rebates alone exceed $30 billion per year and this rule by itself was projected to increase the Federal deficit by about $20 billion per year, which is historic as a single regulation.

This proposal to eliminate rebates was obviously controversial, as reported in the news and evidenced by the facts that the rule was proposed, received almost 26,000 comments from the public, and then this Thursday was withdrawn by HHS.  The proposed rule was complicated because it was a vertical (business-to-business) price control in a market that already has nonlinear pricing, nonlinear and interdependent government subsidies, and longstanding price regulations of various kinds.  The President himself described the rule as requiring a 193 IQ in order to understand its effects:
But prescription drugs, look, it's a rigged system, OK, if I told you how crazy it is, the Web, it's the Web, you need 193 I.Q. to even understand.  This web of geniuses, they put this thing to lower drug prices. It has 19 effects here and 27, so we got it down and we're getting it down further. We have the smartest people, the best people in that world working on it.... (President Donald Trump April 27, 2019, Green Bay WI)



[other of the regulations discussed in that speech were analyzed with Chicago Price Theory too.  Several dozen parts of other CEA reports draw closely on specific pages of Chicago Price Theory].

I agree that it would be essentially impossible to understand the economic effects of this rule in a timely manner without the extensive assistance of Chicago Price Theory and Automated Economic Reasoning. An important tool for analyzing nonlinear pricing is affectionately known as "the Murphy football" among Chicago economics students (and I associate with this Klein and Murphy article about competition with nonlinear pricing; see Chapter 5 and Chapter 13 of the text).

The football picture shows, among other things, the distinction between list price and net price (i.e., list minus rebate), but in order to prevent revealing too much of the answer to one of the new book's homework problems, I show it in more abstract form below.


For the same reason, among others, I will not say what was CEA's projected impact of the rule.  But take Chicago Price Theory and perhaps you can join the "web of geniuses."

There is no other textbook that teaches the "Murphy football."

There is no other textbook that brings its students so immediately to rigorous and timely policy applications of economic theory.


Monday, July 1, 2019

A Brief Summary of Activities in President Trump's Council of Economic Advisers


From July 2018 to June 2019, I served as Chief Economist of the Council of Economic Advisers (CEA). My primary responsibilities were preparing public reports, supervising senior economists and interacting with various groups in the White House and in the relevant agencies on a wide range of topics.  As CEA engaged in topics, they were picked up by Kevin Hassett (especially tax and trade), Tom Philipson (esp. health, infrastructure, student loans), Rich Burkhauser (esp. labor, immigration, and social programs), or me (Affordable Care Act, socialism, regulation, wage growth, macro aspects of trade).  On some of the topics I worked serially or in tandem with Tom (health insurance regulation, Medicare Part D, the Rx CPI, opioid prices) and in tandem with Rich (TROIKA).

The large majority of my time was in various stages of preparing Administration reports for the public, most of which were CEA products although OMB publishes the TROIKA results and the agencies publish rulemaking documents (which CEA sometimes edits).

Typical activities: Supply and Demand
The pace and daily execution of my work closely resembled academic research and economics consulting in litigation matters.  This is probably unusual in the history of the CEA, but was the result of three practices:
  • anticipating the needs of POTUS and EOP economic principals,[1]
  • fitting questions into the catalog of economic theory so that established methods and literatures could be used to quickly obtain reliable answers,[2] and
  • using already-published CEA reports to facilitate accurate and consistent execution of new tasks.

Much public policy discussion is devoid of economics and thereby obscures evaluation of current Federal policies.  Invariably these were policy areas where EOP principals most valued CEA’s work.  Socialism and Medicare for All (see also their updates in the in 2019 Economic Report of the President) are good examples where CEA was able to initiate or at least bend the conversation by assembling results from economic research.

Probably the best tool is the empirical counterpart of the supply and demand picture.  That is, measuring both price and quantity using the best methods available in the academic literature.  CEA is desperately needed to perform this function.  The current CEA already has 11 instances of those pictures in its Economic Reports of the President, as compared to only 8 for all of the combined other Presidential Administrations in U.S. history.[3]  This may seem to be a trivial enterprise, but the “best” in our profession fail at it regularly (see below and here and here).

With the labor market, for example, the public policy community is familiar with methods for measuring quantities (employment, hours, unemployment etc.) but unfamiliar with measuring prices (i.e., wages).  CEA measured wages with attention to composition issues, human capital, taxes, etc., and thereby helped change the factually incorrect narrative that real wages were “stagnating.”

Prescription drugs are another important example.  The typical narrative (including at HHS, which sees itself as the drug-price regulator) was that prescription drug prices were increasing faster than general inflation, as they had for decades. But, with attention to the institutional details of the supply chain, CEA found that the best measures showed an increase in quantities of prescriptions at the same time that prices have been falling over the past two years or so.

With the decades-long opioid epidemic, public policy discussions still ignore prices altogether: another opportunity where CEA made a valuable contribution by following standard economic practice.  (More on this tragic and ongoing story in later posts).

CEA’s trade team executed these methods repeatedly as various tariff rates were changed.

CEA took a similar approach to the Affordable Care Act, where we found that Trump Administration reforms were significantly reducing health insurance prices measured according to a cost of living index.  CEA’s public report on this issue presented the dual of the cost-of-living index, namely cost-benefit analysis.  This spawned many other regulatory impact analyses by CEA that laid the foundation for its report on the economic effects of the Trump Administration’s deregulation portfolio.  I worked closely with Don Kenkel (now the CEA Chief Economist) on these issues throughout the year.

One Economics
I looked carefully and critically at the headline calculations of all CEA reports released during the year (beyond the 12 reports or ERP sections that I edited), as well as the corporate tax reports released at the end of 2017.  The essential methods and sources for these calculations proved to be consistent across reports including, but not limited to, tax, immigration, health, socialism, and regulation.

A three-good version of the neoclassical growth model with taxes was another workhorse that links industry-specific analysis with macroeconomic analysis.  The three goods are leisure and two consumption goods (but see below).  CEA used that to look at Medicare for All (health consumption vs other consumption), tariffs (tariffed goods vs other goods), and dozens of regulations (regulated good vs other consumption goods). We also used it to look at business tax reform and investment regulation, where there is just one consumption good but two capital goods (corporate vs noncorporate).  We typically focused on the steady state and took a broad view of “taxation” that encompasses other market distortions.  We derived quantitative rules of thumb (such as the well-known marginal excess burden of taxation) for the effects of a single sector’s distortion on the aggregate supplies of labor and capital.

I frequently used automated economic reasoning (run with Mathematica with an add-on from the web) to confirm, refine, and extend our reasoning about the three-good model and other applications of logic and economic theory that went beyond the simplest supply-and-demand framework.  I will be adding these examples (which embed hundreds of automatically assembled and decided Tarski formulas) to the library of SAT/quantifier-elimination applications that I maintain with computer scientists.

Working in a Large Organization
The Executive Office of the President (EOP, of which CEA is one of several components), not to mention the entire Federal government, is a large organization with components far more inter-reliant than the components of a university. Professors joining CEA need to be aware that staff meetings are critical for keeping information flowing to the parts of the EOP that need it.  As you attend EOP meetings in your subject area, be prepared to share headlines with the rest of the CEA staff at the staff meetings and listen to others to assess who in CEA or outside CEA might be of help for the next task that arrives.

The EOP has a staff hierarchy for the same reasons, although many employees adhere to it so rigidly that there is a role for organizational entrepreneurs who cross some of those boundaries, which is a role I took on in much of my work.  Otherwise meetings of principals (a.k.a., cabinet-level positions) and deputies (report directly to principals) may not have anyone present who knows first hand the details of the meetings’ subject.  I presume that this is also a problem in large private organizations, but more acute in government where leaks are more of a constraint on determining who is invited to meetings (see, e.g., this memoir’s discussion of leaks in the Obama White House).

CEA Chairman Hassett and COS DJ Nordquist assembled an amazing team of economists.  I also worked closely with six other components of the Federal government: the Office of Management and Budget (OMB), the National Economic Council (NEC), the Office of Information and Regulatory Affairs (OIRA, technically part of OMB), the Domestic Policy Council (DPC), the Department of Labor (DOL), and the Department of Health and Human Services (HHS).

Every day working in the EOP was a pleasure (Mick Mulvaney = Phil Jackson; the Kelly-Collins analogy is imperfect). It is difficult saying goodbye to so many excellent EOP colleagues but by design CEA has high turnover so that others outside can come in with fresh ideas and energy.

[In returning to the University of Chicago, I return to publishing under my own name and take sole responsibility for the analysis and conclusions.  As such, the Federal government is not consulted on my writings.]






[1] E.g., measuring wage growth properly, exposing misconceptions about single-payer systems, highlighting the consequences of deregulation from autos to prescription drugs.
[2] A forthcoming blog post will list more than 50 results from Chicago Price Theory (forthcoming, Princeton University Press) that appear in public CEA reports.
[3] CEA, which is tasked with preparing the Economic Report of the President, dates back “only” to 1946.