[[{“value”:”Almost one year ago, Steve Landsburg tried GPT-4 on one of his exams. It failed, badly. I tried out some of the same questions on Claude 3 Opus, by many accounts now the leading AI. It failed, badly. Steve’s exams are very clever. They aren’t technically difficult but they are tricky in the sense that
The post Claude 3 Opus Also Fails Steve Landsburg’s Economics Exam appeared first on Marginal REVOLUTION.”}]]
Almost one year ago, Steve Landsburg tried GPT-4 on one of his exams. It failed, badly. I tried out some of the same questions on Claude 3 Opus, by many accounts now the leading AI. It failed, badly. Steve’s exams are very clever. They aren’t technically difficult but they are tricky in the sense that the answer you would get by reading the textbook is almost certainly wrong. Not because the textbook is wrong but because you have failed to really understand. Here’s two examples:
Q. Apples are provided by a competitive industry. Pears are provided by a monopolist. Coincidentally, they sell at the same price. You are hungry, and would be equally happy with an apple or a pear. If you care about conserving societal resources, which should you buy?
A. If you care about conserving societal resources, you should buy the apple provided by the competitive industry rather than the pear provided by the monopolist. Here’s why:
In a competitive industry, firms are price takers and must operate efficiently to survive. They produce at the point where marginal cost equals price. This means they are producing at the socially optimal level, using society’s resources efficiently.
In contrast, a monopolist has market power and can set prices above marginal cost. They produce less than the socially optimal quantity in order to drive up prices and maximize profits. This results in a deadweight loss to society – there are potential gains from trade that are not being realized.
Even though the apple and pear are selling at the same price to you as the end consumer, the way that price was arrived at is very different:
The competitive apple price reflects the true marginal cost of production
The monopoly pear price is above the marginal cost of production
By buying the competitively produced apple, more of your money is going towards covering the actual costs of production. By buying the monopoly pear, some of your money is going towards monopoly profits rather than productive use of resources.
Of course, this is a simplification and there are other considerations, like quality, variety, innovation incentives, etc. But in general, competitive industries lead to more efficient outcomes than monopolies. So if your goal is conserving societal resources, buying from competitive producers is usually the better choice. The apple is the way to go in this case.
Claude has read the textbook. It knows that competitive industries are good and that monopolies are bad and it can explain why. Everything it says is correct and yet Claude gets the answer wrong. Do you see why? Pause and check your solution.
In Claude’s defense, I was able to prompt Claude via the Socratic method to the correct answer but it took several rounds of questioning:
The social cost of producing a good is the opportunity cost of the resources used in its production. In a competitive market, price equals marginal cost, so the price reflects the social cost of production. However, in a monopoly market, the price is higher than the marginal cost, so the price does not reflect the social cost.
Given:
The price of apples (Pa) equals the price of pears (Pp)
Apples are produced competitively, so Pa = MCa
Pears are produced by a monopolist, so Pp > MCp
Therefore, MCa = Pa = Pp > MCp
The marginal cost of producing a pear (MCp) is lower than the marginal cost of producing an apple (MCa).
From society’s perspective, the cost of producing an additional pear is lower than the cost of producing an additional apple. The resources needed to produce a pear (MCp) are less than the resources needed to produce an apple (MCa).
Therefore, if you want to minimize the societal cost of your fruit consumption, you should choose the pear. By consuming a pear, you are using up fewer societal resources than if you consume an apple.
The monopoly pricing of the pear is a separate issue from the social cost of production. The higher price of the pear reflects a transfer from consumers to the monopoly producer, but it does not affect the underlying cost of the resources used to produce the pear.
So in conclusion, if your goal is to minimize the cost to society of your fruit consumption, you should buy the monopoly produced pear, as it has a lower marginal cost of production than the competitively produced apple, despite being sold at the same price.
A good answer and very clearly written with nice notation so Claude can “learn” within a session. Still, this isn’t very useful if you don’t already know the answer. Ok, one more.
Q. The town of Mayberry is thinking of expanding its airport. One problem with the expansion is that it would result in more airplane noise. For people who live near the airport, hearing that noise would cause as much unpleasantness as the collective loss of $100,000 every year. True or False: When Mayberry weighs the costs and benefits of its airport expansion, that $100,000 should count as a cost
A. True, the $100,000 in collective unpleasantness experienced by residents near the airport due to increased noise should be counted as a cost when weighing the costs and benefits of the airport expansion. Here’s why:
Externalities: The airplane noise is a negative externality – a cost imposed on third parties (the nearby residents) who are not directly involved in the economic activity (the airport expansion). Externalities lead to market inefficiencies because the full costs are not being accounted for by the decision-makers.
….a bunch more similar reasoning, all correct.
Once again, Claude gives a textbook answer and yet the answer is wrong. In a way this is more surprising than failing the first question because Coase got a Nobel prize for giving the correct answer (and it’s not, bargaining will solve the externality if that is what you are thinking.) I will let commentators work this one out. Do read the question carefully, it’s subtle. Again with Socratic prompting Claude got there eventually.
I wouldn’t underestimate the GPTs, textbook answers can be of great value, especially when the textbooks are long and diverse. Medical diagnostics, legal reasoning and coding are ideal tasks for GPTs. Economic reasoning less so, at this stage.
Congratulations to Steve Landsburg for the excellent questions.
The post Claude 3 Opus Also Fails Steve Landsburg’s Economics Exam appeared first on Marginal REVOLUTION.
Uncategorized
Leave a Reply