[Adapt] [Seminar] Supplement for some questions

[张淳皓]

Hi Adapters,

In the talk about chain of thought this afternoon, there are some details that are not clear. Here are the supplements.

1. For the promptings used in the accuracy test, it is narrated in the paper like this: “As most of the datasets only have an evaluation split, we manually composed a single set of eight generic few-shot exemplars with chains of thought for promptingFigure 1 (bottom) shows one chain of thought exemplar, and the full set of exemplars is given in Table 20 in the Appendix. (These exemplars did not undergo prompt engineering, and we assess robustness to alternative exemplars and chains of thought in Section 3.4.) To investigate whether chain of thought prompting in this form can successfully elicit successful reasoning across a wide range of math word problems, we used this single set of eight chain of thought exemplars for all six datasets.” Table 20 is shown below. And Section3.4 is the chapter about ablation test, which changes the annotator, the examplar set and the propmting style in the experiment, and this part has been introduced in the talk.


For the symbolic reasoning tasks, it is narrated like this: “We again manually compose chains of thought for the few-shot exemplars for each task, which are shown in Table 4 on the next page.” Table 4 is shown below. The full prompts with all examplars are also 8-pair sets, if you are interested in it, you can look for it in the paper.


For commonsence reasoning, it is narrated like this: “For CommonsenseQA and StrategyQA, we randomly selected eight examples from the training set and manually composed chains of thought for them to use as few-shot exemplars. The two BIG-bench tasks do not have training sets, so we selected the first ten examples as exemplars in the evaluation set as few-shot exemplars and report numbers on the rest of the evaluation set.”
Here is a brief example: 


2. About the publication of the papers. 
a) The main paper which introduces CoT: “Chain of Thought Prompting Elicits Reasoning in Large Language Models”. This paper has not been accepted  yet, but it has been cited 99 times till now. This paper is currently updating, the newest version is V5(Mon, 10 Oct 2022 20:21:17 UTC) which may contains more experiment. My talk is based on V2(Wed, 6 Apr 2022 03:51:50 UTC).
b) The paper which introduces zero-shot-CoT is ” Large Language Models are Zero-Shot Reasoners”. This paper has not been accepted also, with citation 44. It is also in an update period. My talk is based on the latest  version [v3] Sun, 2 Oct 2022 07:12:50 UTC
c) The paper which introduces auto-CoT is “Automatic Chain of Thought Prompting in Large Language Models”. This paper also has not been accepted yet. My talk is based on the latest version [v1] Fri, 7 Oct 2022 12:28:21 UTC

Hope you enjot it!

Best regards, 
Chunhao


从 Windows 版邮件发送

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.sjtu.edu.cn/pipermail/adapt/attachments/20221102/6a6c27d8/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: F5DE1BD07B2A4F48957F062E740E21CD.png
Type: image/png
Size: 140574 bytes
Desc: not available
URL: <http://cs.sjtu.edu.cn/pipermail/adapt/attachments/20221102/6a6c27d8/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 57454C6B3FC840CFBDA9FD484DCC6B0F.png
Type: image/png
Size: 208600 bytes
Desc: not available
URL: <http://cs.sjtu.edu.cn/pipermail/adapt/attachments/20221102/6a6c27d8/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DE2FB1E546B84CDD856B07503F73A62B.png
Type: image/png
Size: 292458 bytes
Desc: not available
URL: <http://cs.sjtu.edu.cn/pipermail/adapt/attachments/20221102/6a6c27d8/attachment-0005.png>