In a study of the Theomatic factor 90 relating to references to either one of the sons in the story of the Prodigal Son (Luke 15), the author states in the beginning of section 7: "In the most conservative way possible, we need to find out what the actual odds are for this event occurring." (p.33) The author publishes that the event being considered occurred as follows:
The author observed 46 successes (hits, The author, in calculating the p-values ( WLA of the successful phrases (hits) obtained from it, WLA
. This claim is made upon an assumption that the expected value of _{H}WLA, _{S}M, is not more than _{S}M, the expected value of _{H}WLA._{H}
Assumption: < M_{H}To support this claim, the author states (in uncorrected figures): "Theomatics discovered a total of 46 hits of 2.37 Lk15, p.37) He states further: "Calculating the probability
according to the length of the phrase -- matching the WLA of both the Theomatic hits and the 434 number pool -- is a totally fair, honest, and objective way to figure the p-value." (p.
44) Evidently, no justification is actually proposed by the author; he merely states that he is correct. A formal analysis of the validity of this assumption is warranted. First, we will
complete our explanation of the author's methodology.It so happens that WLA, contrary to the way the author expects
Theomatics to occur normally in such an experiment, so his procedure is simply to remove some of the longer (4-word) phrases from the 4-word sample, regardless whether they were successful trials or not, until _{H}WLA
equals _{S}WLA. Eliminating all of the 4-word phrases reduces the sample from 765 to 434 giving 2.413 for _{H}WLA. Since this is yet
greater than his target of 2.37, he continues removing 3-word phrases from the sample until it yields the appropriate _{S}WLA (p. 35). The author considers this "par for the course," and
states that it is now possible to test his hypothesis that God designed Theomatics (p. 35, emphasis his): "This figure of 434 now constitutes the number pool -- it will now give
us an accurate comparison of Theomatics against the null hypothesis -- and enable us to come up with an accurate p factor."
The author follows this activity by recalculating the p-values without using the
Though the (uncorrected) odds for the 4-word phrases, as noted above, are 1 in 3, the author claims that the p-value for the 4-word phrases without considering The (uncorrected) final p-values ( P) without _{C}WLA, with odds 1:N, and general significance O
(being the average number of random trials needed to get this kind of result if Theomatics does not exist), obtained from the p-values of the hits (P) and clustering (_{H}P), the number of hits (
_{C}H), the phrase sums (S), along with the actual clustering results (0, 1, 2) for each phrase length, are as follows:
In concluding his analysis, the author uses this
In light of the above errors in analysis, the use of the
As stated earlier, the author's claim that it is valid to adjust the sample size based on The reasoning is implied from the very procedure the author employs. Reducing the sample size to something the author would "expect" to be a "normal" or "reasonable" sample size implies that the sample size observed in the experiment is somehow known to be unusual or inappropriate. Instead of discarding the experimental context as inappropriate to display the evident statistical significance of Theomatics, and looking for another scenario that fits more nicely with his expectation, the author conveniently prefers to adjust the sample size. On average, he expects that the
WLA), it should be smaller, so, naturally, he feels the sample obtained in this
particular Theomatic experiment is highly unusual and can be adjusted without adversely affecting the validity of the resulting conclusion. This implies he feels the expected _{H}WLA of the sample (M
) no larger than the expected _{S}WLA of the hits (M), so he is able to justify a presumed sample size that "could have" occurred having a _{H}WLA that is upper-bounded by the _{S}WLA of the hits. Being "conservative" so as not to corrupt his conclusion, he only reduces the sample size so as to obtain an _{H}
equivalent WLA to that of the hits, not making the sample WLA
arbitrarily smaller than this _{S}WLAbound like he expects it to be. _{H }This reasoning clearly implies the author feels that Theomatic events tend to occur more
frequently in longer phrases than in shorter ones, resulting in a WLA of the sample on average. This comprises the author's "justification" to use the _{S}WLA
to reduce the sample size. A summary review of the logic follows:
- Requiring
**WLA**to be_{S }__<__**WLA**is only valid if_{H}**M**_{S}__<__**M**._{H} **M**_{S}__<__**M**is only true if longer phrases tend to give more hits._{H}
The necessary question is this: Is the author's assumption valid? To answer this question, we simply observe, as the author notes, that the (uncorrected) data in this particular experiment indicate otherwise:
Clearly, in each case the WLA of the
hits. The author therefore reduces the sample size in each case to something he expects "could have" been observed, and thus obtains incredible statistical significance for the
Theomatic phenomenon. Again, the author's entire analysis stands or falls based upon this reasoning. Further, we observe the following: the author plainly emphasizes the phrases,
obtaining a higher percentage of hits from them: "The one major factor that makes Theomatics stand tall -- is the shortness of the phrases that produce the Theomatic hits."
(p.35) Again (p. 44), "The real power of Theomatics is the shortness and explicitness of the theomatic phrases and hits. After all, if some sort of Intelligence factor is at work here, then
we would expect short and explicit -- one, two, and three word phrases, to produce the most significant results. smallerIt is when we look at that aspect, that the p-values literally go ballistic
. This is true across the board -- from hundreds of individual studies in my files consisting of thousands of features. The reason for this, is that as one expands outwardly, the patterns dissipate."
In making such statements, the author implies that WLA(or that _{S }M> _{S }M
); the smaller the phrase length being considered, the higher the proportion of hits and the resulting statistical significance of the results. This implies that the percent proportion of hits (_{H}%
) will be larger for the smaller phrases than for the larger ones, so the mean of WLAis less than the mean of _{H }WLA. We can easily observe that this conclusion appears valid here._{S}
The larger the allowed phrase length, the larger the actual difference WLA performance when moving toward longer phrases appears correct.
What is observed in the experiment appears to be appropriate to use as a general assumption about the behavior of Theomatics, being totally consistent with thousands of instances observed by the author.
These facts flatly contradict the author's implicit assumption that WLA. In fact, if _{H}WLA ever _{S}did fall below WLA, this would certainly be an oddity... not the norm... based upon the author's own data and claims._{H}
In addition to granting him the sensational claim of unbelievable statistical odds, the author's
sample reduction procedure conveniently permits him to consider successful trials in his calculation of the p-value that are
The author's reduction of the sample based on
In order to determine the probability of the Theomatic phenomenon observed by the author, one must correct the
The chart shows the phrase length 1:N(_{H }N/ _{H = }1P) the probability of the cluster distribution _{H}P, the total probability of the hits and clustering _{C
}P (P = P X _{H}P), the final odds _{C}1:N(_{ }N = 1/ P), and the representative statistic O
(being the average number of random trials needed to get this kind of result if Theomatics does not exist).The correct way way to determine the statistical significance of the results of any experiment
is to look carefully at what actually occurred in the experiment. Since the author has formally stated that the experiment is to consider results for all phrases of 4 words or less (p.22), the first The author's published claim of odds of 1 in 261 million corresponds to the final p-value for
the 3-word phrases, reducing to The last result, that of the 2-word phrases, is similarly insignificant, as observed in extensive
The above results are all certainly well within what might be expected in a random context: the null hypothesis cannot be rejected... doing so is not even a consideration. The correct conclusion to draw in this Luke 15 analysis is that no Theomatic significance is evident at all. No other conclusion may be deemed correct, much less "conservative." |