Machine Learning (ML) Platforms Can Contradict Dairy Scientists and Feed Firm Websites Regarding Dairy Cattle Performance from Feeding Seaweed Supplements

Artificial intelligence through machine learning applications (hereafter ML) is emerging as a tool in evaluating, comparing, and going beyond human capabilities and knowledge. Despite the potential benefits of ML as a resource for answering scientific questions, such as those included in our analysis, some characteristics of ML-generated responses limit the interpretations of these results—such as ML “hallucinations”—of which researchers should be aware (McIntosh et al., 2023). Nonetheless, ML is quickly becoming a source for authoritative and trusted information on many topics (Knight, 2024; McIntosh et al., 2024), as university-based and other more rigorous research may be behind paywalls or otherwise difficult to access and as pay-to-play journals proliferate. Therefore, it is useful to conduct analyses comparing ML-generated information to traditionally trusted information sources, such as scientists’ observations, and to self-interested commercial information available to the public.

There has been growing interest in the dairy industry for algal feed supplements (AFS), such as Asparagopsis taxiformis, to be used in dairy cattle feed as an effective means of improving cattle health and productivity and reducing methane emissions (Moen, 2024; Tynan et al., 2023). Livestock feed company websites selling AFS list numerous health and environmental benefits from utilizing their dietary supplement in cattle feed. However, it is possible that scientific evidence and support among credentialed experts do not match the claims made on company websites or support for the findings produced by ML platforms. This paper compares the results generated by three commonly used ML platforms to survey results from 100 dairy scientists attending a Cornell Dairy Herd Health and Nutrition Conference in response to questions on the effectiveness of AFS as a supplement to improve herd health and productivity outcomes, and to claims of livestock feed firms on theirwebsites. Findings suggest that while ML may presenta viable resource for information, it should not be the primary source for extracting reliable information. No single source of information regarding seaweed feed supplements for dairy cattle should be the primary source; according to our findings, all forms of information may have some weaknesses.

Table 1. Claims of Dairy Nutritionists
Compared to Machine Learning (ML)
Responses and Firm Claims

NA indicates the ML Platform provided ambiguous
responses

Methods

Web Search and Survey

An initial search for livestock and animal dietary supplement companies was conducted using the Google search engine using combinations of the following key words: seaweed, kelp, algae, livestock supplement, feed supplement, total mixed ration, and feed companies. Google search engine operators were utilized to parse results of feed company websites from news bulletins and unaffiliated website posts. A list of prospective seaweed supplement companies was compiled, and a content analysis was conducted on the substance of the company website. Information on product details was used to create a list of claims made by the companies on their websites or on the packaging of company products. Information for eight firms was tabulated in an Excel spreadsheet.

Data for scientific support among 100 dairy nutritionists were compiled from a Qualtrics survey distributed at a dairy nutrition conference in October 2022, prior to the web search of feed companies. A list of possible effects from feeding AFS was developed from a review of the scientific literature and integrated into the survey. Dairy nutritionists were asked whether the effects listed had “strong scientific support,” “some scientific support,” or “little to no scientific support.” The results of the survey were compiled into an Excel file. Quantitative measures for presence of firm claims were coded as “yes” versus “no” (see Table 1). Each claim made by seaweed supplement companies was compared to the corresponding survey question to determine the validity of the firm claims according to the surveyed dairy nutritionists.

ML Platforms

Questions developed from the survey of dairy nutritionists were created and posed to ML platforms (see Table 1). Questions were phrased in a way that was straightforward and easily comprehensible to the ML platform to ensure clarity of ML responses. The three ML platforms used were ChatGPT, Caktus AI, and Jasper. Questions were formatted to generate a binary “yes/no” response and prompted each ML platform to include an explanation for the selected binary answer choice. For the question regarding “somatic cell counts,”

Jasper replied that it could not answer the question withavailable information. Caktus AI replied the same to the question on “milk fat content.” When this occurred, the query was repeated. In both cases Jasper and Caktus AI answered the follow-up questions with a “yes” or “no.” To test the validity of the responses, we repeated the query, and the opposite answer was provided. In Table 1, we coded these responses as “NA.”Following the series of questions and answers using three ML platforms, each response was compared to the scientific support determined by dairy nutritionists and scientists via the survey as well as whether feed companies made the claim on their websites (see Table 1). Using the reference point of at least 60% of dairy nutritionists indicating some or strong scientific support for a claim or function, we can assess the degree of validity of the ML responses and firm claims. This reference point is arbitrary, so readers may conduct a sensitivity analysis using higher or lower percentages depending on their desired confidence levels.

Findings

• Treats pink-eye infection: Each of the three ML platforms used in our analysis responded with “no” to this claim. The ML platforms suggested that there is little or no scientific support that seaweed supplements are useful in treating pink-eye infections in cattle. This finding matches the claim of little to no scientific support (79.59%) made by scientists and nutritionists. No claim of pink-eye infection treatment was made on reviewed company websites.
• Provides a source of vitamins: Each ML model generated an answer affirming scientific support for seaweed supplements as a good source of vitamins for dairy livestock. Among dairy scientists and nutritionists, 59.65% of survey responses indicated some scientific support for the use of seaweed supplements as a source of vitamins. Half (50%) of feed companies made this claim.
• Provides a source of minerals: Each of the ML models included replied “yes” when asked whether seaweed supplements are a good source of minerals for dairy livestock. Among nutritionists and scientists included in our survey analysis, there were claims for some scientific support (48.39%) as well as strong scientific support (43.55%) that seaweed supplements are a good source of minerals for livestock and dairy cattle. Half (50%) of feed companies made this claim.
• Provides a source of iodine: There is strong scientific support for the use of seaweed supplements as a good source of iodine for dairy cattle among dairy nutritionists and scientists who responded to the survey. In addition, all three ML apps generated a response affirming this claim. One quarter (25%) of feed companies made this claim.
• Reduces somatic cell counts in milk: Among dairy nutritionists, there was little scientific support for the claim that seaweed supplements for dairy cattle are effective in reducing somatic cell counts in milk. Caktus AI had difficulty answering the question definitively. Jasper responded with “yes,” suggesting some level of scientific support. ChatGPT responded with “no,” suggesting there is no scientific support for this claim. Only 12.5% of feed companies made this claim.
• Reduces methane emissions: All of the ML platforms responded yes when asked whether algae feed supplements reduce methane emissions. The scientists and nutritionists asked about this claim affirmed this claim as having some or strong scientific support 90.63% of the time. Only 37.5% of feed companies made this claim.
• Increases weight gain: Among experts who responded to the survey, there was a low level of some strong scientific support for the use of algae feed supplements in increasing weight gain among dairy cattle. However, all three ML models responded with answers indicating some scientific support for this claim. One-quarter (25%) of feed companies made this claim.
• Increases milk yield: Among dairy nutritionists, a little more than one-third believed there was either some support or strong scientific support for this claim. All three ML platforms answered “yes,” highlighting an inconsistency among ML-generated responses and scientific support from survey respondents. In addition, 37.5% of feed companies made this claim.
• Increases milk fat content: About 68% of surveyed scientists indicated there was little or no scientific support for this claim. As reported earlier, Caktus AI responded ambiguously. Jasper and ChatGPT responded with “yes,” indicating at least some level of scientific support. There is an inconsistent response across survey respondents and ML models regarding the role of seaweed supplements in increasing milk fat content of dairy cattle. One-quarter (25%) of feed companies made this claim.
• Improves fatty acid profile of milk: All three ML models replied “yes” when asked about the effect of seaweed supplements on the fatty acid profile of milk. However, based on the survey data, 63.15% of dairy nutritionists found there was little to no scientific support for this claim. This finding highlights the possible limitations of ML-generated responses in producing a measurement of the scientific support for claims made by algae feed supplement companies. No feed companies made this claim.
• Improves calf health: There is an inconsistent response in the level of scientific support indicated by ML platforms compared to dairy nutritionists and scientists for this claim. Although all three ML platforms replied “yes” to this claim, 66% of survey respondents indicated that the effectiveness of algae feed supplements in improving calf health is backed by little to no scientific support. No firms made this claim.
• Helps with fly control: Across all three ML models, responses matched those of survey respondent findings. All three models responded with “no” when asked whether algae feed supplements were effective for fly control among dairy cattle. 89.80% of dairy nutritionists reported that this claim is backed by little or no scientific support. Only 12.5% of feed companies made this claim.
• Helps with cow’s fertility/reproductive issues: Only 5.77% of surveyed dairy nutritionists indicated strong scientific support for this claim. However, all three ML models responded with “yes,” indicating scientific support for the claim. These findings highlight inconsistency in the responses produced by ML platforms compared to survey respondents. Only 25% of feed companies made this claim.
• Enhances immune function: Among survey respondents, 6.9% agreed there was strong scientific support for the claim that algae feed supplements are effective in enhancing immune function of dairy cattle, and 34.48% found this claim is backed by “some” scientific support. All three ML models responded with “yes.” In addition, 62.5% of feed companies made this claim. It may be that the ML models drew from industry websites for their conclusions.

Discussions and Conclusions

We assume for this analysis that the dairy nutritionists’ survey findings are the yardstick for establishing the validity of firm claims and ML-generated answers. Firms are interested in selling products and services and will market product claims they deem most attractive to potential customers rather than claims most supported by dairy scientists. ML platforms scrape information from publicly available sources and do not innately contain expertise on any topic. Rather, the findings they produce come from human-produced knowledge of various types, from any accessible locations. ML platforms “answer” questions by using available knowledge and cannot by themselves assess the accuracy of the knowledge produced. Rather, the platform assesses the degree of agreement or the dominance of findings from the sources it accesses and uses this level of agreement or dominance to produce a definitive or (in some cases, as evidenced by our findings) qualified answers both based on human choices about what has relative or absolute value (Bronson, 2022; McIntosh et al., 2023). Work on ML platforms to improve their performance is ongoing and advances are being made to increase the efficiency and effectiveness of the platforms by “training” them to emphasize certain sources of information, such as textbooks, over others. This could improve the performance of ML platforms but also introduce bias (Bronson, 2022; Knight, 2024).

Our findings indicate that only claims about algae supplements providing sources of vitamins, minerals, and iodine and reducing methane emissions reach the threshold of 60% agreement among dairy nutritionists. Half (50%) of firms made claims about providing vitamins and minerals, while only 25% and 37.5% made claims about iodine and methane reduction, respectively. Again, these data reflect what firms wish to sell, rather than scientific support. The ML platforms were unanimous in their agreement with the nutritionists on these findings. However, the only unanimous “no” finding from the ML platforms was for “helps with fly control,” which about 90% of nutritionists indicated had little or no scientific support. Only12.5% of firms claimed that function. This is the point at which most agreement, positive and negative, ends among the three information sources.

Our findings indicate that ML may currently be useful as a supplementary source of information or as a tool for assessing scientific claims; but ML probably should not be used as a primary source of information on scientific research. It is vital that researchers and educators emphasize to various stakeholders (students, outreach professionals, industry partners, and policy makers) the importance of gathering, triangulating, and interpreting information from expert sources. Efforts to make credible and reliable research more accessible (e.g., open access but not pay-to-play) and readable to laypersons becomes even more important (see Bronson, 2022).

We argue that each format (survey, ML platform, and company website) included in our analysis contains some level of discrepancy and cannot be relied on completely as a single source of information, though it is generally accepted that ChatGPT is the gold standard of ML applications (Knight, 2024). Survey respondents may interpret the meaning of questions differently from each other. Company websites marketing seaweed feed supplements may not be reliable as a primary source of information, as their primary interest may be to encourage dairy farmers to use their products. We emphasize the importance of relying on multiple sources of information and cross-checking them to increase the likelihood of finding reliable information. Also, when using machine learning applications, it is also recommended to pose the questions to them more than once.

For More Information

Bronson, K. 2022. The Immaculate Conception of Data: Agribusiness, Activists, and Their Shared Politics of the Future. McGill-Queens University Press.

Knight, W. 2024. “Inside the Creation of the World’s Most Powerful Open-Source AI Model” Wired, March.

McIntosh, T.R., T. Liu, T. Susnjak, P. Watters, A. Ng, and M.N. Halgamuge. 2023. “A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination.” IEEE Transactions on Artificial Intelligence.

McIntosh, T.R., T. Susnjak, T. Liu, P. Watters, and M.N. Halgamuge. 2024. “Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence.” ArXiv, abs/2402.09880.

Moen, A. 2024. “Stakeholder Opinions of Seaweed Supplements for Dairy Cows.” Project paper for Master of Science degree, Food Systems Graduate Program, University of Vermont.

Tynan, M.K., M.C. Bryant, R. Welsh, and S.L. Greenwood. 2023. “Preliminary Findings of Northeast Organic and

Conventional Dairy Farmers’ Perception of Benefits and Challenges in Feeding Algae.” Renewable Agriculture and Food     Systems 38: e23.