Setting up an Effective Farm Trial

Lee Whittington and LeAnn Johnston of Prairie Swine Research Centre help to explain why farm trial results are not always what we expect and how to improve your odds of success in future trials.
calendar icon 15 December 2010
clock icon 11 minute read

The pork industry is blessed with a number of innovators. All pig farms participate to a greater or lesser extent in experimenting to improve productivity, reduce costs or make management easier. Sometimes the results of such experimentation are as expected. Often however, the time, effort and money required to innovate and experiment results in more questions than answers and does not lead to an innovation being adopted on the farm as part of a new long-term management strategy. This paper will help to explain why results are not always what we expect and how to improve your odds of success in future on-farm trials.


Why You Should Do an On-Farm Trial

Many new technologies come with all the work completed including, the expected change, the confidence in the statistical approach used for analysis and the economic benefit of implementation. So why would you want to take on organizing an on-farm test yourself?

Typically the top reasons given by innovative producers to test something on the farm are:

  1. "My situation is different and I don't believe just because it works elsewhere it will work on my farm."
  2. "The idea is mine and I don't know of anyone else that has tried it so I need to find out for myself."
  3. "I read/heard about this idea from another country and think it might work here."

No two barns are exactly alike, even though they may be designed to operate the same; the people factor adds a unique component that makes a significant difference on the outcome of many practices or products used. Due to the differences between barns there is reason to believe that an on-farm trial would produce a more reliable result than information gathered on other farms.

Most on-farm trials have an economic decision they are trying to address. What is the benefit we are hoping to achieve and what is the cost to achieve it? The cost is often easy to find (example, feed cost per kg, or drug cost per dose) but the performance result in the barn, the statistically tested part, is much more difficult. Not all studies result in a statistically significant conclusion, the product used may not have had a significant result or the experimental test may not be sensitive enough to detect small improvement. If the results are unclear and other information is required to make the decision, perhaps the trial was not designed properly and cannot answer the question you ask.

Why On-Farm Tests Often Fail

The reasons are many but break down into five main categories (modified from Deen, 2009):

  1. The trial design has multiple outcomes. For example, a small improvement in average daily gain, feed efficiency and improvement in one or two carcass features. Do the combined improvements in each area justify the intervention? When the improvement in feed efficiency alone is enough to justify the intervention, adopt the new technology. What if only small gains are made in each area? Likely the study needs to be redesigned to include many more pigs to identify small gains.

  2. If the item being tested has a history of performance under other circumstances (even in species other than pigs) that gives us a clue as to how big a difference we are seeking to measure. What is the variation located within the test herd prior to the test? This knowledge of health status, quality of pig, and variation in key factors such as daily gain are the inherent background 'noise' within the barn. We need to account for this 'noise' to ensure our test can be interpreted.

  3. Danger of believing your test analysis when actually it is worthless. Statistically a negative result of a single study cannot be interpreted as supporting a negative conclusion. This really only means that we are not satisfied ‘beyond a doubt’ that the product performed as expected.

  4. "A micrometer question is often measured with a 'yard-stick'. The scale of the economic benefit required to justify an intervention is much smaller than the capability of the statistical test created." (Deen, 2009). There is so much variation already within the population that it would take a large number of data points (pens of pigs) to sort out the effect of the intervention.

  5. Data collection or the 'people factor'. In order to ensure the trial runs properly, you must get stockpeople on side, arrange additional help to collect information, not fudge data when it is lost, have a backup plan when people unexpectedly leave, have the right measurement tools, and check the intervention procedures regularly to make sure they continue to operate as expected over the trial period. Make sure all the feed is made and tested prior to the start of the test. There are whole lists of other factors such as ventilation error or power failure, out of water events or disease outbreak during the test period, effect of weather patterns, stable parity distribution etc., that may affect the trial.

A sidebar note to the people factor is 'when you start to measure something, it generally begins to improve' (Krueger, 2009). For example, when daily feeder and waterer checks are consistently made and acted on, the results of all groups will likely improve because the 'normal' out-of-feed events do not occur during the test period.

Three 'typical' case studies are shown on page three that demonstrate how easily a simple on-farm trial can be negatively impacted.

How to Avoid Common Pitfalls


  1. Do the math first. How many groups of pigs will it take to have confidence (sufficient power in the statistical test) that the difference I am trying to measure can be assessed from my trial design? This can be the subject of a graduate course but if you have the patience and interest some free software on line can help such as Winepiscope from Europe [click here].

  2. Calculate the likely financial benefit of a successful trial. Will it be sufficient to justify the work and cost of conducting the trial? Most businesses will want a 3:1 return on new investment because they realize that biological systems don't always behave as predicted all the time, so can I expect a $3 return from a $1 intervention?

  3. Get the people involved. Everyone that plays a role needs to be aware of the cost and the large risk of failure to complete the trial as designed.

  4. Use a checklist like the one created by PSC to plan your successful trial implementation.

Conducting On-Farm Trials – The Good, The Bad, and The Ugly

Case Study 1 - Feeding Study


You have a new product that you would like for a good customer to try. You have controlled data and good experiences on other farms that show the same gains on less feed when this product is used in the late finishing period. You have talked to the customer and the barn crew and they seem excited about doing the project. The plans are in place and scales have been checked. The trial starts and everything seems to go well. But at the end of the trial there is no difference between treatments. You talk to the crew at the farm and check the data but everything seems to be in order and nothing such as water outages or illness has occurred. A couple of months after the trial the feed-mill operator calls you to see when you are going to use the feed additive that is still taking up space in his warehouse.

What could you have done to prevent this problem?

While you worked closely with the farm crew to see that everything was correctly done there. You needed to work more closely with the crew at the feed-mill to see that they had everything (diets, ingredient) in place to get the diets made correctly and delivered at the appropriate time.

Case Study 2 – Boar Impact Study


You are considering changing genetic companies and would like to evaluate the potential new company's boar line (Line X). You have examined the data from the new company and have chosen the line that you believe will make the most improvement to your herd. You have established how to track the pigs produced through your herd and have them evaluated for the anticipated improvements in gain and feed efficiency. You received semen from the new company and have carefully tracked the pigs though the system. You get data for 3121 pigs from Old line A, 2988 pigs from Old line B and 342 pigs from New line X. The new line performs much worse in late finishing for feed efficiency than the old lines you were using. This wipes out any previous benefits.

Is there a problem with this data?

On further examination the reason there are so many more pigs from the other genetic lines is that all the pigs from those lines produced over the last year were included in the data set. The pigs from the new line were finished during December, January, and February. Pigs on your farm typically eat more during these cold months and have lower feed efficiency. When the data was compared to the data from only pigs that were finished during the same time period the pigs from the new line did perform better than the old lines you were using. While having more numbers is usually good you also need to compare animals that have been handled in a similar manner.

Case Study 3 – New Product Assessment


You have found a disinfectant that would be a less expensive alternative to what you are currently using. But you want to see how effective the cheaper brand would be in your barns. You have discussed how to organise the test the pens with your consulting veterinarian and have decided to take one swab in each pen using the same three locations in each pen and what to test on the swabs. You arrange to have both the new and old versions of the disinfectants used in several pens that have been randomly selected out of three nursery rooms. You also assign some pens to have no disinfectant and those will be tested as well.

You call the lab that your veterinarian suggested and ask how samples should be packaged and transported and when the best days are for them to receive this type of sample. You discuss the procedure with the rest of the people who work in the barns and pick the nurseries that will be cleaned on days that will best suit the schedule needed to get samples to the lab and avoiding weekends or holidays.

You get both disinfectants and the swabs and packaging materials. You label the bags the swabs will be sent in with the pen, date of collection, and treatment. You and two other people collect the three nurseries using the same manner of collections. You send the samples as you have discussed with the lab.

When the results come back the averages for number of bacterial colonies per swab are: 30, 13, and 11 for the pens with no disinfectant, the old disinfectant, and the new disinfectant, respectively. You discuss the results with your veterinarian and conclude that disinfecting works to decrease the bacteria concentration and that the new disinfect is as good as the old one with reduced cost.

What might be a problem with this study?

Congratulations!! This study was done correctly. Good job on all the planning and correct follow-through. Enjoy the savings from the less expensive disinfectant.

Conclusions

There are many sources of new ideas and technologies awaiting pork producers. Assessing their economic value and appropriateness for your farm should begin with taking the easy route first and looking for third-party verifiable test results that give you confidence the results are repeatable and sufficient to provide a positive economic return under current economic circumstances.

If reliable information does not exist but you believe the potential economic benefit is too great to ignore, and you have adequate resources to design and implement an on-farm test then use PSC's On-Farm Trial Checklist to increase your chances for success.

References

  • Deen, J. 2009. On-farm field trials: the problem of detecting small but economically significant differences, American Association of Swine Veterinarians proceedings, p289-290.
  • Krueger, K. 2009, Proper conduct and interpretation of field trials, Minnesota Nutrition Conference proceedings 248-254.
  • Davies, P. 2010. Field trial design and evaluation, Allan D. Leman Swine Conference pre-conference workshop.
December 2010
© 2000 - 2024 - Global Ag Media. All Rights Reserved | No part of this site may be reproduced without permission.