Ultrasonography in veterinary medicine is safer than other diagnostic modalities, and the devices develop quickly. Therefore, the importance of checkup emerges gradually and the distribution rate of equipment constantly increases. Using improper medical imaging and diagnostic devices may lower diagnostic quality increase repeated examination and may miss the period of treatment as the disease progresses. The biggest advantage of multipurpose phantom evaluation as an image evaluation for ultrasound imaging device is that it is able to acquire data about image diagnosis device easily without any restrictions of time and space. Also, when the multipurpose phantom image has poor image quality, it is expected that the resolution and contrast of clinical image would decline (4,5,15).

The quality control of convex transducer has been standardized with ATS-539 multipurpose phantom (ATS Laboratories, Bridgeport, CT, U.S.A.) (Fig. 1) and widely used for abdominal diagnosis (5,11,13). In the field of veterinary medicine, however, the inspection items and appropriate/inappropriate criteria for ultrasound standard phantom image evaluation have not yet been prepared, and ultrasound imaging devices currently used in Korea have not yet been evaluated.

Figure 1. ATS-539 multipurpose phantom. ATS-539 phantom has been set on the standard for the convex transducer.

As multipurpose phantom provides the most objective data until now (5,7), If the image quality assessment of ultrasonic device is made only by subjective opinion of the observer, the quality control can be inconsistent and varied upon observer’s experience and personnel opinion. In order to supplement this subjective point of view, various quantitative evaluation methods of ultrasound images are being used. Among quantitative evaluation methods, coefficient of variation (COV) is a representative parameter that can analyze the ratio of signal to noise and is widely used in basic medical image analysis. In addition, a parameter, blind referenceless image spatial quality evaluator (BRISQUE), that can analyze image quality has recently been used in various imaging fields, and it is also necessary to analyze its applicability in ultrasound images. We studied, therefore, added image evaluation method that uses COV and BRISQUE to supplement the limit of image evaluation using multipurpose phantom image of low objectivity.

This study aimed to prepare objective, minimum regulations of ultrasound imaging device and examine quantitative and objective new image evaluation method by ATS-539 multipurpose phantom of veterinary medical ultrasound imaging device that are currently used for diagnostic purpose.

Image evaluation was performed on 10 ultrasound imaging devices used by 9 veterinary colleges of Korea. ATS-539 multipurpose phantom was used for micro convex transducer at 4-6 MHz which is most widely used in the field of veterinary medicine.

We used setup value for abdominal examination which is the most widely used with monitor depth at 14cm. We examined 8 items of ATS-539 multipurpose phantom (10) and measured COV and BRISQUE through DICOM (Digital Imaging and Communications in Medicine) image acquired by using phantom according to the standard of measurement.

There are 8 image evaluation items of ultrasound standard multipurpose phantom such as dead zone, vertical and horizontal measurement, axial/lateral resolution, sensitivity, focal zone, functional resolution and gray scale/dynamic range. Among them, we decided dead zone, axial/lateral resolution and gray scale/dynamic range for qualitative evaluation and the remained 5 items for quantitative evaluation. We obtained sequential score according to the number of targets seen for functional resolution, and the suitability of 7 items was measured and evaluated on the basis of 0 in case of fit and -1 point in case of nonconformity (6,11).

Dead zone is the distance from the front of probe to the first identifiable echo between the probe and multipurpose phantom. It consists of total 9 targets, and we evaluated it appropriate when all 9 targets are separately seen by adjusting 5th target come to the center (Fig. 2a).

Figure 2. Dead zone (a) and Axial/lateral resolution (b). (a) These images were obtained by using 4-6 MHz and Nine targets are clearly visualized. (b) Eleven targets are clearly visualized.

Axial/lateral resolution refers to the ability of distinction between two adjacent objects, and it can be divided into axial direction and outer direction. The resolution per sound beam axis is called axial direction resolution, and in this study, we decided it appropriate when 11 targets in vertical measurement image are seen clearly separated (Fig. 2b).

Vertical measurement refers to checking if the actual distance within a medium is correctly expressed along the axis of sound beam; there are 17 targets and the distance between targets is 1.0 ± 0.1 cm. We tried to use multi-focus as much as we could and locate target in the center of the screen so as to acquire the image, and measured 10 cm up to 1-11 cm. 10 ± 1 cm of error tolerance was used in the evaluation (Fig. 3a).

Figure 3. (a) Vertical measurement test. The length indicate the 1th target and the 11th target measurements. (b) Horizontal measurement test. The length indicate the 1th target and the 5th target measurements.

Horizontal measurement is a value vertically acquired to the axis of sound beam; in this study, we measured 8 cm lateral distance in vertical measurement image and evaluated it with 8 cm ± 4 mm of error tolerance (Fig. 3b).

Sensitivity is the depth from which round structures of 8 mm, 6 mm, 4 mm, and 2 mm sizes, which are anechoic targets, can be distinguished. Targets of 8 mm and 6 mm sizes are composed of 8 pieces each, and the other targets are composed of 17 pieces respectively. Acquiring image with targets of 8 mm size arranged in a row in the center, and the targets should be visible up to 5th target and the error tolerance of measuring distance was evaluated to be appropriate at 8.0 ± 0.5 cm (Fig. 4).

Figure 4. Sensitivity measurement. The length indicates the 1th target and the 4th target measurements.

When focal zone strength and outer resolution are at maximum, the most accurate diagnosis information is provided to the area around the focus. We located one focus in 7 cm depth in vertical measurement image, measured lateral distance of a target in focus depth, measured lateral distance of another target 4 cm behind this target so as to see focusing rate of the targets (Fig. 5), and decided the focal rate 75% and above as appropriate (10). The focal rate (%) is defined with the following formula (11).

Figure 5. Focal zone test. The cross marks indicate the 7th target and the 11th target measurements for the distance of the line targets.

$$Focalrate\left(\%\right)=\frac{7cmdeptht\arg etwidth}{11cmdeptht\arg etwidth}\times 100$$

Functional resolution means the ability to express the size, shape and 146 depth of anechoic target in multipurpose phantom. The target arrangement is the same with the structure of sensitivity. The decision was made in the image acquired by using multifocus for 4 mm size target to be vertically located in the center, and the number of targets appropriately observed in 8 mm and 6 mm were scored (Table 1, Fig. 6).

Table 1 . Functional resolution evaluation criterion.

Functional resolution	Appropriate number of targets observed at 8 mm and 6 mm	target: 0, 7 target: –1, 6 target: –2, 5 target: –3, 4 target: –4, 3 target: –5, 2 target: –6, 1 target: –7, 0 target: –8
	Appropriate number of targets observed at 4 mm, 3 mm, and 2 mm	17 target: 0, 16 target: –1, 15 target: –2, 14 target: –3, 13 target: –4, 12 target: –5, 11 target: –6, 10 target: –7, 9 target: –8, 8 target: –9, 7 target: –10, 6 target: –11, 5 target: –12, 4 target: –13, 3 target: –14, 2 target: –15, 1 target: –16, 0 target: –17

Figure 6. Functional resolution measurement. 8, 6, 4, 3, 2 mm cystic structure are well visualized.

Gray scale and Dynamic range means expressing the brightness of image differently according to the magnitude of received echo, and it expresses the brightness from the lowest gray scale level to the maximum brightness level by adjusting the echo signal. 6 targets are all seen in one screen and the center of target is adjusted to be on the center of the screen. Each target should be clearly distinguished and we decided it appropriate when the boundary line maintains more than 4 circles consecutively over 180° (Fig. 7) (11).

Figure 7. Gray scale and dynamic image. The six targets have different values of brightness.

For COV, we measured COV value in ROI area set to evaluate noise level of acquired ultrasound image (Fig. 8). COV can be calculated with the average value of signal intensity and standard deviation ratio. dynamic range measurement image that can calculate COV the most appropriately was used and average value of 4-area data was drawn for each device.

Figure 8. Sample ultrasound image for evaluating COV. The average value was derived by setting 4 ROIs on the image that can measure the dynamic range.

BRISQUE value is able to analyze entire image no-reference basis, and the smaller the BRISQUE value is the better quality of the image is shown (14). We calculated the BRISQUE using the code provided by the MATLAB program.

Phantom image evaluation results of dead zone, vertical and horizontal measurement, axial/lateral resolution, sensitivity, focal zone and gray scale/dynamic range (Table 2).

Table 2 . Phantom image evaluation result.

User*	Dead zone (No)	Vertical measurement (cm)	Horizontal measurement (cm)	Focal zone (%)	Axial/lateral resolution (No)	Sensitivity (No)	Gray scale/dynamic range (No)
A	9	10.3	8.9	10.3	11	4	4
B	9	10.4	9.2	10.4	11	4	4
C	9	10.5	8.9	10.5	11	4	4
D	9	10.3	8.9	10.3	11	4	4
E	9	10.1	8.9	10.1	11	3	4
Fa	9	10.2	8.7	10.2	11	4	4
Fb	9	10	8.5	10	11	4	4
G	9	8.2	8.7	8.2	11	3	4
H	9	10.4	9.4	10.4	11	4	4
I	9	10.4	9.1	10.4	11	4	4

*University is A-I, University F has two equipment a and b..

Dead zone

All the 10 ultrasound imaging devices showed 9 targets separated which are included in the appropriate standard, and so all of them were evaluated with 0 point.

Vertical measurement

9 out of 10 ultrasound imaging devices were included in the 10.00 ± 1.00 cm of error tolerance, and so 9 devices were evaluated with 0 point, and the other 1 device did not show up to 11th depth which is not included in the appropriate standard being evaluated as –1 point. The average of appropriate 9 devices was 10.30 ± 0.10 cm.

Horizontal measurement

All the 10 ultrasound imaging devices were above 8.40 cm and therefore evaluated as –1 point. The average was 8.90 ± 0.25 cm.

Axial/lateral resolution

All the 10 devices showed 11 targets clearly separated which are included in the appropriate standard, and so all of them were evaluated with 0 point.

Sensitivity

8 out of 10 devices had 6.90 ± 0.10 cm of average size up to 4 targets and the other 2 devices showed up to 3 targets with average 5.00 cm measured.

Focal zone

All the 10 devices were above 75% of error tolerance and so all of them were evaluated as 0 point. The lowest focal rate was 75% and the highest focal rate was 100%. Average of the 10 devices was 84.90%.

Functional resolution

With the number of targets observed per size, the device with –2 point showed the highest functional resolution, and another device with –16 point represented the lowest functional resolution with average –9.60 ± 3.63.

Gray scale and dynamic range

All the 10 devices were observed with 4 targets being evaluated as 0 point.

COV image evaluation

As a result of quantitative analysis on COV, average 0.12 ± 0.04 value was drawn. Among 10 devices of 9 veterinary colleges in Korea, the most excellent COV value was 0.05 while the lowest COV value was measured to be 0.17 (Fig. 9a).

Figure 9. COV and BRISQUE (a) Results for coefficient of variation with respect to ultrasound device including average data. (b) Results for blind/referenceless image spatial quality evaluator (BRISQUE) with respect to ultrasound device.

BRISQUE image evaluation

As a result of quantitative analysis, average 47.76 ± 2.78 value was drawn. Among 10 devices of 9 veterinary colleges in Korea, the most excellent BRISQUE value was 43.02 while the lowest BRISQUE value was measured to be 52.34 (Fig. 9b).

In this experiment, the image analysis was performed in both qualitative evaluation and quantitative evaluation. The qualitative evaluation was performed with dead zone, axial and lateral resolutions, and gray scale/dynamic range and the quantitative evaluation was performed with the remained vertical and the horizontal measurements, focal zone, sensitivity, and functional resolution.

In this experiment, all the 10 devices were evaluated to be appropriate in the qualitative evaluation. In the quantitative evaluation of 5 items, however, horizontal measurements showed –1 for all devices which means that the lesion is expected to be measured horizontally longer upon diagnosis in horizontal measurements item.

In the experiment, showing the scannable depth, 3 devices could only see 3 circular structures and, on average, 5 cm, so there was a limit to seeing deep organs. we think that it was caused by the penetrance relatively going down as the frequency domain going up. The frequency adjusts the depth from the patient’s surface (4,11). It does not only adjust the image contrast but also ultrasonic wavelength (9).

In this experiment, the average value was 84.93% which exceeds standard 75% showing high reliability on the focus per depth. The focal zone is where the strength and lateral direction resolution are shown the highest.

In the evaluation result with 8 items by using the existing multipurpose phantom image, the evaluation result with objective minimum regulations of dead zone, axial/lateral resolution, and gray scale/dynamic range was excellent.

The result of multipurpose phantom image can be different by the evaluation standard, and so it would be quite insufficient to evaluate ultrasonic device only with the existing multipurpose phantom image evaluation for accurate and objective evaluation standard.

This study, therefore, used COV and BRISQUE from various image evaluations so as to overcome the limit of subjectivity of image evaluation using multipurpose phantom and suggest quantitative image evaluation method.

The lower the COV and BRISQUE values the better the picture quality. COV and BRISQUE can be measured and evaluated in ultrasound imaging device, and so we measured the image signals along with multipurpose phantom image evaluation consequently evaluating devices more quantitative way.

In clinical thyroid ultrasonic image, COV value was about 0.25, and it was drawn to be about 0.20 even by applying the conventional noise reduction algorithm (2,12). Consequently, the noise level of all ultrasonic devices used by veterinary colleges is thought to be within allowable range for all clinical fields. Among 10 devices of 9 veterinary colleges, the best BRISQUE value was 43.02, and the worst BRISQUE value was measured to be 52.34. In particular, it was verified that the tendencies in COV result and BRISQUE result previously measured are matched.

This study basically used COV evaluation factor which is used the most widely upon evaluating noise level, and additionally applied recently developed BRISQUE for the first time which is an objective evaluation factor for entire image without reference image. BRISQUE has been approved of its accuracy by various researches (14), and it was applied to ultrasonic image for the first time in this study, and we will perform semantic analysis on the numerical values based on the result of this study.

Evaluating 8 items of multipurpose phantom image evaluation is insufficient method for correct ultrasonic device, therefor COV and BRISQUE measurement can be a supplementary method for such insufficiency.

In the field of veterinary medicine, 2.5-3.0 MHz of convex is actually used for large dogs and larger animals but for the smaller animals than those, 4-6 MHz of micro convex is widely used (1,3,8). Therefore, the existing convex with high penetrating power due to low frequency just needs to follow the multipurpose phantom image standard as used in the human medicine, but 4-6 MHz of micro convex, which are used for small and medium sized animals, needs new standard.

This study, therefore, aims to prepare new standard for the multipurpose phantom image evaluation for 4-6 MHz of micro convex probe, which is widely used for small and medium sized animals, and it is meaningful in performing more accurate and reliable diagnosis through systematic quality control of ultrasonic device.

The authors have no conflicting interests.