검색
검색 팝업 닫기

Ex) Article Title, Author, Keywords

## Article

J Vet Clin 2022; 39(3): 114-120

https://doi.org/10.17555/jvc.2022.39.3.114

Published online June 30, 2022

## Quality Evaluation of Ultrasonographic Equipment Using an ATS-539 Multipurpose Phantom in Veterinary Medicine

Young-kwon Cho1 , Youngjin Lee2 , Kichang Lee3

1Department of Radiologic Science, Cheongju University, Cheongju 28503, Korea
2Department of Radiologic Science, Gachon University, Incheon 21936, Korea
3Department of Veterinary Medicine, Jeonbuk National University, Iksan 54596, Korea

Correspondence to:*kclee@jbnu.ac.kr

Received: March 2, 2022; Revised: May 16, 2022; Accepted: May 16, 2022

Copyright © The Korean Society of Veterinary Clinics.

Erratum: J Vet Clin 39(4): 197-197 (2022) https://doi.org/10.17555/jvc.2022.39.4.197

### Abstract

The purpose of this study is to examine the status of quality control using multipurpose phantom of ultrasound equipment used in hospital of veterinary college in South Korea by using ATS-539 multipurpose phantom so as to examine quantitative and objective new image evaluation method. Specialists discussed and analyzed multipurpose phantom images acquired by using convex transducer of 10 ultrasound imaging devices, currently used in 9 veterinary colleges, at 4.0-6.0 MHz. Total 8 items that can be measured with ATS-539 multipurpose phantom including dead zone, vertical and horizontal measurement, axial/lateral resolution, sensitivity, focal zone, functional resolution and gray scale/dynamic range were evaluated. For qualitative evaluation, valid decisions were made based on dead zone, axial/lateral resolution, and gray scale/dynamic range which are resolution index, and coefficient of variation (COV) and blind referenceless image spatial quality evaluator (BRISQUE) were found to increase objectivity. As a result of experiment, all the targeted ultrasonic devices were found appropriate from qualitative evaluation items of dead zone, axial/lateral resolution, and gray scale/dynamic range. In other evaluation items, they were found to be appropriate from focal zone and vertical measurement of quantitative evaluation while inappropriate from horizontal measurement, sensitivity, and functional resolution. COV value was 0.12 ± 0.04, and BRISQUE value was 47.77 ± 2.77, both analysis results show that the noise level of all ultrasonic devices was located within tolerance range. Upon image examination using ATS- 539 multipurpose phantom, they were 100% appropriate with inspection standards of dead zone, axial/lateral resolution, and gray scale/dynamic range, and besides, focal zone and functional resolution can be used as evaluation items. In the field of veterinary medicine, 8 standard items using ATS-539 multipurpose phantom and image evaluation items using COV and BRISQUE can be used as standards for quality control of ultrasonography machine.

Keywords: quality evaluation of ultrasonographic equipment, ATS-539 multipurpose phantom, COV, BRISQUE.

### Introduction

Ultrasonography in veterinary medicine is safer than other diagnostic modalities, and the devices develop quickly. Therefore, the importance of checkup emerges gradually and the distribution rate of equipment constantly increases. Using improper medical imaging and diagnostic devices may lower diagnostic quality increase repeated examination and may miss the period of treatment as the disease progresses. The biggest advantage of multipurpose phantom evaluation as an image evaluation for ultrasound imaging device is that it is able to acquire data about image diagnosis device easily without any restrictions of time and space. Also, when the multipurpose phantom image has poor image quality, it is expected that the resolution and contrast of clinical image would decline (4,5,15).

The quality control of convex transducer has been standardized with ATS-539 multipurpose phantom (ATS Laboratories, Bridgeport, CT, U.S.A.) (Fig. 1) and widely used for abdominal diagnosis (5,11,13). In the field of veterinary medicine, however, the inspection items and appropriate/inappropriate criteria for ultrasound standard phantom image evaluation have not yet been prepared, and ultrasound imaging devices currently used in Korea have not yet been evaluated.

Figure 1.ATS-539 multipurpose phantom. ATS-539 phantom has been set on the standard for the convex transducer.

As multipurpose phantom provides the most objective data until now (5,7), If the image quality assessment of ultrasonic device is made only by subjective opinion of the observer, the quality control can be inconsistent and varied upon observer’s experience and personnel opinion. In order to supplement this subjective point of view, various quantitative evaluation methods of ultrasound images are being used. Among quantitative evaluation methods, coefficient of variation (COV) is a representative parameter that can analyze the ratio of signal to noise and is widely used in basic medical image analysis. In addition, a parameter, blind referenceless image spatial quality evaluator (BRISQUE), that can analyze image quality has recently been used in various imaging fields, and it is also necessary to analyze its applicability in ultrasound images. We studied, therefore, added image evaluation method that uses COV and BRISQUE to supplement the limit of image evaluation using multipurpose phantom image of low objectivity.

This study aimed to prepare objective, minimum regulations of ultrasound imaging device and examine quantitative and objective new image evaluation method by ATS-539 multipurpose phantom of veterinary medical ultrasound imaging device that are currently used for diagnostic purpose.

### Materials and Methods

Image evaluation was performed on 10 ultrasound imaging devices used by 9 veterinary colleges of Korea. ATS-539 multipurpose phantom was used for micro convex transducer at 4-6 MHz which is most widely used in the field of veterinary medicine.

We used setup value for abdominal examination which is the most widely used with monitor depth at 14cm. We examined 8 items of ATS-539 multipurpose phantom (10) and measured COV and BRISQUE through DICOM (Digital Imaging and Communications in Medicine) image acquired by using phantom according to the standard of measurement.

There are 8 image evaluation items of ultrasound standard multipurpose phantom such as dead zone, vertical and horizontal measurement, axial/lateral resolution, sensitivity, focal zone, functional resolution and gray scale/dynamic range. Among them, we decided dead zone, axial/lateral resolution and gray scale/dynamic range for qualitative evaluation and the remained 5 items for quantitative evaluation. We obtained sequential score according to the number of targets seen for functional resolution, and the suitability of 7 items was measured and evaluated on the basis of 0 in case of fit and -1 point in case of nonconformity (6,11).

Dead zone is the distance from the front of probe to the first identifiable echo between the probe and multipurpose phantom. It consists of total 9 targets, and we evaluated it appropriate when all 9 targets are separately seen by adjusting 5th target come to the center (Fig. 2a).

Figure 2.Dead zone (a) and Axial/lateral resolution (b). (a) These images were obtained by using 4-6 MHz and Nine targets are clearly visualized. (b) Eleven targets are clearly visualized.

Axial/lateral resolution refers to the ability of distinction between two adjacent objects, and it can be divided into axial direction and outer direction. The resolution per sound beam axis is called axial direction resolution, and in this study, we decided it appropriate when 11 targets in vertical measurement image are seen clearly separated (Fig. 2b).

Vertical measurement refers to checking if the actual distance within a medium is correctly expressed along the axis of sound beam; there are 17 targets and the distance between targets is 1.0 ± 0.1 cm. We tried to use multi-focus as much as we could and locate target in the center of the screen so as to acquire the image, and measured 10 cm up to 1-11 cm. 10 ± 1 cm of error tolerance was used in the evaluation (Fig. 3a).

Figure 3.(a) Vertical measurement test. The length indicate the 1th target and the 11th target measurements. (b) Horizontal measurement test. The length indicate the 1th target and the 5th target measurements.

Horizontal measurement is a value vertically acquired to the axis of sound beam; in this study, we measured 8 cm lateral distance in vertical measurement image and evaluated it with 8 cm ± 4 mm of error tolerance (Fig. 3b).

Sensitivity is the depth from which round structures of 8 mm, 6 mm, 4 mm, and 2 mm sizes, which are anechoic targets, can be distinguished. Targets of 8 mm and 6 mm sizes are composed of 8 pieces each, and the other targets are composed of 17 pieces respectively. Acquiring image with targets of 8 mm size arranged in a row in the center, and the targets should be visible up to 5th target and the error tolerance of measuring distance was evaluated to be appropriate at 8.0 ± 0.5 cm (Fig. 4).

Figure 4.Sensitivity measurement. The length indicates the 1th target and the 4th target measurements.

When focal zone strength and outer resolution are at maximum, the most accurate diagnosis information is provided to the area around the focus. We located one focus in 7 cm depth in vertical measurement image, measured lateral distance of a target in focus depth, measured lateral distance of another target 4 cm behind this target so as to see focusing rate of the targets (Fig. 5), and decided the focal rate 75% and above as appropriate (10). The focal rate (%) is defined with the following formula (11).

Figure 5.Focal zone test. The cross marks indicate the 7th target and the 11th target measurements for the distance of the line targets.

Focalrate%=7cmdepthtargetwidth11cmdepthtargetwidth×100

Functional resolution means the ability to express the size, shape and 146 depth of anechoic target in multipurpose phantom. The target arrangement is the same with the structure of sensitivity. The decision was made in the image acquired by using multifocus for 4 mm size target to be vertically located in the center, and the number of targets appropriately observed in 8 mm and 6 mm were scored (Table 1, Fig. 6).

Functional resolution evaluation criterion

 Functional resolution Appropriate number of targets observed at 8 mm and 6 mm target: 0, 7 target: –1, 6 target: –2, 5 target: –3, 4 target: –4, 3 target: –5, 2 target: –6, 1 target: –7, 0 target: –8 Appropriate number of targets observed at 4 mm, 3 mm, and 2 mm 17 target: 0, 16 target: –1, 15 target: –2, 14 target: –3, 13 target: –4, 12 target: –5, 11 target: –6, 10 target: –7, 9 target: –8, 8 target: –9, 7 target: –10, 6 target: –11, 5 target: –12, 4 target: –13, 3 target: –14, 2 target: –15, 1 target: –16, 0 target: –17

Figure 6.Functional resolution measurement. 8, 6, 4, 3, 2 mm cystic structure are well visualized.

Gray scale and Dynamic range means expressing the brightness of image differently according to the magnitude of received echo, and it expresses the brightness from the lowest gray scale level to the maximum brightness level by adjusting the echo signal. 6 targets are all seen in one screen and the center of target is adjusted to be on the center of the screen. Each target should be clearly distinguished and we decided it appropriate when the boundary line maintains more than 4 circles consecutively over 180° (Fig. 7) (11).

Figure 7.Gray scale and dynamic image. The six targets have different values of brightness.

For COV, we measured COV value in ROI area set to evaluate noise level of acquired ultrasound image (Fig. 8). COV can be calculated with the average value of signal intensity and standard deviation ratio. dynamic range measurement image that can calculate COV the most appropriately was used and average value of 4-area data was drawn for each device.

Figure 8.Sample ultrasound image for evaluating COV. The average value was derived by setting 4 ROIs on the image that can measure the dynamic range.

BRISQUE value is able to analyze entire image no-reference basis, and the smaller the BRISQUE value is the better quality of the image is shown (14). We calculated the BRISQUE using the code provided by the MATLAB program.

### Results

Phantom image evaluation results of dead zone, vertical and horizontal measurement, axial/lateral resolution, sensitivity, focal zone and gray scale/dynamic range (Table 2).

Phantom image evaluation result

User*Dead zone (No)Vertical measurement (cm)Horizontal measurement (cm)Focal zone (%)Axial/lateral resolution (No)Sensitivity (No)Gray scale/dynamic range (No)
A910.38.910.31144
B910.49.210.41144
C910.58.910.51144
D910.38.910.31144
E910.18.910.11134
Fa910.28.710.21144
Fb9108.5101144
G98.28.78.21134
H910.49.410.41144
I910.49.110.41144

*University is A-I, University F has two equipment a and b.

All the 10 ultrasound imaging devices showed 9 targets separated which are included in the appropriate standard, and so all of them were evaluated with 0 point.

### Vertical measurement

9 out of 10 ultrasound imaging devices were included in the 10.00 ± 1.00 cm of error tolerance, and so 9 devices were evaluated with 0 point, and the other 1 device did not show up to 11th depth which is not included in the appropriate standard being evaluated as –1 point. The average of appropriate 9 devices was 10.30 ± 0.10 cm.

### Horizontal measurement

All the 10 ultrasound imaging devices were above 8.40 cm and therefore evaluated as –1 point. The average was 8.90 ± 0.25 cm.

### Axial/lateral resolution

All the 10 devices showed 11 targets clearly separated which are included in the appropriate standard, and so all of them were evaluated with 0 point.

### Sensitivity

8 out of 10 devices had 6.90 ± 0.10 cm of average size up to 4 targets and the other 2 devices showed up to 3 targets with average 5.00 cm measured.

### Focal zone

All the 10 devices were above 75% of error tolerance and so all of them were evaluated as 0 point. The lowest focal rate was 75% and the highest focal rate was 100%. Average of the 10 devices was 84.90%.

### Functional resolution

With the number of targets observed per size, the device with –2 point showed the highest functional resolution, and another device with –16 point represented the lowest functional resolution with average –9.60 ± 3.63.

### Gray scale and dynamic range

All the 10 devices were observed with 4 targets being evaluated as 0 point.

### COV image evaluation

As a result of quantitative analysis on COV, average 0.12 ± 0.04 value was drawn. Among 10 devices of 9 veterinary colleges in Korea, the most excellent COV value was 0.05 while the lowest COV value was measured to be 0.17 (Fig. 9a).

Figure 9.COV and BRISQUE (a) Results for coefficient of variation with respect to ultrasound device including average data. (b) Results for blind/referenceless image spatial quality evaluator (BRISQUE) with respect to ultrasound device.

### BRISQUE image evaluation

As a result of quantitative analysis, average 47.76 ± 2.78 value was drawn. Among 10 devices of 9 veterinary colleges in Korea, the most excellent BRISQUE value was 43.02 while the lowest BRISQUE value was measured to be 52.34 (Fig. 9b).

### Discussion

In this experiment, the image analysis was performed in both qualitative evaluation and quantitative evaluation. The qualitative evaluation was performed with dead zone, axial and lateral resolutions, and gray scale/dynamic range and the quantitative evaluation was performed with the remained vertical and the horizontal measurements, focal zone, sensitivity, and functional resolution.

In this experiment, all the 10 devices were evaluated to be appropriate in the qualitative evaluation. In the quantitative evaluation of 5 items, however, horizontal measurements showed –1 for all devices which means that the lesion is expected to be measured horizontally longer upon diagnosis in horizontal measurements item.

In the experiment, showing the scannable depth, 3 devices could only see 3 circular structures and, on average, 5 cm, so there was a limit to seeing deep organs. we think that it was caused by the penetrance relatively going down as the frequency domain going up. The frequency adjusts the depth from the patient’s surface (4,11). It does not only adjust the image contrast but also ultrasonic wavelength (9).

In this experiment, the average value was 84.93% which exceeds standard 75% showing high reliability on the focus per depth. The focal zone is where the strength and lateral direction resolution are shown the highest.

In the evaluation result with 8 items by using the existing multipurpose phantom image, the evaluation result with objective minimum regulations of dead zone, axial/lateral resolution, and gray scale/dynamic range was excellent.

The result of multipurpose phantom image can be different by the evaluation standard, and so it would be quite insufficient to evaluate ultrasonic device only with the existing multipurpose phantom image evaluation for accurate and objective evaluation standard.

This study, therefore, used COV and BRISQUE from various image evaluations so as to overcome the limit of subjectivity of image evaluation using multipurpose phantom and suggest quantitative image evaluation method.

The lower the COV and BRISQUE values the better the picture quality. COV and BRISQUE can be measured and evaluated in ultrasound imaging device, and so we measured the image signals along with multipurpose phantom image evaluation consequently evaluating devices more quantitative way.

In clinical thyroid ultrasonic image, COV value was about 0.25, and it was drawn to be about 0.20 even by applying the conventional noise reduction algorithm (2,12). Consequently, the noise level of all ultrasonic devices used by veterinary colleges is thought to be within allowable range for all clinical fields. Among 10 devices of 9 veterinary colleges, the best BRISQUE value was 43.02, and the worst BRISQUE value was measured to be 52.34. In particular, it was verified that the tendencies in COV result and BRISQUE result previously measured are matched.

This study basically used COV evaluation factor which is used the most widely upon evaluating noise level, and additionally applied recently developed BRISQUE for the first time which is an objective evaluation factor for entire image without reference image. BRISQUE has been approved of its accuracy by various researches (14), and it was applied to ultrasonic image for the first time in this study, and we will perform semantic analysis on the numerical values based on the result of this study.

Evaluating 8 items of multipurpose phantom image evaluation is insufficient method for correct ultrasonic device, therefor COV and BRISQUE measurement can be a supplementary method for such insufficiency.

In the field of veterinary medicine, 2.5-3.0 MHz of convex is actually used for large dogs and larger animals but for the smaller animals than those, 4-6 MHz of micro convex is widely used (1,3,8). Therefore, the existing convex with high penetrating power due to low frequency just needs to follow the multipurpose phantom image standard as used in the human medicine, but 4-6 MHz of micro convex, which are used for small and medium sized animals, needs new standard.

This study, therefore, aims to prepare new standard for the multipurpose phantom image evaluation for 4-6 MHz of micro convex probe, which is widely used for small and medium sized animals, and it is meaningful in performing more accurate and reliable diagnosis through systematic quality control of ultrasonic device.

### Conflicts of Interest

The authors have no conflicting interests.

### References

1. Adhikari S. High-frequency transducers for point-of-care ultrasound applications: what is the optimal frequency range? Intern Emerg Med 2014; 9: 463-466.
2. Babu J, Sudha GF. Adaptive speckle reduction in ultrasound images using fuzzy logic on Coefficient of Variation. Biomed Signal Process Control 2016; 23: 93-103.
3. Bok TH, Paeng DK. High frequency ultrasound and its applications to animal and human imaging focusing on vessel and blood. J Acoust Soc Korea 2010; 29: 73-85.
4. Choi JI, Kim PN, Jeong WK, Kim HC, Yang DM, Cha SH, et al. Establishing cutoff values for a quality assurance test using an ultrasound phantom in screening ultrasound examinations for hepatocellular carcinoma: an initial report of a nationwide survey in Korea. J Ultrasound Med 2011; 30: 1221-1229.
5. Dong KR, Bae SH, Lee WH, Kweon DC, Goo EH, Chung WK. Performance evaluation of ultrasonography using ATS-539 multipurpose phantom. J Adv Eng Technol 2010; 3: 133-137.
6. Fabiszewska E, Pasicz K, Grabska I, Skrzyński W, Ślusarczyk-Kacprzyk W, Bulski W. Evaluation of imaging parameters of ultrasound scanners: baseline for future testing. Pol J Radiol 2017; 82: 773-782.
7. Goodsitt MM, Carson PL, Witt S, Hykes DL, Kofler JM Jr. Real-time B-mode ultrasound quality control test procedures. Report of AAPM Ultrasound Task Group No. 1. Med Phys 1998; 25: 1385-1406.
8. Jung JJ, Choi JH, Jung JH. Image quality control for linear transducers with various frequencies by using an ATS-550 phantom. J Korean Phys Soc 2017; 28: 206-212.
9. Kim CS, Kang SS, Kim JH. Evaluation of quantitative image quality using frequency and parameters in the ultrasound image. J Korean Soc Radiol 2016; 10: 247-253.
10. Kim MJ, Lee JS, Ko SJ, Kang SS, Kim JH, Kim DH, et al. Evaluation of image quality using ATS-539 phantom and SNR in the ultrasonographic equipment. J Korea Contents Assoc 2013; 13: 284-291.
11. Kim PN, Lim JW, Kim HC, Yoon YC, Sung DJ, Moon MH, et al. Quality assessment of ultrasonographic equipment using an ATS-539 multipurpose phantom. J Korean Soc Radiol 2008; 58: 533-541.
12. Kim SH, Seo KH, Kang SH, Kim JH, Choi WH, Lee YJ. Feasibility study of improved patch group prior based denoising (PGPD) technique with medical ultrasound imaging system. J Magn 2017; 22: 55-59.
13. Lee S, Choi JI, Park MY, Yeo DM, Byun JY, Jung SE, et al. Intra- and interobserver reliability of gray scale/dynamic range evaluation of ultrasonography using a standardized phantom. Ultrasonography 2014; 33: 91-97.
14. Mittal A, Moorthy AK, Bovik AC. No-reference image quality assessment in the spatial domain. IEEE Trans Image Process 2012; 21: 4695-4708.
15. Thijssen JM, Weijers G, de Korte CL. Objective performance testing and quality assurance of medical ultrasound equipment. Ultrasound Med Biol 2007; 33: 460-471.

### Article

#### Original Article

J Vet Clin 2022; 39(3): 114-120

Published online June 30, 2022 https://doi.org/10.17555/jvc.2022.39.3.114

Copyright © The Korean Society of Veterinary Clinics.

## Quality Evaluation of Ultrasonographic Equipment Using an ATS-539 Multipurpose Phantom in Veterinary Medicine

Young-kwon Cho1 , Youngjin Lee2 , Kichang Lee3

1Department of Radiologic Science, Cheongju University, Cheongju 28503, Korea
2Department of Radiologic Science, Gachon University, Incheon 21936, Korea
3Department of Veterinary Medicine, Jeonbuk National University, Iksan 54596, Korea

Correspondence to:*kclee@jbnu.ac.kr

Received: March 2, 2022; Revised: May 16, 2022; Accepted: May 16, 2022

This is an open access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Erratum: J Vet Clin 39(4): 197-197 (2022) https://doi.org/10.17555/jvc.2022.39.4.197

### Abstract

The purpose of this study is to examine the status of quality control using multipurpose phantom of ultrasound equipment used in hospital of veterinary college in South Korea by using ATS-539 multipurpose phantom so as to examine quantitative and objective new image evaluation method. Specialists discussed and analyzed multipurpose phantom images acquired by using convex transducer of 10 ultrasound imaging devices, currently used in 9 veterinary colleges, at 4.0-6.0 MHz. Total 8 items that can be measured with ATS-539 multipurpose phantom including dead zone, vertical and horizontal measurement, axial/lateral resolution, sensitivity, focal zone, functional resolution and gray scale/dynamic range were evaluated. For qualitative evaluation, valid decisions were made based on dead zone, axial/lateral resolution, and gray scale/dynamic range which are resolution index, and coefficient of variation (COV) and blind referenceless image spatial quality evaluator (BRISQUE) were found to increase objectivity. As a result of experiment, all the targeted ultrasonic devices were found appropriate from qualitative evaluation items of dead zone, axial/lateral resolution, and gray scale/dynamic range. In other evaluation items, they were found to be appropriate from focal zone and vertical measurement of quantitative evaluation while inappropriate from horizontal measurement, sensitivity, and functional resolution. COV value was 0.12 ± 0.04, and BRISQUE value was 47.77 ± 2.77, both analysis results show that the noise level of all ultrasonic devices was located within tolerance range. Upon image examination using ATS- 539 multipurpose phantom, they were 100% appropriate with inspection standards of dead zone, axial/lateral resolution, and gray scale/dynamic range, and besides, focal zone and functional resolution can be used as evaluation items. In the field of veterinary medicine, 8 standard items using ATS-539 multipurpose phantom and image evaluation items using COV and BRISQUE can be used as standards for quality control of ultrasonography machine.

Keywords: quality evaluation of ultrasonographic equipment, ATS-539 multipurpose phantom, COV, BRISQUE.

### Introduction

Ultrasonography in veterinary medicine is safer than other diagnostic modalities, and the devices develop quickly. Therefore, the importance of checkup emerges gradually and the distribution rate of equipment constantly increases. Using improper medical imaging and diagnostic devices may lower diagnostic quality increase repeated examination and may miss the period of treatment as the disease progresses. The biggest advantage of multipurpose phantom evaluation as an image evaluation for ultrasound imaging device is that it is able to acquire data about image diagnosis device easily without any restrictions of time and space. Also, when the multipurpose phantom image has poor image quality, it is expected that the resolution and contrast of clinical image would decline (4,5,15).

The quality control of convex transducer has been standardized with ATS-539 multipurpose phantom (ATS Laboratories, Bridgeport, CT, U.S.A.) (Fig. 1) and widely used for abdominal diagnosis (5,11,13). In the field of veterinary medicine, however, the inspection items and appropriate/inappropriate criteria for ultrasound standard phantom image evaluation have not yet been prepared, and ultrasound imaging devices currently used in Korea have not yet been evaluated.

Figure 1. ATS-539 multipurpose phantom. ATS-539 phantom has been set on the standard for the convex transducer.

As multipurpose phantom provides the most objective data until now (5,7), If the image quality assessment of ultrasonic device is made only by subjective opinion of the observer, the quality control can be inconsistent and varied upon observer’s experience and personnel opinion. In order to supplement this subjective point of view, various quantitative evaluation methods of ultrasound images are being used. Among quantitative evaluation methods, coefficient of variation (COV) is a representative parameter that can analyze the ratio of signal to noise and is widely used in basic medical image analysis. In addition, a parameter, blind referenceless image spatial quality evaluator (BRISQUE), that can analyze image quality has recently been used in various imaging fields, and it is also necessary to analyze its applicability in ultrasound images. We studied, therefore, added image evaluation method that uses COV and BRISQUE to supplement the limit of image evaluation using multipurpose phantom image of low objectivity.

This study aimed to prepare objective, minimum regulations of ultrasound imaging device and examine quantitative and objective new image evaluation method by ATS-539 multipurpose phantom of veterinary medical ultrasound imaging device that are currently used for diagnostic purpose.

### Materials and Methods

Image evaluation was performed on 10 ultrasound imaging devices used by 9 veterinary colleges of Korea. ATS-539 multipurpose phantom was used for micro convex transducer at 4-6 MHz which is most widely used in the field of veterinary medicine.

We used setup value for abdominal examination which is the most widely used with monitor depth at 14cm. We examined 8 items of ATS-539 multipurpose phantom (10) and measured COV and BRISQUE through DICOM (Digital Imaging and Communications in Medicine) image acquired by using phantom according to the standard of measurement.

There are 8 image evaluation items of ultrasound standard multipurpose phantom such as dead zone, vertical and horizontal measurement, axial/lateral resolution, sensitivity, focal zone, functional resolution and gray scale/dynamic range. Among them, we decided dead zone, axial/lateral resolution and gray scale/dynamic range for qualitative evaluation and the remained 5 items for quantitative evaluation. We obtained sequential score according to the number of targets seen for functional resolution, and the suitability of 7 items was measured and evaluated on the basis of 0 in case of fit and -1 point in case of nonconformity (6,11).

Dead zone is the distance from the front of probe to the first identifiable echo between the probe and multipurpose phantom. It consists of total 9 targets, and we evaluated it appropriate when all 9 targets are separately seen by adjusting 5th target come to the center (Fig. 2a).

Figure 2. Dead zone (a) and Axial/lateral resolution (b). (a) These images were obtained by using 4-6 MHz and Nine targets are clearly visualized. (b) Eleven targets are clearly visualized.

Axial/lateral resolution refers to the ability of distinction between two adjacent objects, and it can be divided into axial direction and outer direction. The resolution per sound beam axis is called axial direction resolution, and in this study, we decided it appropriate when 11 targets in vertical measurement image are seen clearly separated (Fig. 2b).

Vertical measurement refers to checking if the actual distance within a medium is correctly expressed along the axis of sound beam; there are 17 targets and the distance between targets is 1.0 ± 0.1 cm. We tried to use multi-focus as much as we could and locate target in the center of the screen so as to acquire the image, and measured 10 cm up to 1-11 cm. 10 ± 1 cm of error tolerance was used in the evaluation (Fig. 3a).

Figure 3. (a) Vertical measurement test. The length indicate the 1th target and the 11th target measurements. (b) Horizontal measurement test. The length indicate the 1th target and the 5th target measurements.

Horizontal measurement is a value vertically acquired to the axis of sound beam; in this study, we measured 8 cm lateral distance in vertical measurement image and evaluated it with 8 cm ± 4 mm of error tolerance (Fig. 3b).

Sensitivity is the depth from which round structures of 8 mm, 6 mm, 4 mm, and 2 mm sizes, which are anechoic targets, can be distinguished. Targets of 8 mm and 6 mm sizes are composed of 8 pieces each, and the other targets are composed of 17 pieces respectively. Acquiring image with targets of 8 mm size arranged in a row in the center, and the targets should be visible up to 5th target and the error tolerance of measuring distance was evaluated to be appropriate at 8.0 ± 0.5 cm (Fig. 4).

Figure 4. Sensitivity measurement. The length indicates the 1th target and the 4th target measurements.

When focal zone strength and outer resolution are at maximum, the most accurate diagnosis information is provided to the area around the focus. We located one focus in 7 cm depth in vertical measurement image, measured lateral distance of a target in focus depth, measured lateral distance of another target 4 cm behind this target so as to see focusing rate of the targets (Fig. 5), and decided the focal rate 75% and above as appropriate (10). The focal rate (%) is defined with the following formula (11).

Figure 5. Focal zone test. The cross marks indicate the 7th target and the 11th target measurements for the distance of the line targets.

$Focal rate%=7cm depth target width11cm depth target width×100$

Functional resolution means the ability to express the size, shape and 146 depth of anechoic target in multipurpose phantom. The target arrangement is the same with the structure of sensitivity. The decision was made in the image acquired by using multifocus for 4 mm size target to be vertically located in the center, and the number of targets appropriately observed in 8 mm and 6 mm were scored (Table 1, Fig. 6).

Functional resolution evaluation criterion.

 Functional resolution Appropriate number of targets observed at 8 mm and 6 mm target: 0, 7 target: –1, 6 target: –2, 5 target: –3, 4 target: –4, 3 target: –5, 2 target: –6, 1 target: –7, 0 target: –8 Appropriate number of targets observed at 4 mm, 3 mm, and 2 mm 17 target: 0, 16 target: –1, 15 target: –2, 14 target: –3, 13 target: –4, 12 target: –5, 11 target: –6, 10 target: –7, 9 target: –8, 8 target: –9, 7 target: –10, 6 target: –11, 5 target: –12, 4 target: –13, 3 target: –14, 2 target: –15, 1 target: –16, 0 target: –17

Figure 6. Functional resolution measurement. 8, 6, 4, 3, 2 mm cystic structure are well visualized.

Gray scale and Dynamic range means expressing the brightness of image differently according to the magnitude of received echo, and it expresses the brightness from the lowest gray scale level to the maximum brightness level by adjusting the echo signal. 6 targets are all seen in one screen and the center of target is adjusted to be on the center of the screen. Each target should be clearly distinguished and we decided it appropriate when the boundary line maintains more than 4 circles consecutively over 180° (Fig. 7) (11).

Figure 7. Gray scale and dynamic image. The six targets have different values of brightness.

For COV, we measured COV value in ROI area set to evaluate noise level of acquired ultrasound image (Fig. 8). COV can be calculated with the average value of signal intensity and standard deviation ratio. dynamic range measurement image that can calculate COV the most appropriately was used and average value of 4-area data was drawn for each device.

Figure 8. Sample ultrasound image for evaluating COV. The average value was derived by setting 4 ROIs on the image that can measure the dynamic range.

BRISQUE value is able to analyze entire image no-reference basis, and the smaller the BRISQUE value is the better quality of the image is shown (14). We calculated the BRISQUE using the code provided by the MATLAB program.

### Results

Phantom image evaluation results of dead zone, vertical and horizontal measurement, axial/lateral resolution, sensitivity, focal zone and gray scale/dynamic range (Table 2).

Phantom image evaluation result.

User*Dead zone (No)Vertical measurement (cm)Horizontal measurement (cm)Focal zone (%)Axial/lateral resolution (No)Sensitivity (No)Gray scale/dynamic range (No)
A910.38.910.31144
B910.49.210.41144
C910.58.910.51144
D910.38.910.31144
E910.18.910.11134
Fa910.28.710.21144
Fb9108.5101144
G98.28.78.21134
H910.49.410.41144
I910.49.110.41144

*University is A-I, University F has two equipment a and b..

All the 10 ultrasound imaging devices showed 9 targets separated which are included in the appropriate standard, and so all of them were evaluated with 0 point.

### Vertical measurement

9 out of 10 ultrasound imaging devices were included in the 10.00 ± 1.00 cm of error tolerance, and so 9 devices were evaluated with 0 point, and the other 1 device did not show up to 11th depth which is not included in the appropriate standard being evaluated as –1 point. The average of appropriate 9 devices was 10.30 ± 0.10 cm.

### Horizontal measurement

All the 10 ultrasound imaging devices were above 8.40 cm and therefore evaluated as –1 point. The average was 8.90 ± 0.25 cm.

### Axial/lateral resolution

All the 10 devices showed 11 targets clearly separated which are included in the appropriate standard, and so all of them were evaluated with 0 point.

### Sensitivity

8 out of 10 devices had 6.90 ± 0.10 cm of average size up to 4 targets and the other 2 devices showed up to 3 targets with average 5.00 cm measured.

### Focal zone

All the 10 devices were above 75% of error tolerance and so all of them were evaluated as 0 point. The lowest focal rate was 75% and the highest focal rate was 100%. Average of the 10 devices was 84.90%.

### Functional resolution

With the number of targets observed per size, the device with –2 point showed the highest functional resolution, and another device with –16 point represented the lowest functional resolution with average –9.60 ± 3.63.

### Gray scale and dynamic range

All the 10 devices were observed with 4 targets being evaluated as 0 point.

### COV image evaluation

As a result of quantitative analysis on COV, average 0.12 ± 0.04 value was drawn. Among 10 devices of 9 veterinary colleges in Korea, the most excellent COV value was 0.05 while the lowest COV value was measured to be 0.17 (Fig. 9a).

Figure 9. COV and BRISQUE (a) Results for coefficient of variation with respect to ultrasound device including average data. (b) Results for blind/referenceless image spatial quality evaluator (BRISQUE) with respect to ultrasound device.

### BRISQUE image evaluation

As a result of quantitative analysis, average 47.76 ± 2.78 value was drawn. Among 10 devices of 9 veterinary colleges in Korea, the most excellent BRISQUE value was 43.02 while the lowest BRISQUE value was measured to be 52.34 (Fig. 9b).

### Discussion

In this experiment, the image analysis was performed in both qualitative evaluation and quantitative evaluation. The qualitative evaluation was performed with dead zone, axial and lateral resolutions, and gray scale/dynamic range and the quantitative evaluation was performed with the remained vertical and the horizontal measurements, focal zone, sensitivity, and functional resolution.

In this experiment, all the 10 devices were evaluated to be appropriate in the qualitative evaluation. In the quantitative evaluation of 5 items, however, horizontal measurements showed –1 for all devices which means that the lesion is expected to be measured horizontally longer upon diagnosis in horizontal measurements item.

In the experiment, showing the scannable depth, 3 devices could only see 3 circular structures and, on average, 5 cm, so there was a limit to seeing deep organs. we think that it was caused by the penetrance relatively going down as the frequency domain going up. The frequency adjusts the depth from the patient’s surface (4,11). It does not only adjust the image contrast but also ultrasonic wavelength (9).

In this experiment, the average value was 84.93% which exceeds standard 75% showing high reliability on the focus per depth. The focal zone is where the strength and lateral direction resolution are shown the highest.

In the evaluation result with 8 items by using the existing multipurpose phantom image, the evaluation result with objective minimum regulations of dead zone, axial/lateral resolution, and gray scale/dynamic range was excellent.

The result of multipurpose phantom image can be different by the evaluation standard, and so it would be quite insufficient to evaluate ultrasonic device only with the existing multipurpose phantom image evaluation for accurate and objective evaluation standard.

This study, therefore, used COV and BRISQUE from various image evaluations so as to overcome the limit of subjectivity of image evaluation using multipurpose phantom and suggest quantitative image evaluation method.

The lower the COV and BRISQUE values the better the picture quality. COV and BRISQUE can be measured and evaluated in ultrasound imaging device, and so we measured the image signals along with multipurpose phantom image evaluation consequently evaluating devices more quantitative way.

In clinical thyroid ultrasonic image, COV value was about 0.25, and it was drawn to be about 0.20 even by applying the conventional noise reduction algorithm (2,12). Consequently, the noise level of all ultrasonic devices used by veterinary colleges is thought to be within allowable range for all clinical fields. Among 10 devices of 9 veterinary colleges, the best BRISQUE value was 43.02, and the worst BRISQUE value was measured to be 52.34. In particular, it was verified that the tendencies in COV result and BRISQUE result previously measured are matched.

This study basically used COV evaluation factor which is used the most widely upon evaluating noise level, and additionally applied recently developed BRISQUE for the first time which is an objective evaluation factor for entire image without reference image. BRISQUE has been approved of its accuracy by various researches (14), and it was applied to ultrasonic image for the first time in this study, and we will perform semantic analysis on the numerical values based on the result of this study.

Evaluating 8 items of multipurpose phantom image evaluation is insufficient method for correct ultrasonic device, therefor COV and BRISQUE measurement can be a supplementary method for such insufficiency.

In the field of veterinary medicine, 2.5-3.0 MHz of convex is actually used for large dogs and larger animals but for the smaller animals than those, 4-6 MHz of micro convex is widely used (1,3,8). Therefore, the existing convex with high penetrating power due to low frequency just needs to follow the multipurpose phantom image standard as used in the human medicine, but 4-6 MHz of micro convex, which are used for small and medium sized animals, needs new standard.

This study, therefore, aims to prepare new standard for the multipurpose phantom image evaluation for 4-6 MHz of micro convex probe, which is widely used for small and medium sized animals, and it is meaningful in performing more accurate and reliable diagnosis through systematic quality control of ultrasonic device.

### Conflicts of Interest

The authors have no conflicting interests.

### Fig 1.

Figure 1.ATS-539 multipurpose phantom. ATS-539 phantom has been set on the standard for the convex transducer.
Journal of Veterinary Clinics 2022; 39: 114-120https://doi.org/10.17555/jvc.2022.39.3.114

### Fig 2.

Figure 2.Dead zone (a) and Axial/lateral resolution (b). (a) These images were obtained by using 4-6 MHz and Nine targets are clearly visualized. (b) Eleven targets are clearly visualized.
Journal of Veterinary Clinics 2022; 39: 114-120https://doi.org/10.17555/jvc.2022.39.3.114

### Fig 3.

Figure 3.(a) Vertical measurement test. The length indicate the 1th target and the 11th target measurements. (b) Horizontal measurement test. The length indicate the 1th target and the 5th target measurements.
Journal of Veterinary Clinics 2022; 39: 114-120https://doi.org/10.17555/jvc.2022.39.3.114

### Fig 4.

Figure 4.Sensitivity measurement. The length indicates the 1th target and the 4th target measurements.
Journal of Veterinary Clinics 2022; 39: 114-120https://doi.org/10.17555/jvc.2022.39.3.114

### Fig 5.

Figure 5.Focal zone test. The cross marks indicate the 7th target and the 11th target measurements for the distance of the line targets.
Journal of Veterinary Clinics 2022; 39: 114-120https://doi.org/10.17555/jvc.2022.39.3.114

### Fig 6.

Figure 6.Functional resolution measurement. 8, 6, 4, 3, 2 mm cystic structure are well visualized.
Journal of Veterinary Clinics 2022; 39: 114-120https://doi.org/10.17555/jvc.2022.39.3.114

### Fig 7.

Figure 7.Gray scale and dynamic image. The six targets have different values of brightness.
Journal of Veterinary Clinics 2022; 39: 114-120https://doi.org/10.17555/jvc.2022.39.3.114

### Fig 8.

Figure 8.Sample ultrasound image for evaluating COV. The average value was derived by setting 4 ROIs on the image that can measure the dynamic range.
Journal of Veterinary Clinics 2022; 39: 114-120https://doi.org/10.17555/jvc.2022.39.3.114

### Fig 9.

Figure 9.COV and BRISQUE (a) Results for coefficient of variation with respect to ultrasound device including average data. (b) Results for blind/referenceless image spatial quality evaluator (BRISQUE) with respect to ultrasound device.
Journal of Veterinary Clinics 2022; 39: 114-120https://doi.org/10.17555/jvc.2022.39.3.114

Table 1 Functional resolution evaluation criterion

 Functional resolution Appropriate number of targets observed at 8 mm and 6 mm target: 0, 7 target: –1, 6 target: –2, 5 target: –3, 4 target: –4, 3 target: –5, 2 target: –6, 1 target: –7, 0 target: –8 Appropriate number of targets observed at 4 mm, 3 mm, and 2 mm 17 target: 0, 16 target: –1, 15 target: –2, 14 target: –3, 13 target: –4, 12 target: –5, 11 target: –6, 10 target: –7, 9 target: –8, 8 target: –9, 7 target: –10, 6 target: –11, 5 target: –12, 4 target: –13, 3 target: –14, 2 target: –15, 1 target: –16, 0 target: –17

Table 2 Phantom image evaluation result

User*Dead zone (No)Vertical measurement (cm)Horizontal measurement (cm)Focal zone (%)Axial/lateral resolution (No)Sensitivity (No)Gray scale/dynamic range (No)
A910.38.910.31144
B910.49.210.41144
C910.58.910.51144
D910.38.910.31144
E910.18.910.11134
Fa910.28.710.21144
Fb9108.5101144
G98.28.78.21134
H910.49.410.41144
I910.49.110.41144

*University is A-I, University F has two equipment a and b.

### References

1. Adhikari S. High-frequency transducers for point-of-care ultrasound applications: what is the optimal frequency range? Intern Emerg Med 2014; 9: 463-466.
2. Babu J, Sudha GF. Adaptive speckle reduction in ultrasound images using fuzzy logic on Coefficient of Variation. Biomed Signal Process Control 2016; 23: 93-103.
3. Bok TH, Paeng DK. High frequency ultrasound and its applications to animal and human imaging focusing on vessel and blood. J Acoust Soc Korea 2010; 29: 73-85.
4. Choi JI, Kim PN, Jeong WK, Kim HC, Yang DM, Cha SH, et al. Establishing cutoff values for a quality assurance test using an ultrasound phantom in screening ultrasound examinations for hepatocellular carcinoma: an initial report of a nationwide survey in Korea. J Ultrasound Med 2011; 30: 1221-1229.
5. Dong KR, Bae SH, Lee WH, Kweon DC, Goo EH, Chung WK. Performance evaluation of ultrasonography using ATS-539 multipurpose phantom. J Adv Eng Technol 2010; 3: 133-137.
6. Fabiszewska E, Pasicz K, Grabska I, Skrzyński W, Ślusarczyk-Kacprzyk W, Bulski W. Evaluation of imaging parameters of ultrasound scanners: baseline for future testing. Pol J Radiol 2017; 82: 773-782.
7. Goodsitt MM, Carson PL, Witt S, Hykes DL, Kofler JM Jr. Real-time B-mode ultrasound quality control test procedures. Report of AAPM Ultrasound Task Group No. 1. Med Phys 1998; 25: 1385-1406.
8. Jung JJ, Choi JH, Jung JH. Image quality control for linear transducers with various frequencies by using an ATS-550 phantom. J Korean Phys Soc 2017; 28: 206-212.
9. Kim CS, Kang SS, Kim JH. Evaluation of quantitative image quality using frequency and parameters in the ultrasound image. J Korean Soc Radiol 2016; 10: 247-253.
10. Kim MJ, Lee JS, Ko SJ, Kang SS, Kim JH, Kim DH, et al. Evaluation of image quality using ATS-539 phantom and SNR in the ultrasonographic equipment. J Korea Contents Assoc 2013; 13: 284-291.
11. Kim PN, Lim JW, Kim HC, Yoon YC, Sung DJ, Moon MH, et al. Quality assessment of ultrasonographic equipment using an ATS-539 multipurpose phantom. J Korean Soc Radiol 2008; 58: 533-541.
12. Kim SH, Seo KH, Kang SH, Kim JH, Choi WH, Lee YJ. Feasibility study of improved patch group prior based denoising (PGPD) technique with medical ultrasound imaging system. J Magn 2017; 22: 55-59.
13. Lee S, Choi JI, Park MY, Yeo DM, Byun JY, Jung SE, et al. Intra- and interobserver reliability of gray scale/dynamic range evaluation of ultrasonography using a standardized phantom. Ultrasonography 2014; 33: 91-97.
14. Mittal A, Moorthy AK, Bovik AC. No-reference image quality assessment in the spatial domain. IEEE Trans Image Process 2012; 21: 4695-4708.
15. Thijssen JM, Weijers G, de Korte CL. Objective performance testing and quality assurance of medical ultrasound equipment. Ultrasound Med Biol 2007; 33: 460-471.

Vol.39 No.4 August, 2022

pISSN 1598-298X
eISSN 2384-0749