Low complexity local dimming algorithm for high quality head up displays in automotive vehicles

Abstract

We present a high image quality head up display (HUD) based on the local dimming liquid crystal display (LCD). The proposed HUD provides high white luminance to improve day-time visibility and low black luminance to cope with the postcard effect during the night by the local dimming technique. To reduce the computational complexity, the pixel-based boosting values for the pixel compensation are computed by simple bi-linear interpolation, complex power functions are implemented by look-up tables, some multiplications are replaced with the sums of shifted values, and the numbers of look-up tables and multipliers are reduced by sequential process. The simulation results of the proposed local dimming algorithm for 640×1280 LCD panel and 10×22 LED backlight ensure higher peak signal-to-noise ratio than 40 dB and larger power reduction performance than 73% over 7 HUD images. In addition, the postcard effect improvement is verified over images on LCD panel as well as windshield. Maximum luminance and contrast ratio on the windshield of the prototype HUD system are measured as 16,088 cd/m${}^2$ and $342\times {10}^6$:1.

Keywords:

• Head up display
• LCD
• local dimming
• postcard effect

1. Introduction

Automotive displays provide drivers and passengers with a wide range of visual information [1]. The instrument cluster displays essential information for the driver, such as speed, engine rotation speed, mileage, fuel status, and various alarms. The center information display is used to control various internal devices, including audio, video, and air conditioner, and also shows the contents of video and global positioning system (GPS). As display technology has developed, all visual information is now provided through electronic displays, and the display area has been extended to the space in front of the passenger seats [2–4].

However, since all of these displays are placed under the driver's line of sight, there is a limit to providing a safe driving experience due to driver distraction [5–9]. To solve this problem, information needs to be displayed on the windshield that matches the driver's gaze, and the head-up display (HUD) is a suitable display technology for this purpose. HUDs are designed as see-through displays, as the driver must be able to see the forward driving environment in addition to the displayed information. Most HUDs are implemented by projecting light on the front windshield to show driving information along with the front scene.

In a see-through display, the ambient illuminance determines the black luminance, or the zero-gray level [10]. As a result, the zero-gray level has a wide range of luminance, appearing very bright during the day and very dark at night. To ensure visibility of visual information even during the day, it is necessary to produce high-luminance images that can be viewed in bright ambient illuminance. Moreover, most HUDs experience significant light loss due to their optical systems composed of mirrors, which requires very high luminance displays as image sources.

Several display technologies can be considered as image sources for HUDs, including liquid crystal display (LCD), organic light-emitting diode (OLED) display, and micro light-emitting diode (micro-LED) display [11–15]. LCDs are composed of a backlight of LED light sources and a liquid crystal (LC) panel that controls each pixel's transmittance, making it easy to implement a high-brightness display that can be manufactured at a low cost. However, perfect black cannot be achieved due to light leakage, resulting in visual artifacts at night [16–18].

OLED displays have a separate light source for each pixel, enabling perfect black luminance. However, the high luminance requirement shortens the display's lifetime, leading to the image-sticking problem caused by accelerated OLED degradation. Micro-LED displays can achieve both high luminance and low black level simultaneously, but their complex manufacturing processes can increase the cost.

LCD is the most suitable display technology for HUDs as it can achieve high brightness and low cost. However, during night driving, light leakage can cause the black area of the display to be distinguishable from the surrounding dark area of the front scene, resulting in a visual artifact known as the postcard effect. To mitigate this issue, it is necessary to lower the zero-gray luminance further in the black area of the display. This can be achieved by dimming the corresponding region of the backlight. Therefore, LCD-based HUDs require a technology that can weaken the postcard effect by partially turning off the backlight in the black area of the image while maintaining the luminance of the backlight over the bright image area to prevent loss of visual information. This backlight control scheme is known as local dimming technology.

Local dimming technology comprises of two blocks: a dimming control block and a pixel compensation block. The dimming control block adjusts the brightness of local regions of a backlight according to the input image, while the pixel compensation block compensates for the effect of the darkened backlight by boosting pixel data to provide equal brightness to the original image. Local dimming technology can also reduce power consumption by dimming the backlight and enhance contrast ratio by lowering the zero-gray luminance further. However, unlike a panel that can be controlled discretely for each pixel, the light sources of the backlight have spread functions, which can lead to image quality issues such as luminance degradation, block artifacts, color distortion, and halo effects [19–21]. Various improvement techniques have been proposed to maximize the advantages of local dimming technology along with the improvement of picture quality [22–28]. However, there is still a problem of rising costs due to the sophisticated calculations involved in implementing these techniques.

On the other hand, since most images to be displayed on the HUD are simple visual information for the driver, static graphic images are often used, the proportion of the area occupied by the visual information is low, and there are large black areas. Thus, the local dimming technology can efficiently improve the postcard effect caused by a large black area, and dramatically reduce power consumption. Due to its relatively simple images compared to general displays like monitors and televisions, it is possible to simplify the overall algorithm further. In this paper, we propose a low complexity local dimming algorithm optimized for HUD applications that can reduce both the power consumption and the postcard effect.

2. Proposed local dimming scheme for LCD HUD

2.1. Proposed algorithm

The proposed algorithm is divided into six modules, as illustrated in Figure 1. The first module, Block Extraction, calculates the region of each block. The second module, Max, computes the maximum gray levels and corresponding dimming values of local blocks. The third module, $3\times 3$ Filtering, widens the dimming distribution by considering the light spread in the basis of a block. The fourth module, Block Value Extraction, extracts boosting weights of blocks to compensate for dimmed LEDs of backlight without consideration of the light spread characteristic. The fifth module, Bi-linear Interpolation, provides pixel-based boosting weights. Finally, the Pixel Compensation module increases pixel values by pixel-based boosting weights and sends boosted pixel data to an LCD panel. The dimming ratios extracted by the Max module are transferred to LED driver circuits to control the luminance of local blocks of a backlight by means of pulse width modulation (PWM).

Figure 1. Overall architecture of proposed local dimming algorithm. All blocks of the proposed one are included in the dotted box and the green boxes represent the external modules controlled by the proposed local dimming algorithm.

When a panel has $H\times W$ pixels and a backlight has $N\times M$ LED blocks, a panel is divided into $N\times M$ local blocks corresponding to LED blocks as illustrated in Figure 2. The $(i,j)$th region, which corresponds to the vertically ith and horizontally jth block, is defined by its center pixel coordinate ($(c{h}^i,c{w}^j)$), height ($2{\sigma }_h$), and width ($2{\sigma }_w$). The values of $(c{h}^i,c{w}^j)$ are calculated using Equations (1(1) $\begin{array}{rl}c{h}^i& =[\frac{H}{N}\cdot (i-1)+\frac{H}{2N}],i=1,\dots ,N\end{array}$(1) ) and (2(2) $\begin{array}{rl}c{w}^j& =[\frac{W}{M}\cdot (j-1)+\frac{W}{2M}],j=1,\dots ,M\end{array}$(2) ), while ${\sigma }_h$ and ${\sigma }_w$ are extracted using Equations (3(3) $\begin{array}{rl}{\sigma }_h& =\lceil \frac{H}{2N}\rceil \end{array}$(3) ) and (4(4) $\begin{array}{rl}{\sigma }_w& =\lceil \frac{W}{2M}\rceil \end{array}$(4) ), respectively. Here, $[x]$ denotes the nearest integer to x and $\lceil x\rceil$ denotes the least integer greater than or equal to x. The pixel coordinates of the upper-left and lower-right corners ($(u{l}^i,u{l}^j)$, $(l{r}^i,l{r}^j)$) are obtained using Equations (5(5) $\begin{array}{rl}u{l}^i& =max(c{h}^i-{\sigma }_h+1,1),i=1,\dots ,N\end{array}$(5) )–(7(7) $\begin{array}{rl}l{r}^i& =min(c{h}^i+{\sigma }_h,H),i=1,\dots ,N\end{array}$(7) ), and Equation (8(8) $\begin{array}{rl}l{r}^j& =min(c{w}^j+{\sigma }_w,W),j=1,\dots ,M\end{array}$(8) ). (1) $\begin{array}{rl}c{h}^i& =[\frac{H}{N}\cdot (i-1)+\frac{H}{2N}],i=1,\dots ,N\end{array}$(1) (2) $\begin{array}{rl}c{w}^j& =[\frac{W}{M}\cdot (j-1)+\frac{W}{2M}],j=1,\dots ,M\end{array}$(2) (3) $\begin{array}{rl}{\sigma }_h& =\lceil \frac{H}{2N}\rceil \end{array}$(3) (4) $\begin{array}{rl}{\sigma }_w& =\lceil \frac{W}{2M}\rceil \end{array}$(4) (5) $\begin{array}{rl}u{l}^i& =max(c{h}^i-{\sigma }_h+1,1),i=1,\dots ,N\end{array}$(5) (6) $\begin{array}{rl}u{l}^j& =max(c{w}^j-{\sigma }_w+1,1),j=1,\dots ,M\end{array}$(6) (7) $\begin{array}{rl}l{r}^i& =min(c{h}^i+{\sigma }_h,H),i=1,\dots ,N\end{array}$(7) (8) $\begin{array}{rl}l{r}^j& =min(c{w}^j+{\sigma }_w,W),j=1,\dots ,M\end{array}$(8) After specifying the local blocks, the maximum gray level ($G_{max}^{(i,j)}$) for each block is obtained using Equation (9(9) $\begin{array}{rl}{G}_{max}^{(i,j)}& =max\left[G_r^{(i,j)},G_g^{(i,j)},G_b^{(i,j)}\right]\end{array}$(9) ), and then the duty ratio ($D{R}^{(i,j)}$) for its backlight block is computed using Equation (10(10) $\begin{array}{rl}D{R}^{(i,j)}& ={\left(\frac{G_{max}^{(i,j)}}{255}\right)}^{2.2}\end{array}$(10) ). Here, $G_r^{(i,j)}$, $G_g^{(i,j)}$, and $G_b^{(i,j)}$ represent the red, green, and blue gray levels, respectively, in the $(i,j)$th local block. The gamma value of the display is given to be 2.2. (9) $\begin{array}{rl}{G}_{max}^{(i,j)}& =max\left[G_r^{(i,j)},G_g^{(i,j)},G_b^{(i,j)}\right]\end{array}$(9) (10) $\begin{array}{rl}D{R}^{(i,j)}& ={\left(\frac{G_{max}^{(i,j)}}{255}\right)}^{2.2}\end{array}$(10) Whereas a panel is controlled in the basis of a pixel, LED blocks cannot prevent light from spreading into the adjacent blocks. To reflect this spreading effect, we perform $3\times 3$ filtering through convolution over the $N\times M$ matrix of duty ratios with zero-padding as shown in Figure 3. The filter coefficients are obtained from the light profile of a LED and the center coefficient is set to 1.0 to avoid clipping artifacts [20]. However, the total sum of the $3\times 3$ filter's elements goes beyond 1.0 and the filtering can result in a larger duty ratio than 1.0. Therefore, the resultant filtered duty ratios ($D{R}_f^{(i,j)}$) are clipped at 1.0.

Figure 2. Local blocks in a panel when a panel has $H\times W$ pixels and a backlight has $N\times M$ LED blocks. These blocks correspond to LED blocks of a backlight.

Figure 3. $3\times 3$ filtering by the convolution with zero-padded $(N+2)\times (M+2)$ duty ratio matrix. The resultant filtered duty ratio matrix has the same size of $N\times M$ as the original duty ratio matrix. The center coefficient of the $3\times 3$ filter is set to be 1.0.

As expressed in Equation (11(11) $\begin{array}{rl}B{R}^{(i,j)}& =\{\begin{array}{cc}{\displaystyle {\left(\frac{1}{D{R}_f^{(i,j)}}\right)}^{\frac{1}{2.2}},}& \mathrm{if}D{R}_f^{(i,j)}\ne 0\\ 0,& \mathrm{if}D{R}_f^{(i,j)}=0\end{array}\end{array}$(11) ), Block Value Extraction produces boosting ratios ($B{R}^{(i,j)}$) of local blocks of the panel based on $D{R}_f^{(i,j)}$. These block boosting ratios are used for the center pixels of local blocks, while the boosting ratios of other pixels ($P{R}^{(h,w)},h=1,2,\dots ,H,w=1,2,\dots ,W$) are generated by simple bi-linear interpolation of adjacent blocks' block boosting ratios to cope with the spreading effects of LEDs, as illustrated in Figure 4.

Figure 4. Bi-linear interpolation to generate pixel-based boosting ratio ($P{R}^{(h,w)}$) from block boosting ratios ($B{R}^{(i-1,j-1)}$, $B{R}^{(i-1,j)}$, $B{R}^{(i,j-1)}$, $B{R}^{(i,j)}$) of adjacent blocks.

If the pixel coordinate is $(h,w)$ in the $(i-1,j)$th block and three adjacent blocks are $(i-1,j-1)$, $(i,j-1)$, and $(i,j)$, then ${\mathrm{\Delta }}_h$ and ${\mathrm{\Delta }}_w$ are calculated by Equations (12(12) $\begin{array}{rl}{\mathrm{\Delta }}_h& =h-c{h}^{i-1}\end{array}$(12) ) and (13(13) $\begin{array}{rl}{\mathrm{\Delta }}_w& =w-c{w}^{j-1}\end{array}$(13) ). After that, $S^{(j-1)}$ and $S^{(j)}$ are obtained by applying linear interpolations vertically between $B{R}^{(i-1,j-1)}$ and $B{R}^{(i,j-1)}$, and between $B{R}^{(i-1,j)}$ and $B{R}^{(i,j)}$ as explained in Equations (14(14) $\begin{array}{rl}{S}^{(j-1)}& =\frac{(2{\sigma }_h-{\mathrm{\Delta }}_h)\cdot B{R}^{(i-1,j-1)}+{\mathrm{\Delta }}_h\cdot B{R}^{(i,j-1)}}{2{\sigma }_h}\end{array}$(14) ) and (15(15) $\begin{array}{rl}{S}^{(j)}& =\frac{(2{\sigma }_h-{\mathrm{\Delta }}_h)\cdot B{R}^{(i-1,j)}+{\mathrm{\Delta }}_h\cdot B{R}^{(i,j)}}{2{\sigma }_h}\end{array}$(15) ).

$P{R}^{(h,w)}$ is produced by their linear interpolation in the horizontal direction as described in Equation (16(16) $\begin{array}{rl}P{R}^{(h,w)}& =\frac{(2{\sigma }_w-{\mathrm{\Delta }}_w)\cdot S^{(j-1)}+{\mathrm{\Delta }}_w\cdot S^{(j)}}{2{\sigma }_w}\end{array}$(16) ). Finally, the compensated pixel data ($C{P}^{(h,w)}$) are generated by multiplying gray levels ($G^{(h,w)}$) of the input image pixels with corresponding pixel boosting ratios ($P{R}^{(h,w)}$) as expressed in Equation (17(17) $\begin{array}{rl}C{P}^{(h,w)}& =P{R}^{(h,w)}\times G^{(h,w)}\end{array}$(17) ) and transferred to the panel. (11) $\begin{array}{rl}B{R}^{(i,j)}& =\{\begin{array}{cc}{\displaystyle {\left(\frac{1}{D{R}_f^{(i,j)}}\right)}^{\frac{1}{2.2}},}& \mathrm{if}D{R}_f^{(i,j)}\ne 0\\ 0,& \mathrm{if}D{R}_f^{(i,j)}=0\end{array}\end{array}$(11) (12) $\begin{array}{rl}{\mathrm{\Delta }}_h& =h-c{h}^{i-1}\end{array}$(12) (13) $\begin{array}{rl}{\mathrm{\Delta }}_w& =w-c{w}^{j-1}\end{array}$(13) (14) $\begin{array}{rl}{S}^{(j-1)}& =\frac{(2{\sigma }_h-{\mathrm{\Delta }}_h)\cdot B{R}^{(i-1,j-1)}+{\mathrm{\Delta }}_h\cdot B{R}^{(i,j-1)}}{2{\sigma }_h}\end{array}$(14) (15) $\begin{array}{rl}{S}^{(j)}& =\frac{(2{\sigma }_h-{\mathrm{\Delta }}_h)\cdot B{R}^{(i-1,j)}+{\mathrm{\Delta }}_h\cdot B{R}^{(i,j)}}{2{\sigma }_h}\end{array}$(15) (16) $\begin{array}{rl}P{R}^{(h,w)}& =\frac{(2{\sigma }_w-{\mathrm{\Delta }}_w)\cdot S^{(j-1)}+{\mathrm{\Delta }}_w\cdot S^{(j)}}{2{\sigma }_w}\end{array}$(16) (17) $\begin{array}{rl}C{P}^{(h,w)}& =P{R}^{(h,w)}\times G^{(h,w)}\end{array}$(17)

2.2. Low complexity implementation

In addition to the simple bi-linear interpolation for the pixel boosting ratio generation, we employ three more methods to reduce the computational complexity such as look-up table (LUT), multiplication approximation based on the sum of shifted values, and sequential operation. First, we implement power functions of DR and BR using LUTs, with input data serving as addresses and pre-calculated function outputs as shown in Figure 5. To optimize the size of LUTs, we also optimize the number of bits for DR and BR.

Second, multiplication operations for the 3×3 filtering with coefficients of floating numbers ($f_1,f_2,f_3$) are replaced with the approximate sum of three or fewer shifted input values by $2^{-{\alpha }_i}$, $2^{-{\beta }_i}$, and $2^{-{\gamma }_i}$ (i = 1, 2, 3) as depicted in Figure 6. Therefore, only adders are used for the 3×3 convolution without any multipliers. Additionally, the divisions by ${\sigma }_h$ and ${\sigma }_w$ in the Bi-linear Interpolation are approximated by the sum of shifted values.

Figure 5. LUT-based implementation for the power functions of DR and BR. The non-linear functions such as gamma and inverse-gamma functions are replaced by LUTs for the hard complexity reduction.

Third, the whole operation is sequentially conducted to reduce the number of LUTs as well as multipliers. Therefore, the number of required LUTs is two for DR and BR and the total number of multipliers is nine that consist of 4 multipliers for $S^{(j-1)}$ and $S^{(j)}$, 2 multipliers for PR and 3 multipliers for red, green, and blue CP values. In contrast, conventional methods [29, 30] require the update of the pixel-based backlight profile to remove block artifacts. These conventional methods require a large-size memory to store the profile of the LED's nonlinear spreading function at all pixel positions of several blocks. This paper considers 9 blocks including one corresponding block and 8 adjacent blocks and each block contains approximately 58×64 pixels, resulting that the memory of the spreading function should store 33,408 values while the memory of the proposed method contains just 3 values of $f_1$, $f_2$, and $f_3$ due to the block-based convolution. In addition, the backlight luminance at a pixel position has to be estimated from the light profiles of 9 LEDs. They require 9 multipliers of LED dimming ratios and LED profiles sine it is too difficult to apply the approximation with shifts and adders to such a large number of coefficients in the conventional methods. Then, PR values in a panel are calculated from the resultant backlight luminance values, however, the proposed method uses only 6 multipliers to estimate them. Furthermore, as the conventional algorithms support the wide range of coefficient values for the light spreading model, the bit depths of the multipliers should be much bigger than the proposed one. As a result, conventional local dimming approaches require much larger memory for the backlight profile model and 3 more large bit-depth multipliers for PR estimation than the proposed scheme that requires only the memory of 3 coefficients for the backlight profile model and 6 multipliers for the bi-linear interpolation at the pixel domain.

Figure 6. Filter coefficient approximations with shifts and adders. ${\alpha }_i$, ${\beta }_i$, and ${\gamma }_i$ (i = 1, 2, 3) are positive integers.

The overall process is described in Figure 7. $G_{max}^{\mathrm{Seq}}$ and $D{R}_f^{\mathrm{Seq}}$ are the data sequences generated by serializing the parallel data of $G_{max}^{(i,j)}$ and $D{R}_f^{(i,j}$ to reduce the number of LUTs. $\mathrm{FI}{N}_1$ to $\mathrm{FI}{N}_9$ are inputs of the 3×3 filter and $\mathrm{IPI}{N}_1$ to $\mathrm{IPI}{N}_4$ are the serialized input sequences for the bi-linear interpolation to reduce the number of multipliers. Yellow, blue, and red colors represent that the data are generated from $(k-1)$th frame, (k)th frame, and $(k+1)$th frame, respectively. The gray areas in $G_{max}^{(i,j)}$ represent the interval where the values are updated to obtain the maximum gray level in the $(i,j)$th local block. The coordinates in the boxes indicate the position of local blocks. Consequently, the current frame data are compensated for by the pixel boosting ratios as well as the backlight dimming ratios obtained from the previous frame to avoid additional frame memory.

Figure 7. Overall sequential process of the proposed local dimming algorithm. Because the process is conducted in a sequential way, the number of LUTs and the number of multipliers are substantially reduced.

3. Experimental results

3.1. Simulation results

Resolution and frame rate of the target LCD display are 640×1280 and 60 Hz. The number of LEDs in a backlight is 10×22, where a single LED is assigned to a block. The light profile is approximated by Equation (18(18) $y(h,w)=\exp \left(-\frac{h^2}{1517.995}\right)\cdot \exp (-\frac{w^2}{1313.641})$(18) ) when the LED is located at the coordinate of (0,0), where h and w are presented in units of lines and pixels, respectively. Therefore, the coefficients of the 3×3 filter are estimated as Table 1. Seven HUD images, as presented in Figure 8, are used for the proposed local dimming algorithm optimization with respect to bit depths. Because HUD usually shows the simple images including the driving information, the specific images are prepared for the evaluation of the proposed local dimming scheme. They are generated using a similar design with a large zero-gray area, but they contain different driving information and include some gradient patterns for evaluating the performance of the local dimming algorithm. (18) $y(h,w)=\exp \left(-\frac{h^2}{1517.995}\right)\cdot \exp (-\frac{w^2}{1313.641})$(18)

Figure 8. 7 HUD images for the test. They have large black areas and the driving information is presented with simple images.

Table 1. Coefficient values of 3×3 filter.

Coefficient	$f_1$	$f_2$	$f_3$
Value	0.078	0.067	0.0052

The baseline performance is obtained in terms of peak signal-to-noise ratios (PSNRs) over the floating point simulation. Then, the bit depths from the LUT of DR values to PR values are optimized with the integer simulation while maintaining the PSNR difference from the baseline within around 1.0 dB. Resultant PSNRs are esimated as shown in Table 2 along with the baseline as well as power reduction ratios, comparing with them of the conventional scheme with the backlight image estimation by the floating-point simulation. The performance gap between conventional and proposed methods are smaller than 1 dB, where the errors in the conventional scheme are caused only by the 8-bit quantization of the compensated panel data. Power reduction ratios (PRR) are calculated by Equation (19(19) $\mathrm{PRR}(\mathrm{\%})=\frac{P_{\mathrm{wo}}-P_w}{P_{\mathrm{wo}}}\times 100$(19) ), where $P_{\mathrm{wo}}$ and $P_w$ are the power consumption without and with the proposed local dimming technique. The power consumption is assumed to be proportional to the luminance of LEDs. The average PSNRs of baseline and integer simulations are 42.516 dB and 41.915 dB, respectively, while the average PRR is 75.15%. Finally, optimized bit depths are illustrated in Figure 9. Coefficients of the 3×3 filter for $D{R}_f^{(i,j)}$ are approximated with the sum of shifted values as shown in Table 3. Because $2{\sigma }_h$ is 64, which is $2^6$, the division by $2{\sigma }_h$ for the interpolation is easily replaced with the 6-bit shifter of $2^{-6}$. On the other hand, since $2{\sigma }_w$ is 1280/22, we have two values of 58 and 59. Thus, the divisions by 58 and 59 are substituted with the sum of shifted values by $2^{-6}+2^{-10}+2^{-11}+2^{-15}$ and $2^{-6}+2^{-10}+2^{-12}+2^{-15}$, respectively. The comparison between the original input images and output images of the proposed local dimming algorithm is presented in Figure 10 with corresponding backlight images. Since the simulation does not include the model of the light leakage, the contrast ratio improvement cannot be perceivable in the simulation results. This enhancement is addressed in the following section. (19) $\mathrm{PRR}(\mathrm{\%})=\frac{P_{\mathrm{wo}}-P_w}{P_{\mathrm{wo}}}\times 100$(19) In addition, to verify the importance of the 3×3 filter for $D{R}_f^{(i,j)}$, we evaluate PSNR values over seven HUD images by the local dimming algorithm without the 3×3 filter. As summarized in Table 4, the scheme with the filter outperforms one without the filter by more than 6 dB because the 3×3 filter alleviates the over-boosting artifacts caused by the large dimming ratio difference between adjacent blocks.

Figure 9. Optimized bit depths for the proposed local dimming scheme. The optimization is achieved by maintaining PSNR values within 1 dB, compared with the floating-point implementation.

Figure 10. Simulation results for 7 HUD Images. Left, center, and right images are original, backlight, and output images for the proposed local dimming scheme, respectively.

Table 2. Performance comparison between floating-point and integer simulation results and power reduction ratios.

Images	#1	#2	#3	#4	#5	#6	#7
Conventional (dB)	42.967	42.996	42.617	42.558	42.889	43.248	42.741
Floating (dB)	42.426	43.653	41.905	42.559	42.042	42.959	42.071
Integer (dB)	41.837	43.146	41.161	42.190	41.670	42.528	40.874
PRR (%)	75.52	73.51	73.46	74.32	76.61	77.48	75.18

Table 3. Approximated coefficients of the 3×3 filter for $D{R}_f^{(i,j)}$.

Coefficient	$f_1$	$f_2$	$f_3$
Approximation	$2^{-4}+2^{-6}$	$2^{-4}+2^{-8}+2^{-10}$	$2^{-8}+2^{-10}+2^{-11}$

Table 4. PSNR (dB) comparison between local dimming schemes without and with the 3×3 filter.

Images	#1	#2	#3	#4	#5	#6	#7	Average
Without	35.870	36.941	35.959	35.315	35.915	35.324	35.217	35.791
With	41.837	43.146	41.161	42.190	41.670	42.528	40.874	41.915

3.2. Measurement results

Our prototype HUD system is composed of 640×1280 LCD panel, 10×22 LED backlight, LED driver board, dimming algorithm board with a field programmable gate array (FPGA) integrated chip (IC), two mirrors, and windshield glass as illustrated in Figure 11. At the viewing distance of 3 m, the display size projected on the windshield is 25-inch and its field of view is ${2.864}^{\circ }\times {11.746}^{\circ }$. The LCD display module is presented in Figure 12(a) and the HUD image is flipped horizontally on the LCD screen as shown in Figure 12(b) because the final image projected on the windshield is horizontally flipped again by mirrors as depicted in Figure 12(c). The specification of the used LCD panel is summarized in Table 5.

Figure 11. Overall architecture of the prototype HUD system. The images on the screen of a backlight dimming LCD are transmitted to the windshield through two mirrors and the images projected on the windshield are transferred to the eyes of drivers.

Figure 12. Photographs of the LCD module. (a) LCD module, (b) Displayed image on LCD screen and (c) Projected image on windshield.

Table 5. LCD panel specification.

Item	Description
Screen size	4.1 in.
Resolution	640(H) × 1280(W) × 3(R/G/B)
Pixel pitch	0.0723 mm
Color depth	16.7M colors
Transmittance	7.2% (typical)
Contrast ratio	1500:1 (typical)

The dimming algorithm board consists of deserializer, LVDS-to-RGB, FPGA, RGB-to-LVDS, and EEPROM as Figure 13(a), where LVDS and EEPROM stand for low voltage differential signaling and electrically erasable programmable read-only memory, respectively. After the serialized data of the input image are converted into the parallel RGB data, FPGA produces the pixel compensated image for the LCD panel and PWM values for 220 LEDs of the backlight. The LED driver board includes 15 LED driver ICs, where one LED driver IC covers 14 or 16 LEDs as depicted in Figure 13(b). The serial peripheral interface (SPI) in the fashion of a daisy chain [31] is used to update PWM values of LED driver ICs. The measurement results on the LCD screen are presented in Figure 14, where the backlight image is produced with the full white image for the panel and the PWM data of the test image for the LED backlight. Apparently, the reduced light leakages on the black area lead to the dramatic improvement on the contrast ratios. In addition, the power consumption of the LCD module including panel, backlight, dimming algorithm board, and LED driver board, is measured over seven HUD images as presented in Table 6. The average measured PRR over seven HUD images is 73.496%.

Figure 13. Board block diagrams. (a) Dimming algorithm board and (b) LED driver board. FPGA takes the input image and then generates the compensated image for a panel as well as the dimming data for a backlight. The backlight consists of 220 LEDs adjusted by 15 LED drivers. LED drivers are connected with FPGA via the SPI daisy chain structure.

Figure 14. Measurement results on the LCD screen for 7 HUD images. While left pictures represent actual images captured on the screen without dimming, center and right pictures are backlight and screen images with dimming. The backlight images are generated by applying a full white image to the panel.

Table 6. Measured power consumption of the LCD module over 7 HUD images.

Image	#1	#2	#3	#4	#5	#6	#7	Average
$P_{\mathrm{wo}}$ (W)	76.943	76.422	76.350	76.523	76.422	76.775	76.619	76.579
$P_w$ (W)	20.107	21.424	21.398	20.775	19.311	18.759	20.294	20.295
PRR (%)	73.87	71.97	71.97	72.85	74.73	75.57	73.51	73.496

The prototype local dimming HUD system supports the maximum luminance of 16,088 cd/m${}^2$ and extends the contrast ratio up to $342\times {10}^6$:1 over the image projected at the windshield. As shown in Figure 15, it is ensured that the postcard effect is improved by the proposed local dimming technology. Because the LED backlight includes LEDs as well as LED driver ICs, the measured reduction ratio is slightly lower than the simulated one.

4. Conclusion

This paper demonstrates the high image quality HUD system based on the local dimming LCD display to cope with postcard effect and large power consumption. By using the HUD characteristics that most images are almost static with driving information and the portion of the dark area is large, the maximum gray level is used to determine the dimming ratio and the pixel boosting ratios for the pixel compensation are extracted by the simple bi-linear interpolation. In addition, the computational complexity is further reduced by means of LUT, multiplication approximation with the sum of shifted values, and sequential process. Therefore, only 2 LUTs and 9 multipliers are deployed for the whole computation. Both simulation and measurement results ensure that the proposed HUD module reduces the postcard effect as well as the power consumption. The proposed local dimming algorithm will provide safe and comfortable experience with drivers and passengers.

Figure 15. Projected HUD image on the windshield without and with the local dimming technology. The local dimming scheme ameliorates the postcard effect.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was supported by Hyundai Mobis Co., Ltd.

Notes on contributors

J. H. Kim

J. H. Kim received the B.S. degree from the Department of Information Display, Kyung Hee University, Seoul, South Korea, in 2022, where he is currently pursuing the M.S. degree. His current research is focused on display electronics and deep learning applications.

Y. J. Pyun

Y. J. Pyun received the B.S. degree from the Department of Information Display, Kyung Hee University, Seoul, South Korea, in 2021, where he is currently pursuing the M.S. degree. His current research is focused on image processing networks based on deep learning.

J. H. Oh

J. H. Oh received M.S. degree in department of EEE from Yonsei University, Seoul, Korea in 2010. And received B.S. degree in department of EECE from University of Seoul, Seoul, Korea in 2003. He worked LG Electronics in 2015 as a senior research engineer and he worked Samsung Electronics in 2003 as a senior engineer. He is currently a principal research engineer at Hyundai Mobis, Yong-in, Korea. His current research is focused on Head-up Displays.

K. H. Song

K. H. Song received the B.S. degree of Mechanical Engineering from Hongik University, Seoul, South Korea, in 2002. He Joined Samsung Electronics in 2003, where he had worked on Active Matrix Liquid-Crystal Displays. After that, he joined LG Electroincs in 2010, where he had worked on Projection Displays. He is currently a principal research engineer at Hyundai Mobis, Yong-in, South Korea. His current research is focused on Head-up Displays and automotive displays.

Y. N. Kim

Y. N. Kim received the B.S. degree from the Department of Information Display, Kyung Hee University, Seoul, South Korea in 2020. She is currently a research engineer at Hyundai Mobis, Yong-in, South Korea. Her current research is focused on Head-up Displays and automotive displays.

J. H. Cho

J. H. Cho received the B.S degree from School of Electronic and Electrical Engineering, Sungkyunkwan University, Seoul, South Korea, in 2012. He has been in charge of HW development of instrument cluster and head up display at Hyundai Mobis.

C. Y. Yoon

C. Y. Yoon received his B.S. and M.S. degrees from Yonsei University, Seoul, Korea in 2002 and 2004, respectively. Following the studies, he joined LG Electronics in 2004 and dedicated the next 11 years to working as an optical engineer with a focus on display technology. In 2015, he joined Hyundai Mobis, where he initially worked as a HUD optical engineer for 5 years. Currently, he is leading the HUD Optics Cell, actively involved in the development of HUD mass production and HUD advanced technologies.

Y. H. Han

Y. H. Han is currently head of electronic convenience & control lab. In Mobis. He is now in charge of development from display product such as center stack display (CSD), HUD and instrument cluster to body domain controller. He majored in RF system at Korea University, and since graduate, has been working for automotive industry for 24 years.

H. Nam

H. Nam received his B.S., M.S. and Ph.D. degrees in EECS from Korea Advance Institute of Science and Technology (KAIST), Daejon Korea, in 1996, 1998, and 2004. He joined Samsung Electronics as a senior engineer in 2005, where he had worked on Active-Matrix Liquid-Crystal Displays. He is currently an associate professor in the department of Information Display at Kyung Hee University, Seoul Korea. His current research interests are low power technologies, integrated circuits, signal/user interfaces for flat panel displays, and machine learning application.