Counting Photons to See Through Fog: Peplography and COLaNoPS

Presented at ICCAS 2025 with Yuma Takahashi (first author), Myungjin Cho, and my advisor Min-Chul Lee: “A Research on Scattering Media Removal and Photon Estimation using COLaNoPS.” ARMS attacks scattering media — fog, smoke, haze — from the angle of “how much of the signal is line-of-sight versus scattered.” This paper attacks a related but distinct problem: what happens when, on top of the scattering, there simply aren’t very many photons to work with in the first place.

Why photon-starved conditions are their own problem

Drive recorders, around-view monitors, security cameras at night, anything shooting through heavy fog with a fog generator turned up too high — these are all cases where the sensor is only catching a trickle of the light that left the object. Under those conditions, treating a pixel value as a continuous, noise-free measurement of scene brightness stops being a reasonable approximation. Photons “rarely exist in unit time and space” under photon-starved conditions, and the accepted way to model that rarity is a Poisson distribution rather than continuous intensity.

That reframing is the whole basis of photon-counting algorithms: instead of asking “how bright is this pixel,” you ask “how many discrete photon-arrival events would we expect here, given a fixed total photon budget for the image.” The photon-counting model is:

\[\lambda(x,y) = \frac{I(x,y)}{\sum_{x=1}^{N_x}\sum_{y=1}^{N_y} I(x,y)}, \qquad C(x,y)\,|\,\lambda(x,y) \sim \text{Poisson}\big[N_p \cdot \lambda(x,y)\big]\]

$I(x,y)$ is the raw pixel value, $\lambda(x,y)$ normalizes it into a probability distribution over the whole image, and $N_p$ is the total number of photons you’re choosing to simulate/estimate across the frame. Each pixel’s final value $C(x,y)$ is then drawn from a Poisson distribution with mean $N_p \cdot \lambda(x,y)$ — brighter pixels get proportionally more expected photon counts, but the actual realized count is still random, which is exactly the shot noise you’d expect from a real photon-starved sensor.

graph LR
    classDef in fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px,color:#000;
    classDef math fill:#fff3e0,stroke:#fb8c00,stroke-width:2px,color:#000;
    classDef out fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000;

    I["Raw pixel value<br/>I(x,y)"]:::in --> Norm["Normalize to<br/>lambda(x,y),<br/>sums to 1 over image"]:::math
    Np["Total photon budget<br/>N_p (e.g. 40,000-130,000)"]:::in --> Mean["Expected count =<br/>N_p * lambda(x,y)"]:::math
    Norm --> Mean
    Mean --> Poisson["Draw C(x,y) ~<br/>Poisson(mean)"]:::math
    Poisson --> Out["Photon-counted<br/>pixel value"]:::out

Lower $N_p$ means fewer total photons to distribute across the image, which means more relative shot noise per pixel — a direct, tunable knob for “how photon-starved is this simulation.”

Peplography: removing scattering, then counting what’s left

Photon-counting alone doesn’t remove fog — it just models how few photons you have to work with. Peplography (Cho & Javidi, Optics Letters, 2016) is the scattering-removal half of the pipeline this paper builds on, and it runs in three steps.

Step 1 — estimate the scattering media locally. Slide a small window (the windowing size, $w_x \times w_y$) across the image, and at each position average the pixel values inside it. That local average is Peplography’s estimate of how much of the signal at that spot is just scattering media (fog/smoke) rather than object:

\[M_{ij} = \frac{1}{w_x w_y}\sum_{m=1}^{w_x}\sum_{n=1}^{w_y} X_{ij}(m,n)\]

Step 2 — subtract it out. $I’ = I - M$. Wherever the original pixel value was close to the local scattering estimate, $I’$ goes to near zero — that’s classified as “just fog.” Wherever the original pixel value was well above the local estimate, $I’$ stays high — that’s the object poking through.

Step 3 — apply photon-counting to $I’$. The difference image still isn’t very visualizable directly, especially under low-light conditions, so the photon-counting model from above is applied to $I’$ to produce the final reconstructed image.

graph TD
    classDef step fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px,color:#000;
    classDef result fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000;

    Orig["Original image I<br/>(peplogram, object + fog)"]:::step --> S1["1. Estimate scattering media M<br/>local windowed average"]:::step
    S1 --> S2["2. Subtract:<br/>I' = I - M"]:::step
    S2 --> S3["3. Apply photon-counting<br/>algorithm to I'"]:::step
    S3 --> Out["Reconstructed image,<br/>object visualized"]:::result

The windowing-size trade-off

The windowing size in Step 1 isn’t a free parameter you can just crank up or down without consequence. Testing windowing sizes of 1/2 and 1/16 of the smaller image dimension on the same photon-starved fog image ($N_p = 100{,}000$) shows the trade-off directly: the large window (1/2 the image) recovers information about the whole object, while the small window (1/16 the image) only recovers edge/detail information. There’s no single windowing size that gets you both — which is precisely the gap this paper’s proposed method is built to close.

graph TD
    classDef win fill:#fff3e0,stroke:#fb8c00,stroke-width:2px,color:#000;
    classDef res fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000;

    W1["Windowing size = 1/2<br/>of smaller image dimension"]:::win --> R1["Whole-object information<br/>coarse, but complete shape"]:::res
    W2["Windowing size = 1/16<br/>of smaller image dimension"]:::win --> R2["Edge/detail information<br/>fine structure, incomplete shape"]:::res

COLaNoPS: spend the photon budget where the object is

Conventional Peplography applies the same photon budget $N_p$ uniformly across the whole image, object and background alike. COLaNoPS (Classify Object Location and the Number of Photons by Section — originally proposed by J.U. Ha, H.-W. Kim, M. Cho, and M.-C. Lee, Sensors, 2025, and applied here) instead asks: why give the same photon budget to background fog as to the object we actually care about?

COLaNoPS runs a rectangular kernel across the image at regular intervals, and for each kernel position does two things.

Object recognition. Compute the average pixel value inside the kernel, $K_{average}(x,y)$. Do this for every kernel position across the whole image, then run K-means on the full set of per-kernel averages to find a threshold $\gamma_K$ that best separates two clusters: “object-like” and “background-like.” Any kernel region averaging at or above $\gamma_K$ is classified as object and gets the full photon budget $N_p$; anything below is classified as background and gets only $N_p/2$:

\[\mu_p = \begin{cases} N_p, & K_{average}(x,y) \ge \gamma_K \\ N_p/2, & K_{average}(x,y) < \gamma_K \end{cases}\]

graph TD
    classDef in fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px,color:#000;
    classDef math fill:#fff3e0,stroke:#fb8c00,stroke-width:2px,color:#000;
    classDef obj fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000;
    classDef bg fill:#ffebee,stroke:#c62828,stroke-width:2px,color:#000;

    Kernel["Kernel slides across image,<br/>average pixel value per position"]:::in --> KM["K-means over all<br/>kernel averages"]:::math
    KM --> Thresh["Threshold gamma_K"]:::math
    Thresh --> Obj["Average >= gamma_K<br/>-> classified OBJECT"]:::obj
    Thresh --> Bg["Average < gamma_K<br/>-> classified BACKGROUND"]:::bg
    Obj --> Pfull["Photon budget = N_p<br/>(full)"]:::obj
    Bg --> Phalf["Photon budget = N_p / 2<br/>(half)"]:::bg

Spatial convolution. Photon-counting is then applied inside each kernel region using its assigned local budget $\mu_p$, and since the kernel slides with overlap between neighboring positions, every pixel ends up covered by multiple overlapping kernel estimates. The final reconstructed value at each pixel is the sum of all overlapping kernel contributions divided by how many kernels overlapped there — an overlap-averaging step that suppresses the random shot noise that a single-pass, non-overlapping estimate would leave in:

\[R(x,y) = \frac{1}{O_{spatial}(x,y)} \sum_i \sum_j \tilde{C}_{kl}\{x - S_i,\, y - S_j\}\]

graph LR
    classDef step fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px,color:#000;
    classDef result fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000;

    K1["Kernel slides across image<br/>at fixed pixel stride"]:::step --> K2["Photon-counting applied<br/>within each kernel,<br/>using its object/background budget"]:::step
    K2 --> K3["Overlapping kernels each<br/>contribute an estimate<br/>per pixel"]:::step
    K3 --> K4["Sum contributions,<br/>divide by overlap count<br/>O_spatial(x,y)"]:::step
    K4 --> Out["Reconstructed image,<br/>random noise averaged down"]:::result

The net effect: the object gets visualized with its full photon allocation (sharper, less noisy where it matters), the background gets a leaner budget (it was never the point), and the overlap-averaging knocks down the Poisson shot noise that a naive per-kernel estimate would otherwise leave speckled across the image.

This paper’s contribution: two-pass, object-detection-driven windowing

Recall the windowing-size trade-off above: a large window gets you the whole object shape, a small window gets you fine detail, and neither gets you both. Rather than picking one windowing size and living with the trade-off, this paper runs Peplography twice and averages the results.

Pass 1 uses a windowing size of 1/2 the smaller image dimension — the assumption being that the object roughly occupies half the frame — visualized with COLaNoPS. Object detection is then run on that first-pass result to get a bounding box around the recovered object.

Pass 2 uses a windowing size of 1/2 the smaller dimension of that bounding box — a much smaller, object-scaled window — visualized with COLaNoPS again.

The final output is the average of the two passes: the coarse pass supplies the overall object shape, the fine pass (now correctly scaled to the object’s actual size, not the whole image) supplies detail that a single fixed windowing size would have missed either way.

graph TD
    classDef step fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px,color:#000;
    classDef detect fill:#fff3e0,stroke:#fb8c00,stroke-width:2px,color:#000;
    classDef result fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000;

    P1["Pass 1: windowing size =<br/>1/2 of smaller image dimension"]:::step --> C1["COLaNoPS<br/>visualization"]:::step
    C1 --> Det["Object detection<br/>-> bounding box"]:::detect
    Det --> P2["Pass 2: windowing size =<br/>1/2 of smaller bounding-box dimension"]:::step
    P2 --> C2["COLaNoPS<br/>visualization"]:::step
    C1 --> Avg["Average Pass 1 + Pass 2"]:::result
    C2 --> Avg
    Avg --> Final["Final reconstructed image:<br/>whole shape + finer detail"]:::result

Experimental setup and results

The setup reproduces scattering media physically rather than synthetically: an object in a water tank, 800mm from the camera, with a fog generator producing the scattering medium. Camera and kernel specs:

Parameter	Value
Camera	NIKON D850
Resolution	3000 × 2000
Sensor size	35.9mm × 23.9mm
Kernel size	1000 × 1000
Kernel shifting pixel	100
Focal length	145mm
ISO	100

At $N_p = 80{,}000$, the conventional method’s whole-image result doesn’t show a recognizable object at all — it’s dominated by noise. The proposed (two-pass + COLaNoPS) method’s result shows the object clearly. Cropping in on the object itself makes the gap even starker: the conventional method’s crop is essentially unreadable random noise, while the proposed method’s crop shows a visibly lower-noise object, distinguishable from the background because object regions consistently carry higher brightness than background regions — a direct consequence of COLaNoPS assigning them a larger photon budget.

Sweeping $N_p$ from 40,000 up to 130,000 and evaluating with three metrics — NormXCorr, SSIM, and PSNR — the proposed method beats the conventional method across the entire sweep, on all three metrics, without exception. NormXCorr and PSNR for the proposed method both climb early in the sweep and then plateau at higher $N_p$; SSIM for the proposed method actually trends slightly downward as $N_p$ increases while still staying well above the conventional method’s flat, near-zero SSIM throughout. The conventional method never catches up on any metric at any tested photon budget.

The honest limitation, stated directly in the paper: even the proposed method’s output isn’t sharp enough to read fine text on the object. That’s flagged as the next problem to solve, not something this paper claims to have already fixed.

Where this sits next to ARMS

ARMS and this paper are really two different axes of the same underlying question: how do you recover a signal when the imaging conditions are actively working against you. ARMS asks it in terms of how much of the light is scattered versus direct (Rician/Rayleigh statistics, Fourier-domain masking). This paper asks it in terms of how few photons do you even have to work with (Poisson statistics, object-aware photon budgeting). Both come out of the same lab, both lean on borrowing well-understood statistical machinery from somewhere else — wireless fading theory in one case, quantum-optics photon statistics in the other — rather than inventing scene-specific heuristics from scratch. That’s turning into a pattern I keep reaching for on every new sensing problem, WiFi CSI included: find the right statistical model for the noise, and the signal-recovery problem gets a lot more tractable.

References:

• Y. Takahashi, J. Jeong, M. Cho, and M.-C. Lee, “A Research on Scattering Media Removal and Photon Estimation using COLaNoPS,” 2025 25th International Conference on Control, Automation and Systems (ICCAS), pp. 60–65, 2025. Supported by Fire and Disaster Management Agency Promotion Program for Scientific Fire and Disaster Prevention Technologies, Grant JPJ000255.
• M. Cho and B. Javidi, “Peplography — a passive 3D photon counting imaging through scattering media,” Optics Letters, vol. 41, no. 22, pp. 5401–5404, 2016.
• J.U. Ha, H.-W. Kim, M. Cho, and M.-C. Lee, “Three-Dimensional Visualization Using Proportional Photon Estimation Under Photon-Starved Conditions,” Sensors, vol. 25, no. 3, p. 893, 2025.
• J. Jeong and M.-C. Lee, “Scattering Medium Removal Using Adaptive Masks for Scatter in the Spatial Frequency Domain,” IEEE Access, vol. 13, pp. 72769–72777, 2025.