Yuanhao Feng

Back

“Earth’s climate sensitivity is very low; doubling atmospheric CO2 would only cause about 1°C of warming.”

To tell whether this is true, you first need evidence. In the project’s evidence corpus, some passages cite the IPCC range of 1.5 to 4.5°C, and some papers estimate 5 to 6°C. Neither figure matches the claim. But before reaching a verdict, you still have to check whether they describe climate sensitivity under the same conditions.

That is how people fact-check, and an automated system works the same way: retrieve evidence first, then decide how the evidence relates to the claim.

This project automates fact-checking for climate claims. Given a climate claim, the system finds a few relevant passages among 1.2 million evidence paragraphs, then assigns the claim one of four labels: SUPPORTS, REFUTES, NOT_ENOUGH_INFO, or DISPUTED.

DISPUTED is the hardest. For SUPPORTS or REFUTES, the question is which side of the claim the evidence is on. For DISPUTED, the question is whether the retrieved passages disagree: among the five retrieved passages, some support the claim and others refute it. No single passage reveals the dispute. The system has to compare all five together. Many of the design choices in retrieval, features, and the classifier below were made with this class in mind.

Climate Claim Fact-Checking Pipeline STAGE 1 RETRIEVAL Claim input BM25 stemming + Lucene Dense Retriever fine-tuned F2LLM-v2 RRF fusion top-100 top-100 from each Evidence corpus: 1.2M passages, one index shared by both retrievers STAGE 2 RERANKING CrossEncoder reranker ms-marco-MiniLM · hard-negative tuned STAGE 3 NLI FEATURES + CLASSIFICATION top-5 evidence DeBERTa-v3 NLI zero-shot · 5 evidence × 3 probs Aggregation 37-dim feature vector XGBoost 300 trees SUPPORTS REFUTES | NEI | DISPUTED rerank scores join the features F = 0.239 (retrieval) A = 0.474 (classification) H = 0.318 (harmonic mean)

A staged pipeline#

Running fine-grained inference on each of 1.2 million passages costs far too much compute. So I split the work into stages: low-cost retrieval methods narrow down the candidates, a reranking model picks five passages, and those five go to the inference steps.

There are four stages.

The first stage is sparse retrieval plus dense retrieval. BM25 matches on term frequency and takes the top 100. A fine-tuned dense retriever does semantic retrieval and also takes the top 100. The two lists are then merged into one ranking with Reciprocal Rank Fusion (RRF).

The second stage is cross-encoder reranking. A fine-tuned CrossEncoder scores each candidate from the fused list, and the top 5 are kept.

The third stage is NLI feature extraction. Each claim is paired with each of its five passages, and every pair goes through natural language inference to produce a probability distribution.

The fourth stage is classification. The NLI probabilities and reranking scores are combined into a 37-dimensional feature vector, and XGBoost does the four-class classification.

Retrieval layer#

BM25: the term-frequency baseline#

BM25 uses the bm25s library with Lucene-style scoring, plus a Stemmer for English stemming. The index is built over 1.2 million passages and cached to disk after the first build.

BM25 alone at top-100: 54.3% recall, only 1.6% precision. Low precision is acceptable here. The job of this stage is to get as much relevant evidence into the candidate set as possible, and filtering is left to the reranker.

Dense retriever: adding semantic retrieval#

BM25 relies on lexical overlap. If a claim says “global warming” and the evidence says “climate change,” the relevant passage may be missed. A dense retriever uses an encoder to map text into a vector space and retrieves by semantic similarity, which can recover passages with different wording.

The base model is codefuse-ai/F2LLM-v2-160M, a 160M-parameter encoder. I chose it because it is small: both inference and fine-tuning fit on a single consumer GPU. Fine-tuning uses MultipleNegativesRanking loss. Positives are claim-evidence pairs from the training set. Negatives are simply the positives of the other claims in the same batch, i.e. in-batch negatives.

After fine-tuning, all 1.2 million passages are pre-encoded into vectors and stored in ChromaDB, so retrieval is a lookup in the vector index. HNSW uses construction_ef and search_ef of 200 and M of 32, trading a higher build cost for index quality. Encoding is the slowest step in the whole pipeline. On a T4 GPU, the first run takes several hours.

Dense retrieval alone at top-100: 69.1% recall, nearly 15 percentage points above BM25.

RRF fusion#

The two top-100 lists are merged with RRF, which scores each candidate by rank: score(d)=∑r1k+rankr(d)\text{score}(d) = \sum_r \frac{1}{k + \text{rank}_r(d)}, with k = 60. In practice I weighted BM25 at 0.5 and the dense retriever at 1.0, leaning toward the semantic results.

After fusion, top-100 recall is 71.0%, higher than either retriever alone. The two retrievers find partly different evidence, so merging recovers some of the passages each one misses on its own.

The marginal value of BM25#

The dense retriever reaches 69.1% recall on its own. Adding BM25 improves recall by less than two percentage points.

I kept BM25 because its additional cost is small. BM25 runs on the CPU, and RRF just merges two lists, which is negligible next to model inference. The recall gain is useful given the small overhead. BM25 can also serve as a fallback when dense retrieval is unavailable.

CrossEncoder reranking#

The fused top-100 needs another round of filtering. BM25 scores by term matching, and the dense retriever encodes the claim and the evidence separately before computing similarity. Both suit retrieval over a large corpus, but neither looks at the claim and the evidence together.

A CrossEncoder takes the claim and the evidence concatenated into one input sequence, so tokens from the two texts can be compared directly in the attention layers, giving a finer relevance judgment. The cost is a separate forward pass for every claim-candidate pair, so it only fits after retrieval.

The base model is cross-encoder/ms-marco-MiniLM-L-6-v2, a 22M-parameter lightweight cross-encoder, which I fine-tuned on the training set. Positives are claim-evidence pairs. Negatives come from the RRF results: for each claim, 10 passages that rank high but are not annotated evidence serve as hard negatives. These passages are on the same topic as the claim and much harder to separate than random negatives. Training on them is what teaches the model to tell passages that are merely on topic from passages that can actually be used to check the claim.

Fine-tuning ran for 3 epochs with a batch size of 16, on about 16,400 samples: 4,122 positives and 12,280 hard negatives.

After reranking, the top 5 are kept, and evidence retrieval F-score rises from 0.041 at the fusion stage to 0.239. Most of the gain comes from precision, which goes from 2.1% to 19.7%.

Precision, Recall, and F-score by Retrieval Stage 1.0 0.8 0.6 0.4 0.2 0 0.03 BM25 0.04 Dense (F2LLM) 0.04 RRF fusion (top-100) 0.20 RRF fusion (top-6) 0.24 Rerank (top-5) 0.25 Rerank (top-3) Precision (P) Recall (R) F-score Fusing BM25 and Dense lifts recall from 54% / 69% to 71%. Cutting to top-6 and reranking lifts precision from 2% to 20%, and F from 0.04 to 0.24. Still, F = 0.24 is low: this is the bottleneck of the whole pipeline.

NLI feature extraction#

At this point, each claim has five reranked passages and their ranking scores. The next problem is how to use this information for four-class classification.

Mapping NLI labels to target classes#

The most direct approach is to run an NLI model on each claim-evidence pair, get the entailment, contradiction, and neutral probabilities, take the largest, and map it to a target label.

That doesn’t work. NLI has only three classes. SUPPORTS, REFUTES, and NOT_ENOUGH_INFO map to entailment, contradiction, and neutral, but DISPUTED has no counterpart.

Deciding DISPUTED means looking at several passages at once and confirming that some support the claim while others refute it. The NLI label of a single passage cannot express that disagreement, and taking the argmax also throws away the size of the probabilities.

So the NLI model serves only as a feature extractor, and the final decision goes to a downstream classifier. Each claim-evidence pair produces a three-dimensional probability vector [p_ent, p_con, p_neu], and the five passages together form a 5×3 matrix. The classifier learns the mapping from this matrix to the four classes.

NLI model#

The model is a DeBERTa-v3-large MNLI checkpoint, specifically MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli. It is a 304M-parameter encoder fine-tuned on five NLI datasets. I use it zero-shot, with no additional training on the current data.

The reason is that the training set has only 1,228 claims. Fine-tuning a 304M-parameter model on that little data carries a high risk of overfitting, so it is better to keep the pretrained model’s existing inference ability.

One implementation detail matters here: different MNLI checkpoints use different label2id orderings. Some models put entailment at id 0, others at id 2. The code reads the model’s label2id configuration at runtime and reorders the logits into a consistent [entailment, contradiction, neutral] order before applying softmax. Without this step, switching to another NLI checkpoint would make the whole pipeline silently produce wrong results.

The 37-dimensional feature vector#

Each of the five passages has three NLI probabilities, and there are five reranking scores. Concatenating them directly gives 20 dimensions. But that layout does not explicitly express the relationships between passages.

Take DISPUTED. The signal I want is a fairly high entailment probability and a fairly high contradiction probability appearing among the same five passages. If the matrix is just flattened, the classifier has to learn these cross-position combinations itself, such as high entailment in passage 1 together with high contradiction in passage 3. With only 1,228 training claims, that is hard to learn.

So I added a few groups of summary statistics that compute such signals directly for the classifier. The final 37-dimensional vector is built from seven blocks:

Composition of the 37-Dimensional Feature Vector NLI output P [5 × 3] ent con neu e1 e2 e3 e4 e5 Rerank scores s [5] 0.92 0.85 0.71 0.63 0.48 flatten(P) NLI probabilities of every evidence, flattened 15 s (raw rerank scores) 5 softmax(s) 5 max-pool (along evidence axis) 3 min-pool (along evidence axis) 3 mean-pool (along evidence axis) 3 rerank-weighted NLI mean sum of P weighted by softmax(s) 3 = 37 XGBoost 4 classes What each group captures flatten: per-evidence inference pattern max + min: conflict across evidence (DISPUTED) rerank scores: retrieval confidence calibration weighted mean: top-ranked evidence counts more Where the DISPUTED signal comes from max(p_ent) and max(p_con) both high = evidence disagrees

  1. flatten(P), 15 dimensions. The 5×3 matrix flattened, keeping each rank position’s inference result.
  2. s, 5 dimensions. Raw reranking scores, telling the classifier how relevant each passage is to the claim.
  3. softmax(s), 5 dimensions. Reranking scores normalized into a probability distribution with softmax.
  4. max-pool, 3 dimensions. The maximum of each NLI probability along the evidence axis. If max(p_ent) and max(p_con) are both high, some passage strongly supports the claim and another strongly refutes it. That is exactly the DISPUTED signal.
  5. min-pool, 3 dimensions. The minimum along the evidence axis. Together with max-pool, it describes how far apart the passages are.
  6. mean-pool, 3 dimensions. The mean NLI probability, reflecting the overall tendency.
  7. rerank-weighted NLI mean, 3 dimensions. A weighted average of the NLI probabilities with softmax(s) as weights, so passages with higher reranking scores count more.

Max-pool and min-pool were added specifically for DISPUTED. They keep the range of each NLI probability across the five passages, so when the classifier judges disagreement, it does not have to rely only on each passage’s most likely label.

Choosing the classification head#

MLP baseline and ablation#

The first version used scikit-learn’s MLPClassifier: one hidden layer of 64 ReLU units, the Adam optimizer, L2 regularization of 10^-4, 10% held out as an internal validation set, and early stopping with patience 20.

MLP V1 used the full 37-dimensional features. On 154 validation claims it got accuracy 0.461, macro-F1 0.315, and DISPUTED F1 of only 0.100.

I wanted to know how much the reranking scores mattered, so I ran an ablation. MLP V2 dropped every rerank-related feature block, namely the raw scores, the softmax scores, and the weighted mean, keeping only the NLI probabilities and positional features. Input dimensionality went from 37 to 26.

The result was unexpected. MLP V2’s accuracy rose to 0.481 and macro-F1 edged up from 0.315 to 0.330, but DISPUTED F1 fell to 0. Macro-F1 held up because REFUTES F1 rose from 0.211 to 0.409, offsetting the decline in DISPUTED F1 in the average.

Both summary metrics went up, yet the model did not recognize a single DISPUTED claim. Those samples were assigned to SUPPORTS and NOT_ENOUGH_INFO. Since then, I have checked per-class metrics whenever I compare models. On imbalanced data, neither accuracy nor macro-F1 replaces a per-class check.

XGBoost classifier#

With the same 37-dimensional features, XGBoost got accuracy 0.474, macro-F1 0.423, and DISPUTED F1 0.303. Macro-F1 is ten percentage points above MLP V1.

The choice of XGBoost also has to do with the features themselves. Among the 37 dimensions, NLI probabilities fall between 0 and 1, while raw reranking scores can be negative. Tree models are insensitive to feature scaling and handle tabular features with mixed value ranges well. With only 1,228 training samples, it was worth trying, and the validation results were indeed better than MLP V1.

The XGBoost configuration: 300 trees, max depth 4, learning rate 0.05, subsample and colsample_bytree both 0.8, L1 regularization 0.1, L2 regularization 1.0, and the multi:softprob objective. DISPUTED is a clear minority class, with only 124 training samples, or 10%. So I gave each sample a class-balance weight computed from inverse class frequency, capped at 3x so the extreme minority class would not dominate the loss.

Early stopping was set to 30 rounds, and training stopped at round 116. It took under 1 second.

Evidence count: Top-3 and Top-5#

I also tried a variant, XGBoost Top-3, which builds features from only the top three passages, with 27 input dimensions. It got accuracy 0.461, macro-F1 0.394, and DISPUTED F1 0.229.

All three metrics are lower than Top-5. At least in this experiment, the fourth- and fifth-ranked passages still help. The weighted-mean features use softmax weights to reduce the influence of low-scoring passages, while the other features keep their full NLI outputs and leave it to the classifier to decide how to use them.

Final results#

ClassifierDimsAccuracymacro-F1DISPUTED F1
MLP V1370.4610.3150.100
MLP V2 (no rerank features)260.4810.3300.000
XGBoost Top-3270.4610.3940.229
XGBoost Top-5370.4740.4230.303

The final system uses XGBoost Top-5.

End-to-end metrics: retrieval F = 0.239, classification accuracy A = 0.474, and their harmonic mean H = 0.318.

End-to-end bottleneck analysis#

H = 2FA / (F + A). The harmonic mean is more sensitive to the smaller of the two values.

Currently F = 0.239 and A = 0.474. Compare raising each by 5 percentage points: with F at 0.289, H is about 0.36; with A at 0.524, H is only about 0.33.

These results suggest prioritizing retrieval improvements.

Retrieval precision is the weakest point. The reranked top-5 has 19.7% precision, meaning that on average only one of the five results is annotated evidence. The other passages are not necessarily irrelevant, but they may not be enough to support a classification. DISPUTED demands more from retrieval, because both supporting and refuting evidence have to be found. If the refuting evidence is missed during retrieval, the classifier downstream has little chance of seeing the disagreement.

For example, suppose a claim’s true label is DISPUTED, but every retrieved passage supports it. The NLI output will lean toward entailment, and the classifier may well label the claim SUPPORTS.

Corrections along the way#

A few attempts directly shaped the final implementation.

At first I used the NLI argmax directly for classification, but DISPUTED has no corresponding NLI label, so four-class classification wasn’t possible. I then switched to having NLI provide only probability features, with a downstream classifier combining multiple passages to decide. The final system reaches a DISPUTED F1 of 0.303.

The MLP V2 ablation also changed my view. I had expected that removing the rerank features would cause only a small drop, but DISPUTED F1 fell straight to 0. With the current model and data, reranking features help a lot in recognizing this class. With them, the classifier can also take into account how relevant each passage is to the claim when it reads the NLI probabilities.

Another problem came from the label order of an NLI checkpoint. After switching checkpoints, the whole pipeline’s accuracy dropped sharply. Only after some digging did I find that the entailment and contradiction logits were swapped. Adding runtime label alignment fixed it.

Next steps#

If I keep working on this, I would try two directions first.

The first is improving retrieval. The current retriever does not look for evidence specifically for DISPUTED. One idea is to build queries from different angles and actively search for passages that might refute the claim. For example, rewrite the claim as its negation, then retrieve with the rewritten sentence to add refuting evidence.

The second is domain adaptation for the NLI model. The current checkpoint is a general-domain one, and it may not reliably distinguish numerical differences in climate text. For example, under the same conditions, “doubling CO2 causes 1°C of warming” and “doubling CO2 causes 2.5°C of warming” are two different estimates, yet the model might label the pair as entailment because the wording is so similar. Fine-tuning on climate-domain NLI data is an option, provided enough reliable annotations can be found.

Climate Fact-Checking: Hybrid Retrieval, NLI Classification
https://me.yuanhaofeng.com/en/blog/climate-fact-check
Author Yuanhao Feng
Published at September 21, 2026