Archives/Athou_commafeed
1
0
Fork 1
mirror of https://github.com/Athou/commafeed.git synced 2026-09-24 04:57:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Use batched IN clause for GUID existence check during feed refresh

Browse Source 
Instead of fetching all existing GUID hashes for a feed from the database only check the GUIDs present in the current feed XML using a SQL IN clause. GUIDs are partitioned into batches of 1000.

Also adds an integration test verifying that refreshing an up-to-date feed does not create duplicate entries.
This commit is contained in:
Ingo Kegel
2026-04-10 10:13:37 +02:00
parent 16036897cb
commit 1d23907652
3 changed files with 41 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
commafeed-server/src/main/java/com/commafeed/backend/feed/FeedRefreshUpdater.java
View File

@@ -129,8 +129,16 @@ public class FeedRefreshUpdater {
Map<FeedSubscription, List<FeedEntry>> insertedUnreadEntriesBySubscription = new HashMap<>();
if (!entries.isEmpty()) {
Set<String> existingGuids = unitOfWork.call(() -> feedEntryDAO.findExistingGuids(feed));
List<Entry> newEntries = entries.stream().filter(e -> !existingGuids.contains(Digests.sha1Hex(e.guid()))).toList();
Map<String, Entry> entriesByGuidHash = new HashMap<>();
for (Entry entry : entries) {
entriesByGuidHash.put(Digests.sha1Hex(entry.guid()), entry);
}
Set<String> existingGuids = unitOfWork.call(() -> feedEntryDAO.findExistingGuids(feed, entriesByGuidHash.keySet()));
List<Entry> newEntries = entriesByGuidHash.entrySet()
.stream()
.filter(e -> !existingGuids.contains(e.getKey()))
.map(Map.Entry::getValue)
.toList();
List<FeedSubscription> subscriptions = null;
for (Entry entry : newEntries) {