37 Commits

Author	SHA1	Message	Date
Athou	ef79cf1748	added comments for refill()	2013-07-29 09:47:21 +02:00
Athou	3dd4f140e2	refactored the way we handle feed refresh queue	2013-07-27 15:45:03 +02:00
Athou	da94880c53	added a little doc	2013-07-26 16:00:02 +02:00
Athou	8845c54d0c	apply formatter	2013-07-25 09:17:33 +02:00
Athou	1e36564ff8	limit batch size	2013-07-05 11:03:56 +02:00
Athou	5530c5fd7e	setting to pause feed crawling	2013-07-05 08:23:18 +02:00
Athou	f230ad74b1	store normalized urls	2013-07-02 16:56:25 +02:00
Athou	d3dd084dd7	better refresh algorithm using queues instead of synchronous database call	2013-06-30 12:35:17 +02:00
Athou	27e03e3038	added in memory cache and redis cache and marked them as application scoped	2013-06-29 11:20:53 +02:00
Athou	158abadf19	use date constructor directly when a calendar is not needed, better performances	2013-06-25 16:57:48 +02:00
Athou	f54e41897d	manually queue feeds for refresh (#316)	2013-06-22 20:06:09 +02:00
Athou	be6db080fd	hibernate tweaks	2013-06-06 09:54:17 +02:00
Athou	f9e9010545	revert to old factor	2013-05-30 15:23:07 +02:00
Athou	63349d6e14	small refactoring	2013-05-30 13:18:26 +02:00
Athou	23c243c8ac	remove potential duplicates	2013-05-28 13:01:02 +02:00
Athou	8adcd35cb4	increment feed update metric only when we actually update the feed	2013-05-25 15:39:54 +02:00
Athou	fc48c64e2b	more reasonable buffer	2013-05-22 16:11:31 +02:00
Athou	97c2c12fe2	don't cap feed count buffer	2013-05-22 15:55:28 +02:00
Athou	f429a11d12	record missing metric	2013-05-22 13:13:05 +02:00
Athou	5bf2dfaa18	increment metric only if queue is not null	2013-05-22 12:25:35 +02:00
Athou	cafcfc4c9f	forgot to empty queues	2013-05-22 10:39:03 +02:00
Athou	15b7d685fe	wip: allow parallel update of feeds	2013-05-22 08:01:05 +02:00
Athou	3dca5d595d	just queue feeds, update them later	2013-05-21 16:40:37 +02:00
Athou	36e39b8c75	delay based on heavy load setting	2013-05-21 10:44:05 +02:00
Athou	d4d95e741f	force feed refresh if updated more than ten minutes ago	2013-05-19 12:11:54 +02:00
Athou	735c2efb34	queue feed for refresh after a subscription	2013-04-25 14:41:56 +02:00
Athou	c71b0aee96	don't batch refresh more than 100 feeds	2013-04-23 16:00:18 +02:00
Athou	83946373d9	use a constant instead	2013-04-23 15:58:56 +02:00
Athou	839beb5d14	small optimizations	2013-04-22 10:23:53 +02:00
Athou	db8fc29d4f	don't update the database if the feed didn't change	2013-04-18 13:22:59 +02:00
Athou	8343434882	remove circular dependency	2013-04-17 07:57:43 +02:00
Athou	743b389808	initial metrics system	2013-04-16 13:52:20 +02:00
Athou	94251d2be7	ability to queue a feed for refresh	2013-04-16 12:50:41 +02:00
Athou	b6d8072090	smarter way of stopping things	2013-04-16 09:29:33 +02:00
Athou	0da3ce1bc5	scale based on number of background threads	2013-04-15 10:50:25 +02:00
Athou	c41c4fe8b9	forgot to refresh lastUpdate field	2013-04-13 10:13:41 +02:00
Athou	281ff6d883	smarter way of getting the next feed to update	2013-04-13 07:02:59 +02:00