Technology for a reliable, persistent stack

General Tech Technology & Software 2 years ago

0 2 0 0 0 tuteeHUB earn credit +10 pts

5 Star Rating 1 Rating

Posted on 16 Aug 2022, this text provides information on Technology & Software related to General Tech. Please note that while accuracy is prioritized, the data presented might not be entirely correct or up-to-date. This information is offered for general knowledge and informational purposes only, and should not be considered as a substitute for professional advice.

Take Quiz To Earn Credits!

Turn Your Knowledge into Earnings.

tuteehub_quiz

Answers (2)

Post Answer
profilepic.png
manpreet Tuteehub forum best answer Best Answer 2 years ago

 

Trying a mental reset here: I tried to create a reliable, persistent stack with MSMQ, didn't work

So in more general terms:

I have producer (a webservice, so multithreaded although "only one") / consumer (multiple processes, as many as needed) setup. The key problems are - The data needs to be consumed/processed in LIFO order (~> stack) - The data needs to be stored/handled in a reliable way (i.e. backed by a disk, message queue, whatever). Bonus points for transaction support. - Interprocess communication involved

Given the points above I struggle to find a neat solution. What I looked at:

  1. Do it yourself Didn't really plan to do that, but initial proof of concepts for this just confirmed that this is hard (for me) and helped me get a better grasp on the many hurdles involved.

  2. MSMQ Would be nice and easy, since it lends itself easily to "reliable", is easy to set up and already part of the target infrastructure. Unfortunately "LIFO"/"Stack" is a killer here. That seems to be impossible to do -> Bzzzt.

  3. Database (SQL Server) I tried to look at a DB based approach, but there are lots of ugly things involved:

    • I'd need to store my data as a blob (since it doesn't lend itself to a column based store easily)
    • Polling a database for work just seems wrong (is it?)
    • Locking with multiple consumers seems to be tricky..

Any suggestion for a technology that I should evaluate? The database based approach seems to be the most "promising" so far, but I still haven't found good examples/success stories of similar usecases.


Updates

  • Windows only
  • For now, I don't even need to do inter-machine communication (i.e. producer/consumer probably will be on one machine for now)
  • The key part in the question, the difficult task for me is: I cannot lose a job/message, even if all processes go down. A DB would give me that "for free", message queues can be set up to be reliable. Map/reduce, while interesting, isn't solving the core issue: How do I make sure that messages/jobs aren't lost?
0 views
0 shares

profilepic.png
manpreet 2 years ago

I'd go with SQL Server for this.

  1. Obviously you'd have to serialize your data to a blob, but any solution would have to do this (at least behind the scenes). You would then just have a table like CREATE TABLE Stack (Id int identity, Data varbinary(MAX))

  2. Polling the database isn't necessary. SQL Server has a query notification service where you just give it a query and it will notify you when the results would be different. Your notification query would just be SELECT * FROM Stack

  3. Locking is the database's problem, not yours. You would just have every consumer run a query (or stored procedure) that uses a transaction to return the most recent entry (the row with the highest Id) and delete it at the same time. If the query returns a result, process it and run it again. If the query returns no results, see #2.

Here's a sample query:

BEGIN TRANSACTION
SELECT Data FROM Stack WHERE Id = (SELECT MAX(Id) FROM Stack)
DELETE FROM Stack WHERE Id = (SELECT MAX(Id) FROM Stack)
COMMIT

Here's a more elegant version that doesn't even require an explicit transaction:

DELETE Stack
OUTPUT DELETED.Data
WHERE Id = (SELECT MAX(Id) FROM Stack)

If you want to do batch processing of 10 items at a time, you would use SQL like this:

DELETE Stack
OUTPUT DELETED.*
WHERE Id IN (SELECT TOP 10 Id FROM Stack ORDER BY Id DESC)

0 views   0 shares

No matter what stage you're at in your education or career, TuteeHub will help you reach the next level that you're aiming for. Simply,Choose a subject/topic and get started in self-paced practice sessions to improve your knowledge and scores.