Locking.

Jun 7, 2013 at 11:34 PM
Edited Jun 7, 2013 at 11:36 PM
Hi Blaze, as You already know I will have a lot of tables. From time to time I have to compact them. Now the problem I have is that if I was just to do:
 using (Transaction transaction = db.Transaction)
   try
   {
    transaction.SynchronizeTables ("someTable");
    db.Compact ("someTable");
    transaction.Commit ();
    }
    catch (Exception ex)
    {
     LOG.DebugFormat (ex.ToString ());
    }
Especially when db.Compact will use: engine.Scheme.RenameTable or engine.Scheme.DeleteTable.
Which will cause exceptions in other threads. I do not want to handle this in every
piece of code where I access dbreeze.

So what I thought was to create something like this:
using (Session2 session = new Session2 (Session2.LockType.EXCLUSIVE, "table1", "table2",...))
 using (Transaction transaction = db.Transaction)
 {
  ...
 }
So I've programmed it like this:
public class Session2 : IDisposable
 {
  #region Declarations
  #region Enumeration
  public enum LockType : byte
  {
   SHARED = 0,
   EXCLUSIVE = 1
  }
  #endregion

  #region Classes
  private class LockCookie
  {
   private long sharedReferences;
   private long exclusiveReference;
   private string name;

   public long SharedReferences
   {
    get
    {
     return sharedReferences;
    }

    set
    {
     sharedReferences = value;
    }
   }

   public long ExclusiveReferences
   {
    get
    {
     return exclusiveReference;
    }

    set
    {
     exclusiveReference = value;
    }
   }

   public string Name
   {
    get
    {
     return name;
    }
   }

   public bool IsFree
   {
    get
    {
     bool rt;

     rt = (0 == sharedReferences) && (0 == exclusiveReference);
     return rt;
    }
   }

   public LockCookie (string name)
   {
    sharedReferences = 0;
    exclusiveReference = 0;
    this.name = name;
   }
  }

  private class DbThreadsGator : IDisposable
  {
   private ManualResetEvent gate;

   public DbThreadsGator ()
   {
    gate = new ManualResetEvent (true);
   }

   public DbThreadsGator (bool gateIsOpen)
   {
    gate = new ManualResetEvent (gateIsOpen);
   }

   public bool PutGateHere ()
   {
    return gate.WaitOne ();
   }

   public bool OpenGate ()
   {
    return gate.Set ();
   }

   public bool CloseGate ()
   {
    return gate.Reset ();
   }

   public void Dispose ()
   {
    gate.Close ();
   }
  }


  #endregion

  #region Delegates
  #endregion

  #region Events
  #endregion

  #region Fields
  private static readonly ILog LOG;
  private static readonly object MONITOR;
  private static readonly DbThreadsGator GATOR;
  private static Dictionary<string, LockCookie> LOCKS;
  private List<LockCookie> myLockCookies;
  private LockType lockType;
  private bool disposed;
  #endregion
  #endregion

  #region Properties
  #endregion

  #region Construction
  static Session2 ()
  {
   LOG = LogManager.GetLogger (typeof (Session2));
   LOCKS = new Dictionary<string, LockCookie> ();
   MONITOR = new object ();
   GATOR = new DbThreadsGator ();
  }

  public Session2 (LockType lockType, params string[] names)
  {
   LockCookie myLockCookie;
   bool block;

   this.disposed = false;
   this.lockType = lockType;

   myLockCookies = new List<LockCookie> ();
   lock (MONITOR)
   {
    foreach (string name in names)
    {
     myLockCookie = null;
     if (LOCKS.ContainsKey (name))
     {
      myLockCookie = LOCKS[name];
     }
     else
     {
      myLockCookie = new LockCookie (name);
      switch (lockType)
      {
       case LockType.SHARED:
        ++myLockCookie.SharedReferences;
        break;

       case LockType.EXCLUSIVE:
        ++myLockCookie.ExclusiveReferences;
        break;
      }
      LOCKS.Add (name, myLockCookie);
     }
     myLockCookies.Add (myLockCookie);
    }
   }
   do
   {
    lock (MONITOR)
     block = IsBlocking ();
    if (block)
     GATOR.PutGateHere ();
    else
    {
     break;
    }
   } while (block);
  }
  #endregion

  #region Methods
  private bool IsBlocking ()
  {
   bool rt;

   rt = false;
   foreach (LockCookie lockCookie in myLockCookies)
   {
    switch (lockType)
    {
     case LockType.SHARED:
      rt = (lockCookie.ExclusiveReferences > 0);
      break;

     case LockType.EXCLUSIVE:
      if (lockCookie.SharedReferences > 0 || lockCookie.ExclusiveReferences > 1)
       rt = true;
      break;
    }
    if (rt)
     break;
   }
   return rt;
  }

  public void Dispose ()
  {
   this.Dispose (true);
   GC.SuppressFinalize (this);
  }

  protected virtual void Dispose (bool disposing)
  {
   if (!disposed)
   {
    if (disposing)
     lock (MONITOR)
     {
      foreach (LockCookie lockCookie in myLockCookies)
      {
       switch (lockType)
       {
        case LockType.SHARED:
         --lockCookie.SharedReferences;
         break;

        case LockType.EXCLUSIVE:
         --lockCookie.ExclusiveReferences;
         break;
       }
       if (lockCookie.IsFree)
        LOCKS.Remove (lockCookie.Name);
       GATOR.OpenGate ();
      }
      myLockCookies.Clear ();
     }
    disposed = true;
   }
  }
  #endregion
 }
As You can notice I've used part of Your code (DbThreadsGator). But I am not sure that this will work as expected. Could You check it out and give me an advice about it ?
Jun 9, 2013 at 12:40 AM
Edited Jun 9, 2013 at 1:05 AM
With the new IO layer replacing can be done much easier.
I have published new sources (a59daececada - Experimental tran.RestoreTableFromTheOtherFile).
Please take them and test as soon as you can...

So, the tesing code can look like this
 private void TestKrome()
        {
//HERE WE JUST CREATE SOME TABLES FOR TEST, NOTE "t2" is filled with the key 1
            using (var tran = engine.GetTransaction())
            {
                tran.Insert<int, byte>("t1", 1, 1);
                tran.Insert<int, byte>("t2", 1, 1);
                tran.Insert<int, byte>("t3", 1, 1);

                tran.Commit();
            }           

//HERE we use LianaTrie directly inside of the OTHER ENGINE and fill table 90000000 with key 2.
//NOTE, we could fill this new LianaTrie with the compacted data from existing "t2"
//NOTE, table name must be always digital, so you can use very HIGH digit, if this compacted table is in the same folder with your real engine tables

            LTrieSettings = new TrieSettings();
            LTrieStorage = new StorageLayer(@"E:\temp\DBreezeTest\DBR1\90000000", LTrieSettings, new DBreezeConfiguration());           

            LTrie = new LTrie(LTrieStorage);
//FILLING WITH KEY 2
            LTrie.Add(((int)2).To_4_bytes_array_BigEndian(),((int)2).To_4_bytes_array_BigEndian());
            LTrie.Commit();
            LTrie.Dispose();

//OK, After LTrie.Dispose(); table "90000000" is free for copying...

            using (var tran = engine.GetTransaction())
            {
                tran.SynchronizeTables("t2");

//!!!!NEW FEATURE
//AT THE MOMENT WHEN FEATURE WORKS NONE OF THREADS IS ABLE TO READ FROM THE TABLE, SO WHILE IT HAPPENS NO MISTAKES CAN BE OCCURED IN DIFF THREADS

                tran.RestoreTableFromTheOtherFile("t2", @"E:\temp\DBreezeTest\DBR1\90000000");              
            }



            using (var tran = engine.GetTransaction())
            {
                foreach (var row in tran.SelectBackward<int,int>("t2"))
                {
//GETTING KEY 2
                    Console.WriteLine("Key: {0}", row.Key);
                }
            }
        }
Looks good, but there is an architectural problem.
If I have a reading thread where I read keys:
foreach(var row in tran.SelectForward("t2"))
{
   //Here I got object row, which already contains key, but it doesn't contain value only physical reference to the value in the file.
   //If we say
   byte[] val = row.Value;
   //only at this moment DBreeze will refer to the DISK to get new value

   //So, if before command row.Value other thread has run RestoreTableFromTheOtherFile  and we have restored other file, this physical link to value
   //can bring us not to the value, but to somewhere else....what is absolutely not good.

   //So, may be when row object is received it must also hold smth like DateTime of table fixation. Fixation will be changed when table is replaced and 
   //row.Value will lead in any case to exception.

   //Or just to block reading threads within compaction time, what is not good for APP smoothness.

   //That's a point of discussion - please, participate.
   //As I understand you want to lock even reading threads, while compaction, using extra decoration around transaction.


}
Jun 10, 2013 at 1:50 AM
It's nice what You did. But as You said it yourself it can be a problem for reading threads.
[Yes I was blocking reading threads as well in my aproach]. But I bet You could do it better than me in the engine itself.
Biggest problem was that:

using (var tran = engine.GetTransaction())
{
tran.SynchronizeTables("someTable");
Compact ("someTable");
tran.commit ();
}

Compact (tableName)
{
Create temporary table.
Copy tableName structure to the temporary table.
Rename Temporary table to tableName.
}

Now If i have reading threads there will be a problem, because of rename.
Is it possible for You to introduce a mechanism, which will entirely lock table for the current transaction ?

for example:
tran.SynchronizeTables (bool fullLock,, names,...);

So after this lock is gained, no other readers will be able to access it until compacting transaction completes,
therefore no exceptions in other threads.

I think that would be a much better approach, since it wouldn't be so low-level.
  • Your approach involves using another engine instance. Would be better to avoid such practice IMO.
Jun 10, 2013 at 10:32 AM
I am actually not ready right now for such deep changes. There are many caveats.
That's why let's make so, I start to think on this idea, may be one day I will grow up to it.
And you use your approach - btw, I like it, because you explicitly setup in the program all bottleneck places for reading and writing threads.

If you really need to compact the data and you can't leave without these lockings?

The scheme of exception is following:

When the time of compaction comes, you lock table(s) for write using tran.SynchronizeTables.
You start to copy data to other engine table, but main engine table still exists and consuming threads can read data satisfying read-customers.
Copying will take every time more and more time, because table grows up, but is available for reads at this moment.
When copy is finished, we need some seconds to execute File.Move, replacing old table file with the new one. At this moment no exceptions.
And after that in some reading threads could be raised DBreeze TABLE_WAS_CHANGED_LINKS_ARE_NOT_ACTUAL exception.

Google Mail also sometimes says "on the server occurred mistake, please try again"...Actually all systems must be ready to data storage access exception handling.

For the compaction case, if u can handle exceptions your system stays always responsive (only small delay, while File.Move), but if u lock reading threads, system will stay unresponsive during copying time (every time longer and longer) + File.Move time.
Jun 11, 2013 at 12:10 AM
Ok can you help me review my code atleast ? The one on top of this thread. I would like to be absolutely sure that it works as expected.
Jun 11, 2013 at 2:27 PM
As a first draft.

There will be two extra files:

DBSessionLocker.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using DBreeze.Utils;
using DBreeze.SchemeInternal;

namespace DBreeze
{
    public enum eDBSessionLockTypes
    {
        EXCLUSIVE,
        SHARED
    }

    //!!!!!!!!!!!!!!!!   IN DBREEZE.ENGINE.DISPOSE, after disposing all elemetns, call DBSessionLocker.Dispose()

    internal static class DBSessionLocker
    {   
        static Dictionary<int, internSession> _waitingSessions = new Dictionary<int, internSession>();
        static Dictionary<int, internSession> _acceptedSessions = new Dictionary<int, internSession>();
        static List<int> _waitingSessionSequence = new List<int>();

        static DbReaderWriterLock _sync = new DbReaderWriterLock();

        static object lock_disposed = new object();
        static bool disposed = false;


        class internSession
        {
            public string[] tables;
            public eDBSessionLockTypes lockType= eDBSessionLockTypes.EXCLUSIVE;
            public DbThreadsGator gator = null;          
        }

        /// <summary>
        /// 
        /// </summary>
        /// <param name="lockType"></param>
        /// <param name="tables"></param>
        /// <returns>false if thread grants access, false if thread is in a queue</returns>
        public static bool AddSession(eDBSessionLockTypes lockType, string[] tables)
        {            
            lock (lock_disposed)
            {
                if (disposed)
                    return true;
            }

            internSession iSession = null;
            bool ret = true;

            _sync.EnterWriteLock();
            try
            {
                foreach (var ses in _acceptedSessions)
                {
                    if (DbUserTables.TableNamesIntersect(ses.Value.tables.ToList(), tables.ToList()))
                    {
                       if (ses.Value.lockType == eDBSessionLockTypes.EXCLUSIVE || lockType == eDBSessionLockTypes.EXCLUSIVE)
                        {
                            //Lock
                            ret = false;
                            break;
                        }
                    }
                }

                if (!ret)
                {
                    internSession xSes = null;
                    foreach (var ses in _waitingSessionSequence)
                    {

                        if (ses == System.Threading.Thread.CurrentThread.ManagedThreadId)
                            break;

                        _waitingSessions.TryGetValue(ses, out xSes);

                        if (DbUserTables.TableNamesIntersect(xSes.tables.ToList(), tables.ToList()))
                        {
                            if (xSes.lockType == eDBSessionLockTypes.EXCLUSIVE || lockType == eDBSessionLockTypes.EXCLUSIVE)
                            {
                                //Lock
                                ret = false;
                                break;
                            }
                        }
                    }
                }

                if (_waitingSessions.TryGetValue(System.Threading.Thread.CurrentThread.ManagedThreadId, out iSession))
                {
                    //This session was in the waiting list once
                    if (ret)
                    {
                        //We have to take away session from waiting list
                        iSession.gator.Dispose();
                        iSession.gator = null;
                        _waitingSessions.Remove(System.Threading.Thread.CurrentThread.ManagedThreadId);
                        _waitingSessionSequence.Remove(System.Threading.Thread.CurrentThread.ManagedThreadId);
                    }
                    else
                    {
                        iSession.gator.CloseGate();
                    }
                }
                else
                {
                    //Creating new session
                    iSession = new internSession()
                    {
                        lockType = lockType,
                        tables = tables
                    };

                    if (!ret)
                    {
                        iSession.gator = new DbThreadsGator(false);
                        _waitingSessions.Add(System.Threading.Thread.CurrentThread.ManagedThreadId, iSession);
                        _waitingSessionSequence.Add(System.Threading.Thread.CurrentThread.ManagedThreadId);
                    }
                }

                if (ret)
                {
                    //Adding into accepted sessions                    
                    _acceptedSessions.Add(System.Threading.Thread.CurrentThread.ManagedThreadId, iSession);
                }
            }
            finally
            {
                _sync.ExitWriteLock();
            }

            if (!ret)
            {
                //putting gate
                iSession.gator.PutGateHere();
            }

            return ret;

        }

        /// <summary>
        /// 
        /// </summary>
        public static void RemoveSession()
        {
            lock (lock_disposed)
            {
                if (disposed)
                    return;
            }

            internSession iSession = null;
            List<int> ws = null;

            _sync.EnterWriteLock();
            try
            {
                if (!_acceptedSessions.TryGetValue(System.Threading.Thread.CurrentThread.ManagedThreadId, out iSession))
                    return;

                if (iSession.gator != null)
                {
                    iSession.gator.Dispose();
                }

                _acceptedSessions.Remove(System.Threading.Thread.CurrentThread.ManagedThreadId);

                ws = _waitingSessionSequence.ToList();
            }
            finally
            {
                _sync.ExitWriteLock();
            }


            if (ws != null && ws.Count() > 0)
            {
                foreach (int wsId in ws)
                {

                    _sync.EnterReadLock();
                    try
                    {

                        if (!_waitingSessions.TryGetValue(wsId, out iSession))
                            continue;

                    }
                    finally
                    {
                        _sync.ExitReadLock();
                    }

                    try
                    {
                        if (iSession.gator != null)
                            iSession.gator.OpenGate();
                    }
                    catch
                    {
                    }

                }
            }


        }


        /// <summary>
        /// MUST BE CALLED BY ENGINE DISPOSE (After all other DBreeze disposes)
        /// </summary>
        public static void Dispose()
        { 
            lock (lock_disposed)
            {
                if (disposed)
                    return;
                disposed = true;
            }

            foreach (var ses in _waitingSessions)
            {
                if (ses.Value.gator != null)
                {
                    ses.Value.gator.OpenGate();
                    ses.Value.gator.Dispose();
                }
            }

             foreach (var ses in _acceptedSessions)
            {
                if (ses.Value.gator != null)
                {
                    ses.Value.gator.OpenGate();
                    ses.Value.gator.Dispose();
                }
            }

            _acceptedSessions.Clear();
            _waitingSessions.Clear();
            _waitingSessionSequence.Clear();

        }


    }
}
DBSession.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;



namespace DBreeze
{
    public class DBSession : IDisposable
    {
        public DBSession(eDBSessionLockTypes lockType, params string[] tables)
        {
            //Console.WriteLine("I: {0}", System.Threading.Thread.CurrentThread.ManagedThreadId);
            while (true)
            {
                if (DBSessionLocker.AddSession(lockType, tables))
                    break;
            }
        }

        public void Dispose()
        {
            //var x = new DBSession(eDBSessionLockTypes.EXCLUSIVE, "t1", "t*s", "t$");

            DBSessionLocker.RemoveSession();
        }
    }
}
In DBreezeEngine Dispose function, after all disposed objects we must dispose DBSessionLocker.Dispose();
Jun 11, 2013 at 2:29 PM
Testing code:
using DBreeze.Utils.Async;

private void testF_003()
        {

            Action t2 = () =>
            {
                ExecF_003_2();
            };

            t2.DoAsync();


            Action t1 = () =>
            {
                ExecF_003_1();
            };

            t1.DoAsync();

           


            Action t3 = () =>
            {
                ExecF_003_3();
            };

            t3.DoAsync();
        }


  private void ExecF_003_1()
        {
            using (DBSession ses = new DBSession(eDBSessionLockTypes.EXCLUSIVE, "t1"))
            {
                //Console.WriteLine("T1 {0}> ", DateTime.Now.ToString("HH:mm:ss.ms"));
                Console.WriteLine("T1 {0}> {1}; {2}", DateTime.Now.Ticks, System.Threading.Thread.CurrentThread.ManagedThreadId, DateTime.Now.ToString("HH:mm:ss.ms"));
                Thread.Sleep(2000);
            }
        }

        private void ExecF_003_2()
        {
            using (DBSession ses = new DBSession(eDBSessionLockTypes.SHARED, "t1"))
            {
                //Console.WriteLine("T2 {0}> ", DateTime.Now.ToString("HH:mm:ss.ms"));
                //Console.WriteLine("T2 {0}> ", DateTime.Now.Ticks);
                Console.WriteLine("T2 {0}> {1}; {2}", DateTime.Now.Ticks, System.Threading.Thread.CurrentThread.ManagedThreadId, DateTime.Now.ToString("HH:mm:ss.ms"));
               Thread.Sleep(400);
            }
        }

        private void ExecF_003_3()
        {
            using (DBSession ses = new DBSession(eDBSessionLockTypes.SHARED, "t1"))
            {
                //Console.WriteLine("T3 {0}> ", DateTime.Now.ToString("HH:mm:ss.ms"));
                //Console.WriteLine("T3 {0}> ", DateTime.Now.Ticks);
                Console.WriteLine("T3 {0}> {1}; {2}", DateTime.Now.Ticks, System.Threading.Thread.CurrentThread.ManagedThreadId, DateTime.Now.ToString("HH:mm:ss.ms"));
                //Thread.Sleep(200);
                Thread.Sleep(3000);
            }
        }
Jun 11, 2013 at 2:55 PM
Edited Jun 11, 2013 at 3:17 PM
This new code is integrated into DBreeze table names patterns and solves the problem of sequential threads execution.
Check it.

Theoretically DBSession can return a transaction instance.

The only con is that you must know in advance table names which you want to lock, before running session.
With SynchronizeTable you can use reading instructions before locking.

Such session decoration can be combined with SynchronizeTables instruction.
But SynchronizeTable has no connection to DBsession.

Now in scenarios where it's necessary to put full lock on the tables, we can make smth. like this:

Proc1 - inserting keys
 using (var tran = engine.GetTransaction())
            {
                using (DBSession ses = new DBSession(eDBSessionLockTypes.SHARED, "t1","p*"))
                {
                     //can be added if we want to change one of those tables.
                     tran.SynchronizeTable("t1","p*");

                    tran.Insert..."t1"
                    tran.Insert..."p127"

                    tran.Commit();
                }
            }
Proc2 - reading keys
 using (var tran = engine.GetTransaction())
            {
                using (DBSession ses = new DBSession(eDBSessionLockTypes.SHARED, "t1"))
                {
                    
                    foreach(var row in tran.SelectForward("t1")
                      ....
                }
            }
Proc3 - Removing Keys from t1 with the table recreation
 using (var tran = engine.GetTransaction())
            {
                using (DBSession ses = new DBSession(eDBSessionLockTypes.EXCLUSIVE, "t1"))
                {
                    
                    tran.RemoveKeys("t1",true);
                    tran.Commit();
                }
            }
Jun 11, 2013 at 7:11 PM
Wow nice :) I will try as soon as possible. I didn't have time lately to test stuff, I am on a verge of production code. But will test asap.
Jun 12, 2013 at 3:14 PM
Btw when will you integrate these into the source ?
Jun 12, 2013 at 3:15 PM
I will, did u test it?
Jun 12, 2013 at 4:28 PM
Not yet, will test tommorow. I was really busy these days.
Jun 12, 2013 at 4:34 PM
Also, Would it be possible to extend something else:

from documentation:

//SETTING UP ALTERNATIVE FOLDER FOR TABLE t11
conf.AlternativeTablesLocations.Add("t11",@"D:\temp\DBreezeTest\DBR1\INT");
//SETTING UP THAT ALL TABLES STARTING FROM “mem_” must reside in-memory
conf.AlternativeTablesLocations.Add("mem_*", String.Empty);
//SETTING UP Table pattern to reside in different folder

conf.AlternativeTablesLocations.Add(“t#/Items", @"D:\temp\DBreezeTest\DBR1\EXTRA");

This is all nice and fine, but what if some tables are not known at run time ?


Rather than this, use some sort of a resolver ? Like a callback. For example
Engine.TablePathResolver += Engine.Resolver ();

private string Resolver (object sender, SomeEventArgs e)
{

}

e - Would pass name of the Table.
and string returned would be a path for the table.

Is this possible at all ? I haven't thought about everything here, just an idea. I know I am gonna have a lot of tables.
Jun 12, 2013 at 4:55 PM
Step by step, I have also not so much time for now.

What is not clear.... may be you gonna have many tables like p1,p2,....pN.... and you don't know how many N's u will have, but prefix "p" you know, that's why u use patterns: for "p*" location must be X1, for "t*" - location X2. It must be more then enough.
Jun 13, 2013 at 11:05 AM
So, new release with locks. We can make following
private void ExecF_003_1()

       {

           using (var tran = engine.GetTransaction(eTransactionTablesLockTypes.EXCLUSIVE, "t1", "p*", "c$"))

           {

               Console.WriteLine("T1 {0}> {1}; {2}", DateTime.Now.Ticks, System.Threading.Thread.CurrentThread.ManagedThreadId, DateTime.Now.ToString("HH:mm:ss.ms"));

               tran.Insert<int, string>("t1", 1, "Kesha is a good parrot");

               tran.Commit();

               Thread.Sleep(2000);

           }

       }

       private void ExecF_003_2()

       {

           List<string> tbls = new List<string>();

           tbls.Add("t1");

           tbls.Add("v2");

           using (var tran = engine.GetTransaction(eTransactionTablesLockTypes.SHARED, tbls.ToArray()))

           {

               Console.WriteLine("T2 {0}> {1}; {2}", DateTime.Now.Ticks, System.Threading.Thread.CurrentThread.ManagedThreadId, DateTime.Now.ToString("HH:mm:ss.ms"));

               foreach (var r in tran.SelectForward<int, string>("t1"))

               {

                   Console.WriteLine(r.Value);

               }              

           }

       }

       private void ExecF_003_3()

       {

           using (var tran = engine.GetTransaction(eTransactionTablesLockTypes.SHARED, "t1"))

           {

               Console.WriteLine("T3 {0}> {1}; {2}", DateTime.Now.Ticks, System.Threading.Thread.CurrentThread.ManagedThreadId, DateTime.Now.ToString("HH:mm:ss.ms"));

               //This must be used in any case, when Shared threads can have parallel writes

               tran.SynchronizeTables("t1");

               tran.Insert<int, string>("t1", 1, "Kesha is a VERY good parrot");

               tran.Commit();

               foreach (var r in tran.SelectForward<int, string>("t1"))

               {

                   Console.WriteLine(r.Value);

               }

           }

       }





using DBreeze.Utils.Async;


       private void testF_003()

       {

           Action t2 = () =>

           {

               ExecF_003_2();

           };

           t2.DoAsync();


           Action t1 = () =>

           {

               ExecF_003_1();

           };

           t1.DoAsync();


           Action t3 = () =>

           {

               ExecF_003_3();

           };

           t3.DoAsync();

       }
Jun 13, 2013 at 5:19 PM
Man that's nice I got to test it ASAP. I will try before end of the weekend.
Jun 14, 2013 at 4:20 AM
Edited Jun 14, 2013 at 4:23 AM
Ok I've tested it a bit today:
private static void Compact ()
{
   int i;
   Row<int, string> row;
   string value;

   engine.Scheme.DeleteTable ("tmp");
   using (var tran = engine.GetTransaction (eTransactionTablesLockTypes.EXCLUSIVE, "TestTable", "tmp"))
   {
    for (i = 0; i < 100000; ++i)
    {
     if ((i % 1000) == 0)
      Console.WriteLine ("Working @{0}", i);
     row = tran.Select<int, string> ("TestTable", i);
     if (row.Exists)
     {
      value = row.Value;
      tran.Insert<int, string> ("tmp", i, value);
     }
    }
    tran.Commit ();
    engine.Scheme.RenameTable ("tmp", "TestTable");  
   }
}
This waits forever. Because of RenameTable.

I've tried it both in single thread as multithread. Same result.
Jun 14, 2013 at 4:33 AM
Edited Jun 14, 2013 at 4:38 AM
Also:
using (var tran = engine.GetTransaction ()) 
{
 Access "Table1".
 ..
 ..
}
Where in other thread:
using (var tran = engine.GetTransaction (eTransactionTablesLockTypes.EXCLUSIVE, "Table1"))
{
 Access "Table1"
}
The first thread, ignores that we actually have Exclusive lock on that table in second thread. Is this intentional ?
I mean I am going to use SHARED/EXCLUSIVE variant in my entire code. But shouldn't all tables now by default
if not specified otherwise be eTransactionTablesLockTypes.SHARED ?



Also, I've noticed that if you:
  • Start 50 threads that are accessing db with eTransactionTablesLockTypes.SHARED.
  • Start one that is accessing with eTransactionTablesLockTypes.EXCLUSIVE
  • Start another 50 threads that are accessing db with eTransactionTablesLockTypes.SHARED.
The one with exclusive access will ALWAYS be the last one to execute for some reason. so basically if you have a
busy system, that exclusive one will never get a chance to do it's work ?
Jun 14, 2013 at 7:57 AM
Edited Jun 14, 2013 at 8:01 AM
  • about rename table issue will look later
  • in docu is written that either you use engine.GetTransaction for specific tables or engine.GetTransaction (eTransactionTablesLockTypes.EXCLUSIVE... WITHIN THE WHOLE PROGRAM.
  • actually implemented algorithm includes sequential thread execution, so if you start 50 threads SHARED, then EXCLUSIVE, then 50 shared.... they must be executed 50-1-50 .. if it's not so, be sure that you didn't start 50-50-1
  • BTW, just for your note, if u use SELECT-INSERT chain (like in your RenameTable issue example), SELECT better to use with AsReadVisibilityScope = true, it will create second trie instance and operation will be much-much faster.
Jun 14, 2013 at 3:07 PM
Edited Jun 14, 2013 at 3:14 PM
  1. Problem is with opentableholder which tranasaction holds until it is disposed.
    But if i go out of transaction and rename. There is a possibility that another thread
    already writing the old table where data loss will occur. That's why I previously I blocked
    things above transaction.
  2. Ok.
  3. I did start 50-1-50
  4. Yup but this was just a simple test.
Jun 14, 2013 at 3:32 PM
about 3

you've run sequentially 3 threads 50 - 1 - 50, but r u sure that they were run in such sequence? when thread starts make Console.Writeline starttime.ticks.
Jun 14, 2013 at 3:35 PM
about 1 - may be try our new RestoreTableFromFile with the second engine.
Jun 14, 2013 at 5:51 PM
I will attempt it with restoretablefromfile, what's the deal with table PHYSICAL file names btw ? You said to pick some high number ?
Jun 15, 2013 at 8:06 PM
Edited Jun 15, 2013 at 8:09 PM
krome wrote:
I will attempt it with restoretablefromfile, what's the deal with table PHYSICAL file names btw ? You said to pick some high number ?
When restoretablefromfile, after restoration, procedure tries to delete restoration-source-file, so if you use the same engine this removal can be problematic, due to file lock. Solution, either use other engine instance or remove from the code restoration-source-file deletion instruction (probably this I will make in the neareast release).

About high numbers - forget it, not interesting in current implementation.
Jun 17, 2013 at 5:52 PM
Is this ok ?
private static void Compact ()
  {
   int i;
   Row<int, string> row;
   string value;
   ulong count;
   byte[] btkey;
   byte[] btval;

   using (var tran = engine.GetTransaction (eTransactionTablesLockTypes.EXCLUSIVE, "TestTable", "tmp"))
   {
    tran.SynchronizeTables ("TestTable");
    count = tran.Count ("TestTable");
    Console.WriteLine ("~ Compact thread");

    TrieSettings settings;
    StorageLayer storageLayer;
    LTrie trie;

    settings = new TrieSettings ();
    storageLayer = new StorageLayer (@"test\90000000", settings, new DBreezeConfiguration ());
    trie = new LTrie (storageLayer);

    
    for (i = 0; i < (int) count; ++i)
    {
     if ((i % 100000) == 0)
      Console.WriteLine ("Working @{0}", i);
     row = tran.Select<int, string> ("TestTable", i, true);
     if (row.Exists)
     {
      btkey = DataTypesConvertor.ConvertKey<int> (row.Key);
      btval = DataTypesConvertor.ConvertValue<string> (row.Value);
      trie.Add (btkey, btval);
     }
    }
    trie.Commit ();
    trie.Dispose ();

    tran.RestoreTableFromTheOtherFile ("TestTable", @"test\90000000");
    tran.Commit ();
   }
   Console.WriteLine ("Compact Thread finished");
  }
Jun 18, 2013 at 9:44 AM
  • if u use eTransactionTablesLockTypes.EXCLUSIVE you don't need tran.SynchronizeTables ("TestTable");
  • you can use new instance of DBreeze (located in the other folder) with all benefits, instead of low level LTrie.
Jun 18, 2013 at 10:29 AM
Sure but I don't really need it. I only need small operation, that is copy table. Btw what happens, if RestoreTable is interrupted ? before completion ? Like if machine reboots, power fails etc ?
Jun 18, 2013 at 10:44 AM
krome wrote:
what happens, if RestoreTable is interrupted ? before completion ? Like if machine reboots, power fails etc ?
That's a good question... for now there is no defense will be a special entry in transaction log with a destination file path for emergency restoration... but this all a bit later.
Jun 18, 2013 at 12:12 PM
If I use low level such as above , for compaction, does it have to have a DIGITAL name or not ? Can I use some temporary name like generated from guids etc ?
Jun 18, 2013 at 12:15 PM
krome wrote:
If I use low level such as above , for compaction, does it have to have a DIGITAL name or not ? Can I use some temporary name like generated from guids etc ?
must be digital... it's in source code, only TransactionJournal and Scheme can have their own names.
Jun 19, 2013 at 2:07 AM
Edited Jun 19, 2013 at 5:09 AM
-
Jun 21, 2013 at 2:22 PM
Edited Jun 21, 2013 at 2:40 PM
Something weird is going on blaze.




Simplified version that i tested today:
public void CompactAnalyticalTable (string temporaryDirectory)
{
   string tableName;
   string tmpTableName;
   string tmpTablePathname;
   DBreezeConfiguration tmpConfiguration;
   DBreezeEngine tmpEngine;
   
   tableName = "someTable";
   using (Transaction transaction = engine.GetTransaction (eTransactionTablesLockTypes.EXCLUSIVE, tableName))
    try
    {
     temporaryDirectory = Path.GetFullPath (Path.Combine (temporaryDirectory, "engine", Guid.NewGuid ().ToString ()));
     tmpConfiguration = new DBreezeConfiguration ()
     {
      DBreezeDataFolderName = temporaryDirectory,
      Storage = DBreezeConfiguration.eStorage.DISK,
     };
     tmpEngine = new DBreezeEngine (tmpConfiguration);
     transaction.SynchronizeTables (tableName);

     tmpTableName = String.Format ("temporary/{0}", Guid.NewGuid ().ToString ());
     using (Transaction tmpTransaction = tmpEngine.GetTransaction (eTransactionTablesLockTypes.EXCLUSIVE, tmpTableName))
     {
      tmpTransaction.SynchronizeTables (tmpTableName);

      tmpTransaction.Insert<byte, byte[]> (tmpTableName, 1, new byte[] { 0x1, 0x2, 0x3, 0x4, 0x5 });
      tmpTransaction.Insert<byte, byte[]> (tmpTableName, 2, new byte[] { 0x6, 0x7, 0x8, 0x9, 0xa });

      tmpTransaction.Commit ();
      tmpTablePathname = tmpEngine.Scheme.GetTablePathFromTableName (tmpTableName);
     }
     tmpEngine.Dispose ();
   
     transaction.RestoreTableFromTheOtherFile (tableName, tmpTablePathname);
     transaction.Commit ();
    }
    catch (Exception ex)
    {
     LOG.ErrorFormat (ex.ToString ());
    }
  }
The 'someTable' table, contained 7 entries (nested tables). As You can see all I do is insert 2 keys.
Table is transferred with RestoreTableFromOtherFile from other engine.
But there is some weird problem, transaction.Count ("someTable"), will still show that there are 7 entries.

As far as i understand, after restoretableFromOtherFile & transaction.commit () transaction should now reflect the new table (restored from other egine) ?

Especially I had problems when I inserted nested tables.

Is something else not cleared maybe when restorefromotherfile is used ?

I would really kindly ask You to help me with this, I am at final stages. Where I need to deploy.
Jun 21, 2013 at 2:49 PM
Edited Jun 21, 2013 at 2:51 PM
This is especially true if i replace those 2x inserts in above code with:
NestedTable nt1 = transaction.InsertTable<byte> (tableName, (byte) 1, 0)
NestedTable nt2 = tmpTransaction.InsertTable<byte> (tmpTableName, (byte) 1, 0);

foreach (Row<byte[], byte[]> row in nt1.SelectForward<byte[], byte[]> (true))
      {
       if (!row.Exists)
        continue;
       nt2.Insert<byte[], byte[]> (row.Key, row.Value);
      }
I found out that if i leave out: transaction.Commit (); the results are different.
Jun 21, 2013 at 3:03 PM
Try to get rid of following lines in the "simplified version":
  • transaction.SynchronizeTables (tableName);
  • tmpTransaction.SynchronizeTables (tmpTableName);
  • transaction.Commit (); - this one we don't need at all, at this place.
Jun 21, 2013 at 3:08 PM
Yup, apperently transaction.commit () is problematic after restorefromotherfile. Could you investigate it plz ?
Jun 21, 2013 at 3:10 PM
also for me is not clear why tmpEngine is open with EXLUSIVE transaction....must be just GetTransaction()
Jun 21, 2013 at 3:12 PM
krome wrote:
Yup, apperently transaction.commit () is problematic after restorefromotherfile. Could you investigate it plz ?
transaction.commit () SHOLUD NOT be after restorefromotherfile.
Why do you need it there?
Jun 21, 2013 at 3:14 PM
Edited Jun 21, 2013 at 3:14 PM
I didn't know that. After I debugged Your code, I found it weird. Btw why not exclusive on tmpEngine ? Is there any harm ? It's a temporary table. Btw thanks for quick reply as always :)
Jun 21, 2013 at 3:28 PM
No extra harm, but there is a sense to open temporary DBreeze engine instance in the beginning of the application together with the working instance. And dispose it in the end of application. Temp engine logical table names could be the same as in working instance.
First you copy from working engine (WE) into temporary engine (TE), then execute restorefromotherfile, then execute tran.RemoveKeys(withFIleRecreation = true) on temp engine. You lock exclusively tables only in working engine transaction, and temp engine tables will be "logically" locked.

For now we can assume that restorefromotherfile command is a Commit and we don't need any other commit.
Problem of the Commit after restorefromotherfile , probably, exists because of

NestedTable nt1 = transaction.InsertTable<byte> (tableName, (byte) 1, 0)
insetead of
NestedTable nt1 = transaction.SelectTable<byte> (tableName, (byte) 1, 0)

... but it should not happen if commit is not executed after restorefromotherfile, smth. like this for now.
Jun 21, 2013 at 3:37 PM
Edited Jun 21, 2013 at 3:38 PM
Good idea, but at the beginning I designed everything around it to use single engine. I don't wanna clutter the code for now. Maybe when I release second version.

About the commit, fine I explored it You helped clarify. But in the future I think You should do something about it. Otherwise other people will have problems. Or at least for now document it ? :)