Class Rufus::Tokyo::Cabinet

  1. lib/rufus/tokyo/cabinet/abstract.rb
Parent: Object

A ‘cabinet’, ie a Tokyo Cabinet [abstract] database.

Follows the abstract API described at :

http://tokyocabinet.sourceforge.net/spex-en.html#tcadbapi

An usage example :

db = Rufus::Tokyo::Cabinet.new('test_data.tch')
db['pillow'] = 'Shonagon'

db.size # => 1
db['pillow'] # => 'Shonagon'

db.delete('pillow') # => 'Shonagon'
db.size # => 0

db.close

Included modules

  1. HashMethods
  2. Transactions
  3. Outlen

Public class methods

new (name, params={})

Creates/opens the cabinet, raises an exception in case of creation/opening failure.

This method accepts a ‘name’ parameter and an optional ‘params’ hash parameter.

‘name’ follows the syntax described at

http://tokyocabinet.sourceforge.net/spex-en.html#tcadbapi

under tcadbopen(). For example :

db = Rufus::Tokyo::Cabinet.new('casket.tch#bnum=100000#opts=ld')

will open (eventually create) a hash database backed in the file ‘casket.tch’ with a bucket number of 100000 and the ‘large’ and ‘deflate’ options (opts) turned on.

Note that there is an open method similar to File#open for openening a db and closing it when it’s no longer needed :

Rufus::Tokyo::Cabinet.new('data.tch') do |db|
  db['key'] = value
end

database name

From tokyocabinet.sourceforge.net/spex-en.html#tcadbapi :

‘If it is “*”, the database will be an on-memory hash database. If it is

"+", the database will be an on-memory tree database. If its suffix is
".tch", the database will be a hash database. If its suffix is ".tcb",
the database will be a B+ tree database. If its suffix is ".tcf", the
database will be a fixed-length database. If its suffix is ".tct", the
database will be a table database.'

You’re supposed to give a path to the database file you want to use and Cabinet expects you to give the proper prefix.

db = Rufus::Tokyo::Cabinet.new('data.tch') # hash database
db = Rufus::Tokyo::Cabinet.new('data.tcb') # B+ tree db
db = Rufus::Tokyo::Cabinet.new('data.tcf') # fixed-length db

will result with the same file names :

db = Rufus::Tokyo::Cabinet.new('data', :type => :hash) # hash database
db = Rufus::Tokyo::Cabinet.new('data', :type => :btree) # B+ tree db
db = Rufus::Tokyo::Cabinet.new('data', :type => :fixed) # fixed-length db

You can open an in-memory hash and an in-memory B+ tree with :

h = Rufus::Tokyo::Cabinet.new(:mem_hash) # or
h = Rufus::Tokyo::Cabinet.new('*')

t = Rufus::Tokyo::Cabinet.new(:mem_tree) # or
t = Rufus::Tokyo::Cabinet.new('+')

parameters

There are two ways to pass parameters at the opening of a db :

db = Rufus::Tokyo::Cabinet.new('data.tch#opts=ld#mode=w') # or
db = Rufus::Tokyo::Cabinet.new('data.tch', :opts => 'ld', :mode => 'w')

most verbose :

db = Rufus::Tokyo::Cabinet.new(
  'data', :type => :hash, :opts => 'ld', :mode => 'w')

mode

* :mode    a set of chars ('r'ead, 'w'rite, 'c'reate, 't'runcate,
           'e' non locking, 'f' non blocking lock), default is 'wc'

other parameters

‘On-memory hash database supports “bnum”, “capnum”, and “capsiz”.

On-memory tree database supports "capnum" and "capsiz".
Hash database supports "mode", "bnum", "apow", "fpow", "opts",
"rcnum", and "xmsiz".
B+ tree database supports "mode", "lmemb", "nmemb", "bnum", "apow",
"fpow", "opts", "lcnum", "ncnum", and "xmsiz".
Fixed-length database supports "mode", "width", and "limsiz"'

 * :opts    a set of chars ('l'arge, 'd'eflate, 'b'zip2, 't'cbs)
            (usually empty or something like 'ld' or 'lb')

 * :bnum    number of elements of the bucket array
 * :apow    size of record alignment by power of 2 (defaults to 4)
 * :fpow    maximum number of elements of the free block pool by
            power of 2 (defaults to 10)
 * :mutex   when set to true, makes sure only 1 thread at a time
            accesses the table (well, Ruby, global thread lock, ...)

 * :rcnum   specifies the maximum number of records to be cached.
            If it is not more than 0, the record cache is disabled.
            It is disabled by default.
 * :lcnum   specifies the maximum number of leaf nodes to be cached.
            If it is not more than 0, the default value is specified.
            The default value is 2048.
 * :ncnum   specifies the maximum number of non-leaf nodes to be
            cached. If it is not more than 0, the default value is
            specified. The default value is 512.

 * :xmsiz   specifies the size of the extra mapped memory. If it is
            not more than 0, the extra mapped memory is disabled.
            The default size is 67108864.

 * :capnum  specifies the capacity number of records.
 * :capsiz  specifies the capacity size of using memory.

 * :dfunit  unit step number. If it is not more than 0,
            the auto defragmentation is disabled. (Since TC 1.4.21)

NOTE :

On reopening a file, Cabinet will tend to stick to the parameters as set when the file was opened. To change that, have a look at the man pages of the various command line tools coming with Tokyo Cabinet.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 179
179:     def initialize (name, params={})
180: 
181:       @db = lib.tcadbnew
182: 
183:       name = '*' if name == :mem_hash # in memory hash database
184:       name = '+' if name == :mem_tree # in memory B+ tree database
185: 
186:       if type = params.delete(:type)
187:         name += { :hash => '.tch', :btree => '.tcb', :fixed => '.tcf' }[type]
188:       end
189: 
190:       @path = name
191: 
192:       name = name + params.collect { |k, v| "##{k}=#{v}" }.join('')
193: 
194:       (lib.tcadbopen(@db, name) == 1) || raise(
195:         TokyoError.new("failed to open/create db '#{name}' #{params.inspect}"))
196: 
197:       self.default = params[:default]
198:       @default_proc ||= params[:default_proc]
199:     end
new_hash (params={})

Returns a new in-memory hash. Accepts the same optional params hash as new().

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 223
223:     def self.new_hash (params={})
224: 
225:       self.new(:hash, params)
226:     end
new_tree (params={})

Returns a new in-memory B+ tree. Accepts the same optional params hash as new().

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 231
231:     def self.new_tree (params={})
232: 
233:       self.new(:tree, params)
234:     end
open (name, params={}) {|db| ...}

Same args as initialize, but can take a block form that will close the db when done. Similar to File.open

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 204
204:     def self.open (name, params={})
205: 
206:       db = self.new(name, params)
207: 
208:       if block_given?
209:         yield db
210:         nil
211:       else
212:         db
213:       end
214: 
215:     ensure
216: 
217:       db.close if block_given? && db
218:     end

Public instance methods

[]= (k, v)

No comment

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 252
252:     def []= (k, v)
253: 
254:       k = k.to_s; v = v.to_s
255: 
256:       lib.abs_put(@db, k, Rufus::Tokyo.blen(k), v, Rufus::Tokyo.blen(v))
257:     end
add_double (key, inc=1)

Alias for incr

add_int (key, inc=1)

Alias for incr

adddouble (key, inc=1)

Alias for incr

addint (key, inc=1)

Alias for incr

clear ()

Removes all the records in the cabinet (use with care)

Returns self (like Ruby’s Hash does).

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 303
303:     def clear
304: 
305:       lib.abs_vanish(@db)
306: 
307:       self
308:     end
close ()

Closes the cabinet (and frees the datastructure allocated for it), returns true in case of success.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 320
320:     def close
321: 
322:       result = lib.abs_close(@db)
323:       lib.abs_del(@db)
324: 
325:       (result == 1)
326:     end
compact_copy (target_path)

Copies the current cabinet to a new file.

Does it by copying each entry afresh to the target file. Spares some space, hence the ‘compact’ label...

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 342
342:     def compact_copy (target_path)
343: 
344:       @other_db = Cabinet.new(target_path)
345:       self.each { |k, v| @other_db[k] = v }
346:       @other_db.close
347:     end
copy (target_path)

Copies the current cabinet to a new file.

Returns true if it was successful.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 332
332:     def copy (target_path)
333: 
334:       (lib.abs_copy(@db, target_path) == 1)
335:     end
defrag ()

Triggers a defrag run (TC >= 1.4.21 only)

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 482
482:     def defrag
483: 
484:       raise(NotImplementedError.new(
485:         "method defrag is supported since Tokyo Cabinet 1.4.21. " +
486:         "your TC version doesn't support it"
487:       )) unless lib.respond_to?(:tctdbsetdfunit)
488: 
489:       call_misc('defrag', Rufus::Tokyo::List.new)
490:     end
delete (k)

Removes a record from the cabinet, returns the value if successful else nil.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 283
283:     def delete (k)
284: 
285:       k = k.to_s
286: 
287:       v = self[k]
288: 
289:       (lib.abs_out(@db, k, Rufus::Tokyo.blen(k)) == 1) ? v : nil
290:     end
delete_keys_with_prefix (prefix)

Deletes all the entries whose keys begin with the given prefix

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 411
411:     def delete_keys_with_prefix (prefix)
412: 
413:       call_misc(
414:         'outlist', lib.abs_fwmkeys(@db, prefix, Rufus::Tokyo.blen(prefix), -1))
415:           # -1 for no limits
416: 
417:       nil
418:     end
incr (key, inc=1)

Increments the value stored under the given key with the given increment (defaults to 1 (integer)).

Accepts an integer or a double value.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 460
460:     def incr (key, inc=1)
461: 
462:       key = key.to_s
463: 
464:       v = inc.is_a?(Fixnum) ?
465:         lib.addint(@db, key, Rufus::Tokyo.blen(key), inc) :
466:         lib.adddouble(@db, key, Rufus::Tokyo.blen(key), inc)
467: 
468:       raise(TokyoError.new(
469:         "incr failed, there is probably already a string value set " +
470:         "for the key '#{key}'"
471:       )) if v == Rufus::Tokyo::INT_MIN || (v.respond_to?(:nan?) && v.nan?)
472: 
473:       v
474:     end
keys (options={})

Returns an array with all the keys in the databse

With no options given, this method will return all the keys (strings) in a Ruby array.

:prefix --> returns only the keys who match a given string prefix

:limit --> returns a limited number of keys

:native --> returns an instance of Rufus::Tokyo::List instead of
  a Ruby Hash, you have to call #free on that List when done with it !
  Else you're exposing yourself to a memory leak.
[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 370
370:     def keys (options={})
371: 
372:       outlen = nil
373: 
374:       if pre = options[:prefix]
375: 
376:         l = lib.abs_fwmkeys(
377:           @db, pre, Rufus::Tokyo.blen(pre), options[:limit] || -1)
378: 
379:         l = Rufus::Tokyo::List.new(l)
380: 
381:         options[:native] ? l : l.release
382: 
383:       else
384: 
385:         limit = options[:limit] || -1
386:         limit = nil if limit < 1
387: 
388:         l = options[:native] ? Rufus::Tokyo::List.new : []
389: 
390:         lib.abs_iterinit(@db)
391: 
392:         outlen = FFI::MemoryPointer.new(:int)
393: 
394:         loop do
395:           break if limit and l.size >= limit
396:           out = lib.abs_iternext(@db, outlen)
397:           break if out.address == 0
398:           l << out.get_bytes(0, outlen.get_int(0))
399:         end
400: 
401:         l
402:       end
403: 
404:     ensure
405: 
406:       outlen.free if outlen
407:     end
ldelete (keys)

Given a list of keys, deletes all the matching entries (in one sweep).

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 448
448:     def ldelete (keys)
449: 
450:       keys = keys.collect { |k| k.to_s }
451: 
452:       call_misc('outlist', Rufus::Tokyo::List.new(keys))
453:     end
lget (keys)

Given a list of keys, returns a Hash { key => value } of the matching entries (in one sweep).

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 423
423:     def lget (keys)
424: 
425:       keys = keys.collect { |k| k.to_s }
426: 
427:       Hash[*call_misc('getlist', Rufus::Tokyo::List.new(keys))]
428:     end
lib ()

Using the cabinet lib

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 238
238:     def lib
239: 
240:       CabinetLib
241:     end
lput (hash)

Alias for merge!

merge! (hash)

Merges the given hash into this Cabinet (or Tyrant) and returns self.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 432
432:     def merge! (hash)
433: 
434:       call_misc(
435:         'putlist',
436:         hash.inject(Rufus::Tokyo::List.new) { |l, (k, v)|
437:           l << k.to_s
438:           l << v.to_s
439:           l
440:         })
441: 
442:       self
443:     end
path ()

Returns the path to this database.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 245
245:     def path
246: 
247:       @path
248:     end
putkeep (k, v)

Like put but doesn’t overwrite the value if already set. Returns true only if there no previous entry for k.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 262
262:     def putkeep (k, v)
263: 
264:       k = k.to_s; v = v.to_s
265: 
266:       (lib.abs_putkeep(
267:         @db, k, Rufus::Tokyo.blen(k), v, Rufus::Tokyo.blen(v)) == 1)
268:     end
size ()

Returns the number of records in the ‘cabinet’

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 294
294:     def size
295: 
296:       lib.abs_rnum(@db)
297:     end
sync ()

“synchronize updated contents of an abstract database object with the file and the device“

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 352
352:     def sync
353: 
354:       (lib.abs_sync(@db) == 1)
355:     end
tranabort ()

Warning : this method is low-level, you probably only need to use transaction and a block.

Direct call for ‘transaction abort’.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 521
521:     def tranabort
522: 
523:       #check_transaction_support
524: 
525:       libcall(:tcadbtranabort)
526:     end
tranbegin ()

Warning : this method is low-level, you probably only need to use transaction and a block.

Direct call for ‘transaction begin’.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 497
497:     def tranbegin
498: 
499:       #check_transaction_support
500: 
501:       libcall(:tcadbtranbegin)
502:     end
trancommit ()

Warning : this method is low-level, you probably only need to use transaction and a block.

Direct call for ‘transaction commit’.

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 509
509:     def trancommit
510: 
511:       #check_transaction_support
512: 
513:       libcall(:tcadbtrancommit)
514:     end
weight ()

Returns the ‘weight’ of the db (in bytes)

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 312
312:     def weight
313: 
314:       lib.abs_size(@db)
315:     end

Protected instance methods

call_misc (function, list_pointer)

Wrapping tcadbmisc or tcrdbmisc (and taking care of freeing the list_pointer)

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 540
540:     def call_misc (function, list_pointer)
541: 
542:       list_pointer = list_pointer.pointer \
543:         if list_pointer.is_a?(Rufus::Tokyo::List)
544: 
545:       begin
546:         l = do_call_misc(function, list_pointer)
547:         raise "function '#{function}' failed" unless l
548:         Rufus::Tokyo::List.new(l).release
549:       ensure
550:         Rufus::Tokyo::List.free(list_pointer)
551:       end
552:     end
do_call_misc (function, list_pointer)

Calls the tcadbmisc function

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 556
556:     def do_call_misc (function, list_pointer)
557: 
558:       lib.tcadbmisc(@db, function, list_pointer)
559:     end
get (k)

(The actual #[] method is provided by HashMethods

[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 272
272:     def get (k)
273: 
274:       k = k.to_s
275: 
276:       outlen_op(:abs_get, k, Rufus::Tokyo.blen(k))
277:     end
libcall (lib_method, *args)
[show source]
     # File lib/rufus/tokyo/cabinet/abstract.rb, line 561
561:     def libcall (lib_method, *args)
562: 
563:       (eval(%{ lib.#{lib_method}(@db, *args) }) == 1) or \
564:         raise TokyoError.new("call to #{lib_method} failed")
565:     end