mongodb - read from secondary with mgo.Monotonic -

i trying configure reading primary , 2 secondary nodes of mongo replica set provide better load balancing. each of 3 nodes on different machines ip addresses: ip1, ip2, ip3.

my golang site, martini web server 2 urls /insert , /get:

package main  import (     "github.com/go-martini/martini"     "gopkg.in/mgo.v2"     "gopkg.in/mgo.v2/bson"     "net/http" )  const (     dialstr        = "ip1:port1,ip2:port2,ip3:port3"     dbname         = "test"     collectionname = "test"     elementscount  = 1000 )  var mainsessionforsave *mgo.session  func connecttomongo() {     var err error     mainsessionforsave, err = mgo.dial(dialstr)     mainsessionforsave.setmode(mgo.monotonic, true)     if err != nil {         panic(err)     } }  func getmgosessionperrequest() *mgo.session {     var sessionperrequest *mgo.session     sessionperrequest = mainsessionforsave.copy()     return sessionperrequest }  func main() {     connecttomongo()     preparemartini().run() }  type element struct {     int `bson:"i"` }  func preparemartini() *martini.classicmartini {     m := martini.classic()     sessionperrequest := getmgosessionperrequest()     m.get("/insert", func(w http.responsewriter, r *http.request) {         := 0; < elementscount; i++ {             e := element{i: i}             err := collection(sessionperrequest).insert(&e)             if err != nil {                 panic(err)             }         }         w.write([]byte("data inserted successfully"))     })     m.get("/get", func(w http.responsewriter, r *http.request) {         var element element         const findi = 500         err := collection(sessionperrequest).find(bson.m{"i": findi}).one(&element)         if err != nil {             panic(err)         }         w.write([]byte("get data successfully"))      })      return m }  func collection(s *mgo.session) *mgo.collection {     return s.db(dbname).c(collectionname) } 

i run golang site command go run site.go , prepare experiment requested http://localhost:3000/insert - after minute test data inserted.

then started test reading secondary , primary nodes in attacker.go:

package main  import (     "fmt"     "time"      vegeta "github.com/tsenart/vegeta/lib" )  func main() {      rate := uint64(4000) // per second     duration := 4 * time.second     targeter := vegeta.newstatictargeter(&vegeta.target{         method: "get",         url:    "http://localhost:3000/get",     })     attacker := vegeta.newattacker()      var results vegeta.results     res := range attacker.attack(targeter, rate, duration) {         results = append(results, res)     }      metrics := vegeta.newmetrics(results)     fmt.printf("99th percentile: %s\n", metrics.latencies.p99) } 

running go run attacker.go requested url http://localhost:3000/get 4000 times per second. while attacker working opened 3 servers , run htop command watch resources consumption. primary node shows under high load cpu 80%. secondaries calm.

why?

as used mgo.monotonic ...

mainsessionforsave.setmode(mgo.monotonic, true) 

... expected read nodes: ip1, ip2, ip3 , expected watch nodes under equal load , equal cpu consumption. not so. did configure wrong? in fact mgo.monotonic not working in case , read primary node.

the sessionperrequest created once: preparemartini called @ server startup, , sessionperrequest set then. closures passed m.get() access variable. then, after first write (during test setup), mgo access primary:

monotonic consistency start reading slave if possible, load better distributed, , once first write happens connection switched master.

(if mgo continued reading secondary after writing primary, read wouldn't reflect write made, pain. , switching primary should newer data getting secondary, never older, preserves monotonicity. that's how should ideally work, anyway; see "open issues" link below more.)

the solution push creating session down handlers, e.g., remove sessionperrequest , put explicit atop each handler, like

coll := mainsessionforsave.copy().db(dbname).collection(collname) 

all consistency promises should read in light of open issues mongodb consistency: right now, during network partitions, reads can see old data , writes later rolled back, when mgo trying read primary. (compare-and-set doesn't have issue, of course that's larger slower operation.) it's worth perusing post discussion of consistency levels , descriptions of how different database behaviors might manifest app's end-users.