Postgres doesn’t support active-active replication natively. As of this writing, we have to rely on 3rd party tools. I decided to go with BDR. I didn’t want to spin up multiple VMs. So, the obvious choice is docker. Make sure you have docker on mac & docker compose. Step1: Running 2 Postgres instances using docker container Thanks to jgiannuzzi, who created a docker image with Postgres and BDR. docker-compose.yml file content

This is a guest post by Suvam Prasad (https://www.linkedin.com/in/suvamprasad/). I am very glad to have his post. This book is very special to me because it helped me to discover new perspectives of web designing mythology. The book is written by Steve Krug(a famous user experience professional). The book contained hardly 216 pages that give you enough knowledge and idea to understand web design usability, user accessibility, and how to be a great user experience designer.

TLDR; A short book on Product Development and especially explains how B2C (especially social media) companies make their users hooked on their products. Though Product owners, designers, and founders can use this book in improving their products, every user can read this to understand why some of the popular apps do things (like infinite scroll, weekly digest emails, notifications) which we use day-to-day. Hooked also overlaps with Nir Eyal’s other book Undistractable

Disclaimer: This book was published in 1989. So, we have to time travel to understand a few things, but overall the concepts are still relevant. Also, read this during COVID-19 (middle of a global pandemic). This review is a little different than other books. Going to share most of the notes as points. Peter Lynch is America’s money manager. He was Vice-Chairman of the Fidelity Management & Research Company. He was the portfolio manager of Fidelity Magellan Fund, which was the best performing fund in the world from May 1977 to May 1990.

Part 1 is about Master Internal Triggers which talks about that distractions are mainly caused by discomfort. Most people don’t want to acknolwedge the uncomfortable truth that distraction is always an unhealthy escape from reality. Being indistractable means striving to do what you say you will do. We are compelled to reach for things we supposedly need but really don’t. (eg. checking emails) We tend to blame things like television, junk food, social media etc - but these are all promiate causes of our distraction, but not the root cause.

Here is a list of books which I really enjoyed reading. They are mostly around the topics such as Self Development, Management, Finance, Technology. Give and Take by Adam Grant Atomic Habits by James Clear A random walkdown wall street Designing Data Intensive Application surely you’re joking mr. feynman How To Win Friends and Influence People Lean Startup The Go programming languate Make your bed Hard thing about hard things One Up On Wall Street Learn to Earn Indistractable Thinking fast and slow (I couldn’t complete this book.

Chapter 1 is Good Returns which explains the Dangers and Rewards of Giving more than you get with multiple examples. One of the things which striked me You can’t just ignore someone because you don’t think they’re important enough. I can’t agree more on that. I have seen first class that founders making this mistake. Its so easy to just ignore, but think about the long term or big picture.

Admiral William H. McRaven served as Navy SEAL for 37 years. As a Four-Start Admiral, his last assignment was as Commander of all U.S. Special Operations Forces. He attended the University of Texas, Austin. This book is based on his commencement speech at his alma mater. He served as the Chancellor for the University of Texas system between 2015 and 2018. Summary Chapter 1: Start Your Day With a Task Completed If you want to change the world… start off by making your bed.

Summary Chapter 1: Tutorial Introduction to various major compoents of Golang Topics such as data types, functions, packages, concurrency in Go was illustrated using simple programs. Chapter 2: Program Structure For every type T, there is a corresponding conversion operation T(x) that converts value of x to type T (if applicable)

I was trying to play with Phil’s pgtam and the first step is that to install Postgres version 16. Sounds fairly innocous. When I ran configure, I was getting the below error: checking for icu-uc icu-i18n... no configure: error: ICU library not found If you have ICU already installed, see config.log for details on the failure. It is possible the compiler isn't looking in the proper directory. Use --without-icu to disable ICU support.

Libpq-Postgres-Go After I started working on Postgres, I have heard this term libpq enough times, but never had a good grasp of it. After digging around this topic for a couple of days, here is my understanding. From the Postgres doc https://www.postgresql.org/docs/9.5/libpq.html, libpq is the C application programmer’s interface to PostgreSQL. libpq is a set of library functions that allow client programs to pass queries to the PostgreSQL backend server and to receive the results of these queries

closure is a function inside another function where you can reference variable defined in the outer function. I have seen them useful in dealing with packages where a function expects an type func() as an argument where the func() doesn’t take any argument. example from package github.com/cenkalti/backoff signature of backoff.Retry is as below backoff.Retry(o backoff.Operation, b backoff.Backoff) where backoff.Operation is of type func() error type Operation func() error func main() { func retryWrapper(imp string) func() error{ return func() { funcThatCouldFail(imp) retun nil } } err := backoff.

I was writing an article for Infoq on the topic less spoken costs of managed databases and one question that the editor asked is how much margin RDS makes compared to running a Postgres instance on EC2? That intrigued me and honestly, I never did the math so far. I have been using AWS-managed databases since 2016 and I thought what’s a better time to do an analysis of Cloud cost than today ?

In Google Cloud Next 2019, they introduced a new product called Cloud Run. I’ve been using it from the EAP (Early Access Program) days. [As the name suggests, basically it runs your docker image. You might be familiar with other serverless products such as Cloud Function or lambda where you provide your source code instead of a docker image. In my opinion, Cloud Run is more flexible than functions. Let me explain why.

This is part of my Today I Learnt series where I share whatever I am learning something new. After having issues with port forwarding in Xfinity I decided to look for alternative solutions. I have used Cloudflare tunnel (used to be called Argo Tunnel) in the past to expose websites running on my laptop to the Internet. So, I decided to try it out for Ghost blogging site which I am setting up for my dad.

This aricle is inspired by Give and Take by Adam Grant. I was wondering how can I help my friends and network. Searching a job is something stressful and we all have been through. Once my wife and I searched for a year and finally we moved to San Francisco. So far its been a challenge for my network to know whether I can refer to a particular company. To solve this, I am publishing a list of companies where I can refer.

Mark my word. We had so many competitions in the past. Eg. vim vs emacs, tabs vs spaces etc. The new comparison which is added in this decade is functions vs containers Both the functions and containers can be serverless. So, please avoid the notion of serverless means functions. In fact, predominantly all the functions (like lambda, cloud functions) are running behind the screen as containers. (there are few products like Cloudflare workers are running without containers)

In this article we will see how to convert a simple python application to a containerized (Docker) one Go to the profile of vignesh ravichandran vignesh ravichandran Sep 23, 2017 TL;DR -> Containerize a python app. Push the image and execute from Google Cloud. On a leisurely Saturday afternoon, I thought its nice to play with containers. I was fascinated with the idea of containerizing an app and running it across different environments.

Citus: Distributed PostgreSQL for Data-Intensive Applications paper can be downloaded here. Recently, our team got a request to provide a solution to shard Postgres. One of the solutions that we discussed was Citus. I have heard about the product and seen their blogs related to Postgres in the past but never used it. I thought it would be fun to read about its internal workings. If you find something wrong on the notes, please send a pull request.

In Google Cloud Next 2019, they introduced a new product called Cloud Run. I’ve been using it from the EAP (Early Access Program) days. [As the name suggests, basically it runs your docker image. You might be familiar with other serverless products such as Cloud Function or lambda where you provide your source code instead of a docker image. In my opinion, Cloud Run is more flexible than functions. Let me explain why.

Step 1: Setting up HAProxy I hope you gone through Part1 of this series. Perhaps, one thing you might have noticed is that I’ve to connect to the specific master database. In our case, since both the databases are running on docker, only the localhost port is different. (In a production environment, most likely you going to run the databases on a different host). The main reason for active-active replication is high availability.

Postgres doesn’t support active-active replication natively. As of this writing, we have to rely on 3rd party tools. I decided to go with BDR. I didn’t want to spin up multiple VMs. So, the obvious choice is docker. Make sure you have docker on mac & docker compose. Step1: Running 2 Postgres instances using docker container Thanks to jgiannuzzi, who created a docker image with Postgres and BDR. docker-compose.yml file content

In this article we will see how to convert a simple python application to a containerized (Docker) one Go to the profile of vignesh ravichandran vignesh ravichandran Sep 23, 2017 TL;DR -> Containerize a python app. Push the image and execute from Google Cloud. On a leisurely Saturday afternoon, I thought its nice to play with containers. I was fascinated with the idea of containerizing an app and running it across different environments.

I am writing this around the time where we are at the beginning of a global pandemic. Everyone is trying to figure out how to keep their business up and running during this challenging period. Most of the employers have mandated work from remote policy. One of the main challenges is how to connect to your applications remotely. Traditionally the answer is VPN. There are so many challenges wrt scalability, availability, and performance of these traditional VPN software.

When I first heard GPG it was little daunting. I haven’t used it exactly, even though I was familar with the concept. So, I decided to dig deeper and understand how to use it. GnuPG (GPG) is an implementation of PGP The simple idea is that you generate a key pair: secret or private key and a public key. You can provide your public key to anyone. They can then encrypt either a secret or file using your public key.

We all at some point in time thought about having a personal website. It may be for writing about your travel trips, posting pictures of food that you cooked or ranting about the tech which you don’t like. I will leave the reason to you Step1: Buying a domain This websites domain name is viggy28.dev. You want a human friendly name, so that your friends, family and rest of the internet can reach your website.

In Google Cloud Next 2019, they introduced a new product called Cloud Run. I’ve been using it from the EAP (Early Access Program) days. [As the name suggests, basically it runs your docker image. You might be familiar with other serverless products such as Cloud Function or lambda where you provide your source code instead of a docker image. In my opinion, Cloud Run is more flexible than functions. Let me explain why.

I kept hearing about the term wire protocol especially Postgres wire protocol in the recent days (Looking at you cockroachdb, yugabytedb - in a good way) but never really quite understood it. Decided to implement something simple in Go to understand it better. As always, if you find anything wrong or I misunderstood please correct me. In simple terms, “wire” - something over network generally (but PG also supports over domain sockets)

I saw this pattern of a function taking optional parameters (variadic arguments) and I never understood why they do that until today. There is a bit of a background on why I need to use functional pattern. Let’s say I don’t use functional option. Then, all the arguments need to be passed. Also, difficult to set default values and implement logic for default values. package main import ( "fmt" "log" ) type Config struct { Host string Port uint Visibility bool } type clientOption func(c *Config) error func newConfig(host string, opts .

double pointers A pointer (whose value is another variable’s address) which points to another pointer copying a pointer to another pointer copies the address copying a de-referenced pointer to another de-referenced pointer updates the value of the one to another https://gist.github.com/viggy28/604cda186ff1b872d562b31ca3976732 package main import "fmt" type DB struct { Name string } // double pointer // a pointer which points to another pointer func wrapperFunc(config **DB) func() error { return func() error { cfg := &DB{} (*cfg).

context is a standard package. Context is an interface in context package. type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} } There are 4 methods in Context interface. Most of the times, we don’t define our own types which satisfy the interface (which off course we can do), but we mostly use the factory functions provided by the context package itself.

go run command go run command is a quick way to run go program. However, what it does behind the screen is it compiles the program and runs an executable and removes it once the program is stopped. byte is a built-in alias of uint8. rune is a built-in alias of int32. The semicolon insertion rule In Go, I thought there is no need for semicolon at the end of the statement.

closure is a function inside another function where you can reference variable defined in the outer function. I have seen them useful in dealing with packages where a function expects an type func() as an argument where the func() doesn’t take any argument. example from package github.com/cenkalti/backoff signature of backoff.Retry is as below backoff.Retry(o backoff.Operation, b backoff.Backoff) where backoff.Operation is of type func() error type Operation func() error func main() { func retryWrapper(imp string) func() error{ return func() { funcThatCouldFail(imp) retun nil } } err := backoff.

defer is one of the unique keywords in Go. What does defer do? As the name suggests, it defers something. It defers function calls. A deferred function gets invoked right after the surrounding function returns. func outer() { defer inner() log.Println("outer got called") } func inner() { log.Println("inner got called") } outer got called prints first then inner got called. What the main use of defer statement? It helps mainly with actions that need to happen but not immediately.

I came across this piece of code from The Go Programming Language book. loop: for { select { case size, ok := <-fileSizes: if !ok { break loop // fileSizes was closed } nfiles++ nbytes += size case <-tick: printDiskUsage(nfiles, nbytes) } } The complete program is available in Github Named break statement allows to break perhaps multiple control statements. In this example, break loop breaks the select and for statement.

I was going through The Go Programming Language and in chapter 5 there was a section Caveat: Capturing Iteration Variables. The example is like this: var rmdirs []func() for _, dir := range tempDirs() { // 1 os.MkdirAll(dir, 0755) // 2 rmdirs = append(rmdirs, func() { os.RemoveAll(dir) // NOTE: incorrect! // 3 }) } dir is the same variable. i.e. the address of the variable is same dir’s value evaluated at this point is different at each iteration IMPORTANT dir is not evaluated here.

Concurrency channels are like typed pipes, Where you can send something from one side and receive from other side. using channel direction operator <-. Depends on the direction of the variables around the operator, its either sending or receiving. x := <-c // receive from c c <- sum // send sum to c fmt.Println(“channel receive happening”, <-c) channels are reference type. similar to maps. So when we pass channel as an argument to a function or copy we are passing the reference.

In Go, copying generally does a copy by value. So modifying the destination, doesn’t modify the source. However, that holds good only if you are copying value types not reference types (i.e. pointer, slice, map etc.) package main import ( "fmt" ) func main() { type Cat struct { age int name string Friends []string //reference type } wilson := Cat{7, "Wilson", []string{"Tom", "Tabata", "Willie"}} // shallow copy nikita := wilson // modifies both nikita and wilson nikita.

Disclaimer: Whatever I write here is my learnings. So, don’t take these words as granted or reference. Also, please lmk if there are any errors. I have often come across this term JWT. I knew the definition - Json Web Token, but honestly that’s all I knew for a veryyyy longgg time. Recently, I came across a problem where I have been told that JWT might be a good solution for it.

Libpq-Postgres-Go After I started working on Postgres, I have heard this term libpq enough times, but never had a good grasp of it. After digging around this topic for a couple of days, here is my understanding. From the Postgres doc https://www.postgresql.org/docs/9.5/libpq.html, libpq is the C application programmer’s interface to PostgreSQL. libpq is a set of library functions that allow client programs to pass queries to the PostgreSQL backend server and to receive the results of these queries

Understanding context in Golang through Postgres I was trying to learn Go package context especially with respect to Postgres. On a very high level context provides context to the operation. Yeah, I agree, the previous statement doesn’t really add much value, but hold on I don’t really know how to explain it, rather let’s go over some code. Sometimes its easier to understand by seeing it in action :)

Initializing embedded field: The field name of an anonymous field during a struct instantiation is the name of the struct (Time) type Event struct { ID int time.Time } event := Event{ ID: 1234, Time: time.Now(), } If an embedded field type implements an interface, the struct containing the embedded field will also implement this interface

program1 package main import ( "log" ) func main() { done := make(chan struct{}) go func() { log.Println("done") }() <-done //wait for background goroutine to finish } program2 run nc -l 8000 on another terminal package main import ( "log" "net" ) func main() { conn, err := net.Dial("tcp", "localhost:8000") if err != nil { log.Fatal(err) } conn.Close() done := make(chan struct{}) go func() { log.Println("done") }() <-done //wait for background goroutine to finish } Both of those examples have a receive statement from a channel that no one ever sends to.

Step 1: Setting up HAProxy I hope you gone through Part1 of this series. Perhaps, one thing you might have noticed is that I’ve to connect to the specific master database. In our case, since both the databases are running on docker, only the localhost port is different. (In a production environment, most likely you going to run the databases on a different host). The main reason for active-active replication is high availability.

I am writing this around the time where we are at the beginning of a global pandemic. Everyone is trying to figure out how to keep their business up and running during this challenging period. Most of the employers have mandated work from remote policy. One of the main challenges is how to connect to your applications remotely. Traditionally the answer is VPN. There are so many challenges wrt scalability, availability, and performance of these traditional VPN software.

When I first heard GPG it was little daunting. I haven’t used it exactly, even though I was familar with the concept. So, I decided to dig deeper and understand how to use it. GnuPG (GPG) is an implementation of PGP The simple idea is that you generate a key pair: secret or private key and a public key. You can provide your public key to anyone. They can then encrypt either a secret or file using your public key.

We all at some point in time thought about having a personal website. It may be for writing about your travel trips, posting pictures of food that you cooked or ranting about the tech which you don’t like. I will leave the reason to you Step1: Buying a domain This websites domain name is viggy28.dev. You want a human friendly name, so that your friends, family and rest of the internet can reach your website.

I am writing this around the time where we are at the beginning of a global pandemic. Everyone is trying to figure out how to keep their business up and running during this challenging period. Most of the employers have mandated work from remote policy. One of the main challenges is how to connect to your applications remotely. Traditionally the answer is VPN. There are so many challenges wrt scalability, availability, and performance of these traditional VPN software.

When I first heard GPG it was little daunting. I haven’t used it exactly, even though I was familar with the concept. So, I decided to dig deeper and understand how to use it. GnuPG (GPG) is an implementation of PGP The simple idea is that you generate a key pair: secret or private key and a public key. You can provide your public key to anyone. They can then encrypt either a secret or file using your public key.

We all at some point in time thought about having a personal website. It may be for writing about your travel trips, posting pictures of food that you cooked or ranting about the tech which you don’t like. I will leave the reason to you Step1: Buying a domain This websites domain name is viggy28.dev. You want a human friendly name, so that your friends, family and rest of the internet can reach your website.

Disclaimer: Whatever I write here is my learnings. So, don’t take these words as granted or reference. Also, please lmk if there are any errors. I have often come across this term JWT. I knew the definition - Json Web Token, but honestly that’s all I knew for a veryyyy longgg time. Recently, I came across a problem where I have been told that JWT might be a good solution for it.

From the wiki, In computing, data storage, and data transmission, character encoding is used to represent a repertoire of characters by some kind of encoding system that assigns a number to each character for digital representation. Okay, they mean converting character to some number for storing and transmitting data. There are two popular character sets 1. ASCII 1. American Standard Code for Information Interchange 2. Pretty much all the characters and symbols in modern keyboard comes under ASCII 3.

There are two types of variable declaration. recursively expanded variables - foo is assigned with bar which inturns assigned to ugh whose value is Huh? foo = $(bar) bar = $(ugh) ugh = Huh? all:;echo $(foo) $ make $ Huh? simply expanded variables - foo is assigned with bar however bar is not defined yet. so it outputs to nothing. foo := $(bar) bar := $(ugh) ugh := Huh?

This is a crosspost from my TIL What the heck is .PHONY in makefile? I have seen this for so long but I always shrugged off. Whatever that is. Well, finally I decided to learn about it. A bit of introduction to Makefile syntax. target: prerequisites <TAB> recipe The first target in a Makefile will be executed by default when we call make. blah: blah.o cc blah.o -o blah # Runs third blah.

There are two types of variable declaration. recursively expanded variables - foo is assigned with bar which inturns assigned to ugh whose value is Huh? foo = $(bar) bar = $(ugh) ugh = Huh? all:;echo $(foo) $ make $ Huh? simply expanded variables - foo is assigned with bar however bar is not defined yet. so it outputs to nothing. foo := $(bar) bar := $(ugh) ugh := Huh?

This is a crosspost from my TIL What the heck is .PHONY in makefile? I have seen this for so long but I always shrugged off. Whatever that is. Well, finally I decided to learn about it. A bit of introduction to Makefile syntax. target: prerequisites <TAB> recipe The first target in a Makefile will be executed by default when we call make. blah: blah.o cc blah.o -o blah # Runs third blah.

TL;DR: Connect with people whom you admire. Things can take longer so stay in touch. At the right moment, don’t hesitate to ask. I recently became an Engineering Manager after being an Individual Contributor for almost 8 years. If you are interested to know more about the reasoning behind the change. As like many other jobs in Tech, there is no formal training for an Engineering Manager. Reminds me of my dad’s saying “How do you teach one to swim?

Last week, I got promoted to Engineering Manager (EM) of the Database team at Cloudflare. I joined the team almost 3 years ago. A senior team member and myself were hired to form a then dedicated database team. I have been enjoying it and the team is amazing to work with. Great people. Opportunities came to either advance on the Individual Contributor (IC) or the leadership side. I picked the latter.

TL;DR: Connect with people whom you admire. Things can take longer so stay in touch. At the right moment, don’t hesitate to ask. I recently became an Engineering Manager after being an Individual Contributor for almost 8 years. If you are interested to know more about the reasoning behind the change. As like many other jobs in Tech, there is no formal training for an Engineering Manager. Reminds me of my dad’s saying “How do you teach one to swim?

Capturing thoughts from https://twitter.com/viggy28/status/1530800893842444289 When you are doing major DML changes, other than locks one more thing to keep in mind is replication lag. Especially if you use your replicas in hot standby mode. When you need to delete most of the records in a massive table, its better to create a new table and just copy the records that you need to preserve. When you need to delete all the records in a massive table, just truncate it instead of deleting them.

This aricle is inspired by Give and Take by Adam Grant. I was wondering how can I help my friends and network. Searching a job is something stressful and we all have been through. Once my wife and I searched for a year and finally we moved to San Francisco. So far its been a challenge for my network to know whether I can refer to a particular company. To solve this, I am publishing a list of companies where I can refer.

There are tools like https://tehnoblog.org/ip-tools/ip-address-in-cidr-range/ which does the job. It doesn’t support IPv6. One of my colleagues today taught me how to do it by hand quickly. Say we have an IP 2b06:4600:1101:0:abcd:efa:dbd:ea60:f5a6 is in the CIDR 2b06:4600:1101::/64 First step is to check what’s the mask bits. Here it is 64. So you take the address, take the first 64 bits and see if they are the same as the CIDR.

This article is part of my Today I Learnt. I have been trying to share whatever I learnt newly on the day. Some of them are basic, but still I want to write it down for two reasons. 1. To help me understand the concepts better 2. Potential help other people who running in to same issue. Alright, It’s all started when I bought a new domain (synergy.net) for my Dad.

In v6, /48 is the minimum prefix that can be advertised by the routers in the Internet. I believe in v4, its /24. Until yesterday I wasn’t aware of that. I might have thought you could even advertise one IP (ie. /128 or /32) but that’s not possible. Ref: https://blog.apnic.net/2020/06/01/why-is-a-48-the-recommended-minimum-prefix-size-for-routing

Last week, I got promoted to Engineering Manager (EM) of the Database team at Cloudflare. I joined the team almost 3 years ago. A senior team member and myself were hired to form a then dedicated database team. I have been enjoying it and the team is amazing to work with. Great people. Opportunities came to either advance on the Individual Contributor (IC) or the leadership side. I picked the latter.

This is something new for this site. In fact, First of its kind. Mostly, I write about Tech, Finance, books etc. Recently, I thought about writing/journaling more of my personal life and experiences. Also, interactions with the professional people whom I met. If you have known Sri or myself personally, we moved to San Francisco from Boston (Sri) and Los Angeles (myself) in 2019. We don’t have that many friends or family here in the Bay area.

double pointers A pointer (whose value is another variable’s address) which points to another pointer copying a pointer to another pointer copies the address copying a de-referenced pointer to another de-referenced pointer updates the value of the one to another https://gist.github.com/viggy28/604cda186ff1b872d562b31ca3976732 package main import "fmt" type DB struct { Name string } // double pointer // a pointer which points to another pointer func wrapperFunc(config **DB) func() error { return func() error { cfg := &DB{} (*cfg).

I was writing an article for Infoq on the topic less spoken costs of managed databases and one question that the editor asked is how much margin RDS makes compared to running a Postgres instance on EC2? That intrigued me and honestly, I never did the math so far. I have been using AWS-managed databases since 2016 and I thought what’s a better time to do an analysis of Cloud cost than today ?

I was trying to play with Phil’s pgtam and the first step is that to install Postgres version 16. Sounds fairly innocous. When I ran configure, I was getting the below error: checking for icu-uc icu-i18n... no configure: error: ICU library not found If you have ICU already installed, see config.log for details on the failure. It is possible the compiler isn't looking in the proper directory. Use --without-icu to disable ICU support.

Abstract On a high level, the talk addressed following questions: What is Edge and why are the benefits of edge? What are the challenges of implementing Postgres at the edge? How we implemented it at Cloudflare? Where Postgres at the edge is heading? Slides from the presentation Link from Developerweek Cloudx Link from Developerweek Cloudx virtual

Abstract We talked about the importance of having a good toolset, of practicing incidents and of internally advocating database best practices to a large engineering organization. Recordings of the presentation

Abstract With the advent of cloud/managed offerings, running Postgres on their own hardware is becoming rare. It’s great that a lot of the Ops work is taken care of by the provider, however, understanding a layer or two beneath these abstractions will be useful for anyone in strengthening their knowledge. For eg. When you can’t connect to an instance how do you find where the problem is? a. Are you able to reach the server i.

Abstract This talk explores how Cloudflare uses HAProxy for health checks, load balancing and reading traffic among nodes set up with Postgres streaming replication in hot standby mode. Cloudflare operates multiple Postgres Clusters across four data centers, and all of these clusters are made up of six nodes. During a primary failure, Cloudflare’s high availability system promotes a replica to become a primary, and HAProxy makes sure there is no write traffic between two primaries to avoid a split-brain scenario.

I kept hearing about the term wire protocol especially Postgres wire protocol in the recent days (Looking at you cockroachdb, yugabytedb - in a good way) but never really quite understood it. Decided to implement something simple in Go to understand it better. As always, if you find anything wrong or I misunderstood please correct me. In simple terms, “wire” - something over network generally (but PG also supports over domain sockets)

I use pg_rewind for rewind a Postgres cluster which is ahead of the primary. Yesterday, after the rewind Postgres didn’t start. It failed with the below error. This is using 9.6 version pg_rewind client. Sep 2 20:06:14.371503 productiondb[1]: [1-1] time=2022-09-02 20:06:14.338 GMT,pid=78197,user=admin,db=postgres,client=[local],appname=[unknown],vid=,xid=0 FATAL: the database system is starting up Sep 2 20:06:14.371611 productiondb[1]: [1-1] time=2022-09-02 20:06:14.370 GMT,pid=78184,user=,db=,client=,appname=,vid=,xid=0 LOG: entering standby mode Sep 2 20:06:14.371991 productiondb[1]: [1-1] time=2022-09-02 20:06:14.371 GMT,pid=78184,user=,db=,client=,appname=,vid=,xid=0 PANIC: could not open file "pg_replslot/ae0750b1/state": No such file or directory Sep 2 20:06:14.

Citus: Distributed PostgreSQL for Data-Intensive Applications paper can be downloaded here. Recently, our team got a request to provide a solution to shard Postgres. One of the solutions that we discussed was Citus. I have heard about the product and seen their blogs related to Postgres in the past but never used it. I thought it would be fun to read about its internal workings. If you find something wrong on the notes, please send a pull request.

Capturing thoughts from https://twitter.com/viggy28/status/1530800893842444289 When you are doing major DML changes, other than locks one more thing to keep in mind is replication lag. Especially if you use your replicas in hot standby mode. When you need to delete most of the records in a massive table, its better to create a new table and just copy the records that you need to preserve. When you need to delete all the records in a massive table, just truncate it instead of deleting them.

Before I forget let me write this down here. $$ log_stament - none - all log_min_duration_statment - millisecond value When you set log_statement=none and log_min_duration_statement=1 then any statement which takes longer than 1 millisecond will be logged. When you set log_statement=all and log_min_duration_statement=1 then all statements are logged; however it only shows duration on statements longer than 1 millisecond.

From the wiki, In computing, data storage, and data transmission, character encoding is used to represent a repertoire of characters by some kind of encoding system that assigns a number to each character for digital representation. Okay, they mean converting character to some number for storing and transmitting data. There are two popular character sets 1. ASCII 1. American Standard Code for Information Interchange 2. Pretty much all the characters and symbols in modern keyboard comes under ASCII 3.

I have know about the postgres wire protocol, but first I ran into streaming replication protocol. I was looking at this code in stolon replConnParams["replication"] = "1" db, err := sql.Open("postgres", replConnParams.ConnString()) if err != nil { return nil, err } defer db.Close() rows, err := query(ctx, db, "IDENTIFY_SYSTEM") if err != nil { return nil, err } I was wondering what is this IDENTIFY_SYSTEM. A G-search pointed me to this

Libpq-Postgres-Go After I started working on Postgres, I have heard this term libpq enough times, but never had a good grasp of it. After digging around this topic for a couple of days, here is my understanding. From the Postgres doc https://www.postgresql.org/docs/9.5/libpq.html, libpq is the C application programmer’s interface to PostgreSQL. libpq is a set of library functions that allow client programs to pass queries to the PostgreSQL backend server and to receive the results of these queries

Understanding context in Golang through Postgres I was trying to learn Go package context especially with respect to Postgres. On a very high level context provides context to the operation. Yeah, I agree, the previous statement doesn’t really add much value, but hold on I don’t really know how to explain it, rather let’s go over some code. Sometimes its easier to understand by seeing it in action :)

Step 1: Setting up HAProxy I hope you gone through Part1 of this series. Perhaps, one thing you might have noticed is that I’ve to connect to the specific master database. In our case, since both the databases are running on docker, only the localhost port is different. (In a production environment, most likely you going to run the databases on a different host). The main reason for active-active replication is high availability.

Postgres doesn’t support active-active replication natively. As of this writing, we have to rely on 3rd party tools. I decided to go with BDR. I didn’t want to spin up multiple VMs. So, the obvious choice is docker. Make sure you have docker on mac & docker compose. Step1: Running 2 Postgres instances using docker container Thanks to jgiannuzzi, who created a docker image with Postgres and BDR. docker-compose.yml file content

In this article we will see how to convert a simple python application to a containerized (Docker) one Go to the profile of vignesh ravichandran vignesh ravichandran Sep 23, 2017 TL;DR -> Containerize a python app. Push the image and execute from Google Cloud. On a leisurely Saturday afternoon, I thought its nice to play with containers. I was fascinated with the idea of containerizing an app and running it across different environments.

This is part of my Today I Learnt series where I share whatever I am learning something new. After having issues with port forwarding in Xfinity I decided to look for alternative solutions. I have used Cloudflare tunnel (used to be called Argo Tunnel) in the past to expose websites running on my laptop to the Internet. So, I decided to try it out for Ghost blogging site which I am setting up for my dad.

Mark my word. We had so many competitions in the past. Eg. vim vs emacs, tabs vs spaces etc. The new comparison which is added in this decade is functions vs containers Both the functions and containers can be serverless. So, please avoid the notion of serverless means functions. In fact, predominantly all the functions (like lambda, cloud functions) are running behind the screen as containers. (there are few products like Cloudflare workers are running without containers)

I have used many signals with kill like SIGHUP, SIGKILL etc. Today I came across USR2. Did a Google search and found the manual page from GNU. According to it, These signals are used for various other purposes. In general, they will not affect your program unless it explicitly uses them for something. Another one from the BSD mailing list USR2 is a "user defined signal" (from "man signal") It doesn't "mean" anything by definition.

I have seen this term here and there. I remember fiddling with this in pain of setting up port forwarding in xfinity/. DHCP - Dynamic Host Configuration Protocol (DHCP) is a network management protocol used on Internet Protocol (IP) networks for automatically assigning IP addresses and other communication parameters to devices connected to the network using a client–server architecture (from wiki ) client-server architecture? In a home network, router is the server and client is my PCs, laptops, Raspberry Pis etc.

namespaces is a Linux Kernel feature. There are different types of namespaces - Partitions Kernel resources such that one set of Processes see a different resources while others different - Apartment complex analogy - 7 different namespaces - PID namespace - Watching TV show - Net namespace - each namespace gets an unique IP address s list of port. - apartment address analogy - Uts namespace - host names for Ip address - Telling taxi driver apartment name instead of address - User namespace - Files are associated with VID Euser Identification - Mailbox associated to apartment unit # not with the name of the person - Mit namespace o to isolate mount points such that processes in different namespaces cannot view each others files.