With 9.4 came logical decoding but what is it and how can it be used? Besides being a precursor to bi-directional replication there are plenty of use cases for this and many don't even require you to implement a plugin. We'll look at trigger-less auditing, partial replication and full statement replication.
The document discusses upcoming changes and new features in MySQL 5.7. Key points include:
- MySQL 5.7 development has focused on performance, scalability, security and refactoring code.
- New features include online DDL support for additional DDL statements, InnoDB support for spatial data types, and cost information added to EXPLAIN output.
- Benchmarks show MySQL 5.7 providing significantly higher performance than previous versions, with a peak of 645,000 queries/second on some workloads.
Devrim Gunduz gives a presentation on Write-Ahead Logging (WAL) in PostgreSQL. WAL logs all transactions to files called write-ahead logs (WAL files) before changes are written to data files. This allows for crash recovery by replaying WAL files. WAL files are used for replication, backup, and point-in-time recovery (PITR) by replaying WAL files to restore the database to a previous state. Checkpoints write all dirty shared buffers to disk and update the pg_control file with the checkpoint location.
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...StreamNative
Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.
If you are like most test driven developers, you write automated tests for your software to get fast feedback about potential problems. Most of the tests you write will verify the functional behaviour of the software: When we call this function or press this button, the expected result is that value or that message.
But what about the non-functional behaviour, such as performance: When we perform this query the expected speed of getting results should be no more than that many milliseconds. It is important to be able to write automated performance tests as well, because they can give us early feedback about potential performance problems. But expected performance is not as clear-cut as expected results. Expected results are either correct or wrong. Expected performance is more like a threshold: If the performance is worse than this, we want the test to fail.
GPORCA is query optimizer used inside Greenplum database, the first open source MPP solution based on PostgreSQL.
These are slides presented at the PGConf Seattle 2017. It introduced the internals of GPORCA, and provide OSS developers context to contribute back to the project.
This document discusses logical replication with pglogical. It begins by explaining that pglogical performs row-oriented replication and outputs replication data that can be used in various ways. It then covers the architectures of standalone PostgreSQL, physical replication, and logical replication. The rest of the document discusses key aspects of pglogical such as its output plugin, selective replication capabilities, performance and future plans, and examples of using the output with other applications.
MySQL Time Machine by replicating into HBase - Slides from Percona Live Amste...Boško Devetak
At Booking.com we have complex MySQL installations, with very large tables in different servers. We’ve encountered the following question that we could not address with MySQL alone: How did some data/table look at a specific point in time?
Answering this question is needed for many things, from observing trends in data changes and deriving insight, to fixing data after problems. Sadly, mistakes happen, discovery of those is not always immediate, and fixing the data later is hard if you do not know how it looked at a specific point in time.
All of these become easy to solve if we have an easily accessible stream/history of data changes. However, having this is difficult because non-trivial problems need to be solved: dealing with schema changes, failover and establishing data quality guarantees.
MySQL Time Machine was made to solve these problems.
Debugging Planning Issues Using Calcite's Built-in LoggersStamatis Zampetakis
Wrong results, high memory usage (OutOfMemoryError, GC pauses, etc), unresponsive server, infinite planning time, are some common issues that may arise when using Calcite and in general a query processor in production systems.
In this talk, we will demonstrate how we can exploit Calcite's built-in loggers (notably CALCITE-4704, and CALCITE-4991) to debug such issues through use-cases from Apache Hive.
Yahoo: Experiences with MySQL GTID and Multi Threaded ReplicationYashada Jadhav
Our experiences running GTID and parallel replication in production, advantages, limitations, lessons learnt and tips and tricks we discovered along the way at Oracle Openworld 2015
This document compares PLpgSQL and PL/SQL programming languages. PLpgSQL is based on PL/SQL but has some differences in implementation and features. It focuses on simplicity and integration with PostgreSQL rather than compatibility with Oracle. While some functionality is similar, PLpgSQL is designed for the PostgreSQL environment rather than being a pure clone of PL/SQL. The author discusses the history and design of PLpgSQL.
Exploring plsql new features best practices september 2013Andrejs Vorobjovs
The document discusses exploring new features and best practices in PL/SQL for better performance. It covers topics like parsing time, bulk binding, PL/SQL function result cache, subprogram inlining, finer grained dependencies, and new features in Oracle Database 12c. The presentation provides an overview of Oracle SQL Developer and guidelines for writing efficient and readable PL/SQL code.
MySQL 5.6 - Operations and Diagnostics ImprovementsMorgan Tocker
This document discusses MySQL 5.6 and its improvements to operational and diagnostic capabilities. Key enhancements include online DDL operations that do not block reads or writes, buffer pool dump and restore for faster startup, import/export of partitioned tables, and transportable tablespaces. Diagnostic tools were improved with EXPLAIN showing more details, the ability to EXPLAIN updates and deletes, optimizer tracing, and the performance schema providing detailed query level instrumentation and monitoring by default.
This document provides an overview and programming tips for using SQL procedural language (SQL PL) stored procedures on DB2 for z/OS. It discusses various features and enhancements for SQL PL including compound blocks, templates, dynamic SQL, XML support, array data types, global variables, and autonomous transactions. The document also provides examples and best practices for writing SQL procedures, including handling naming resolution, using templates for readability, and working with arrays and dynamic SQL.
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Jaime Crespo
Tutorial delivered at Percona MySQL Conference Live London 2013.
It doesn't matter what new SSD technologies appear, or what are the latest breakthroughs in flushing algorithms: the number one cause for MySQL applications being slow is poor execution plan of SQL queries. While the latest GA version provided a huge amount of transparent optimizations -specially for JOINS and subqueries- it is still the developer's responsibility to take advantage of all new MySQL 5.6 features.
In this tutorial we will propose the attendants a sample PHP application with bad response time. Through practical examples, we will suggest step-by-step strategies to improve its performance, including:
* Checking MySQL & InnoDB configuration
* Internal (performance_schema) and external tools for profiling (pt-query-digest)
* New EXPLAIN tools
* Simple and multiple column indexing
* Covering index technique
* Index condition pushdown
* Batch key access
* Subquery optimization
Travelling in time with SQL Server 2016 - Damian WideraITCamp
SQL Server 2016 comes up with a very exciting feature called Temporal tables. You can make queries to historical data lot easier by using this feature. The mechanism is very simple however you all should know it in depth to make sure you can use it efficiently. And this is exactly what I am going to do during this session – show you how to create temporal tables, how to use and manage them.
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGPablo Garbossa
Most database products have their own auditing functionalities or plugins but they always involve overhead which means they end up having them turned off or with the bare minimum enabled.
In this workshop we will show how to get reliable logging for mysql and mongodb servers in a scalable and non intrusive way, its drawbacks and how we can build our own open source tools to achieve results similar to most commercial products.
Tools to sniff, process and act upon queries will be shared and we will show how simple is to set up and monitor a database environment so it can be replicated and grow horizontally. All the code needed will be published.
This is the speech Max Liu gave at Percona Live Open Source Database Conference 2016.
Max Liu: Co-founder and CEO, a hacker with a free soul
The slide covered the following topics:
- Why another database?
- What kind of database we want to build?
- How to design such a database, including the principles, the architecture, and design decisions?
- How to develop such a database, including the architecture and the core technologies for TiKV and TiDB?
- How to test the database to ensure the quality and stability?
The document discusses a transaction log prototype created by the GDS Publishing platform team to address bugs and issues in their system. It logs every change or transaction as a master copy for auditing purposes. Derived representations then allow future requirements to be easily supported by adding new representations based on the log. Examples shown include a bank account transaction log and how publishing documents could utilize derived representations like current/historical documents and editorial comments. The prototype implemented an event log, commands, queries to derived data, and was connected to their policy publisher. It used PostgreSQL for its transactions and JSON support. Remaining questions cover editing workflows, validation, performance, and snapshotting/replay capabilities.
This document summarizes a case study on using Exadata, Oracle Data Integrator (ODI), and parallel data loading to improve ETL performance. It describes how the environment had performance issues with ODI loads due to temporary tablespace usage. Tuning included setting dynamic sampling to 0, adding rollup tables and materialized views to pre-aggregate data, and indexing to reduce sorting and aggregation during loads. These changes led to significant improvements in elapsed time.
Similar to Fun Things to do with Logical Decoding (20)
From Warehouses to Lakes: The Value of StreamsMike Fowler
You have a beautifully modelled warehouse and a lake with all your business data but you’re still looking at the past and making decisions from what happened yesterday. In this talk we’ll look at how you can really get value from your data and make decisions as events happen and even before they do.
From Warehouses to Lakes: The Value of StreamsMike Fowler
Every business has a wealth of data but getting value from data is hard. We've tried Data Warehouses and Data Lakes, and while both give us insights we are after, they present their own challenges. Perhaps most challenging of all is making decisions based on yesterday's data. In this talk we'll look at how you can start using your data to make decisions as events happen in your business and how we can even make predictions too. Best of all, we can populate our Data Lakes and Data Warehouses at the same time keeping all the historic analytics in place.
Getting Started with Machine Learning on AWSMike Fowler
Machine Learning (ML) is an exciting field that Cloud Computing has helped to accelerate. AWS has played a big part in this with it’s continually expanding range of services from the simply named Machine Learning through to SageMaker. But how do you get started? Thankfully you don’t need to become an expert in linear algebra or statistics, all you need to begin is good idea of the life-cycle of a ML project and a passing familiarity with these AWS services. In this talk we’ll outline a typical ML project and review services such as SageMaker and Rekognition so that you can begin to make use of them in your own projects.
This talk looks at converting an existing GCP serverless application into one build using Firebase. Firebase helps to simplify deployment, particularly around simple web hosting. The talk also looks at how easy it is to use GCP services integrated with Firebase such as authentication and Cloud Firestore.
Reducing Pager Fatigue Using a Serverless ML BotMike Fowler
Being woken up at 3 am by the pager is never fun but seeing an incident resolve before you’ve even left the bed is maddening. Sleepily the next day you tune the alert for a better night’s sleep yet more untuned alerts sing to you in your sleep. After a few rounds of alert-tuning whack-a-mole you wonder: Could I predict if an incident will resolve itself?
This is the story of how a weary engineer used a Cloud ML model with Cloud Functions to reduce pager noise. Recounting some of the challenges faced, we’ll explore training a model with a limited data set & continual training in a serverless environment. We’ll also explore the implications of using a bot as a first responder to a pager.
Have you got data in AWS but don’t know how to get started with Machine Learning? My talk will help you make sense of AWS’ offerings and show you how to use them without having to become a mathematician first. See the full talk on YouTube: https://youtu.be/3phjk1CxhXM
Debezium is a Kafka Connect plugin that performs Change Data Capture from your database into Kafka. This talk demonstrates how this can be leveraged to move your data from one database platform such as MySQL to PostgreSQL. A working example is available on GitHub (github.com/gh-mlfowler/debezium-demo).
Leveraging Automation for a Disposable InfrastructureMike Fowler
Moving from the Iron Age to the Cloud Age in computing is supposed to save us money, but many migrations seem to cost more in the long run and result in an infrastructure that is as complex to manage as the one we had before. This is often due to the so called “lift & shift” approach many take – it’s a short term win that doesn’t address why you wanted to move to the cloud in the first place.
The Cloud Age affords us the opportunity to not treat our infrastructure as something special, but as something disposable. By applying the practices of Continuous Integration and delivery to our infrastructure and configuration management, we can build truly scalable infrastructures to host our application’s wildest dreams.
In this talk, we will look at the tools and processes that can be adopted to truly make use of the possibilities of the Cloud.
Managing your own PostgreSQL servers is sometimes a burden your business does not want. In this talk we will provide an overview of some of the public cloud offerings available for hosted PostgreSQL and discuss a number of strategies for migrating your databases with a minimum of downtime.
Terraform is an open source tool that helps you control your infrastructure configuration through code. This talk will serve as a primer showing how to build a basic infrastructure in the Google Cloud and how we can re-use our code to construct multiple, identical environments.
Managing your own PostgreSQL servers is sometimes a burden your business does not want. In this talk we will provide an overview of some of the public cloud offerings available for hosted PostgreSQL and discuss a number of strategies for migrating your databases with a minimum of downtime.
Managing your own PostgreSQL servers is sometimes a burden your business does not want. In this talk we will provide an overview of some of the public cloud offerings available for hosted PostgreSQL and discuss a number of strategies for migrating your databases with a minimum of downtime.
Moving from the Iron Age to the Cloud Age in computing is supposed to save us money yet many migrations seem to cost more in the long run and result in infrastructures as complex to manage as what we had before. This is often the result of the so called “lift & shift” approach many take – it’s a short term win that doesn’t address why you wanted to move to the cloud in the first place.
The Cloud Age affords us the opportunity not to treat our infrastructure as something special, instead as something disposable. By applying the practices of continuous integration and delivery to our infrastructure and configuration management we can built truly scalable infrastructures to host our application’s wildest dreams.
In this talk we will look at the tools and processes that can be adopted to truly make use of the possibilities of the Cloud.
These days you can't go far without encountering XML or JSON and in the world of the web these data types are ubiquitous. Since version 8.3 XML has been supported as a data type and JSON support was introduced in 9.2. We'll be looking at what advantages there are in storing your data with these data types and how we can query and manipulate our data once it's stored.
IN Dubai [WHATSAPP:Only (+971588192166**)] Abortion Pills For Sale In Dubai** UAE** Mifepristone and Misoprostol Tablets Available In Dubai** UAE
CONTACT DR. SINDY Whatsapp +971588192166* We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai** Sharjah** Abudhabi** Ajman** Alain** Fujairah** Ras Al Khaimah** Umm Al Quwain** UAE** Buy cytotec in Dubai +971588192166* '''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol** Cytotec” +971588192166* ' Dr.SINDY ''BUY ABORTION PILLS MIFEGEST KIT** MISOPROSTOL** CYTOTEC PILLS IN DUBAI** ABU DHABI**UAE'' Contact me now via What's App… abortion pills in dubai Mtp-Kit Prices
abortion pills available in dubai/abortion pills for sale in dubai/abortion pills in uae/cytotec dubai/abortion pills in abu dhabi/abortion pills available in abu dhabi/abortion tablets in uae
… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all** Cytotec Abortion Pills are Available In Dubai / UAE** you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pills in Dubai** UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if it's beyond 6 months. Our Abu Dhabi** Ajman** Al Ain** Dubai** Fujairah** Ras Al Khaimah (RAK)** Sharjah** Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical** medical and surgical abortion methods for early through late second trimester** including the Abortion By Pill Procedure (RU 486** Mifeprex** Mifepristone** early options French Abortion Pill)** Tamoxifen** Methotrexate and Cytotec (Misoprostol). The Abu Dhabi** United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used** 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need for surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi** United Arab Emirates** uses the latest medications for medical abortions (RU-486** Mifeprex** Mifegyne** Mifepristone** early options French abortion pill)** Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi** United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our
Are you wondering how to migrate to the Cloud? At the ITB session, we addressed the challenge of managing multiple ColdFusion licenses and AWS EC2 instances. Discover how you can consolidate with just one EC2 instance capable of running over 50 apps using CommandBox ColdFusion. This solution supports both ColdFusion flavors and includes cb-websites, a GoLang binary for managing CommandBox websites.
Responsibilities of Fleet Managers and How TrackoBit Can Assist.pdfTrackobit
What do fleet managers do? What are their duties, responsibilities, and challenges? And what makes a fleet manager effective and successful? This blog answers all these questions.
Cultural Shifts: Embracing DevOps for Organizational TransformationMindfire Solution
Mindfire Solutions specializes in DevOps services, facilitating digital transformation through streamlined software development and operational efficiency. Their expertise enhances collaboration, accelerates delivery cycles, and ensures scalability using cloud-native technologies. Mindfire Solutions empowers businesses to innovate rapidly and maintain competitive advantage in dynamic market landscapes.
Attendance Tracking From Paper To DigitalTask Tracker
If you are having trouble deciding which time tracker tool is best for you, try "Task Tracker" app. It has numerous features, including the ability to check daily attendance sheet, and other that make team management easier.
NBFC Software: Optimize Your Non-Banking Financial CompanyNBFC Softwares
NBFC Software: Optimize Your Non-Banking Financial Company
Enhance Your Financial Services with Comprehensive NBFC Software
NBFC software provides a complete solution for non-banking financial companies, streamlining banking and accounting functions to reduce operational costs. Our software is designed to meet the diverse needs of NBFCs, including investment banks, insurance companies, and hedge funds.
Key Features of NBFC Software:
Centralized Database: Facilitates inter-branch collaboration and smooth operations with a unified platform.
Automation: Simplifies loan lifecycle management and account maintenance, ensuring efficient delivery of financial services.
Customization: Highly customizable to fit specific business needs, offering flexibility in managing various loan types such as home loans, mortgage loans, personal loans, and more.
Security: Ensures safe and secure handling of financial transactions and sensitive data.
User-Friendly Interface: Designed to be intuitive and easy to use, reducing the learning curve for employees.
Cost-Effective: Reduces the need for additional manpower by automating tasks, making it a budget-friendly solution. Benefits of NBFC Software:
Go Paperless: Transition to a fully digital operation, eliminating offline work.
Transparency: Enables managers and executives to monitor various points of the banking process easily.
Defaulter Tracking: Helps track loan defaulters, maintaining a healthy loan management system.
Increased Accessibility: Cutting-edge technology increases the accessibility and usability of NBFC operations. Request a Demo Now!
React and Next.js are complementary tools in web development. React, a JavaScript library, specializes in building user interfaces with its component-based architecture and efficient state management. Next.js extends React by providing server-side rendering, routing, and other utilities, making it ideal for building SEO-friendly, high-performance web applications.
1. Migrating Rant & Rave to PostgreSQL
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2014
Fun Things to do with Logical Decoding
Mike Fowler
PGDay UK 2015
2. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Overview
● What is Logical Decoding?
● Replications Slots
● Output Plugins
● Enabling Logical Decoding
● What does Logical Decoding make possible?
3. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
About Me
● Head of Systems Engineering at Rant & Rave
● Been using PostgreSQL for over 10 years (since 7.4)
● Contributed some XML support
– XMLEXISTS/xpath_exists()
– xml_is_well_formed()
● Contributed a number of bugfixes to the JDBC
driver
4. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
What is Logical Decoding?
“The process of extracting all persistent changes to
a database's tables into a coherent, easy to
understand format which can be interpreted
without detailed knowledge of the database's
internal state”
“Implemented by decoding the contents of the
write-ahead log, which describe changes on a
storage level, into an application-specific form such
as a stream of tuples or SQL statements”
5. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
No, really - what is Logical Decoding?
● The results of logical decoding give a perspective on the
logical changes made during a transaction
● The process of logical decoding allows human
understanding of the physical changes made during a
transaction
● Achieved by creating a replication slot with a plugin to
produce data for a receiver
– Basic example plugin available in contrib
– New utility pg_recvlogical to consume plugin output
– Can roll your own to match your requirements
6. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Some PostgreSQL Basics
● PostgreSQL is a transactional database
● All changes in PostgreSQL are recorded in a Write
Ahead Log (WAL) before any data on disk is actually
changed
● The WAL is the key to replication
– Essentially log based binary replication
– Master runs the walsender to transmit new segments
– Slaves run walreceivers to receive the segments
7. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
A Basic Example
We'll use a simple table called sample:
CREATE TABLE sample (
id SERIAL PRIMARY KEY,
data JSON,
captured TIMESTAMP
DEFAULT CURRENT_TIMESTAMP
);
8. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
A Basic Example
INSERT INTO sample (data)
VALUES
('{"beverage":"coffee","temp":96}');
● The backend takes our INSERT and calculates the
values for the omitted columns id and captured
● The backend also determines exactly what needs to
be changed on disk and commits this to the WAL
9. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Replication Slots
● 9.4 introduced replication slots
● Capability to retain WAL segments on the server
instead of relying on continuous archiving
● Allows master to retain segments even when a
slave is offline
● There are two types of replication slot
– Physical
– Logical
10. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Logical Replication Slots
● Entries in a logical slot are WAL segments that have
been decoded by means of an output plugin
● Output plugins are responsible for transforming the
data from the WAL into the format the consumer of
a logical replication slot desires
● contrib/test_decoding is a working example of the
interface and can be used to play
11. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Output Plugin Rules
● Only transactions that have been successfully
flushed to disk are decoded
● Transactions that are rolled back never get decoded
● Changes to UNLOGGED or TEMPORARY tables are
also not decoded
● Concurrent transactions are decoded in commit
order
12. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Making an Output Plugin
● Consists of building a shared library that implements a number of
functions as detailed in section 46.6 of the manual
– An initialization function
– Transaction begin, row change and transaction end callbacks
– Optional plugin start & stop callbacks
– 9.5 also adds an optional origin filter callback
● For a given transaction the begin & end is called once with the row
change called for each changed row even if multiple rows were
changed in a single statement
13. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Output Plugin Considerations
● Keep the plugin simple
– It's running as part of your database
– It's going to process all your changes
● Offload complex filter/analysis to the slot consumer
● Drop the slot if it's no longer in use to stop using
server resourcesSELECT
pg_drop_replication_slot('pgday_slot');
14. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Enabling Logical Decoding
● Enable replication slots (will require restart)
– wal_level must be set to 'logical'
– max_replication_slots must be at least 1
● Create a replication slot
pgday15=# SELECT * FROM
pg_create_logical_replication_slot('pgday_slot',
'test_decoding');
slot_name | xlog_position
------------+---------------
pgday_slot | 0/16C9A78
(1 row)
15. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Revisiting the Basic Example
We used a simple table called sample:
CREATE TABLE sample (
id SERIAL PRIMARY KEY,
data JSON,
captured TIMESTAMP
DEFAULT CURRENT_TIMESTAMP
);
16. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Revisiting the Basic Example
INSERT INTO sample (data)
VALUES ('{"beverage":"coffee","temp":96}');
● The backend took our INSERT and calculated the values for
the omitted columns id and captured
● The backend also determined exactly what needed to be
changed on disk and committed it to the WAL
● This time the test_decoding plugin was invoked when the
WAL segment entered the logical replication slot
17. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Consuming the Plugin Output
● All data from the plugin will remain in the slot until
consumed
● Data can be consumed in two ways
– SQL interface
● Useful for experimentation and demonstration
● Can be used to form a basic programmatic integration with non-C
applications (e.g. Java using JDBC)
– pg_recvlogical
● Similar to pg_receivexlog
● Can be used to output to a file for later consumption
18. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
The Output from our Example
pgday15=# SELECT * FROM
pg_logical_slot_get_changes('pgday_slot', NULL, NULL);
location | xid | data
-----------+-----+----------------------------------
0/16C9ED0 | 723 | BEGIN 723
0/16C9ED0 | 723 | table public.sample: INSERT:
id[integer]:1
data[json]:'{"beverage":"coffee","temp":96}'
captured[timestamp without tz]:'2015-07-03 19:52:39.497161'
0/16CA000 | 723 | COMMIT 723
(3 rows)
19. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
pg_recvlogical
● Use pg_recvlogical to record to a single file
– -f option to specify output file
– -F to specify frequency of fsync in seconds
● Run pg_recvlogical on a separate server
– -h to specify hostname or ip
– -p port
● Can also create/delete slots
20. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Some simple LD Use Cases
● Trigger-less auditing
– Keep a record of all changes in transaction order
– Change only query log
● Trigger auditing
● Basic throughput analysis
21. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Limitations of test_decoding
● Only details the new/changed row
● No detail of a WHERE clause
● No timestamps in output to help cross reference
● Changed indexes don't make for easy reading
table public.sample: UPDATE: old-key: id[integer]:3 new-tuple: id[integer]:4
data[json]:'{"beverage":"coffee","temp":96}' captured[timestamp without time
zone]:'2015-07-07 00:07:56.305964'
22. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Enhancing test_decoding
● Plugins can use most of backend infrastructure
– Can use output functions
– Read only access to relations
– Can not use anything that requires a transaction
● Information logged to WAL can be tuned with
REPLICA IDENTITY
– Can be adjusted per table with ALTER TABLE
– DEFAULT keeps old value of PRIMARY KEY
– USING INDEX will keep values of columns covered
23. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Example: decoder_raw
● Enhanced version of test_decoding by Michael Paquier
● Appends WHERE clauses based on data derived from
REPLICA IDENTITY
– If DEFAULT, PRIMARY KEY column(s) appear in WHERE
● Removes the commit number from BEGIN and COMMIT
● Available on Git Hub
https://github.com/michaelpq/pg_plugins/tree/master/decoder_r
aw
24. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
decoder_raw Output
=# SELECT slot_name
FROM
pg_create_logical_replication_slot('custom_slot',
'decoder_raw');
slot_name
--------------
custom_slot
(1 row)
=# CREATE TABLE aa (a int primary key, b text);
CREATE TABLE
=# INSERT INTO aa VALUES (1, 'aa'), (2, 'bb');
INSERT 0 2
=# UPDATE aa SET b = 'cc' WHERE a = 1;
UPDATE 1
=# DELETE FROM aa WHERE a = 1;
DELETE 1
25. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
decoder_raw Output
=# SELECT data
FROM pg_logical_slot_peek_changes('custom_slot',
NULL, NULL, 'include-transaction', 'on');
data
----------------------------------------------------
BEGIN;
COMMIT;
BEGIN;
INSERT INTO public.aa (a, b) VALUES (1, 'aa');
INSERT INTO public.aa (a, b) VALUES (2, 'bb');
COMMIT;
BEGIN;
UPDATE public.aa SET a = 1, b = 'cc' WHERE a = 1 ;
COMMIT;
BEGIN;
DELETE FROM public.aa WHERE a = 1 ;
COMMIT;
(12 rows)
26. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Partial Replication
● Can be achieved in two ways
– Plugin can filter out unneeded tables
– Slot consumer can discard unneeded tables
● Useful if you have write notes independent of a
cluster
● Streaming partial replication is possible
– If output of plugin is well structured pg_recvlogical
could be piped straight into a psql client
27. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Full Statement Replication
● Changes can be streamed to other DBMS platforms
– Allows replication to non-PostgreSQL servers
– Useful for keeping a backup during a migration to
PostgreSQL
● Differs from most versions of statement replication
– The master computes the statement and determines the
changes
– Values of columns have already been computed so will be
identical (e.g. CURRENT_TIMESTAMP)
28. Fun Things to do with Logical Decoding
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2015
Conclusion
● Logical Decoding exposes meta-data about
committed transactions
● Plugins can be written to enrich and format the
meta-data for placement in replication slots
● Data in replication slots can be consumed by
external programs using SQL or
pg_recvlogical to perform a number of useful
tasks
29. Migrating Rant & Rave to PostgreSQL
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2014
Questions?
Mike Fowler
PGDay UK 2015
30. Migrating Rant & Rave to PostgreSQL
Mike Fowler, mike@mlfowler.com mike.fowler@rantandrave.com
PGDayUK 2014
Thank you!
Mike Fowler
PGDay UK 2015