Trends and issues impacting database management systems circa 2004 included increasing complexity, lack of resources, and rapid changes in technology. New database management system versions were being released frequently with new features enabled for the internet and real-time usage. Emerging technologies like Java, .NET, and XML were becoming more widely adopted and database systems were taking on additional functionality beyond traditional querying and storage. The internet was driving changes requiring database administrators to have new skills to support increasingly complex enterprise infrastructure and applications.
Report
Share
Report
Share
1 of 65
More Related Content
Db trends final
1. Trends and Issues Impacting
Database Management Systems
Circa 2004
Craig S. Mullins
Director, Technology Planning/BMC Software, Inc.
http://www.craigsmullins.com
2. General Industry Trends
• Complexity
• Heterogeneity
• Speed-to-Market
• Rapid change • Buy versus Build
• Consolidation • Follow the Leader
• Lack of Resources
• Skilled Technicians
– A lot of PFCSKs and IROCs
• Time
– Already overworked
• Budget
3. DBMS Industry Trends
• Rapid DBMS Versioning
• Enabling for the Internet
• Online, real-time …
• Java and .Net
• XML
• Multimedia
• Procedural logic
• ERP and CRM
• Mo’ data! Mo’ data!
• Integration
4. New DBMS Versions
• Analysis of New Features
– Check all Requirements
• Hardware and Software
• Planning the Upgrade
– Impact to system, applications
– Scheduling
• Fallback Strategy
• Migration Verification
10. Internet Infrastructure Weaknesses
Problem Symptom Effect
Sporadic crashes Unplanned
Unreliable for no apparent outages
reason
Operators do not Simple problems
Complex
understand how to result in long
resolve problems outages
Fragile IT mgrs. Must Long debugging
debug innocuous cycles for new
changes releases
Systems must be
Vulnerable Viruses and bugs rolled back to
attack all systems clean backups
at once
Source: Forrester Research
12. E-Business Applications Fail to
Deliver Service Because...
Unplanned outages - Unplanned Outages
Planned and Planned outages -
driven by problems hr. workdays
x Staff Shortages - 18 driven by change
x Short Implementation Times x Management
x Application failure
x Executive Demands for Web Presence
x Maintenance
x
x Element Failure Scalability
Management Tool x Migrations
x Performance Keep Up With Own Scripts
x Unable to
x
x Capacity limitsWeb Load/Growth Version
Unpredictable
x
x Transaction Success of Site Management
x Unknown
Backouts x Propagation
x Customized Environments
x Need Flexibility of Management
13. Where is Application Downtime?
30% of
• Planned vs. Unplanned Outages
– Planned outages represents 70% of
Outages
70% of application
downtime.
– Just 30% is due to
unplanned outages and
50% of the unplanned
downtime is due to
problems during planned
downtime.
14. Impact on eDBAs
• Downtime is not tolerated
– Downtime was never “good” but it was tolerated
– No longer, though as we move from availability to
e-vailability with intelligent techniques & solutions
• Avoiding downtime with automated tools
– “On the fly” operational tuning
• ALTER SYSTEM - Oracle9i
• SET SYSPARM - DB2 V7
– Monitor performance across multiple platforms
– Redundant systems – HACMP, RAID, etc.
15. Example: Keeping Your Systems Up!
Monitor
SYSPLEX
ARCHIVE FAILED?
Action DB2A ADD LOG
NOTIFY AML
Change ZPARM DB2B
Monitor DB2C
Action
EDM POOL FAILED? LOGS
LOGS
LOGS + LOG
16. Example: Recover with no Down Time!
LOGS
LOGS
LOGS
Undo/Redo SQL
Log
Analysis
DB2A
DB2B
No need to
take the data Data
off-line to run
DB2C
SQL against it!
17. Online and Real-time
• The need for more and more availability drives
online and real-time maintenance
– The DBMS begins to allow for more changes to be
made during normal operations
– The DBMS begins to gather statistics and performance
metrics during normal operations
– ISVs deliver more online, real-time features and
functionality that the DBMS does not yet deliver
– Less manual-intervention required
18. Database Design and Web Time
• When the Web is involved everything becomes
“rush-rush” - do it now!
• Don’t let database design suffer - take your time
and do it right.
• Apps are temporary but data is forever!
– If you do not believe this, then consider: “How
often has your organization re-entered or re-
keyed data into a new database when the data
already exists elsewhere?”
19. Prepare for Global Scope
Who is accessing the
database?
Internal and External users
Local, National, and
International
Now versus then
24 x 7 x 365¼
20. Impact on the DBA
Where is the performance problem?
Most experts agree that 75% to 80% of
performance problems in relational
applications is caused by poor SQL or
gateway HTML
application code, but on the web . . .
ISP
XML init.ora ZPARMs
connection CGI
3GL bridge/router/hub
DB2 Java operating system
Connect DNS
ASP hardware
HTTP
network cabling
Java application code
SQL
SQL*Net applet
database schema
network software
22. Why Java?
• Portability
• Make web pages active and dynamic
• Like C/C++ with a smaller footprint
• Lower cost of change (DLLs)
• Download changes from the web
• But
– slower - interpreted, not compiled
23. Java and Databases:
Two Methods
• JDBC
– Enables Dynamic SQL from Java
– Uses API (CLI)
• SQLJ
– Enables Static SQL for Java
– Uses embedded SQL
24. Java Alphabet Soup
• J2EE - Java 2 Enterprise Edition
– Standard services and specifications for making
Java highly available, secure, reliable, and
scalable for enterprise adoption
• EJB - Enterprise Java Beans
– Components that contain the business logic for
a J2EE application
25. Impact of Java on DBA
• Application tuning
– Must understand Java
• To provide guidance during design reviews
– Is the problem in the SQL or the application
• How can you tune the application if you do not
understand the language (Java)?
– Optimizing SQL is not enough since it may be embedded
in poor application code
– Must understand the SQL techniques used
• JDBC and SQLJ
26. Microsoft .NET
• ... is a set of Microsoft technologies for
connecting people, systems, and devices
• ... allows Internet Servers to expose
functions to any client named as .NET web
services
• … enables software to be delivered as a
service over the web
• … is designed to let many different services
and systems interact
27. Java versus .Net
• ...designed to enable • …designed to enable
applications to be development in
deployed on any multiple languages as
platform as long as long as the application
they are written in is deployed on
Java Windows
29. What is XML?
• XML stands for eXtensible Markup Language.
– Like HTML, XML is based on SGML
(Standard Generalized Markup Language)
– HTML uses tags to describe the appearance of data on a page,
whereas XML uses tags to describe the data itself, instead of its
appearance.
– Allows documents to be self-describing, through the specification
of tag sets and the structural relationships between the tags.
30. XML is a Meta Language
• XML is actually a meta language - a language used to
define other languages.
– These languages are collected in dictionaries called Document
Type Definitions (DTDs).
– The DTD stores definitions of tags for specific industries or fields
of knowledge. So, the meaning of a tag must be defined in a DTD
before it can be used.
– The DTD for an XML document can be either part of the
document or stored in an external file.
31. Sample XML DTD
<!DOCTYPE CUSTOMER [
<!ELEMENT CUST (first_name, middle_initial, last_name,
company_name, street_address, city,
state, zip_code, country*)>
<!ELEMENT first_name (#PCDATA)>
<!ELEMENT middle_initial (#PCDATA)>
<!ELEMENT last_name (#PCDATA)>
<!ELEMENT company_name (#PCDATA)>
<!ELEMENT street_address (#PCDATA)>
<!ELEMENT city (#PCDATA)>
<!ELEMENT state (#PCDATA)>
<!ELEMENT zip_code (#PCDATA)>
<!ELEMENT country (#PCDATA)>
]
33. Why is XML Important?
• XML is used for exchanging and sharing data
– Inter- and intra-organization
• XML may be used to define database structures
– and vice versa
DOCTYPE
XML definition
e DDL
lorem ipsum . . . Generat Physical
Database
e XML
Generat
34. XML to DBMS
s Add XML functionality to the DBMS
• Extender capabilities similar to IBM DB2 video, image, audio,
and other multimedia data types
• Combine UDT, UDFs, and triggers for functionality
s XML document stored in a column -or-
s XML components stored as parts of multiple columns
in multiple tables
s Formulate XML documents from existing tables
s Search XML documents text and sections
s XQuery capabilities
35. XML and Data Management
• Middleware: Software called from your • Wrappers: Software that treats XML
application to transfer data between XML documents as a source of relational data.
documents and databases. For data- These products typically query XML
centric applications. documents using SQL. For data-centric
• XML-Enabled Databases: Databases applications.
with extensions for transferring data • Content Management Systems:
between XML documents and Applications built on top of native XML
themselves. Primarily for data-centric databases and/or the file system for
applications. content/document management. For
• Native XML Databases: Databases that document-centric applications.
store XML in "native" form. For data- and • XML Query Engines: Standalone engines
document-centric applications. that can query XML documents. For
• XML Servers: XML-aware J2EE servers, data- and document-centric applications.
Web application servers, integration • XML Data Binding: Products that can
engines, and custom servers. For data- bind XML documents to objects. Some of
and document-centric applications. these can also store/retrieve objects from
the database. For data-centric
applications.
Source: XML Database Products by Ronald Bourret
http://www.rpbourret.com/xml/XMLDatabaseProds.htm
36. Impact of XML on DBA
• Database definition
– Perhaps a new way to create databases
• DTDs to build a database schema
• Create DTD from a database schema
• Interface to data modeling tools
• Data access
– Database extenders
– XML defines the data it contains
• New XML DBMS products?
– Tamino (Software AG)
– Xperanto (IBM)
• Over-enthusiasm!
38. Triggers vs. Functions vs. Procs
U
P I
N D
D Code
Code
Code
S E
A
E L
T Triggers
R E
E
T T
UDFs
E function( )
if this then that
else
do this stuff
return x
Stored end
Procedures
→ Code
Code
Code SQL
39. Procedural DBA Duties
Stored
Procedures Performance
Monitors
D
E D
V E
Admin. DB2
E Triggers B Process
L U
O G
P External
Design Libraries
Review
Functions
40. Role of the Procedural DBA
DBCO Impl
(COMMIT in
em entation Ensuring
proc, write o
r guide) Reuse
n
m stratio
ini set) Des
i
CO Ad rder, proc
DB r firing o Rev gn
igge
iews
(tr EX
An PLAI
On Call a ly
sis
N Coding
for DBCO Complex
Abends Queries
QL
i n gS Debuggi
ng
Schema un SQL
T
R esolution
DBCO = Database Code Object
42. “Universal” Data
Complex
Compound Docs
Graphics
Video
Design Data
Spatial Data
Images
Temporal Data
Text
Unstructured Structured
Audio
Existing
Databases
Seismic Data
Simple
Source: Gartner Group
43. Multimedia = Big Databases
Object Typical Size
HD TV 200 MB/second
Feature-length,
high-resolution movie 5-6 GB
High-resolution video 3 GB/hour
Feature-length movie 2 GB
Video 1 GB/hour
Radiologic image 40-60 MB
Color image 20-40 MB
Large image 200 KB-3 MB
Text 30-40 KB/page
Check image 45 KB
44. Integration and Federation
• DBMSs are adapting to “handle” more
types of non-traditional data
– Spreadsheets
– Word documents
– Presentations
• How?
– Integrate the data into the DBMS
– Federate and manage the data “where it lies”
45. DBMS to Manage
All Kinds of Data
Federated DBMS
• A federated
approach allows
the DBMS to
manage data
where it exists
Spreadsheet
46. DBMS to Manage
All Kinds of Data
Or integration…
• An alternate
approach “sucks”
the non-relational
data into the
DBMS to be
managed
Spreadsheet
47. Autonomic Computing
Real Time Self
Statistics Managing
Intelligent
Automation
Virtual
Reorg
Database Correct and
Wizards Notify
49. Phenomenal Data Growth
• “Global 2000 companies double the amount of data they
own every year, while the average dot-com’s data doubles
every 90 days.”
– Mike Ruettgers, CEO of EMC Corp., Oracle Open World 2001
• “Inside IBM we talk about 10 times more connected
people, 100 time more network speed, a 1000 times more
devices and a million times more data.”
– Lou Gerstner, CEO of IBM Corp., eBusiness Expo 2000
• A recent research note from Giga Information Group
estimates that there are about 201,000TB, or about 197
petabytes, on the planet. Of course, this is just an estimate that
Giga deems to be accurate within an order of magnitude (that is,
within a factor of 10).
• McKinsey & Company reports a CAGR of 76% for data
storage.
51. Database Size Issues
• Technology enables larger databases
• Web, multimedia, data warehousing, and data mining
drive up database size
• Disk drives increase in capacity but
speed of access does not keep up with
capacity increases
• Cost of storage decreasing; so
why not store more data? But...
– What data do users need to store?
– How long must it be maintained?
– What are they willing
to pay? 0010
001010100
101011101011
101011010010101
01001001101010111000011
11001010100100101000100101
52. The Database Environment
• A lot of choices!
– Vendor, platform, and architecture of DBMS
MVS, OS/390, z/OS Enterprise
Windows NT / 2000 / XP - Parallel Edition
Unix Departmental
AIX Personal
Sun Solaris
Mobile (PDA)
HP-UX
Linux
others?
Others (VSE, VMS, MPE, OS/400, etc.)
Desktop OS
Windows 98 / ME / XP
Linux
Mac?
Adabas, Teradata, PostgreSQL, Supra, Compaq Non-Stop SQL, Ingres, IMS, IDMS, Datacom, Teradata, others...
53. Heterogeneity Plus!
• The DBA’s
knowledge must span
the entire enterprise
– Operating Systems
– Networking Protocols
– Programming
Languages
– Business Objectives
– And so on…
54. Impact on the DBA
• Unrealistic Expectations
– Impossible to master everything
• Education is the first thing cut!
– Impossible to specialize in a heterogeneous shop
• Reactive mode is encouraged
– Even though proactive mode is optimal
– Who looks for more problems when they don’t have enough time
in the day to solve the problems of those complaining the loudest.
– DBA uses YBWJ method
55. Worldwide Spending for ERP
Packaged Software
$30
$25
$20
Manufacturing
Distribution
$15
Human Resources
Financial
$10
$5
$0
1995 1996 1997 1998 1999 2000 2001 2002
Source: IDC
56. The Computing “Platform” Evolution
This is the current
“Customer Service Apps” battleground,
NOT the DBMS
2000+
“Base” Applications SAP, Oracle, Baan, PeopleSoft
The DBMS DB2, Oracle, SQL Server
Network (LAN/WAN) TCP/IP, Novell, WinNT
Communications SNA, TCP/IP, NFS
The Operating System MVS, UNIX, WIN (NT,95)
Proprietary (IBM+) to Open
60’s Hardware (Wintel, HP/Intel)
57. The Data Continuum
o n
pti
Perce
ence
eri
Wisdom
p
Ex Knowledge
Integration
ext › Multiple points
Cont Information
Integration of knowledge
› Systems Understanding
› People › Knowledge
Business Systems applied
Data › Data in › Technology to
Context store and report
People
Simple
› Numbers › Identify
› Characters › Capture
› Date › Maintain
› Time
Complex
› BLOBs
› CLOBs
Varied
› Content
Management
58. The Data Continuum
Wisdom
Knowledge
Knowledge +
Application
Information Technology + Systems
People
Data Models
Knowledge
Data Data Dictionary
Management
Repository
DBMS
Files
Spreadsheets
Graphics
Folders
Etc.
59. The Data Continuum
o n
pti
An Example: rce
Pe
ence
eri
Wisdom
Exp Knowledge
ext
Cont Information
Data
60. Other DBMS Market Trends
• ODBMS – no longer a threat
to become “mainstream” but…
• “Post-Relational” DBMS?
– XML DBMS
– The Associative Model
– Multivalue DBMS
• In Memory DBMS
61. Enter the DBA
• The job of database
administration is getting
increasingly more difficult
as database technology
rapidly advances adding
new functionality, more
options, and more
complex and comp-
licated capabilities...
62. The DBA is a “Jack of all Trades”
OS/390 C++ Windows V$ Tables
Linux
Oracle SQL XML Java
DB2 Informix applet
Unix MQ
Connect DB2
DNS
gateway HTML
VTAM CGI TCP/IP
ISP
connection ZPARMs
3GL database schema
COBOL
ASP operating system
Java
HTTP hardware
SQL Server network software
bridge/router/hub application code
VB network cabling
CICS SQL*Net
JCL
63. So What is Needed?
• Intelligent automation of DBA tasks
– because no one has all the skilled resources
they need
– frees up more DBA time
They work hard so you
don’t have tooooo…
“Scrubbing Bubbles”
Intelligent Automated
64. A DBA Control Panel
DBA-focused on DBA tasks, not focused on systems management
tasks like framework products… but integrated with systems
management functionality where it makes sense.
As we discussed, the infrastructure that may be needed to provide a specific service or create a new business model can be very complex. This complexity can be evident both physically and logically. In addition, this infrastructure may change rapidly as the business changes. The first steps in managing a critical business service surround identifying the specific infrastructure that the service needs to run. Essentially, we need to map out the path that the business transactions travel along to better understand how to manage the service. (next)
Depending on what business model is being adopted, a company may be required to integrate into a “virtual” enterprise to deliver specific services or business models. For example: If an organization has acquired an EProcurement application such as Ariba, the “service” will extend into more than just the “in-house” enterprise. You would want to manage the supplier’s system (as it relates to the transaction). You would want to have information from the Shipping company, etc.. No longer can we only think in terms of the “enterprise” we must always look further to determine if the service in question requires integration to a virtual enterprise and determine where management makes sense. Now, we will present the BMC Software approach to Systems Management in this new Era. (next)
Internet commerce services fail many times because there are weak infrastructures supporting them. 1.) Unreliability - The best of the largest retail e-commerce sites achieve 99.4% availability (Keynote Systems, benchmarker). The worst sites achieve only 86.7% availability and costs retailers millions of dollars in revenues. 2.) Complexity - If something crashes, it can take IT personnel hours to figure out where the problem is - routers, servers, operating systems, middleware, etc. 3.) Fragile - Simple upgrades and bug fixes can leave systems in limbo for weeks while IT personnel figure our which configurations won't crash every few hours. 4.) Vulnerable - Infrastructure today still has weaknesses in which a single event can bring down an entire system. For example, standards on single machine types, like NT, leave whole web sites vulnerable to viruses.
Unplanned outages - unexpected problems. End up having to recover the environment: the longer the recovery, the longer the unplanned outage becomes. Point to customer : Industry is focused on unplanned outages on open systems and trying to minimize number of surprises in the environment. Most companies say that more outage time is spent on planned outages than on unplanned outages. Planned outages are driven by change in the environment. This is becoming a much larger issue for customers. If you could reduce the amount of time on the planned outages, you can have a significant impact on the amount of downtime. BMC is the only vendor in industry that focuses on both planned and unplanned outages, and can reduce the overall impact of outages from an application perspective for a customer.
Unplanned outages - driven by problems Element failure Performance degradation Capacity limitation Application logic error Transaction backout Data corruption Planned outages - driven by change Database maintenance Application migrations Configuration upgrade Data propagation Source of 70% and 30% is CIO feed back during Customer briefings and meetings.
There are two methods of accessing your DB2 databases using Java: JDBC SQLJ
XML is getting a lot of publicity these days. If you believe everything you read, then XML is going to solve all of our interoperability problems, completely replace SQL, and possibly even deliver world peace. Okay, that last one is an exaggeration, but you get the point. In actuality, XML stands for eXtensible Markup Language. The need for extensibility, structure, and validation is the basis for the evolution of the web towards XML. XML, like HTML, is based upon SGML (Standard Generalized Markup Language) which allows documents to be self-describing, through the specification of tag sets and the structural relationships between the tags. HTML is a small, specifically defined set of tags and attributes, enabling users to bypass the self-describing aspect for a document. XML, on the other hand, retains the key SGML advantage of self-description, while avoiding the complexity of full-blown SGML.
XML provides a standard for data interchange. By incorporating XML into DB2 you can more directly and quickly access the XML documents. With DB2 you can search and store entire XML documents using SQL. You also have the option of combining XML documents with traditional data stored in relational tables. When you store or compose a document you can invoke DBMS functions to trigger an event to automate the interchange of data between applications.
However, there are some problems with XML. For example, standard web browsers do not currently understand the descriptive tags. This problem will be alleviated in time as XML-capable web browsers come to market. Another problem with XML is not really the fault of XML, but of market hype. There is a lot of confusion surrounding XML in the industry. Some folks believe that XML will provide metadata where none currently exists or that XML will replace SQL as a data access method for relational data. Neither of these assertions are true. There is no way that any technology, XML included, can conjure up information that does not exist. Humans must create the metadata tags in XML for the data to be described. XML enables self-describing documents. It does not describe your data for you. And XML does not do what SQL does. Hence, XML cannot replace SQL. SQL is the standard access method for relational data. It is used to “tell” a relational DBMS what data is to be retrieved. XML is a document description language. It describes the contents of data. XML may be useful for defining databases, but not for accessing them.