SlideShare a Scribd company logo

Dynamics CRM high volume systems - lessons from the field

•Download as PPTX, PDF•
•
S
Stéphane Dorrekens

Three field stories from companies describe their experiences with high volume CRM implementations: a financial institution with 8,000 users and 350GB of data across two implementations; a financial institution with 2,000 users, 2,500GB of data across two implementations; and a financial institution with 1,000 users and over 450GB of data across six implementations, with 50GB added per month for the largest one. The document discusses lessons learned from these implementations regarding infrastructure design, functional design, and performance testing to support high volume systems.

Related slideshows

Optimizing Alert Monitoring with Oracle Enterprise Manager
Optimizing Alert Monitoring with Oracle Enterprise ManagerOptimizing Alert Monitoring with Oracle Enterprise Manager
Optimizing Alert Monitoring with Oracle Enterprise Manager
 
Tuning OEM Templates
Tuning OEM Templates Tuning OEM Templates
Tuning OEM Templates
 
Mastering SAP Monitoring - Workload Monitoring
Mastering SAP Monitoring - Workload MonitoringMastering SAP Monitoring - Workload Monitoring
Mastering SAP Monitoring - Workload Monitoring
 
1 of 40
High Volume Systems Lessons from the Field Stéphane Dorrekens, RealDolmen Philip Verlinden, RealDolmen
Session Objectives And Takeaways •Session Objectives: • Demystify High Volumes • Stories from the Field ! •Key Takeaways • Key attention points for high volumes • Draft design considerations
Field Stories • Financial Institution (8.000 users, 2 implementations, 350Gb) • Financial Institution (2.000 users; 2 implementations; 2500Gb) • Financial Institution (1.000 users, 6 implementations, 450Gb+50Gb/Month for biggest implementation ) • Public Transport (500 users, 1 implementation, ) • Pharmaceutical (400 users , 2 implementations, 300Gb)
What is High Volume ?
What is High Volume ? • Typical inflexion points (where system behavior is impacted) • High Volume Data • >1 Mio for base record tables (ie: Contact, Account,..) • >5000 activities/day or >1Mio/year • High Volume Transactions • High Volume Users/Security • >300 concurrent users • >1.000 teams, BU • High Volume vs Enterprise Scale • Small/Medium companies can have huge amounts of data (ex: associations/volunteers) • Small/Medium companies can have huge amounts of transactions/sec (ex: web interactions) • Only Enterprise has huge number of users and complex security • CRM Online vs On Premise • Data Cost & Max GB (<1Tb)
Impact of High Volume •The Bad News Where’s the high-volume switch? High-Volume setting (on/off): •The Good News It does work … with some effort…
Impact of High Volume • Less Room for “Improvisation” or “Lack of Design” • Every point not taken care of or thought over will be an issue • Some out of the box features are not designed for high volume • Charts limits of 50.000 records • Team membership > 1.000 teams • No out of the box archiving • Reporting and Transactions share the same Database • Queries are locking per default • … • Don’t load millions of Contacts into the system and expect it to just work.
Trending • Omni –channel + More Social + More Customer Interactions Challenge the customer on possible: - Data Security Walls <> Performance • Customer Centric • Customer expectations • CRM & CEM
Infrastructure Design
Infrastructure Design High Volume of Data • SQL Server is the Key point • Get the numbers ! (Size, Growth, Repartition of data between tables/entities ) • Data Layer IOPS check (especially SAN); Disk is more important than RAM at that level as the Disk/RAM ratio is lower than usual. • RAM (Absolute Minimum 10%; Recommended Minimum 30%) • SQL 2014 Std maximum is 128Gb (64 for 2012), SQL Enterprise maximum is 4Tb • Index but not over the top • Disks Configuration • Tempdb usage/location (Considers SSD) • Consider using a separate reporting database • set the MaxServerMemory setting in SQL SQL to reserve 4/6 Gb of system memory for the OS and other applications • Not to DO • Use non dedicated LUNs • Use an old SAN infrastructure (IOPS check) • Forget to reserve failover host resources • Forget Testing Maintenance (see last section) MDF SQL Server 2014 1x4 Cores 250 GB RAM LDF LUN A LUN B 1024 Gb
Infrastructure Design High Volume of Data • … but SQL is not the only point impacted • Data Upload • Front-end server sizing (even temporary) • Bandwidth • a shared 1Gb/s can be overloaded • Deployment • Backup/Restore to another environment can takes literally days • Virtualization • Specific issues : Host reservation (RAM), Dedicated LUNs, .. • Not to Do • Upload 50 Mio Records on a massive web server farm and not warning the network team. • Hint: maybe they have automatic Denial of Service protection?
Infrastructure Design High Volume of Users • Performance vs Stability • Minimum of a pair front-end with LB • LB is not enough, rapid failover is needed • App pools • Limits • Recycle • Separate roles • Separate back-end servers • Separate endpoints • To do • Test Failover by hanging an app pool, don’t expect LB to work OOTB • Reboot Front-End Daily Front-End Server 1x2 Cores 8 GB RAM Organization Web Service Role 1x2 Cores 8 GB RAM Front-End Server 1x2 Cores 8 GB RAM Organization Web Service Role 1x2 Cores 8 GB RAM CRM.Contoso.local CRMServices.Contoso.local Back End Role 1x4 Cores 8 GB RAM
Functional Design
Functional Design • 80/20 rule • OOTB versus Code • Where to invest and where not ? • All Recommendations which follows are interlinked, example : • Deleting workflows history is good but faster if no POA record exists
Normalise but… • Normalise your datamodel • Map it on the CRM datamodel as much as possible • Get and map the numbers • And reflect… • Security Design (CRM) • Volumes • B2C in Germany (+80 Mio or Belgium +11 Mio) • Functional (MUI) • Online-proof: FetchXML <> FilteredViews
Security, from performant to less performant… from standard to exception • Business Unit / Roles • Access Teams • Hierarchical Security • Team Ownership • Sharing (User/Team) Field story on Teams numbers • Non-Scalable: Strictly limit the number of Owner teams a user is a member of • Scalable: Access Teams but think numbers and maintenance Field story on Sharing: keep the POA under control • Disable the “Share record with the previous owner” • Parental Relationships to Referential or remove ‘Reparent’ cascade • Workflows ownership: Async Service user • Keep monitoring the POA Size
Views 1/2 • Only show the minimum of records needed for the feature • Uses preemptive filtering instead of after the fact filtering (user based) • The rationale is that the Top X records are shown and ordered by AFTER the security model is applied. This combination forbids use of a non-clustered index and we cannot change the clustered index (sequential GUIDs) • Thus the less records you have to sort; the fastest • Specify a period if possible (ie: Last Year, Last 6 Months) • This avoids degradation overtime and limits #Records to sort • Remove sort order if not needed (via SDK, update FetchXML ) • Uses the database order (sequential GUID) if sort is not needed • The rationale is to remove the order by impact • Don’t use the ‘ALL’ views as default • Make the default the most restrictive view needed (ie: ‘My’ or ‘My in that period’)
Views 2/2 • Don’t use a specific view for search reasons, search is ALWAYS global. • If only search is needed, use a view returning no records • The optimal is to search for a non existing record; non existing value on an indexed field • If you want to combine search and a default list; use the most restrictive list needed. • Explain to the users the concept of global search • Find Fields • Use the most restrictive list of ‘find fields’ as ALL those fields will be searched • Those principles are not only valid for ‘standard views’, they should be applied on: • Dashboard views • Lookups views • Search Result View
QuickFind Filtering • How to search in a view: • Simple: OOTB dashboard List • Harder? Field Story on Quicfkfind with 10 Mio contacts? • Filter with • Datamodel • User preference (custom entitiy: POS or global) • PreRetrieveMultiple add “where”-clause • Button to change scope
Plug-ins • Limit the Plug-ins registered per Events Only one plug-in execute should fire for a given event; thus the plug- in class naming convention should be : xx.Plugins.<Entity>.<Message>.<Type>. xx.Plugins.Account.Create.Sync Only one plug-in dll should exists for a given entity: xx.Plugins.Account This also allows sharing of information between the different steps of the same pipeline (ie: SharedVariables) • Make sure to define the filtering attributes for Plug-ins. Because CRM uses now the auto save every 30 seconds, they will be triggered frequently. • Implement Tracing Some issues only occurs under heavy stress (ie: bad instantiations); good tracing will helps you tremendously Use FetchXML aggregation when possible (especially count) • Do not use any kind of static instantiations, it will fail under heavy load
The sandbox tax 1st Test with Registration:Sandbox [2015-03-30 20:14:34.858] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 8 > OrganizationSdkService starts processing request for user:0d57f89f-bf5e-4e4c-b4ab-8ad6d9b2e039 Request Xml:Update<UpdateRequest [2015-03-30 20:14:34.987] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 8 |Category: Sandbox |User: 0d57f89f-bf5e-4e4c-b4ab-8ad6d9b2e039 |Level: Info |ReqId: 631e39a9-8527-4397-8fe1-275f1f9e1102 | SandboxCodeUnit.Execute ilOffset = 0x394 > SandboxPlugin.Execute: exit [2015-03-30 20:14:34.987] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 8 |Category: Platform.Sdk |User: 0d57f89f-bf5e-4e4c-b4ab-8ad6d9b2e039 |Level: Verbose |ReqId: 631e39a9-8527-4397-> MessageProcessor start executing step 58e202dd-4b49-417a-b732- f95957ef99b0 of type 'Microsoft.Crm.ObjectModel.MultiCurrencyPlugin' synchronously for message 'Update' for entity 'contact' 129msec to complete the call 2nd Test with Registration:None [2015-03-30 20:20:44.900] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 33 > OrganizationSdkService starts processing request for user:0d57f89f-bf5e-4e4c-b4ab- 8ad6d9b2e039 As user:5c0e5ad5-8cd6-e311-93f9-00155d1a7308 Request Xml:Update<UpdateRequest [2015-03-30 20:20:44.963] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 33 |Category: Platform.Sdk |User: 0d57f89f-bf5e- > MessageProcessor start executing step 58e202dd-4b49-417a-b732-f95957ef99b0 of type 'Microsoft.Crm.ObjectModel.MultiCurrencyPlugin' synchronously for message 'Update' for entity 'contact' 63 msec to complete the call
The sandbox issue Update Event Pipeline Retrieve . Pre Stage PreEvent Plugin Sandbox Sandbox PostEvent Plugin Retrieve Event Pipeline Pre Stage Platform Post Stage 30 msec Sandbox 30 msec 30 msec 30 msec
Async Workflows • Use with care • Strictly limit the number of workflows registered per event, to avoid race conditions and system overload. • Purge completed workflows as much as possible, but beware of the “delete tax”
Audit • Restrict Scope • Do not audit when doing the initial import • Do not audit read only data (especially, if having a daily or more update frequency) • Do not audit unneeded fields • Manage Growth • Delete Audit information is only by date, not by type • Choose SQL Server version wisely • SQL Enterprise audit uses Quarterly Partitions, SQL Standard doesn't • Allows splitting the audit logs into different filegroups for better I/O • Upgrade from Standard to Enterprise will not create partitions! • Audit is always async so triggers high usage of back-end
Locks & Deadlocks • The SQL Server Default is locking* • By default, define the queries with the no-lock hint • no-lock='true‘ in FetchXML • QueryExpression.NoLock • With (Nolock) in T-SQL • That applies everywhere (ie: Views, Charts, Reports, Plugins, JS, etc..) • Minimize lock contention when doing updates (ie: cascade restrictions) • Export Excel is (currently) locking so give the right with care. A good ref.: Microsoft Dynamics CRM 2015 Performance and Scalability * You can change the SQL Server default (like to Snapshot isolation) but it’s not the recommended best practice and triggers its own issues to manage (ie: Tempdb usage in Snapshot isolation)
Integration & GUIDs • Don't let external systems generate CRM GUIDs, as they won't be sequentials, even for Read-Only, back end originated data (cfr http://blogs.msdn.com/b/crminthefield/archive/2015/01/19/the-dangers- of-guid-newguid.aspx) • Using a sequential .NET generated GUID (cfr http://blogs.msdn.com/b/dbrowne/archive/2012/07/03/how-to-generate- sequential-guids-for-sql-server-in-net.aspx), is dangerous as you need to be sure only one system/integration generates those GUIDs; which is implicit if generated by the Database Server.
Integration & Performance • Web Services are not the bottleneck if handled properly • Data Injection needs to be massively multi-threaded (Bulk API can be helpful as well) and scaled out • Front-End processing need to be massively scaled out; the bottleneck being the IIS CPU Usage (+/- 50 ops/sec/core) • Disable everything unnecessary (not only for initial load): Audit, WF, Plugins, etc.. • The true bottleneck is Network bandwidth and IOPS Tip : When you use the Update message, do not set the OwnerId attribute on a record unless the owner has actually changed
Performance Test • Why? • Sizing is always based on assumptions which may not be valid (ie: Bandwith) • The system behaves differently under heavy stress • You will find issues in configuration, code, etc.. unseen in normal tests • How? • End-Users Stress • Choose the 5 most used scenarios, script them (see tool) and randomize the script. • Execute the scenarios with accelerated users and after first initial load. • Back-End Stress • Simulate a peak load from all real-time integration points • Even if the test is a success, ramp up to failure to identify your weak points • With what? • Visual Studio Web Performance Project (2013 Ultimate) • Stress Stimulus (Fiddler Extension) • HP LoadRunner • CRM Performance Toolkit (CRM2011)
After Go Live
Maintenance • Maintenance Service • Maintenance Job Editor to Reschedule the jobs • Check for failure and timeouts; move to a SQL Maintenance Job if needed (usually Reindex <> Shrink) • Backups • Avoid Hourly Transactions Logs backup if not needed • Don’t change the Connection Pooling (100 max) • if you need more you do have an issue with non closing connections (reentrant code) • Regulary checks for • Long Running Queries (Sys.dm_exec_query_stats, Sys.dm_exec_query_plan ,Sys.dm_exec_sql_text) • Missing Indexes (min. 4 weeks) (Sys.dm_db_missing_index_group_stats,Sys.dm_db_missing_index_grou ps,Sys.dm_db_missing_index_details,Sys.dm_db_index_usage_stats) • Wait Statistics • Deadlocks (Sys.dm_tran_locks)
Application Performance Management
Manage Growth • Move BloBs out of the CRM database • SharePoint Integration • Others Binary repositories (ie: Documentum, PDF in File Servers, etc..) • Database Purge • Define and Automate Data cleanup • Bulk Delete is fine but : • Define indexes and split in small chunks as Bulk Delete easily timeout • Truncate table is unsupported but can be used if you need a fast cleanup (ie: audits, jobs,..). The DB contraints limit the risks
Archiving, Delete rights and Restore • No OOTB archiving but keeping it all is not an option in high growth systems. • Even if storage cost decreases fast, the overall sizing cost and impacts does not (think about RAM, backups, etc..) • Usually a lot of data is already “archived” as part of BAU; think about DHW, BI, Legal Archives, Disaster Recovery,… • Archiving is not the expensive part, Accessing/Restoring the archived data is, also prevent accidental Delete! • From easy to complex, implement a restore: • Archive CRM Organizations (Live) • Archive CRM Instances (Offline or Live); Sandbox on CRM Online • Backup CRM Database (Offline) • Reports Snaphot history (Live/Offline but degraded access to information) • …
Email Router Server-Side Sync • Email Router • Adjust Batch Size, the Default is 5; so 200 calls per 1000 users • Server-Side Sync • Settings in Deployment Properties for Throttling (Mailbox, EC items) • Think of dedicating one server for Server-Side Sync "Email Integration Service Role"
Takeaways
Session Objectives And Takeaways • Session Objective(s): • Demystify High Volumes • Stories from the Field ! • Key Takeaways • Key attention points for high volumes • Draft design considerations
QUESTIONS
Stephane.Dorrekens@RealDolmen.com Philip.Verlinden@RealDolmen.com Please remember to fill out your session evaluation survey online! The link to the survey was emailed to you, or go to: eXtremeCRM.com/eXtremeCRM2015Madrid/Surveys/tabid/1632/Default.aspx Complete prior to the closing session to be included in today’s drawing!
Other sources • http://blogs.msdn.com/b/emeadcrmsupport/archive/2014/07/16/dy namics-crm-quick-find-performance-part-ii.aspx • https://code.msdn.microsoft.com/windowsdesktop/Multi- Language-Lookups-in-18fadc80 • https://www.microsoft.com/en-us/download/details.aspx?id=45905 • http://blogs.msdn.com/b/crminthefield/archive/2014/09/11/codeple x-dynamics-crm-quickfind-on-selected-view.aspx • http://dynamics4.com/changing-search-behavior-in-microsoft- crm-2011-part-1/
http://crm.realdolmen.com We’re looking forward to meet you ! The team at eXtremeCRM

More Related Content

Dynamics CRM high volume systems - lessons from the field

  • 1. High Volume Systems Lessons from the Field Stéphane Dorrekens, RealDolmen Philip Verlinden, RealDolmen
  • 2. Session Objectives And Takeaways •Session Objectives: • Demystify High Volumes • Stories from the Field ! •Key Takeaways • Key attention points for high volumes • Draft design considerations
  • 3. Field Stories • Financial Institution (8.000 users, 2 implementations, 350Gb) • Financial Institution (2.000 users; 2 implementations; 2500Gb) • Financial Institution (1.000 users, 6 implementations, 450Gb+50Gb/Month for biggest implementation ) • Public Transport (500 users, 1 implementation, ) • Pharmaceutical (400 users , 2 implementations, 300Gb)
  • 5. What is High Volume ? • Typical inflexion points (where system behavior is impacted) • High Volume Data • >1 Mio for base record tables (ie: Contact, Account,..) • >5000 activities/day or >1Mio/year • High Volume Transactions • High Volume Users/Security • >300 concurrent users • >1.000 teams, BU • High Volume vs Enterprise Scale • Small/Medium companies can have huge amounts of data (ex: associations/volunteers) • Small/Medium companies can have huge amounts of transactions/sec (ex: web interactions) • Only Enterprise has huge number of users and complex security • CRM Online vs On Premise • Data Cost & Max GB (<1Tb)
  • 6. Impact of High Volume •The Bad News Where’s the high-volume switch? High-Volume setting (on/off): •The Good News It does work … with some effort…
  • 7. Impact of High Volume • Less Room for “Improvisation” or “Lack of Design” • Every point not taken care of or thought over will be an issue • Some out of the box features are not designed for high volume • Charts limits of 50.000 records • Team membership > 1.000 teams • No out of the box archiving • Reporting and Transactions share the same Database • Queries are locking per default • … • Don’t load millions of Contacts into the system and expect it to just work.
  • 8. Trending • Omni –channel + More Social + More Customer Interactions Challenge the customer on possible: - Data Security Walls <> Performance • Customer Centric • Customer expectations • CRM & CEM
  • 10. Infrastructure Design High Volume of Data • SQL Server is the Key point • Get the numbers ! (Size, Growth, Repartition of data between tables/entities ) • Data Layer IOPS check (especially SAN); Disk is more important than RAM at that level as the Disk/RAM ratio is lower than usual. • RAM (Absolute Minimum 10%; Recommended Minimum 30%) • SQL 2014 Std maximum is 128Gb (64 for 2012), SQL Enterprise maximum is 4Tb • Index but not over the top • Disks Configuration • Tempdb usage/location (Considers SSD) • Consider using a separate reporting database • set the MaxServerMemory setting in SQL SQL to reserve 4/6 Gb of system memory for the OS and other applications • Not to DO • Use non dedicated LUNs • Use an old SAN infrastructure (IOPS check) • Forget to reserve failover host resources • Forget Testing Maintenance (see last section) MDF SQL Server 2014 1x4 Cores 250 GB RAM LDF LUN A LUN B 1024 Gb
  • 11. Infrastructure Design High Volume of Data • … but SQL is not the only point impacted • Data Upload • Front-end server sizing (even temporary) • Bandwidth • a shared 1Gb/s can be overloaded • Deployment • Backup/Restore to another environment can takes literally days • Virtualization • Specific issues : Host reservation (RAM), Dedicated LUNs, .. • Not to Do • Upload 50 Mio Records on a massive web server farm and not warning the network team. • Hint: maybe they have automatic Denial of Service protection?
  • 12. Infrastructure Design High Volume of Users • Performance vs Stability • Minimum of a pair front-end with LB • LB is not enough, rapid failover is needed • App pools • Limits • Recycle • Separate roles • Separate back-end servers • Separate endpoints • To do • Test Failover by hanging an app pool, don’t expect LB to work OOTB • Reboot Front-End Daily Front-End Server 1x2 Cores 8 GB RAM Organization Web Service Role 1x2 Cores 8 GB RAM Front-End Server 1x2 Cores 8 GB RAM Organization Web Service Role 1x2 Cores 8 GB RAM CRM.Contoso.local CRMServices.Contoso.local Back End Role 1x4 Cores 8 GB RAM
  • 14. Functional Design • 80/20 rule • OOTB versus Code • Where to invest and where not ? • All Recommendations which follows are interlinked, example : • Deleting workflows history is good but faster if no POA record exists
  • 15. Normalise but… • Normalise your datamodel • Map it on the CRM datamodel as much as possible • Get and map the numbers • And reflect… • Security Design (CRM) • Volumes • B2C in Germany (+80 Mio or Belgium +11 Mio) • Functional (MUI) • Online-proof: FetchXML <> FilteredViews
  • 16. Security, from performant to less performant… from standard to exception • Business Unit / Roles • Access Teams • Hierarchical Security • Team Ownership • Sharing (User/Team) Field story on Teams numbers • Non-Scalable: Strictly limit the number of Owner teams a user is a member of • Scalable: Access Teams but think numbers and maintenance Field story on Sharing: keep the POA under control • Disable the “Share record with the previous owner” • Parental Relationships to Referential or remove ‘Reparent’ cascade • Workflows ownership: Async Service user • Keep monitoring the POA Size
  • 17. Views 1/2 • Only show the minimum of records needed for the feature • Uses preemptive filtering instead of after the fact filtering (user based) • The rationale is that the Top X records are shown and ordered by AFTER the security model is applied. This combination forbids use of a non-clustered index and we cannot change the clustered index (sequential GUIDs) • Thus the less records you have to sort; the fastest • Specify a period if possible (ie: Last Year, Last 6 Months) • This avoids degradation overtime and limits #Records to sort • Remove sort order if not needed (via SDK, update FetchXML ) • Uses the database order (sequential GUID) if sort is not needed • The rationale is to remove the order by impact • Don’t use the ‘ALL’ views as default • Make the default the most restrictive view needed (ie: ‘My’ or ‘My in that period’)
  • 18. Views 2/2 • Don’t use a specific view for search reasons, search is ALWAYS global. • If only search is needed, use a view returning no records • The optimal is to search for a non existing record; non existing value on an indexed field • If you want to combine search and a default list; use the most restrictive list needed. • Explain to the users the concept of global search • Find Fields • Use the most restrictive list of ‘find fields’ as ALL those fields will be searched • Those principles are not only valid for ‘standard views’, they should be applied on: • Dashboard views • Lookups views • Search Result View
  • 19. QuickFind Filtering • How to search in a view: • Simple: OOTB dashboard List • Harder? Field Story on Quicfkfind with 10 Mio contacts? • Filter with • Datamodel • User preference (custom entitiy: POS or global) • PreRetrieveMultiple add “where”-clause • Button to change scope
  • 20. Plug-ins • Limit the Plug-ins registered per Events Only one plug-in execute should fire for a given event; thus the plug- in class naming convention should be : xx.Plugins.<Entity>.<Message>.<Type>. xx.Plugins.Account.Create.Sync Only one plug-in dll should exists for a given entity: xx.Plugins.Account This also allows sharing of information between the different steps of the same pipeline (ie: SharedVariables) • Make sure to define the filtering attributes for Plug-ins. Because CRM uses now the auto save every 30 seconds, they will be triggered frequently. • Implement Tracing Some issues only occurs under heavy stress (ie: bad instantiations); good tracing will helps you tremendously Use FetchXML aggregation when possible (especially count) • Do not use any kind of static instantiations, it will fail under heavy load
  • 21. The sandbox tax 1st Test with Registration:Sandbox [2015-03-30 20:14:34.858] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 8 > OrganizationSdkService starts processing request for user:0d57f89f-bf5e-4e4c-b4ab-8ad6d9b2e039 Request Xml:Update<UpdateRequest [2015-03-30 20:14:34.987] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 8 |Category: Sandbox |User: 0d57f89f-bf5e-4e4c-b4ab-8ad6d9b2e039 |Level: Info |ReqId: 631e39a9-8527-4397-8fe1-275f1f9e1102 | SandboxCodeUnit.Execute ilOffset = 0x394 > SandboxPlugin.Execute: exit [2015-03-30 20:14:34.987] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 8 |Category: Platform.Sdk |User: 0d57f89f-bf5e-4e4c-b4ab-8ad6d9b2e039 |Level: Verbose |ReqId: 631e39a9-8527-4397-> MessageProcessor start executing step 58e202dd-4b49-417a-b732- f95957ef99b0 of type 'Microsoft.Crm.ObjectModel.MultiCurrencyPlugin' synchronously for message 'Update' for entity 'contact' 129msec to complete the call 2nd Test with Registration:None [2015-03-30 20:20:44.900] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 33 > OrganizationSdkService starts processing request for user:0d57f89f-bf5e-4e4c-b4ab- 8ad6d9b2e039 As user:5c0e5ad5-8cd6-e311-93f9-00155d1a7308 Request Xml:Update<UpdateRequest [2015-03-30 20:20:44.963] Process: w3wp |Organization:e1f759d5-8cd6-e311-93f9-00155d1a7308 |Thread: 33 |Category: Platform.Sdk |User: 0d57f89f-bf5e- > MessageProcessor start executing step 58e202dd-4b49-417a-b732-f95957ef99b0 of type 'Microsoft.Crm.ObjectModel.MultiCurrencyPlugin' synchronously for message 'Update' for entity 'contact' 63 msec to complete the call
  • 22. The sandbox issue Update Event Pipeline Retrieve . Pre Stage PreEvent Plugin Sandbox Sandbox PostEvent Plugin Retrieve Event Pipeline Pre Stage Platform Post Stage 30 msec Sandbox 30 msec 30 msec 30 msec
  • 23. Async Workflows • Use with care • Strictly limit the number of workflows registered per event, to avoid race conditions and system overload. • Purge completed workflows as much as possible, but beware of the “delete tax”
  • 24. Audit • Restrict Scope • Do not audit when doing the initial import • Do not audit read only data (especially, if having a daily or more update frequency) • Do not audit unneeded fields • Manage Growth • Delete Audit information is only by date, not by type • Choose SQL Server version wisely • SQL Enterprise audit uses Quarterly Partitions, SQL Standard doesn't • Allows splitting the audit logs into different filegroups for better I/O • Upgrade from Standard to Enterprise will not create partitions! • Audit is always async so triggers high usage of back-end
  • 25. Locks & Deadlocks • The SQL Server Default is locking* • By default, define the queries with the no-lock hint • no-lock='true‘ in FetchXML • QueryExpression.NoLock • With (Nolock) in T-SQL • That applies everywhere (ie: Views, Charts, Reports, Plugins, JS, etc..) • Minimize lock contention when doing updates (ie: cascade restrictions) • Export Excel is (currently) locking so give the right with care. A good ref.: Microsoft Dynamics CRM 2015 Performance and Scalability * You can change the SQL Server default (like to Snapshot isolation) but it’s not the recommended best practice and triggers its own issues to manage (ie: Tempdb usage in Snapshot isolation)
  • 26. Integration & GUIDs • Don't let external systems generate CRM GUIDs, as they won't be sequentials, even for Read-Only, back end originated data (cfr http://blogs.msdn.com/b/crminthefield/archive/2015/01/19/the-dangers- of-guid-newguid.aspx) • Using a sequential .NET generated GUID (cfr http://blogs.msdn.com/b/dbrowne/archive/2012/07/03/how-to-generate- sequential-guids-for-sql-server-in-net.aspx), is dangerous as you need to be sure only one system/integration generates those GUIDs; which is implicit if generated by the Database Server.
  • 27. Integration & Performance • Web Services are not the bottleneck if handled properly • Data Injection needs to be massively multi-threaded (Bulk API can be helpful as well) and scaled out • Front-End processing need to be massively scaled out; the bottleneck being the IIS CPU Usage (+/- 50 ops/sec/core) • Disable everything unnecessary (not only for initial load): Audit, WF, Plugins, etc.. • The true bottleneck is Network bandwidth and IOPS Tip : When you use the Update message, do not set the OwnerId attribute on a record unless the owner has actually changed
  • 28. Performance Test • Why? • Sizing is always based on assumptions which may not be valid (ie: Bandwith) • The system behaves differently under heavy stress • You will find issues in configuration, code, etc.. unseen in normal tests • How? • End-Users Stress • Choose the 5 most used scenarios, script them (see tool) and randomize the script. • Execute the scenarios with accelerated users and after first initial load. • Back-End Stress • Simulate a peak load from all real-time integration points • Even if the test is a success, ramp up to failure to identify your weak points • With what? • Visual Studio Web Performance Project (2013 Ultimate) • Stress Stimulus (Fiddler Extension) • HP LoadRunner • CRM Performance Toolkit (CRM2011)
  • 30. Maintenance • Maintenance Service • Maintenance Job Editor to Reschedule the jobs • Check for failure and timeouts; move to a SQL Maintenance Job if needed (usually Reindex <> Shrink) • Backups • Avoid Hourly Transactions Logs backup if not needed • Don’t change the Connection Pooling (100 max) • if you need more you do have an issue with non closing connections (reentrant code) • Regulary checks for • Long Running Queries (Sys.dm_exec_query_stats, Sys.dm_exec_query_plan ,Sys.dm_exec_sql_text) • Missing Indexes (min. 4 weeks) (Sys.dm_db_missing_index_group_stats,Sys.dm_db_missing_index_grou ps,Sys.dm_db_missing_index_details,Sys.dm_db_index_usage_stats) • Wait Statistics • Deadlocks (Sys.dm_tran_locks)
  • 32. Manage Growth • Move BloBs out of the CRM database • SharePoint Integration • Others Binary repositories (ie: Documentum, PDF in File Servers, etc..) • Database Purge • Define and Automate Data cleanup • Bulk Delete is fine but : • Define indexes and split in small chunks as Bulk Delete easily timeout • Truncate table is unsupported but can be used if you need a fast cleanup (ie: audits, jobs,..). The DB contraints limit the risks
  • 33. Archiving, Delete rights and Restore • No OOTB archiving but keeping it all is not an option in high growth systems. • Even if storage cost decreases fast, the overall sizing cost and impacts does not (think about RAM, backups, etc..) • Usually a lot of data is already “archived” as part of BAU; think about DHW, BI, Legal Archives, Disaster Recovery,… • Archiving is not the expensive part, Accessing/Restoring the archived data is, also prevent accidental Delete! • From easy to complex, implement a restore: • Archive CRM Organizations (Live) • Archive CRM Instances (Offline or Live); Sandbox on CRM Online • Backup CRM Database (Offline) • Reports Snaphot history (Live/Offline but degraded access to information) • …
  • 34. Email Router Server-Side Sync • Email Router • Adjust Batch Size, the Default is 5; so 200 calls per 1000 users • Server-Side Sync • Settings in Deployment Properties for Throttling (Mailbox, EC items) • Think of dedicating one server for Server-Side Sync "Email Integration Service Role"
  • 36. Session Objectives And Takeaways • Session Objective(s): • Demystify High Volumes • Stories from the Field ! • Key Takeaways • Key attention points for high volumes • Draft design considerations
  • 38. Stephane.Dorrekens@RealDolmen.com Philip.Verlinden@RealDolmen.com Please remember to fill out your session evaluation survey online! The link to the survey was emailed to you, or go to: eXtremeCRM.com/eXtremeCRM2015Madrid/Surveys/tabid/1632/Default.aspx Complete prior to the closing session to be included in today’s drawing!
  • 39. Other sources • http://blogs.msdn.com/b/emeadcrmsupport/archive/2014/07/16/dy namics-crm-quick-find-performance-part-ii.aspx • https://code.msdn.microsoft.com/windowsdesktop/Multi- Language-Lookups-in-18fadc80 • https://www.microsoft.com/en-us/download/details.aspx?id=45905 • http://blogs.msdn.com/b/crminthefield/archive/2014/09/11/codeple x-dynamics-crm-quickfind-on-selected-view.aspx • http://dynamics4.com/changing-search-behavior-in-microsoft- crm-2011-part-1/
  • 40. http://crm.realdolmen.com We’re looking forward to meet you ! The team at eXtremeCRM

Editor's Notes

  1. High-Volume switch: doesn't fully work out of the box Reasons/Examples/Challenges…
  2. *Omni-channel More data Less security *Customer Centric -think from the client’s perspective, how does he experience your brand? -is the experience/interaction consistent  COMMON PURPOSE: align the products and services to the needs of the customer and ensuring the business interaction is as smooth and consistent as possible
  3. *80/20: aim for higher but keep it under control *OOTB vs Code: Microsfot rapid release cycle –think about templating, race to keep up to date *Good Investement Areas: Webresources -complexity: databanding -perceived loading time and the actual loading time, every millisecond matters. -technical design Protoype correctly for re-use *Bad Investement Areas: Workflow -don’t go for an all workflow approach, doesn’t work, make the decision effort . NET or OOTB, sync or not? Unsupported
  4. *Normalize - Every textbook… *OOTB – so MAP (80/20) *Numbers: activities?! REFLECT -Security: POA, Access Teams maintenance,.. -Volumes POA, Access Teams,… -Functional MUI: Scott Durrow name parsing <> Order
  5. BUH Since version 1.0 Limit 1,000 BU’s Access Teams 10,000,000 records and 5,000 users Managebility
  6. CRM 2015 Update 0.1 FTS
  7. SQL Clustered Index: If a column where new values are not higher than previous is used for a clustered index, adding each new row would require re-ordering, i.e. moving the whole row and placing it to its proper location in accordance with clustered index ordering, thus splitting data pages and affecting SQL Server performance. If such clustered index is created on a table with frequent inserts and updates, it can cause performance degradation.
  8. APM -DynaTrace -App Dynamics
  9. STEPHANE
  10. Why numbers? Numbers are as relevant for infrastructure sizing as for other sizings . A special case could be the CRM data security design, that's why we always what to get fairly good estimate of the expected volume so you can align the these requirements as much as possible with the performant OOTB security features.
  11. PHILIP
  12. PHILIP Typical problems: JQuery Comparision: performance counters, new TurboForms