VSTS Automated Load Testing Cloud Based Agent – On Premises

I know this title may cause you your head to spin if you’re not working this daily. but .. let us get to the details .
Microsoft Visual Studio Team Services  VSTS (formerly known as VSO Visual Studio online ) and future known as (Visual Studio …..)  is a wonderful collection of tools that run hosted as a SaaS offering ,delivering most of the tooling that you would get within your local Visual Studio IDE.  — [PS I love VS ]

VSTS offers , CI/CD cycle as most of online ALM tooling products.

The most common build cycle / pipeline did not have running load tests for the longest time. now we are blessed to have this most recent addition to the list of tools.  Yay….

The client I am working for now , decided to use this feature, but as with all corporate minded IT they do not want to execute build on SaaS or PaaS machines, they want to build on own custom built agents and custom build on-premises  . Microsoft has an App for that !!!

The problem :

Webtest with data bound calls to local data source ( like CSV files and other tabular flat files) fail to execute with the following error message.

2018-07-24T20:10:56.1174689Z ##[warning]Could not run load test ‘myTEST’ on agent ‘MyCLTAgentAPP02’: Could not access table ‘MyTableData#csv’ in data source ‘MyTableData’ of test ‘mywebtest,0df341b5a5-ddb6-43dc-8f75-36583bcd77fd’: The Microsoft Jet database engine could not find the object ‘MyTableData.csv’. Make sure the object exists and that you spell its name and the path name correctly.

The error points that the correct location of the file is C:\Windows\System32\MyTableData.csv      which is not correct.

 

The Solution :

1. First understand how this runs when you invoke it from your Visual Studio IDE.
2. VS bundles the folder structure you have on your scaffold project into a single flat folder with all the files in it at one single location ( means you need to have a very good naming convention to keep files from overwriting each other).
3. The Files on your VS project mus be labeled ( Alway Copy ) as a “Copy Output Directory” property.
4. include the .testsettings files into your project and make sure they are copied as well.
5. Within your build cycle use the VSTS native [FILE COPY]  task  to copy the test folder
6. check the “Flatten Folders” Under advanced settings

the rest should be as documented online .

Here is what flatten folders do

C:\MySolution\vsts.runsettings

 C:\MySolution\WebTest\MyWebTest.webtest

 C:\MySolution\loadTests\MyLoadTest.loadtest

 C:\MySolution\TestData\MyTable.csv

 

After flattening this it will be

C:\MySolution\vsts.runsettings

C:\MySolution\MyWebTest.webtest

C:\MySolution\MyLoadTest.loadtest

C:\MySolution\MyTable.csv

 

Happy load testing y’all 🙂 .

VSTS Error Configuring Agents – or VS Test manager IDE Hung on ” Configuring Agents “

VSTS ( Visual Studio Team Services ) Cloud Load Testing

Symptom : Error Configuring Agents – or UI Hung on “ Configuring Agents “
Diagnosis : visiting any of the agents machines Application log you find this message.
(Microsoft.VisualStudio.TestService.TmiExecutionHost.exe, PID 452, Thread 5) HKEY_LOCAL_MACHINE\\EnterpriseTools\QualityTools\TestTypes Key cannot be found!

The cause : unknown

It is mainly an Agents configuration problem , mostly during an agent refresh
Solution : reinstall the agent service and restart the service

use the .\ManageVSTSCloudLoadAgent.ps1  per this documentation , and make sure your proxy configuration is updated ( on the IE side of the agent machine) the winhttp proxy is ignored by this windows service .

Best of Luck.

Workflow Manager Fail with “Trusted provider is missing” error

I encountered this problem with a customer where one of our workflow heavy applications stopped to function .

The error I got within ULS Logs look like

00000003-0000-0ff1-ce00-000000000000  trusted provider is missing

Turns out it is a security change to the IIS Application for the “Workflow Management Site” where the Authentication provider included

Asp.Net Impersonation Enabled 
Along with other changes to that IIS application .

 

This customer is using SharePoint 2013 Enterprise on premises with NTLM only in a windows integrated mode , no SSL .

After narrowing down the issue to “Authentication problem” I did setup a new WFM with OOB configuration.

Asp.Net Impersonation Disabled 

I have checked all those settings below this will fix this issue after you restart the application pool.

See Below

Best of Luck.

 

 

 

 

SharePoint 2013 TypeError: Unable to get property ‘replace’ of undefined or null referenceTypeError

You get the following error when you attempt to edit list items or list views.

TypeError: Unable to get property ‘replace’ of undefined or null referenceTypeError: Unable to get property ‘replace’ of undefined or null reference

TypeError: Unable to get property ‘replace’ of undefined or null referenceTypeError: Unable to get property ‘replace’ of undefined or null referenceTypeError: Unable to get property ‘_events’ of undefined or null reference.

Problem :

this happened after an automatic windows update installed the January 2016 SharePoint Server cumulative updates.

Solution :

First , try to run the SharePoint Products upgrade wizard on the SharePoint host with administrative rights.

Second Install the January 2016 SharePoint foundation ( I know that was not required before but for some hideous reason , this is coming back again )

You can get this download from here ..

https://technet.microsoft.com/en-us/library/dn789211(v=office.14).aspx

After the installation  ,please re run the SharePoint upgrade wizard it might leave the server in an unstable state ( some components may take the update and some don’t , that is why you need to run this wizard).

Best of luck

PS: This is a deja vu  from a similar issue in 2007 …. no more comments

SharePoint ULS Log analysis using ELK (ElasticSearch LogStash and Kibana)

SharePoint ULS Log analysis using ELK (ElasticSearch LogStash and Kibana)

A Solution to Multi-Tenant systems Log Access.

By: George Gergues

 

Introduction

SharePoint is a large platform that is always growing, and changing, and as with large application platforms that hosts many components, the complexity is always manifested in the platform log (ULS Logs) and log management. There are several ways to trace and debug issues using those logs, and tooling in this area has not been keeping up with the speed SharePoint. The other major complexity is multi tenancy, as the cost to own and operate a single SharePoint farm has been on the rise, many companies started to offer a multitenant farm system, mainly Office 365 by Microsoft, along with other hosting vendors in the same domain. Multitenancy reduces the cost of owning into a cost to lease or operate, but takes away platform specific facilities like Audit Log and platform log, as those log containers are not partitioned at the same tenancy level but are singletons per host in the case of ULS Log or per Content Database in the case of Audit Log table.

 

The Solution

In this posting we are proposing the use of the ELK platform (ElasticSearch, LogStash and Kibana) as a tool to ingest the ULS logs, leverage the fast index and search capabilities and make use of the event correlation features that come by aggregating logs and other factors.

Such tool used by system engineers and hosting staff can be very useful in responding to first level support incidents, or to relay the events to the end user without compromising security.

Document level links: The system engineer can share a document level link, to the event to show the full details of the event, those are mainly

Also this tool can show trending of events and repeated problems or patterns that might be cyclical in nature and provide better picture of farm level performance and common problems.

With little development, you can expose those search portal to the end users to directly perform those tenant level searches without exposing the full log to the end user or tenant admin.

The next few sections will show in detail how to build and customize such system.

What is ELK

ELK = [ E + L + K ]

ELK is the combination of three open source solutions to three problems and merged together, they form the best solution for large log and large dataset analytics. All three systems operate using REST.

E: ElasticSearch:

Index and search server base on the Lucene open source project, storing documents based items (JSON objects), the storage is clusterable and highly redundant through shards and multiple distributed nodes. Elastic Search is the Java ported version of Lucene and runs inside a JVM.

L: LogStash

is a Log capture and processing framework that works in a Capture-> Process -> Store cycle per single event on the log source. It is open source project that evolved a lot in the past year to operate with a long list filters, modules and plugins. LogStash is very powerful when used to construct schema JSON documents from unstructured text inside log files using GROK filters. LogStash is written mainly in Ruby and running inside JVM using JRuby.

K: Kibana

is a “node.js” data visualization layer that is tightly integrated with the elasticsearch index and can build charts and dashboards that represents data in the elastic search index very fast. It is making use of the fast REST api and is very responsive inside the browser even with large datasets. The Kibana configuration is stored as a JSON document in the elastic search index, along with all the charting and dashboard scripts that it produces.

The Challenge

1. Multitenancy
   

    As mentioned in the introduction, with multitenancy comes the lack of access to platform ULS logs due to security restrictions to OS Filesystem level. The only way SharePoint shows application and system exceptions is via a custom application error pages, and they are very obscure, normally assigned a session or transaction id ( uuid ) named (correlation Id)and that is the only piece of information the user gets along with the timestamp.

Example of Correlation Id “53fed7f1-cf35-1253-0000-000050f7b00c”

Cannot give all tenants access to the shared logs as they main contain information regarding custom application or extensions, it might also expose other tenants content or exceptions depending on the debug level.

1. Log Size and Log Cycling
   

   The other problem is Log Sizes and Log cycling. As with any large enterprise deployment, you are required to maintain some weeks of log history, either on the same system or offline. The minute those files leave the system, they are harder to manage and /or correlate.
   

2. Multi Log Events
   

   Some events can span the end of one log file and continue onto the next log, and some of the current tools like ULS viewer could not handle easily. With such system, the log parsing is already done, and the queries are much faster.            
   

3. Aggregate Log Sources
   

   Using such system, an engineer would have the ability to shows events from multiple aggregate sources, and to analyze events that may have started and caused by other dependencies, while manifested only at SharePoint where the error occurs. Some easy log aggregation candidates would include the IIS server logs per host along with windows event logs and SQL server logs.
   

ULS Logs

The Unified Logging System (ULS log) is the standard logging format used for SharePoint farms.

Location is Configurable by admin but default installation point to the SharePoint hive C:\Program files\Common Files\Microsoft Shared\Web Server Extensions\14 or 15 or 16 \LOGS\*.log

Process: Host
Executable running and causing the event.

TID: Process thread Id on that host.

Area: Component or application generating the event.

Event ID: Internal category Id per application.

The files rotate every 30 minutes by default.

SharePoint Application Error Page

The SharePoint custom error page for 2010, 2013 and Office 365 looks like this.

Only a correlation Id is visible to the users and they are to report it to the support technician.

This can be very frustrating as a developer as you need to wait for hours and in some cases days before you get an answer .

Setting up ELK

1. Install Java Runtime (JRE)
   
2. Download the packages
   (zip files) for each of the following and place each on a separate folder. Logstash , ElasticSearch, and Kibana.
   
3. Edit the configuration file for each package for your environment (Development configuration )
   

For simplicity we opted to use the file input module to perform in our limited development environment, yet you can steam the log files using from multiple source using any plugin, LumberJack is the most commonly used log streaming plugin.

 

 

 

Details on the GROK filter syntax

    The Grok filter is the main core component that tries to understand the schema of your unstructured log text and generates a temporary schema based object to be stored, indexed and later on queried upon.

 

There are a few formats for the log file and the date timestamp variations that puts a bit of a burden to collect construct a pattern to match all the value.

 

Sample Pattern

SP_ULS_FMT1 (?<sptimestamp>%{MONTHNUM}/%{MONTHDAY}/%{YEAR}%{HOUR}:%{MINUTE}:%{SECOND}\*?*)%{SPACE}%{PROG:sp_process}\(%{BASE16NUM:sp_pid}\)%{SPACE}%{BASE16NUM:sp_tid}%{SPACE}\t+%{DATA:sp_area}%{SPACE}\t+%{DATA:sp_category}%{SPACE}\t+%{DATA:sp_eventid}\t+%{SPACE}%{WORD:severity}%{SPACE}%{DATA:sp_eventmessage}%{SPACE}%{UUID:correlationid}%{SPACE}

You can see the full configuration script on the GitHub repo
https://github.com/Gergues/SharePoint-ELK

Building the Dashboard: Kibana Visualization

The Kibana configuration is fairly simple and all constructed through the UI. The Configuration of each of those elements is stored in the elasticSearch storage as an index named (.kibana).

The dashboard is made of some smaller components, mainly charts, and widgets, that target a specific measurement against the index.

The raw data of each element can be displayed and extracted as csv file.

A Severity chart, is simply a vector of the unique elements in the field SP_SEVERITY describing the event severity level.

The whole dashboard is dynamically rendered and filtered via the queries you select

 

This sample shows the event correlation id with showing the severity level and the total count of events that happened along with the components

 

 

 

 

 

 

Full Configuration scripts

You can find the full configuration scripts to get you started here.

https://github.com/Gergues/SharePoint-ELK

Limitations

ELK inside JVM instances (Java Virtual Machine), so you are bound by the limits of that particular instance. As with any Java based process you can tweak
free memory and heap allocation parameters to get the best throughput, yet the optimal performance will be achieved by using clusters of the service, and that is where shard storage shine.

Improvements

1. Aggregate Log sources in dashboards
   

As a next natural enhancement for this project, IIS Logs on all servers, windows log and SQL Server logs as well as firewall and or load balancers.

1. Security
   

   1. You can start implementing some security over this system, not that in our sample system. Initially users don’t have access to the Search API directly but only via Kibana (the visualization layer) this is the first measure you can take.
      
   2. You can also implement an IIS or Apache proxy to implement authentication and authorization with any access control.
      
   3. You can use one of the commercial products (Shield and Marvel) from Elastic.Co
      

2. Scalability
   

   This system is designed to be very highly distributed and available, with multiple node and clusters, (only keep the elastic nodes in the same geography to reduce latency). But you can have multiple hosts the logstash role, kibana role or all of them.
   

References

1. Elasticsearch : https://www.elastic.co/products/elasticsearch
   
2. LogStash : https://www.elastic.co/products/logstash
   
3. Kibana : https://www.elastic.co/products/kibana
   
4. ELK guides : https://www.elastic.co/guide/en/found/current/elk-and-found.html
   
5. Guides on using Kibana : https://www.timroes.de/2015/02/07/kibana-4-tutorial-part-1-introduction/
   
6. Grok Debug Tool : http://grokdebug.herokuapp.com/
   
7. The full configuration scripts: https://github.com/Gergues/SharePoint-ELK
   

SharePoint audit in action

The Triangle SharePoint User Group  ( TriSPUG) meeting for Tuesday  03JUN2014 at the Microsoft building in Durham .
I finally finished the presentation and the slides and the code samples on codeplex.

 

Please find attached

The code samples are here https://spauditdemo.codeplex.com/

 

Let me know if you have any questions.

 

Best of luck

SharePoint audit in action

SPAuditDemo

Splash

 

Secure Store Service did not performed the operation

Error : Secure Store Service did not performed the operation”

The Error : Along with the grammar issue , ( not that English is my third language ) is a valid one,

With all sorts of errors on the SP log , System and application logs.

This was only a symptom to a bigger problem

Database log file growing out of size (in our case filled the disk space)

 

Those failed write to the DB file from the Log, happen often in slow disk environments (not sure why)

In some case it will show this error on the backup logs (: The transaction log for database ‘Name of DB’ is full)

How to solve this short term problem (PS : this is how I solved it but I am not sure if this is the correct way)

1. Do no panic, it is a SQL problem.
2. Go to the SQL Management tools, change the DB Options Recovery mode from Full to Simple.
3. Perform a full DB Backup to any other disk.
4. Change the DB Options Recovery mode from Simple back to full.
5. You should be back online by now.

    

HAVING MULTIPLE USERPROFILE SYNC SERVICES

Having Multiple UserProfile Sync services

In an architecture I worked on I was isolating two service groups , I wanted to isolate a service group that service some SP Applications from others , with security , information leakage and other concerns in mind ( like having own managed metadata services , and profile services etc.)

The problem was having two or More UserProfile Synchronization services, While many might say , why , and you don’t need another one , or it can’t be done.

Simply the problem is that User Profiles are always tied to the mySite host and that is a global configuration setting on the service application (your first provisioned UPS application ).

The opted design:

You can have multiple UserProfile Sync services but they can each reside on a single SharePoint hos within your farm.

You cannot have multiples on the same host , as it is simply a Forefront Identify management engine ( Geneva ) and given the complexity of this thing , I would not even think about running two of them on the same host.

The main problem :

After declaring success on this front and having the two sync service application , each with multiple and different AD connections , I noticed that the sync works fine , to all sites and content . yet the farm backup job fails sporadically with this error on the log.

[8/21/2013 9:11:08 PM] FatalError: Object SP-USER-PROFILE-SERVICE failed in event OnBackup.

For more information, see the spbackup.log or sprestore.log file located in the backup directory.

SPDuplicateObjectException: An object of the type Microsoft.Office.Server.Administration.ProfileSynchronizationUnprovisionJob

named “ProfileSynchronizationUnprovisionJob” already exists under the parent Microsoft.SharePoint.Administration.SPTimerService named “SPTimerV4”. Rename your object or delete the existing object.

The main solution:

The cause of the problem is the incremental backup that took longer than usual and got into the AD Sync cycle of one of the Sync Services.

The solution is simple, Stop both SYNC services, start both sync services ( on each respective server) , push the backup window a few hours apart.

That solved my problem. Hope is solves yours.

Best of Luck

The closed type does not have a corresponding settable property

I have encountered this error with a BCS application that I built and was working for some time.

The Exception / error is

"The closed type does not have a corresponding  <PropertyName> settable property"

The BCS application was working inside a farm connection to a list in another network and anther farm for sure.
 

The List column names were changed ( at least one of them  ) and that did break the proxy that was compiled for that BCS model to work  with

 

How to detect such problem :

Browse to the service wcf interface that should show the fields ( you will need to see the source of the page to see the field name from the atom feed)

match those to your Model mapped name ( that is normally dynamically configured by VS ) 

see the mismatched fields, fix your code and rebuild and redeploy
( and call the list owner let them know of the change and don’t get angry )

 

Lesson Learned : Cannot depend on uncontrolled lists as solid schemas.

I think the main problem is that the feed sends the Fields display name and not the Internal Name of the list which is unique and static

 

Microsoft Please have a fix (or guidance)

 

Best of luck

 

 

 

Error with User Profile services

Error with User Profile services

The server encountered an unexpected error in the synchronization engine:

“BAIL: MMS(268): eafam.cpp(1510): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): eafam.cpp(901): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): eafam.cpp(1013): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): amexec.cpp(1701): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): amexec.cpp(2086): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): eaf.cpp(1417): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): eaf.cpp(657): 0x80230304 (The image or dimage already has an attribute with that name.)

ERR: MMS(268): synccoreimp.cpp(5266): 0x80230304 – export-flow failed 0x80230304

BAIL: MMS(268): synccoreimp.cpp(5267): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): synccoreimp.cpp(4858): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): synccoreimp.cpp(10873): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): synccoreimp.cpp(10557): 0x80230304 (The image or dimage already has an attribute with that name.)

BAIL: MMS(268): synccoreimp.cpp(2545): 0x80230304 (The image or dimage already has an attribute with that name.)

ERR: MMS(268): synccoreimp.cpp(6483): 0x80230304 – MV to CS synchronization failed 0x80230304: [{F81CD149-ADC9-4720-89E2-E9CBD2CE39A9}]

BAIL: MMS(268): synccoreimp.cpp(6486): 0x80230304 (The image or dimage already has an attribute with that name.)

ERR: MMS(268): syncmonitor.cpp(2515): SE: Rollback SQL transaction for: 0x80230304

MMS(268): SE: CS image begin

MMS(268): SE: CS image end

Forefront Identity Manager 4.0.2450.34″

The Microsoft article does not actually describe the problem nor the solution, yet it is very simple .

The FIM engine [That is the Sync engine described in the message] is the Forefront Identity Manager

The problem is : One or more properties are being overwritten by mistake to the temp storage by the Sync engine.

In my case ( and the most common one ) it was the AD Attributes (Both with Import direction )

AD Attribute “Title” – > SP Profile “Title”

AD Attribute “Title” – > SP Profile “Job Title”

Solution : If the Microsoft solution does not do it for you (like it didn’t work for me)

1. Simply remove both mappings.
   
2. Do a full Profile Sync.
   
3. Add the first mapping and Do full profile sync (monitor errors)
   
4. Add the Second mapping and do full profile sync (monitor errors)
   

Thank you Google ….. and Microsoft.

Best of luck.