SOLVED!... Anyone Try AutoSPInstaller Multi-Server Install on DOD or other "more secured" Servers??

Jul 14, 2012 at 12:45 AM

Objective: 3 Tier, Small farm (2 Web Front Ends and an App Server)

Planned Configuration:  Central Admin and Query Elements on the 2 WFE's, with all other services and the Crawler on the App Server. Will not use Office Web Apps.

All three servers began as the exact clean image and all are on the same domain as the SQL server. Being DOD, I expect the image has several security tweaks in it, after I found the "Secondary Logon" service disabled on them and changed them to run. (I'm not sure what all services need to be running on the SP servers or the SQL server for the scripts to run without issues.)

When problems occurred, I researched and verified all my common names, all the correct access priveleges on the SQL Server (server roles and db roles), local machine roles. It seems that some of the things the script said it couldn't do actually get done when I investigate. Several issues are seemingly related to the script updating to run as the general "spservices" service account - though I ensured the "Secondary Logon" service is running on all servers.  POTENTIAL CLUE: Each time an issue occurred, there was an error in my ULS logs: It says it's a SQL server issue, that the account "[HOSTNAME]$" was unable to login."

I'm hoping that someone might know more than me about possible server hardening, different agents or other security issues that might lead to the pattern I'm seeing.

Other than the fact that after the SharePoint binaries are installed on all servers, the SharePoint Timer Service somehow is always "Disabled" (and I have to manually fix it) -

WFE-1 installation comes off nearly without a hitch - It freezes at "Creating Configuration Database" (When timer stops in ULS log, I re-run and the database is there); and I notice one message near the end, around where the PDF search stuff is being configured that said, WARNING: - No Search Applications found.

However, when I moved to the App Server, the script has the following issues:

Freezes at "Attempting to Join Farm" (Just like above, I wait until ULS shows timer stops, Re run and it is already joined)

Script Aborts when it updates to run as my service account & it errors out trying to start the "Sandboxed Code Service"; I go in and change my config input file to not install it and re-run...

It then aborts when it updates to run as my service account and errors out while "Waiting for User Profile Sync Service...Cannot find an overload for "SetSynchronizationMachine"... I change my input file to go around User Profile, and re-run...

Script then aborts when it updates to run as my service account and errors out with "Exception calling "Deploy" with "0" argument(s): explaining an object already exists......." I edit my input file to go arount this too and...

Script aborts after it reports "Secure Store already provisioned" then errors out while "Creating the Master Key" indicating a problem with the parameter argument for 'ServiceApplicationProxy' is null.  I edit the input file and re-run and...

Script aborts after Updating Web Analytics Data Processing Service to run as the spservice account and indicates an error with "0" arguments again here.

The last issue I see is when 'Setting the administration component..." on Enterprise Search. It complains a "timer job can only be run on a server wher the timer service is installed."

The second WFE, freezes attempting to join the farm like the others. It then has a few errors but doesn't abort. First, it says when attempting to create a local Central Admin site that "an adminvs service instance could not be found on local machine because it's null";  It then struggles trying to create a search crawl extension.

Jul 14, 2012 at 7:50 AM

Hi,

any chance you could upload your initial config file? Honestly the errors above really look more then strange. You got snapshots of your machines to rollback to? One thing to try as there are so many different errors. Did you try to just manually install SharePoint and run through the builtin wizzards to see if this reports similar issues? If so, this is nothing with AutoSPInstaller but somehow your servers are srewed.

Stefan

Jul 14, 2012 at 11:45 PM

Thanks for your reply Stefan. I've been working through the logs and investigating the problem(s) to a point where my frustration level is off the charts. I really need some help. I have wiped the servers clean and retried several times. These issues arise each time - An error in my ULS log about how the server I'm configuring not able to login to SQL server (with HOSTNAME$ - the dollar sign is there). I hoped that might be a clue as to what the malfunction is.

Actually, I'm inclined to agree with you that it's likely something with my server image - but the constraints in DOD are very strict and we HAVE to use this "approved" image for the servers. What I'm hoping above hope to find is the information or pointed to the right direction/reference where I can investigate to understand all critical server settings for my VMs running SharePoint and the one running SQL Server. (i.e. For SharePoint Server 2010 to be installed/configured to run, which MUST be set to what?  Meaning, which services, component services, local server polies and/or any other attributes.)

Here is my original config-input file...

<?xml version="1.0" encoding="utf-8"?>
<Configuration Environment="Prod" Version="3.0.3">

<Install> 
<ConfigFile>config.xml</ConfigFile> 
<OfflineInstall>true</OfflineInstall>
<PauseAfterInstall>false</PauseAfterInstall>
<RemoteInstall>false</RemoteInstall>
<ParallelInstall>false</ParallelInstall>
<Disable>
<CertificateRevocationListCheck>true</CertificateRevocationListCheck>
</Disable>
</Install>

<Farm>
<Passphrase>FarmPa55PhRa53!$</Passphrase>
<Account AddToLocalAdminsDuringSetup="true" LeaveInLocalAdmins="true">
<Username>Domain\svc.sp10FarmAdmin</Username>
<Password>SvcAcctPassword</Password>
<Email>WebMaster@domain.mil</Email>
</Account>
<CentralAdmin Provision="SP10_WebFrontEnd_1 SP10_WebFrontEnd_2">
<Database>AdminContent</Database>
<Port>2010</Port>
<UseSSL>false</UseSSL>
</CentralAdmin>
<Database>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
<DBPrefix>Portal_SP10</DBPrefix>
<ConfigDB>Config</ConfigDB>
</Database>
<Services>
<SandboxedCodeService Start="SP10_AppServer" />
<ClaimsToWindowsTokenService Start="SP10_AppServer" />
<SMTP Install="false" />
<OutgoingEmail Configure="true">
<SMTPServer>smtp.domain.mil</SMTPServer>
<EmailAddress>WebMaster@domain.mil</EmailAddress>
<ReplyToEmail>WebMaster@domain.mil</ReplyToEmail>
</OutgoingEmail>
</Services>
<ManagedAccounts>
<ManagedAccount CommonName="spservice">
<username>Domain\svc.sp10service</username>
<Password>SvcAcctPassword</Password>
</ManagedAccount>
<ManagedAccount CommonName="portalapppool">
<username>Domain\svc.sp10portalapppool</username>
<Password>SvcAcctPassword</Password>
</ManagedAccount>
<ManagedAccount CommonName="mysiteapppool">
<username>Domain\svc.sp10profileapppool</username>
<Password>SvcAcctPassword</Password>
</ManagedAccount>
<ManagedAccount CommonName="searchservice">
<username>Domain\svc.sp10search</username>
<Password>SvcAcctPassword</Password>
</ManagedAccount>
</ManagedAccounts>
<ObjectCacheAccounts>
<SuperUser>Domain\svc.sp10csu</SuperUser>
<SuperReader>Domain\svc.sp10csr</SuperReader>
</ObjectCacheAccounts>
<Logging>
<IISLogs Compress="true">
<Path></Path>
</IISLogs>
<ULSLogs Compress="true">
<LogLocation></LogLocation>
<LogDiskSpaceUsageGB>50</LogDiskSpaceUsageGB>
<DaysToKeepLogs></DaysToKeepLogs>
<LogCutInterval></LogCutInterval>
</ULSLogs>
<UsageLogs Compress="true">
<UsageLogDir></UsageLogDir>
<UsageLogMaxSpaceGB></UsageLogMaxSpaceGB>
<UsageLogCutTime></UsageLogCutTime>
</UsageLogs>
</Logging>
</Farm>

<WebApplications AddURLsToHOSTS="true">
<WebApplication type="Portal" name="Portal Home" applicationPool="Portal Home App Pool" applicationPoolAccount="Domain\svc.sp10portalapppool" url="http://SP10_WebFrontEnd_1" port="80" AddURLToLocalIntranetZone="true" databaseName="PORTAL_CONTENT" useClaims="false" useBasicAuthentication="false" useOnlineWebPartCatalog="false">
<Database>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
<SiteCollections>
<SiteCollection siteUrl="http://SP10_WebFrontEnd_1" owner="Domain\svc.sp10portalapppool" name="Portal Home" description="Portal Home Site" SearchUrl="http://SP10_WebFrontEnd_1/search" CustomTemplate="false" Template="SPSPORTAL#0" LCID="1033" Locale="en-us" Time24="false"></SiteCollection>
</SiteCollections>
</WebApplication>
<WebApplication type="MySiteHost" name="MySite Host" applicationPool="MySites" applicationPoolAccount="Domain\svc.sp10profileapppool" url="http://SP10_WebFrontEnd_1" port="8080" AddURLToLocalIntranetZone="true" databaseName="MYSITES_CONTENT" useClaims="false" useBasicAuthentication="false" useOnlineWebPartCatalog="false">
<Database>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
<ManagedPaths>
<ManagedPath relativeUrl="personal" explicit="false" />
<ManagedPath relativeUrl="sites" delete="true" />
</ManagedPaths>
<SiteCollections>
<SiteCollection siteUrl="http://SP10_WebFrontEnd_1:8080" owner="Domain\svc.sp10profileapppool" name="My Site Host" description="My Site Host" SearchUrl="http://SP10_WebFrontEnd_1:8080/search" CustomTemplate="false" Template="SPSMSITEHOST#0" LCID="1033" Locale="en-us" Time24="false"></SiteCollection>
</SiteCollections>
</WebApplication>
</WebApplications>
<ServiceApps>
<ManagedMetadataServiceApp Provision="SP10_AppServer" Name="Managed Metadata Service" ProxyName="Managed Metadata Service">
<Database>
<Name>METADATA</Name>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
</ManagedMetadataServiceApp>
<UserProfileServiceApp Provision="SP10_AppServer" Name="User Profile Service Application" ProxyName="User Profile Service Application" EnableNetBIOSDomainNames="false" StartProfileSync="true" CreateDefaultSyncConnection="false" SyncConnectionAccount="" SyncConnectionAccountPassword="">
<Database>
<ProfileDB>PROFILE</ProfileDB>
<SyncDB>SYNC</SyncDB>
<SocialDB>SOCIAL</SocialDB>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
</UserProfileServiceApp>
<EnterpriseSearchService Provision="SP10_AppServer" ContactEmail="WebMaster@domain.mil" ConnectionTimeout="60" AcknowledgementTimeout="60" ProxyType="Default" IgnoreSSLWarnings="false" InternetIdentity="Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot)" IndexLocation="C:\Program Files\Microsoft Office Servers\14.0\Data\Office Server\Applications" PerformanceLevel="PartlyReduced" Account="Domain\svc.sp10search" Password="SvcAcctPassword" ShareName="SearchIndex">
<EnterpriseSearchServiceApplications>
<EnterpriseSearchServiceApplication Name="Search Service Application" DatabaseServer="SP10_SQL_Server" DatabaseName="SEARCH" FailoverDatabaseServer="" Partitioned="false" Partitions="1" SearchServiceApplicationType="Regular" ContentAccessAccount="Domain\svc.sp10service" ContentAccessAccountPassword="SvcAcctPassword">
<ApplicationPool Name="SharePoint Search Application Pool" Account="Domain\svc.sp10search" Password="SvcAcctPassword" />
<CrawlServers>
<Server Name="SP10_AppServer" />
</CrawlServers>
<QueryServers>
<Server Name="SP10_WebFrontEnd_1 SP10_WebFrontEnd_2" />
</QueryServers>
<SearchQueryAndSiteSettingsServers>
<Server Name="SP10_WebFrontEnd_1 SP10_WebFrontEnd_2" />
</SearchQueryAndSiteSettingsServers>
<AdminComponent>
<Server Name="SP10_AppServer" />
<ApplicationPool Name="SharePoint Search Application Pool" Account="Domain\svc.sp10search" />
</AdminComponent>
<Proxy Name="Search Service Application" Partitioned="false">
<ProxyGroup Name="Default" />
</Proxy>
</EnterpriseSearchServiceApplication>
</EnterpriseSearchServiceApplications>
</EnterpriseSearchService>
<StateService Provision="SP10_AppServer" Name="State Service" ProxyName="State Service">
<Database>
<Name>STATE_SERVICE</Name>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
</StateService>
<WebAnalyticsService Provision="SP10_AppServer" Name="Web Analytics Service Application">
<Database>
<ReportingDB>WEBANALYTICS_REPORTING</ReportingDB>
<StagingDB>WEBANALYTICS_STAGING</StagingDB>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
</WebAnalyticsService>
<SPUsageService Provision="SP10_AppServer" Name="Usage and Health Data Collection">
<Database>
<Name>USAGE_HEALTH</Name>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
</SPUsageService>
<SecureStoreService Provision="SP10_AppServer" Name="Secure Store Service" ProxyName="Secure Store Service">
<Database>
<Name>SECURE_STORE</Name>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
</SecureStoreService>
<BusinessDataConnectivity Provision="SP10_AppServer" Name="Business Data Connectivity Service" ProxyName="Business Data Connectivity Service">
<Database>
<Name>BDC</Name>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
</BusinessDataConnectivity>
<WordAutomationService Provision="SP10_AppServer" Name="Word Automation Services" ProxyName="Word Automation Services">
<Database>
<Name>WORD_AUTOMATION</Name>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
</WordAutomationService>
</ServiceApps>
<EnterpriseServiceApps>
<ExcelServices Provision="SP10_AppServer" Name="Excel Services Application" UnattendedIDUser="svc.sp10Unattended" UnattendedIDPassword="SvcAcctPassword"></ExcelServices>
<VisioService Provision="SP10_AppServer" Name="Visio Graphics Service" ProxyName="Visio Graphics Service" UnattendedIDUser="svc.sp10Unattended" UnattendedIDPassword="SvcAcctPassword"></VisioService>
<AccessService Provision="false" Name="Access Services" ProxyName="Access Services"></AccessService>
<PerformancePointService Provision="true" Name="PerformancePoint Service" ProxyName="PerformancePoint Service" UnattendedIDUser="svc.sp10Unattended" UnattendedIDPassword="SvcAcctPassword">
<Database>
<Name>PerformancePoint</Name>
<DBServer>SP10_SQL_Server</DBServer>
<DBAlias Create="false" DBInstance="SERVER\INSTANCE" DBPort="" />
</Database>
</PerformancePointService>
</EnterpriseServiceApps>
<OfficeWebApps Install="false" ConfigFile="config-OWA.xml">
<ExcelService Provision="false" Name="Excel Web App" ProxyName="Excel Web App" UnattendedIDUser="" UnattendedIDPassword=""></ExcelService>
<WordViewingService Provision="false" Name="Word Viewing Service" ProxyName="Word Viewing Service"></WordViewingService>
<PowerPointService Provision="false" Name="PowerPoint Service Application" ProxyName="PowerPoint Service Application"></PowerPointService>
</OfficeWebApps>
<AdobePDF>
<iFilter Install="SP10_WebFrontEnd_1 SP10_WebFrontEnd_2" />
<Icon Configure="true" />
<MIMEType Configure="true" />
</AdobePDF>
<ForeFront Install="false" ConfigFile="answerfile-ForeFront.xml" />
</Configuration>

Jul 15, 2012 at 10:06 AM

Hi,

so, this looks like a tough one. But what I found first, you removed some of the the config lines after the install tag, so it is missing those:

<LoopbackCheck>true</LoopbackCheck>
<UnusedServices>true</UnusedServices>
<IEEnhancedSecurity>true</IEEnhancedSecurity>

But this really shouldn't be an issue. Also in your first post you write about the search not being setup properly. Reason is that search needs some different config. Above config shows:

<QueryServers> 
      <Server Name="SP10_WebFrontEnd_1 SP10_WebFrontEnd_2" /> 
</QueryServers> 
<SearchQueryAndSiteSettingsServers> 
      <Server Name="SP10_WebFrontEnd_1 SP10_WebFrontEnd_2" /> 
</SearchQueryAndSiteSettingsServers> 

Instead you have to enter each server indiviually like that:

<QueryServers> 
      <Server Name="SP10_WebFrontEnd_1" /> 
      <Server Name="SP10_WebFrontEnd_2" /> 
</QueryServers> 
<SearchQueryAndSiteSettingsServers> 
      <Server Name="SP10_WebFrontEnd_1" /> 
      <Server Name="SP10_WebFrontEnd_2" /> 
</SearchQueryAndSiteSettingsServers>

As I don't know the DOD hardening guides I have no idea what changed on the systems. Usually there should be no logins to your database with the hostname unless something runs as "local system". To really see if you're environment is having some issues I suppose to see this article about required permissions to SQL server: http://sharepointgeorge.com/2010/installing-sharepoint-2010-privilege-service-accounts/ and also please see if a manual installation using the wizzards is working.

Also, you wite you wiped the servers, does this include the database server as well? If not did you remove all the databases before rerunning the scripts? Otherwise the config will already be there.

Stefan

Jul 15, 2012 at 7:12 PM

Thank you Stefan.  The info on the Search settings is hugely helpful - it's tough to know which elements support a list of host names separated by spaces on your first use of the scripts.  Also, the insight that something is using the local system account gives me a place to begin investigating.  FYI - Yes, I wiped out the databases each time I uninstalled SPS 2010.

I did attempt a manual configuration after using the AutoSPInstaller to install the binaries and update - I just set it to "Pause after Install" then ran the Product Config Wizard.  I ran into some issues, but didn't have time to dig into the logs as it was late Friday and I had to leave (one of the constraints is that I can't even connect over VPN).  Are you recommending not to even use AutoSPInstaller to install SharePoint, but instead to use the Wizard for that part too?

I so appreciate your help - I have to get this resolved.  By the way, I'm not sure how the other lines about what to disable got deleted - but they are in my script. I must have accidentally deleted them when I cleaned the comments out for easier viewing.

Kirk

Jul 19, 2012 at 7:15 PM

So, after much frustration and tediously combing ULS logs and Application & Security Event logs, I identified a pattern that clearly indicated that SOMETHING was interfering with communication between the VM's to become SP servers and the SQL server.  Though I had indicated this as my suspicion over two weeks earlier, the evidence I presented (plus the escalation from including others in the CC list on the email) caused action to be taken.

I was then informed that while I was out for the 4th of July break the organization received orders from DoD to fully implement a combination HIPS/HBSS security solution. For anyone not familar with the acronyms, they are "Host Intrusion Prevention System" and "Host Based Security System". The culprit was the HBSS, which according to several DoD websites was developed to address known exploit traffic using an Intrusion Prevention System (IPS) and host firewall. See, the IPS uses algorithms to identify known definitions but also to identify possible new patterns. So, why did this system block the AutoSPInstaller from doing its work?...

Well, it appears by looking in the HIPS log that it recognized the use of "Wordpad.exe" to write the transcript of the installation as an old vulnerability where malicious software could use buffer overruns to take over systems. Apparently the HBSS built a profile of the AutoSPInstaller's PowerShell script and secretly stored the "NEW" definition of an attack in its host-based firewall - Jumping alive and "coming to the rescue" each time the AutoSPInstaller would attempt to use the FarmAdmin account to access my Configuration Database.  It took so long because for whatever reason, the personnel here simply would not act on my "suspicion" though it's all I had since the logging was completely vague.

Finally, once these two tools were set to be 'dormant' the scripts zipped through and setup the entire farm in under an hour.  I have been working on dry-runs of the migration of the current portal to the new environment, and gained a lot of credibility with the client now that my near three week old initial claim of what was likely wrong was spot on.  Thanks to Stefan for attempting to help, and who indeed provided me with confidence to press when he told me that these errors definitely indicated something wrong with the servers!!

Cheers!

Jul 20, 2012 at 6:40 AM

Glad, you finally got it resolved and thanks for the credits!

Stefan