tag:blogger.com,1999:blog-72498272734683080812019-02-22T23:14:07.468+13:00Daniel Ballinger's FishOfPrey.compublic class FishOfPrey : IDevelop
{
}FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.comBlogger445125tag:blogger.com,1999:blog-7249827273468308081.post-21608575637214622572018-10-03T22:40:00.000+13:002018-10-05T07:39:59.932+13:00Dreamforce 2018 Roundup / Summary<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-84lXt-lgzf0/W7LLCdFdt9I/AAAAAAAD-Xo/ejSzgtFwQZQuO2zbMW0ASO5HYSOpxIPswCKgBGAs/s1600/20180925_151729.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-84lXt-lgzf0/W7LLCdFdt9I/AAAAAAAD-Xo/ejSzgtFwQZQuO2zbMW0ASO5HYSOpxIPswCKgBGAs/s800/20180925_151729.jpg" width="750" data-original-width="1600" data-original-height="900" /></a></div>
<a href="https://github.com/afawcett/df18-dx-demo"></a>
<p>The recap for this Dreamforce is going to come across as a bit of a photo journal as I go back through my phones timeline to piece back together what I got up to. Somehow, even on my 5th time attending this conference things still went past at a frantic pace.</p>
<!--div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ZHcXO870RDs/W7LLCaAqGOI/AAAAAAAD-Xo/3AI2AjORsyICqwKOFOi_MtIxsBmw0lKNQCKgBGAs/s1600/20180925_151651.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-ZHcXO870RDs/W7LLCaAqGOI/AAAAAAAD-Xo/3AI2AjORsyICqwKOFOi_MtIxsBmw0lKNQCKgBGAs/s800/20180925_151651.jpg" width="750" data-original-width="1600" data-original-height="900" /></a></div-->
<h2 style="clear: both;">Table of Contents</h2>
<ol>
<li><a href="#preconference">Preconference</a></li>
<li><a href="#day1">Day One</a>
<ul>
<li>My Presentation</li>
<li>Main Keynote</li>
</ul>
</li>
<li><a href="#day2">Day Two</a>
<ul>
<li>The Lightning Platform Roadmap</li>
<li>Dreamfest</li>
</ul>
</li>
<li><a href="#day3">Day Three</a>
<ul>
<li>Developer Keynote</li>
<li>Meet the Developers</li>
<li>True to the Core</li>
</ul>
</li>
<li><a href="#day4">Day Four</a>
<ul>
<li>Mulesoft</li>
</ul>
</li>
<li><a href="#deploymentFish">Deployment Fish</a></li>
</ol>
<hr/>
<h2 style="clear: both;" id="preconference">Preconference</h2>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-5YZrikmghBg/W7ZeE6jEzJI/AAAAAAAD-ak/Cd0uMcM8ajc9FzdQKpsS7DBua6zE7RwPQCKgBGAs/s1600/20180924_155221.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-5YZrikmghBg/W7ZeE6jEzJI/AAAAAAAD-ak/Cd0uMcM8ajc9FzdQKpsS7DBua6zE7RwPQCKgBGAs/s320/20180924_155221.jpg" width="180" height="320" data-original-width="900" data-original-height="1600" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://3.bp.blogspot.com/-5hsto9sO_a0/W7PMLqR4siI/AAAAAAAD-YQ/2wNtlKbmFLkd17zgKH5k2Xg0jzb9dQlqgCKgBGAs/s1600/20180924_184545.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-5hsto9sO_a0/W7PMLqR4siI/AAAAAAAD-YQ/2wNtlKbmFLkd17zgKH5k2Xg0jzb9dQlqgCKgBGAs/s320/20180924_184545.jpg" width="180" height="320" data-original-width="900" data-original-height="1600" /></a></div>
<p>A visit to the Salesforce Tower for the Tooling Partner meeting. Then a quick visit to the Salesforce Park (<a href="https://sf.curbed.com/2018/9/25/17903758/transbay-transit-center-closed-salesforce-park">the day before it was closed due to structural damage</a>).</p>
<h2 id="day1" style="clear: both;">Day One</h2>
<h3><a href="https://success.salesforce.com/0693A000006jVIT">Testing Lightning Flow Automations</a></h3>
<ul>
<li>Usage of <code>Test.loadData()</code> to prep records for testing from a static resource.</li>
<li>Starting auto launched flows from a testing context.</li>
<li>In Winter'19 - Use the <code>FlowTestCoverage</code> and <code>FlowElementTestCoverage</code> from the Tooling API to get coverage details for flows. See <a href="https://releasenotes.docs.salesforce.com/en-us/winter19/release-notes/rn_forcecom_flow_test_coverage.htm">Track Process and Flow Test Coverage</a></li>
</ul>
<h3><a href="https://success.salesforce.com/myagenda?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WUY5QAO">DX Super Session</a> <a href="https://www.youtube.com/watch?v=AkkyKBfLQr8">(Video)</a></h3>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">New scratch org snapshots. In pre-release it is using the 0TT key prefix. Which is a Trailforce Template.<a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://t.co/d7EqiEO3yA">pic.twitter.com/d7EqiEO3yA</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1044644000316026880?ref_src=twsrc%5Etfw">September 25, 2018</a></blockquote>
<p>New scratch org "snapshots" to use as the starting point for creating additional scratch orgs. The snapshot can be used to pre-configure the scratch org in a known state. E.g. with the dependent packages installed and test data loaded. They are currently using the <a href="http://www.fishofprey.com/2011/09/obscure-salesforce-object-key-prefixes.html#0TT">0TT Trailforce Template</a> keyprefix. Note that any orgPreferences are additive over what is already present in the snapshot.</p>
<!--Snapshots for scratch orgs. Create a scratch org in a known initial shape. E.g. Sample data pre-loaded. Package dependencies already installed and configured.-->
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-jdspp-upOgI/W7FpAujYfwI/AAAAAAAD-Wk/XBsK4taphnEqCLgnfuVVlc-0DhLEHzQcQCKgBGAs/s1600/20180925_103803.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-jdspp-upOgI/W7FpAujYfwI/AAAAAAAD-Wk/XBsK4taphnEqCLgnfuVVlc-0DhLEHzQcQCKgBGAs/s320/20180925_103803.jpg" width="320" height="180" data-original-width="1600" data-original-height="900" alt="spinners" /></a></div>
<p>FORCE_SHOW_SPINNER and FORCE_SPINNER_DELAY <a href="https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_cli_env_variables.htm">environment options</a> that can make automating the CLI easier. They prevent the spinner coming back on the standard output.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-Bezh3zdM8uw/W7FpAl1kRXI/AAAAAAAD-Wk/sbUn6QbXZosAuIQYeG3j3r_czJ0FXDJDgCKgBGAs/s1600/20180925_110422.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-Bezh3zdM8uw/W7FpAl1kRXI/AAAAAAAD-Wk/sbUn6QbXZosAuIQYeG3j3r_czJ0FXDJDgCKgBGAs/s320/20180925_110422.jpg" width="320" height="180" data-original-width="1600" data-original-height="900" /></a></div>
<p><a href="https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_reference.meta/sfdx_cli_reference/cli_reference_force_package.htm#cli_reference_create">Naming</a> for unlocked package versions.</p>
<ul style="clear: both;">
<li>Source for the demo - <a href="https://github.com/afawcett/df18-dx-demo">https://github.com/afawcett/df18-dx-demo</a></li>
</ul>
<h3 style="clear: both;">My Presentation</h3>
<p>I've done a whole separate blog post to cover my talk - <a href="http://www.fishofprey.com/2018/09/happysoupmix.html">Dreamforce 2018 Presentation - Understand your Org shape via visualization of Metadata Component Dependencies</a>.</p>
<h3>Main keynote</h3>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">It’s Keynote time! <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://twitter.com/hashtag/MVP?src=hash&amp;ref_src=twsrc%5Etfw">#MVP</a> Thanks <a href="https://twitter.com/hollygfirestone?ref_src=twsrc%5Etfw">@hollygfirestone</a> <a href="https://twitter.com/lexpisani?ref_src=twsrc%5Etfw">@lexpisani</a> &amp; team for herding us into our seats 😍 <a href="https://t.co/Onh6W3bDIF">pic.twitter.com/Onh6W3bDIF</a></p>&mdash; Kristi Campbell Guzman (@KristiForce) <a href="https://twitter.com/KristiForce/status/1044703378637385729?ref_src=twsrc%5Etfw">September 25, 2018</a></blockquote>
<p>Demo's of Einstein voice to convert speech, to text, to actions within Salesforce.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Einstein Voice is FREE!!!! Control Salesforce using your voice!!</p>&mdash; David K. Liu (@dvdkliu) <a href="https://twitter.com/dvdkliu/status/1044727090224877568?ref_src=twsrc%5Etfw">September 25, 2018</a></blockquote>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/MK_mrT7Kmw9ygMrhhLWO-rM-h-vRsTiKmi2t5-f-1-hybZ_42ZOF5wCpDOqm1kYJ84EByYyOYNtvZ35lIfse7w6P2OhZLX6949Nz1m2maBhwtSX1EkWaBSzypTzKkfLfoMFT123c" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/MK_mrT7Kmw9ygMrhhLWO-rM-h-vRsTiKmi2t5-f-1-hybZ_42ZOF5wCpDOqm1kYJ84EByYyOYNtvZ35lIfse7w6P2OhZLX6949Nz1m2maBhwtSX1EkWaBSzypTzKkfLfoMFT123c" width="320" height="218" data-original-width="666" data-original-height="454" /></a></div>
<p>Customer 360 - Create a canonical view of the customer across Marketing Cloud, Commerce Cloud, and Service Cloud.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Oh. I need one of those for a current project. Where can I get a <a href="https://twitter.com/hashtag/df18?src=hash&amp;ref_src=twsrc%5Etfw">#df18</a> Customer 360 balloon. <a href="https://t.co/B56R3GCu9r">pic.twitter.com/B56R3GCu9r</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1044722909086867457?ref_src=twsrc%5Etfw">September 25, 2018</a></blockquote>
<h3>MVP Party</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-kpqEbg8-rrg/W7PtiYSvqhI/AAAAAAAD-Yc/mAJqfnMJBZoPmkwOsuj1VSb4RFwqVkhSwCKgBGAs/s1600/20180925_201212.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-kpqEbg8-rrg/W7PtiYSvqhI/AAAAAAAD-Yc/mAJqfnMJBZoPmkwOsuj1VSb4RFwqVkhSwCKgBGAs/s800/20180925_201212.jpg" width="750" data-original-width="1600" data-original-height="764" /></a></div>
<p>This year it was at the Tonga Room &amp; Hurricane Bar. The band on the boat was 👌.</p>
<h2 id="day2">Day Two</h2>
<h3>The Lightning Platform Roadmap - <a href="https://success.salesforce.com/myagenda?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WUYBQA4">Part I</a> and <a href="https://success.salesforce.com/myagenda?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WUYCQA4">Part II</a></h3>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Changes coming to Apex from the roadmap <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> session. Yay for FLS/CRUD enforcement getting better native support. <a href="https://t.co/Eyp6VVwXqH">pic.twitter.com/Eyp6VVwXqH</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1044978778684841984?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Support for non scratch org dev from the CLI. Plus pre and post command hooks. <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://t.co/TflpFCzmKU">pic.twitter.com/TflpFCzmKU</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1044979203920166912?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Apex extract method and other refactoring support. Preview of an entirely web based IDE. <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://t.co/4rgQrDOVMF">pic.twitter.com/4rgQrDOVMF</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1044979555130257408?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Sandbox cloning. Targeting both a sandbox and scratch of to a specific release version. <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://t.co/B35dMvA12e">pic.twitter.com/B35dMvA12e</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1044979906717732869?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">The ability to create a snapshot of a Scratch org to use as a template for creating new scratch org. Useful to get all the pre setup and data load done.</p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1044980428640145408?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">2nd generation packages for managed packages, along with a migration path. <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://t.co/fnVUNFzOFK">pic.twitter.com/fnVUNFzOFK</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1044980814713249793?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Integration roadmap via <a href="https://twitter.com/extraidea?ref_src=twsrc%5Etfw">@extraidea</a> at <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://t.co/K3cCTDi4iq">pic.twitter.com/K3cCTDi4iq</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1044991921561583616?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<h2 style="clear: both;" id="day3">Day Three</h2>
<h3><a href="https://youtu.be/C6Cp2P5Eb7Y">Developer Keynote</a></h3>
<p>Lightning Developer Pro Sandboxes - Spin up faster than a traditional Sandbox. Approx 5 minutes to activate (versus a few hours to even days).</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-9RmM2g8mrOU/W6_JkIi_dyI/AAAAAAAD-TA/sj-AxTcFNV0gKDszdPzfYZWJEWltyS7agCLcBGAs/s1600/20180927_134951.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-9RmM2g8mrOU/W6_JkIi_dyI/AAAAAAAD-TA/sj-AxTcFNV0gKDszdPzfYZWJEWltyS7agCLcBGAs/s640/20180927_134951.jpg" width="640" height="251" data-original-width="1600" data-original-height="628" /></a></div>
<p>Another example of a named package version.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Using <a href="https://twitter.com/MuleSoft?ref_src=twsrc%5Etfw">@MuleSoft</a> anypoint to aggregate multiple APIs into a single endpoint at the <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> dev keynote. <a href="https://t.co/ems78qtGXS">pic.twitter.com/ems78qtGXS</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1045411745420926976?ref_src=twsrc%5Etfw">September 27, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Spring 19 roadmap from the <a href="https://twitter.com/SalesforceDevs?ref_src=twsrc%5Etfw">@SalesforceDevs</a> <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> keynote <a href="https://t.co/KVGoIP8nqN">pic.twitter.com/KVGoIP8nqN</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1045416474335006720?ref_src=twsrc%5Etfw">September 27, 2018</a></blockquote>
<!--h3>Development Roadmap for Spring '19</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-E57S2qYFg0k/W6_IkVfbhAI/AAAAAAAD-Sw/gJeroGxNw4QRZKI4pKXp35__4wmkp66DgCLcBGAs/s1600/20180927_135305.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-E57S2qYFg0k/W6_IkVfbhAI/AAAAAAAD-Sw/gJeroGxNw4QRZKI4pKXp35__4wmkp66DgCLcBGAs/s640/20180927_135305.jpg" width="640" height="348" data-original-width="1600" data-original-height="871" /></a></div-->
<h3>True to the Core</h3>
<p>Lots of coverage of how Salesforce will work towards reinvigorating the idea exchange to handle the current scale. In particular, limiting each users ability to vote of every possible idea. Forcing them to focus on what is most important to them. Salesforce will then commit to working on the highest rated ideas.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">So everyone who saw True to the Core at <a href="https://twitter.com/Dreamforce?ref_src=twsrc%5Etfw">@Dreamforce</a> today, what do you think of our IdeaExchange plan? Send me feedback! CC <a href="https://twitter.com/parkerharris?ref_src=twsrc%5Etfw">@parkerharris</a> <a href="https://twitter.com/jennyst0?ref_src=twsrc%5Etfw">@jennyst0</a> <a href="https://twitter.com/mike945778?ref_src=twsrc%5Etfw">@mike945778</a></p>&mdash; Bret Taylor (@btaylor) <a href="https://twitter.com/btaylor/status/1045128979886133249?ref_src=twsrc%5Etfw">September 27, 2018</a></blockquote>
<h3>Everything that's Awesome with Apex</h3>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-jV-rbrO2gow/W7QEnRI5PMI/AAAAAAAD-Yo/7CLOBTRdB3IQlHROGNs9USYelDv8GuX_wCLcBGAs/s1600/slide_14.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-jV-rbrO2gow/W7QEnRI5PMI/AAAAAAAD-Yo/7CLOBTRdB3IQlHROGNs9USYelDv8GuX_wCLcBGAs/s320/slide_14.jpg" width="320" height="180" data-original-width="1024" data-original-height="577" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-ubRtQE72ZxY/W7QEnryOSBI/AAAAAAAD-Ys/wHPFZ4fYlD4ScZoqdVSSB1OcdZzj4uicwCLcBGAs/s1600/slide_15.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-ubRtQE72ZxY/W7QEnryOSBI/AAAAAAAD-Ys/wHPFZ4fYlD4ScZoqdVSSB1OcdZzj4uicwCLcBGAs/s320/slide_15.jpg" width="320" height="180" data-original-width="1024" data-original-height="577" /></a></div>
<p>(<a href="https://speakerdeck.com/ca_peterson/df18-everything-thats-awesome-with-apex">Slides</a>)</p>
<p>Roadmap for Spring '19 and Summer '19</p>
<ul>
<li>Exposing the Org Limit details in Apex. E.g. Daily Asysnc Apex executions.</li>
<li>Support for creating Scratch Orgs with access to Platform Cache.</li>
<li>Support for "rename symbol" refactoring in Apex via the Apex Language Services, and by extension Visual Studio Code. Longer term, this may expand to implementing interfaces and extracting variables.</li>
<li style="clear: both;"><div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-ww-ikVTu5f8/W7QEnov-UZI/AAAAAAAD-Yw/VKBluii18o8Am1o_VLNEdOgOugbGpLBbQCLcBGAs/s1600/slide_20.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-ww-ikVTu5f8/W7QEnov-UZI/AAAAAAAD-Yw/VKBluii18o8Am1o_VLNEdOgOugbGpLBbQCLcBGAs/s320/slide_20.jpg" width="320" height="180" data-original-width="1024" data-original-height="577" /></a></div>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-qZPCSpLLQ-U/W7QEoSGaB9I/AAAAAAAD-Y0/aFO_1CcMgBERA-zH_mvQmjI2zb1lsz7DACLcBGAs/s1600/slide_21.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-qZPCSpLLQ-U/W7QEoSGaB9I/AAAAAAAD-Y0/aFO_1CcMgBERA-zH_mvQmjI2zb1lsz7DACLcBGAs/s320/slide_21.jpg" width="320" height="180" data-original-width="1024" data-original-height="577" /></a></div>Improved support for enforcing Field Level Security (FLS) and Create,Read,Update,Delete (CRUD) in Apex. Goal is to reduce the amount of processing that was historically required to do this. It's particularly important for those publishing on the App Exchange.
https://twitter.com/FishOfPrey/status/1044978778684841984</li>
</ul>
<h3 style="clear: both;">Quip Party</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-hfiBaxh0MQ8/W6_GtHIhVsI/AAAAAAAD-SQ/O0oQ5KxPO3w9D7z42W0Q4gdhJxWk-dD4ACLcBGAs/s1600/20180927_190607.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-hfiBaxh0MQ8/W6_GtHIhVsI/AAAAAAAD-SQ/O0oQ5KxPO3w9D7z42W0Q4gdhJxWk-dD4ACLcBGAs/s640/20180927_190607.jpg" width="640" height="360" data-original-width="1600" data-original-height="900" /></a></div>
<p>The August Hall was a great venue. Including a bowling alley and mini arcade downstairs.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Some bowling with <a href="https://twitter.com/shunkosa?ref_src=twsrc%5Etfw">@shunkosa</a> and <a href="https://twitter.com/jan_vdv?ref_src=twsrc%5Etfw">@jan_vdv</a> at the <a href="https://twitter.com/quip?ref_src=twsrc%5Etfw">@Quip</a> party before Macklemore comes on. <a href="https://t.co/dAyWjbutjj">pic.twitter.com/dAyWjbutjj</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1045517455408357378?ref_src=twsrc%5Etfw">September 28, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Macklemore with special guest <a href="https://twitter.com/SteveMoForce?ref_src=twsrc%5Etfw">@SteveMoForce</a> <a href="https://t.co/z7P5SibBCZ">pic.twitter.com/z7P5SibBCZ</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1045536856849281025?ref_src=twsrc%5Etfw">September 28, 2018</a></blockquote>
<h2 id="day4">Day 4</h2>
<h3>MuleSoft</h3>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Taking a RAML definition in <a href="https://twitter.com/MuleSoft?ref_src=twsrc%5Etfw">@MuleSoft</a> anypoint and creating a mock web service. Outputs an endpoint URL to call. Can also convert definition to the Open API format. <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://t.co/XPzo7uuPLT">pic.twitter.com/XPzo7uuPLT</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1045724665409224704?ref_src=twsrc%5Etfw">September 28, 2018</a></blockquote>
<hr/>
<h2 id="deploymentFish">Deployment Fish Bottle Openers</h2>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">I said I&#39;d create a bespoke <a href="https://twitter.com/deploymentfish?ref_src=twsrc%5Etfw">@deploymentfish</a> bottle opener if <a href="https://twitter.com/SalesforceDevs?ref_src=twsrc%5Etfw">@SalesforceDevs</a> delivered the switch statement for Apex last year at <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a>. They delivered, so I did too at <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a>. Thanks <a href="https://twitter.com/slettehaugh?ref_src=twsrc%5Etfw">@slettehaugh</a><br>,<a href="https://twitter.com/ca_peterson?ref_src=twsrc%5Etfw">@ca_peterson</a>, and team. <a href="https://t.co/OdFH505GMZ">https://t.co/OdFH505GMZ</a> <a href="https://t.co/9ulWwlKL2M">https://t.co/9ulWwlKL2M</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1045077751403663360?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Highlights of <a href="https://twitter.com/Dreamforce?ref_src=twsrc%5Etfw">@Dreamforce</a> day 1 included meeting <a href="https://twitter.com/FishOfPrey?ref_src=twsrc%5Etfw">@FishOfPrey</a> and getting my own deployment fish. <br><br>What were your top <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> moments? <a href="https://t.co/CdYh1DbxZJ">pic.twitter.com/CdYh1DbxZJ</a></p>&mdash; Carl Brundage (@carlbrundage) <a href="https://twitter.com/carlbrundage/status/1044819624611762177?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">TIL that <a href="https://twitter.com/deploymentfish?ref_src=twsrc%5Etfw">@deploymentfish</a> are UV sensitive. <a href="https://t.co/0HZQYjHW3m">pic.twitter.com/0HZQYjHW3m</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1045390243841507329?ref_src=twsrc%5Etfw">September 27, 2018</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Forgot this in my shirt pocket. TSA initially thought it was some kind of blade or razor. Another reason I will always treasure my <a href="https://twitter.com/hashtag/DeploymentFish?src=hash&amp;ref_src=twsrc%5Etfw">#DeploymentFish</a>. Thanks, <a href="https://twitter.com/FishOfPrey?ref_src=twsrc%5Etfw">@FishOfPrey</a>! <a href="https://t.co/INHcjkU3za">pic.twitter.com/INHcjkU3za</a></p>&mdash; peter chittum (@pchittum) <a href="https://twitter.com/pchittum/status/1045860337319333889?ref_src=twsrc%5Etfw">September 29, 2018</a></blockquote>
<h2>To catch up on</h2>
<ul>
<li><a href="https://success.salesforce.com/myagenda?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A0000022F5tQAE">Be An Efficient Salesforce Developer with VS Code</a> (<a href="https://www.youtube.com/watch?v=hw9LBvjo4PQ">Video</a>)</li>
<li>Change Data Capture</li>
</ul>
<h2>See Also</h2>
<ul>
<li><a href="https://www.salesforce.com/blog/2018/10/dreamforce-18-ultimate-content-guide.html">The Dreamforce 2018 Ultimate Content Guide</a></li>
</ul>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-62968412111201789492018-09-26T08:40:00.000+12:002018-10-09T08:49:53.996+13:00Dreamforce 2018 Presentation - Understand your Org shape via visualization of Metadata Component Dependencies<p>At Dreamforce this year I gave a <a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WWXOQA4">theater presentation</a> on using the Metadata Component Dependency Pilot API along with <a href="https://gephi.org/">Gephi</a> to explore ways of untangling the "Happy Soup" that makes up a Salesforce org.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Thanks <a href="https://twitter.com/FishOfPrey?ref_src=twsrc%5Etfw">@FishOfPrey</a> for your epic talk about visualization of metadata component dependencies <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://twitter.com/absi_nv?ref_src=twsrc%5Etfw">@absi_nv</a> <a href="https://t.co/IwMdgG6YZf">pic.twitter.com/IwMdgG6YZf</a></p>&mdash; Samuel Moyson (@SamuelMoyson) <a href="https://twitter.com/SamuelMoyson/status/1044758028568805376?ref_src=twsrc%5Etfw">September 26, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>In this post I'll cover the contents of that talk and expanded on some areas that I'd like to cover in more depth.</p>
<h2>Session recording</h2>
<iframe id="ytplayer" type="text/html" width="640" height="360"
src="https://www.youtube.com/embed/6sIRJSdoCqo?autoplay=0&origin=http://www.fishofprey.com"
frameborder="0"></iframe>
<h2>Why Visualize an Orgs Metadata?</h2>
<div class="separator" style="clear: both; text-align: center;">
<figure style="clear: right; float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://en.wikipedia.org/wiki/Anscombe's_quartet" imageanchor="1" style="clear: margin-bottom: 1em;"><img border="0" src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Anscombe's_quartet_3.svg/425px-Anscombe's_quartet_3.svg.png" width="320" height="233" data-original-width="425" data-original-height="309"></a>
<figcaption>All four sets are identical when examined using simple<br>summary statistics, but vary considerably when graphed - <a href="https://en.wikipedia.org/wiki/Anscombe's_quartet">Source</a></figcaption>
</figure>
</div>
<p>If you've ever seen the <a href="https://en.wikipedia.org/wiki/Anscombe's_quartet">Anscombe's quartet</a> you'd know a compelling reason to visualize information in addition to straight out numerical analysis. Visualizations can reveal patterns and features in the data that otherwise might not be readily apparent.</p>
<p>Additionally, when we start drilling into <b>all </b>the metadata in an org and the relationships within that metadata we can very quickly be overwhelmed by the share volume of it. There can be thousands of metadata items and many more relationships between them.</p>
<h3 style="clear: both;">The Soup and the Polar Bear</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-Hy6z8jzgmWI/W6MuoI2Mj4I/AAAAAAAD9oU/qqfa7PZYtnU_bdcg60PenFKJSPR1TTqhACLcBGAs/s1600/HappySoupSlide.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-Hy6z8jzgmWI/W6MuoI2Mj4I/AAAAAAAD9oU/qqfa7PZYtnU_bdcg60PenFKJSPR1TTqhACLcBGAs/s800/HappySoupSlide.png" width="750" data-original-width="1213" data-original-height="679"></a></div>
<p>The concept of the metadata in your production org existing as a <a href="https://youtu.be/Pf33nrsqZOc?t=5m37s">"Happy Soup"</a> was introduced at TrailheadDX 2017 by Wade Wegner. It describes the scenario where all the conceptual apps that make up your production org intermingle without any strongly defined boundaries between them. Essentially, the org becomes one big melting pot of metadata. This has been the status quo for some time now (if you ignore managed packages and the relatively new addition of Packaging 2.0).</p>
<div class="separator" style="clear: both; text-align: center;">
<figure style="clear: right; float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://3.bp.blogspot.com/-SOyzwjABkN8/W6NoNtgt1sI/AAAAAAAD9og/YpaPKt6j5xYdYoy4Uot4Y51nAPUxe4bDwCLcBGAs/s1600/MarvellousMedicine.jpg" imageanchor="1" style="clear: margin-bottom: 1em;"><img border="0" src="https://3.bp.blogspot.com/-SOyzwjABkN8/W6NoNtgt1sI/AAAAAAAD9og/YpaPKt6j5xYdYoy4Uot4Y51nAPUxe4bDwCLcBGAs/s400/MarvellousMedicine.jpg" width="400" height="400" data-original-width="628" data-original-height="628"></a>
<figcaption>Soup chef at work on your production org - a Marvelous Metadata Mixing Pot</figcaption>
</figure>
</div>
<p>Now, take one of the production orgs that I work in day-to-day as an example. It was forged <b>11 years ago</b> when the seasonal release logo was a polar bear and the API version was v8.0. Ever since then it's been accumulating metadata changes as multiple people have come and gone. Each person <!--(Soup Chef, if you will)--> leaving a distinct mark on the metadata that makes up that org.</p>
<p>Back to our soup metaphor, that org is a decade old soup. Different soup chefs have been mixing ingredients in via a pinch of declarative UI changes, a dash of changes sets, a smidgen of metadata API deploys, and maybe the odd unmanaged package thrown in for good measure. Over the years an org can seem less like a carefully conceived soup recipe and more like a marvelous concoction with everything thrown in and reacting together in weird and wonderful ways.</p>
<p>Which kind of leaves you wondering - If you were presented with just the resulting happy soup, how would you identify the ingredients that went into it and how they need to interact so you could replicate it again? How can we unmake the happy soup?</p>
<p>The goal of our metadata visualizations will be to use the minds ability to quickly perceive and derive meaning from details such as size, color, position, and proximity to find insights that might not be apparent in tabular data. That might sound complicated, but the objective is to make otherwise complicated relationships readily apparent.</p>
<blockquote>“By visualizing information, we turn it into a landscape that you can explore with your eyes. A sort of information map. And when you’re lost in information, an information map is kind of useful.” - <a href="https://www.ted.com/talks/david_mccandless_the_beauty_of_data_visualization/transcript?language=en">David McCandless</a></blockquote>
<p>So, we know we want to make visualizations of the metadata and the dependencies within, but where do we start gathering that data?</p>
<h2 style="clear: both;">The Metadata Component Dependency API</h2>
<p>Luckily for us the new Metadata Dependencies pilot introduced in Summer '18 at TrailheadDX allows for the use of Tooling API SOQL queries to quickly find dependencies between metadata components. You can signup for the pilot via your AE as per the release notes: <a href="https://releasenotes.docs.salesforce.com/en-us/summer18/release-notes/rn_metadata_metadatacomponentdependency.htm">Untangle Your Dependencies with MetadataComponentDependency Queries (Pilot)</a></p>
<p>The benefit of this pilot is that it is <i>much</i> faster and easier than trying to assemble the same information yourself. I've tried extracting similar details via the <code>ApexClassMember.SymbolTable</code>. While possible, it is a path that is thwart with multiple API calls per ApexClass. And at the end of all that you will still only have a fraction of the possible relationships covered.</p>
<p>Once you are in the pilot it is pretty simple to use. A single SOQL query can bring back the majority of relationships for an org. For example, using the Tooling API via the sfdx cli:</p>
<pre style="white-space: pre-wrap;">sfdx force:data:soql:query --usetoolingapi --query "SELECT MetadataComponentId, MetadataComponentName, MetadataComponentType, RefMetadataComponentId, RefMetadataComponentName, RefMetadataComponentType FROM MetadataComponentDependency Where RefMetadataComponentType = 'CustomField'" -u devorg</pre>
<a href="https://1.bp.blogspot.com/-y6AbNR8jDT8/W6Nu1SVju-I/AAAAAAAD9o0/Zab1lOrcnLkX-u3kLdqVjv3zogqhnf4KwCLcBGAs/s1600/SoqlQuery.png" imageanchor="1"><img border="0" src="https://1.bp.blogspot.com/-y6AbNR8jDT8/W6Nu1SVju-I/AAAAAAAD9o0/Zab1lOrcnLkX-u3kLdqVjv3zogqhnf4KwCLcBGAs/s1600/SoqlQuery.png" width="760" data-original-width="1600" data-original-height="540"></a>
<!--a href="https://3.bp.blogspot.com/-rVFJu0qF1fs/W6Nt-vBU5GI/AAAAAAAD9os/swGPked4138b5LTpVIRzrzrONan2TskRwCLcBGAs/s1600/SOQLUsage.gif" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-rVFJu0qF1fs/W6Nt-vBU5GI/AAAAAAAD9os/swGPked4138b5LTpVIRzrzrONan2TskRwCLcBGAs/s1600/SOQLUsage.gif" width="760" data-original-width="1600" data-original-height="612" /></a-->
<h3>Supported Dependencies</h3>
<p>As this is a pilot feature the types of dependencies it can reveal isn't 100% yet. Below is a small subset of what can and can't be tracked: (as at the time of posting) </p>
<table border="1">
<tbody><tr><th>Type</th><th>Used Id</th><th>Supported</th></tr>
<tr><td>CustomObject</td><td>Apex Class</td><td>✅</td></tr>
<tr><td>CustomObject</td><td>Process Builder</td><td>✅</td></tr>
<tr><td>CustomObject</td><td>Lookup Relationship Field</td><td>❌</td></tr>
<tr><td>CustomObject</td><td>Reports</td><td>❌</td></tr>
<tr><td>CustomField</td><td>Layout</td><td>✅</td></tr>
<tr><td>CustomField</td><td>Reports</td><td>❌</td></tr>
<tr><td>Flow</td><td>Process Builder</td><td>✅</td></tr>
<tr><td>Flow</td><td>Flow</td><td>❌</td></tr>
</tbody></table>
<p>You can see the complete list of supported metadata relationships in <a href="https://github.com/afawcett/dependencies-sample">https://github.com/afawcett/dependencies-sample</a></p>
<h3>Augmenting the Graph</h3>
<p>Using the results from the Metadata Component Dependency query as a starting point we can then augment the resulting graph with additional metrics. Here are some of the additional details that can be merged in as meta-metadata.</p>
<ul>
<li>CustomField.Metadata - can be used to show lookup relationships</li>
<li>ApexClass.ApiVersion - useful for finding Apex classes with mismatched API versions</li>
<li>The code coverage results for each Apex class</li>
<li>The number of lines of code in each Apex Class</li>
<li>Where available, the created date for the Metadata</li>
<li>The number of records in a CustomObject</li>
<li>The number of Triggers per sObject</li>
</ul>
<h2>Tooling to create the Dependency Graph</h2>
<a href="https://1.bp.blogspot.com/-jtXayVZO5aU/W6NwPk5L1wI/AAAAAAAD9pA/5yAlK09TVoYy9Gud7EUTdayt35ugvsHewCLcBGAs/s1600/PathToVisualization.png" imageanchor="1"><img border="0" src="https://1.bp.blogspot.com/-jtXayVZO5aU/W6NwPk5L1wI/AAAAAAAD9pA/5yAlK09TVoYy9Gud7EUTdayt35ugvsHewCLcBGAs/s800/PathToVisualization.png" width="760" data-original-width="1600" data-original-height="900"></a>
<p>The first part of the visualization process will be to query the MetadataComponentDependency records and convert the results into a graph. I didn't go into this step in any detail in my talk as it isn't very interesting to watch. At it's simplest, you can use one of the following options.</p>
<h3>CLI Generation of the Gephi graph</h3>
<pre style="white-space: pre-wrap;">NOTE: At the time of publishing the Github repo wasn't fully up to date with the latest public build. I'll aim to have this done shortly after Dreamforce.</pre>
<p>Run the following command:</p>
<pre>sfdx fitdx:dependencies:report -u DeveloperOrg &gt; DeveloperOrg.gexf</pre>
<p>And that's pretty much it. The command from the <a href="https://github.com/FuseInfoTech/FitCLI">FitCLI</a> plugin will run the query, build the graph, and export it ready for opening in Gephi. You only need to point it at the correct Salesforce Org and redirect the output into a file.</p>
<p>If you want to add the augmented graph metrics you can supply the <code>-a</code> option. Note that this will take <i>significantly longer</i> to complete as it involves a number of additional tooling API queries. It's also beneficial to run all the Apex Tests before exporting the graph so it has complete coverage details.</p>
<h3>GUI option for creating the Gephi file</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-gdlsZmwCNt4/W6RHkrDitrI/AAAAAAAD9pw/uokoF4GU0f42f97bPXyTQ9WBPS3xEJcCgCLcBGAs/s1600/MetadataDependencies.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-gdlsZmwCNt4/W6RHkrDitrI/AAAAAAAD9pw/uokoF4GU0f42f97bPXyTQ9WBPS3xEJcCgCLcBGAs/s800/MetadataDependencies.PNG" data-original-width="852" data-original-height="154" width="750"></a></div>
<p>As an alternative to using the command line you can also using the <a href="https://www.fuseit.com/explorer">FuseIT SFDC Explorer</a> from v3.10 onwards. This now includes the <b>Metadata Dependencies</b> tab to generate Gephi file. It's pretty much exactly the same as the command line version and will prompt for where you want to save the resulting file.</p>
<p>Ideally with either the CLI or the GUI options the target org will have access to <code>MetadataComponentDependency</code> records. If that isn't the case you can still use the augmented option to get some graph details. It won't be as fast or as complete, but you will get a good deal of the relationships based on other data exposed via the Tooling API.</p>
<!--p>Note that you can still use the graph generation tooling below without access to the pilot. It won't be as fast or as complete, but you will get a good deal of the relationships based on other data exposed via the Tooling API.</p-->
<h2>Examples</h2>
<p>Now we can get into the interesting part of loading the graphs into <a href="https://gephi.org/">Gephi</a> to see what we can discover. You can open the <code>.gexf</code> files directly into Gephi and then start manipulating them. </p>
<!--details on the filters being applied-->
<p>This video quickly goes through the Field Usage and Packages examples that are expanded on below.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/tDerY7OpSp4" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
<h3>Field Usage</h3>
<p>Lets say we want to find what will be impacted if changes are made to the <code>Property__c.Address__c</code> field in the <a href="https://github.com/forcedotcom/sfdx-dreamhouse">Dreamhouse app</a>. The steps would be as follows:</p>
<ol>
<li>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-WCEm_Ho0K6E/W6RWRFxmFII/AAAAAAAD9p8/4fyBHy8h8bMRm5i-z5m16wc9aFcuqk5QgCLcBGAs/s1600/FIeldUsage_PartitionByType.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-WCEm_Ho0K6E/W6RWRFxmFII/AAAAAAAD9p8/4fyBHy8h8bMRm5i-z5m16wc9aFcuqk5QgCLcBGAs/s400/FIeldUsage_PartitionByType.PNG" width="221" height="400" data-original-width="251" data-original-height="455"></a></div>
Partition the Node color by Type. This will make it more apparent what type of nodes you are looking at.
</li>
<li style="clear: both;">
<div class="separator" style="text-align: center;">
<a href="https://3.bp.blogspot.com/-gK6bc_pjE-s/W6RX5SGtp1I/AAAAAAAD9qI/dlMMIVw0nSwyJ2VA_ZUNQmqUQE9Kc2q6gCLcBGAs/s1600/FIeldUsage_NetworkDiameter.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-gK6bc_pjE-s/W6RX5SGtp1I/AAAAAAAD9qI/dlMMIVw0nSwyJ2VA_ZUNQmqUQE9Kc2q6gCLcBGAs/s400/FIeldUsage_NetworkDiameter.PNG" width="400" height="360" data-original-width="229" data-original-height="206"></a>
<a href="https://2.bp.blogspot.com/-lWEQP5uQp-E/W6RYgi72PwI/AAAAAAAD9qQ/vrmkB_RTHZ4DGOu9qJ5h4u4KavJ1PyFiACLcBGAs/s1600/FIeldUsage_FieldSize.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-lWEQP5uQp-E/W6RYgi72PwI/AAAAAAAD9qQ/vrmkB_RTHZ4DGOu9qJ5h4u4KavJ1PyFiACLcBGAs/s400/FIeldUsage_FieldSize.PNG" width="400" height="232" data-original-width="252" data-original-height="146"></a></div>
Size the nodes based on the <b>Betweenness Centrality</b>. Which is a fancy way of saying "make the nodes that appear on the shortest paths between other nodes bigger". This is one possible indication of how important a node is to all the other nodes in the graph. Before we can do this we need to run the <b>Network Diameter</b> statistics.
</li>
<li style="clear: both;">
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-1Yqkue5Rp-0/W6RaJp6iVYI/AAAAAAAD9qc/hrLz24WGtfU0ekvMEGPxEwGCIurwMXdXgCLcBGAs/s1600/FIeldUsage_Layout.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-1Yqkue5Rp-0/W6RaJp6iVYI/AAAAAAAD9qc/hrLz24WGtfU0ekvMEGPxEwGCIurwMXdXgCLcBGAs/s400/FIeldUsage_Layout.PNG" width="306" height="400" data-original-width="249" data-original-height="325"></a></div>
Apply a layout to arrange the nodes. I've used the "Force Atlas" layout with the <i>Repulsion strength</i> set to 1500 and <i>Adjust by Sizes</i> checked.
</li>
<li>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-LPa8WPRoRb0/W6Rbzpku4sI/AAAAAAAD9qo/cgHM5EwZgpQ5LIrCiSA_vgcdTiwFDq-8QCLcBGAs/s1600/FIeldUsage_ShowNodeLabels.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-LPa8WPRoRb0/W6Rbzpku4sI/AAAAAAAD9qo/cgHM5EwZgpQ5LIrCiSA_vgcdTiwFDq-8QCLcBGAs/s400/FIeldUsage_ShowNodeLabels.PNG" width="400" height="38" data-original-width="544" data-original-height="52"></a></div>
Show the Node Labels and scale to fit
</li>
<li><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-VUGNBT9yfaA/W6Rcgelh51I/AAAAAAAD9qw/qW4ISAqsdMUJhHM4_I6TjNJ7c9I9ZkANACLcBGAs/s1600/FieldUsage_Address.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-VUGNBT9yfaA/W6Rcgelh51I/AAAAAAAD9qw/qW4ISAqsdMUJhHM4_I6TjNJ7c9I9ZkANACLcBGAs/s400/FieldUsage_Address.PNG" width="400" height="74" data-original-width="1038" data-original-height="192"></a></div>
Switch to the <b>Data Table</b> window and filter to the Address field.
</li>
<li><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-_lMfPZ1YWMo/W6RdkQ3p9PI/AAAAAAAD9q8/SL98icAn6vUD_FBrjruJHFeKKyQYc1bswCLcBGAs/s1600/FieldUsage_AddressUsage.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-_lMfPZ1YWMo/W6RdkQ3p9PI/AAAAAAAD9q8/SL98icAn6vUD_FBrjruJHFeKKyQYc1bswCLcBGAs/s400/FieldUsage_AddressUsage.PNG" width="400" height="215" data-original-width="693" data-original-height="373"></a></div>
Zoom out a bit, and you will be able to see the the Metadata that references the Property__c.Address__c field.</li>
</ol>
<!-- TODO Link to Dreamhouse example gexf file -->
<h3 style="clear: both;">Potential Packages</h3>
<ol>
<li>Run the "Modularity" Statistics. This algorithm will look for metadata communities based on the dependencies between them. You can adjust the <b>Resolution</b> up or down if you want more or less communities.</li>
<li>Change the appearance of the Nodes to be partitioned by the resulting "Modularity Class". With the correct resolution and layout it will given you a reasonable guide to where potential packages are located. You will likely need to make some finer adjustments along the boundaries.</li>
<li>One way to export the nodes for the package is to select them and then copy to a new workspace. The new data table can then be exported as a csv.<br/> <a href="https://4.bp.blogspot.com/-BdSNob_hLmA/W6YQYsK24eI/AAAAAAAD9ss/Z5NcbI32W2cE1LPvNh3BlkBBmUmaXolEACLcBGAs/s1600/ExportSelectedNodes.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-BdSNob_hLmA/W6YQYsK24eI/AAAAAAAD9ss/Z5NcbI32W2cE1LPvNh3BlkBBmUmaXolEACLcBGAs/s320/ExportSelectedNodes.png" width="320" height="189" data-original-width="536" data-original-height="316" /></a> </li>
</ol>
<h3 style="clear: both;">API Versions</h3>
<iframe width="560" height="315" src="https://www.youtube.com/embed/vfj0tNEWBsA" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
<ol>
<li><a href="https://3.bp.blogspot.com/-PmnxosGrZ00/W6YScZOIHhI/AAAAAAAD9s4/9y26xhsPlvI86o8TcH6TzOo6rflDjDNRgCLcBGAs/s1600/ApiVersionFilter.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-PmnxosGrZ00/W6YScZOIHhI/AAAAAAAD9s4/9y26xhsPlvI86o8TcH6TzOo6rflDjDNRgCLcBGAs/s320/ApiVersionFilter.png" width="225" height="320" data-original-width="251" data-original-height="357" /></a>
Filter to just those bits of metadata that have an API version defined. One way to do this is to drag a "Non-null (ApiVersion)" filter down onto the Queries.</li>
<li>Under <b>Appearance > Nodes</b>, set the node size to rank by <b>Degree</b>. The degree is a measure of how many edges go in or out of the node. So more connected nodes will be larger.</li>
<li>Set the Nodes color to be ranked by API version. Choose a color pallete that emphasizes lower API versions as being problematic.</li>
<li>Set the Node Label to "API Version" and size as required.</li>
</ol>
<h3 style="clear: both;">Code Coverage</h3>
<iframe width="560" height="315" src="https://www.youtube.com/embed/bUL4Cr4fIns" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
<ol>
<li>Size the nodes by "LengthWithoutComments" to give an indication of how much code is in each Apex class.</li>
<li>Color the nodes by ranking of the "PercentCovered". Use the slider to roughly indicate code with 75% coverage.</li>
<li>It can be useful to filter to just the test classes and assign them a different node color. They won't have any coverage themselves.</li>
<li>Set the node label to the "PercentCovered"</li>
<li>A circular layout sorted by the coverage percent can be useful to separate out the problem nodes. Move the text classes into a seperate ring so you can see the relationships between the tests and the classes they cover.</li>
</ol>
<h3 style="clear: both;">Classes with excessive lines of code</h3>
<p>As an experiment I drastically increased the amount of node scaling that was possible. This can really emphasis the differences in the size of Apex classes.</p>
<a href="https://4.bp.blogspot.com/-VtKLxBNCbhE/W6YS7aV09sI/AAAAAAAD9tA/-aTgnz6dpDMa2OIu9SA3jNUhzGgsACV3ACLcBGAs/s1600/LengthWithoutComments.PNG" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-VtKLxBNCbhE/W6YS7aV09sI/AAAAAAAD9tA/-aTgnz6dpDMa2OIu9SA3jNUhzGgsACV3ACLcBGAs/s320/LengthWithoutComments.PNG" width="280" height="320" data-original-width="703" data-original-height="802" /></a>
<!--
TODO:
import Org health report details
import lightning readiness report
# of records on custom objects
import actual live records rather than just metadata
Visualize Test Case failures - see areas where failures are occurring
Triggers per sObject - aim for only one
Track cascading impacts of a change to a field.
Paint Tool for neighoubous and neibours of neigbours.
Heatmap tool
Metric - test execution time
-->FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-68125349216410115512018-08-24T12:13:00.001+12:002018-09-22T21:17:28.254+12:00Dreamforce 2018 Session picks<p>Here are some of my current picks for Dreamforce 2018 sessions. I'm aiming for a mix of developer related topics in areas I want to learn more about plus anything that sounds informative. It isn't an exhaustive list and there are certainly some other session that I'll be adding.</p>
<h1>Most important session that I'm definitely going to attend</h1>
<p>I might be biased, but this session is on my must attend list.</p>
<p>Tell your friends, tell your neighbors, tell your family, bring your mum along. There will be something for everyone!<br/>
Who doesn't like pretty pictures, metadata, and a sprinkling of graph theory?</p>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WWXOQA4">Understand your Org shape via visualization of Metadata Component Dependencies</a>
<!--blockquote>When moving to Salesforce DX unlocked packages you need a fast and easy way to identify the modules that make up a Salesforce org and how they are related to each other. This also applies when needing insight into how a specific field is used by other components.
By using MetadataComponentDependency queries along with Gephi, an open-source visualization and exploration tool for graphs, we can rapidly untangle the dependencies via a range of layouts and metrics. These interactive visualizations aid developers in understanding the shape of a Salesforce org, leading to benefits as migrating Visualforce pages to Lightning components and identifying potential packages from complex metadata relationships.</blockquote-->
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Learn how to visualize the ingredients in your <a href="https://twitter.com/hashtag/Salesforce?src=hash&amp;ref_src=twsrc%5Etfw">#Salesforce</a> Orgs &quot;Happy Soup&quot; via my <a href="https://twitter.com/Dreamforce?ref_src=twsrc%5Etfw">@Dreamforce</a> session using MetadataComponentDependency queries:<br><br>&quot;Understand your Org shape via visualization of Metadata Component Dependencies&quot;<br><br> <a href="https://twitter.com/hashtag/DF18?src=hash&amp;ref_src=twsrc%5Etfw">#DF18</a> <a href="https://t.co/0W4KM1ald1">https://t.co/0W4KM1ald1</a> <a href="https://t.co/tKxyqWy4gM">pic.twitter.com/tKxyqWy4gM</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/1031345907877392384?ref_src=twsrc%5Etfw">August 20, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</li>
</ul>
<h2>Keynotes</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WYWsQAO">Salesforce for Developers Keynote</a></li>
</ul>
<h2>Meet The *'s</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WYXRQA4">Parker Harris' True to the Core: What's Next for Our Core Products</a> - Expect a focus on reinvigorating the idea exchange.</li>
<li>Meet the Developers - currently MIA?</li>
<li>Meet the Apex Engineering and Product Team - currently MIA?</li>
</li>
</ul>
<h2>Apex</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WZTXQA4">Everything that's Awesome with Apex</a>
<blockquote>Get a sneak peak into Apex plans, roadmap and how we are making secure development easier. We'll share some of our newest concepts <b>(async Apex anyone?)</b> and get your feedback on hot IdeaExchange features.</blockquote>
👀Async Apex, maybe something to help with FLS and CRUD in managed packages. Yes please!
</li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WXVIQA4">Test Your Visual Workflow and Process Builder Automations</a>. I'm assuming there will be some Apex involved in this testing process. </li>
</ul>
<h2>APIs</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WZU7QAO">Creating Massive Adoption for Your APIs</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WVcQQAW">The New User Interface API</a></li>
</ul>
<h2>Salesforce DX</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WUY5QAO">Super Session: Salesforce DX</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WVyWQAW">Advanced Techniques To Adopt Salesforce DX Unlocked Packages</a></li>
</ul>
<h2>IDE</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WZTjQAO">Building Powerful Tooling for IDEs Through Language Servers</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WZTbQAO">Banish the Bugs with the Apex Replay Debugger</a></li>
</ul>
<!--h2>Peeking under the hood</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytGEQAY">New Apex Compiler Rollout</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001pwnDQAQ">How We Do the Magic We Do - A Peek into Multi-tenancy</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yuLjQAI">SOQL: Performance Explained</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytGCQAY">How Salesforce does Incremental Code Deploys for internal orgs</a></li>
</ul-->
<h2>Platform Events</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WZTtQAO">Synchronize Data and Orchestrate Workflows in Real-Time with Change Data Capture</a></li>
</ul>
<h2>Platform</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WUYBQA4">The Lightning Platform Roadmap - Part I</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WUYCQA4">The Lightning Platform Roadmap - Part II</a></li>
</ul>
<h2>Einstein / AI</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WYhfQAG">BREAKING NEWS: The Latest Developments and Roadmap for Einstein</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WWiCQAW">Einstein Vision - How To Train Your AI Like An Expert!</a></li>
</ul>
<h2>Security</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WX4qQAG">Avoiding Common Security Mistakes</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WYwMQAW">Automated Code Quality & Security with CodeScan</a></li>
</ul>
<h2>Fun</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00001XoCSUUA3#/session/a2q3A000001WZCpQAO">Flood the Trailblazer Community Cove: Swag & Sticker Meetup</a><br/>Got too much swag or something you can't get home - swap it! Last year I gave a bunch of stuff away that I'd collected but couldn't actually put in my luggage. I'm looking at you oversized Codey. And the rose seed in soil that NZ customs would have opinions on.</li>
</ul>
<h2>Lightning</h2>
<ul>
<li>TODO - Need something to fill out this area to be a well rounded developer. Lightning Roadmap maybe?</li>
<li></li>
</ul>
<!--René Winkelmeyer-->
<h2>See also</h2>
<ul>
<li>My <a href="http://www.fishofprey.com/2016/08/dreamforce-2016-session-picks-and.html#tips">General Dreamforce Tips</a> from last year</li>
<li>
Salesforce Engineering: <a href="https://engineering.salesforce.com/your-dreamforce-agenda-for-developers-118d7b6ad90">Your Dreamforce [2018] Agenda for Developers</a></li>
</ul>
FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-7212195425370311862018-07-23T12:16:00.000+12:002018-07-23T12:17:09.871+12:00FuseIT SFDC Explorer 3.9.18190.1 - Summer '18<p>Another roundup of some of the changes to the <a href="http://www.fuseit.com/explorer">FuseIT SFDC Explorer</a> since the <a href="http://www.fishofprey.com/2017/08/fuseit-sfdc-explorer-36171841-and.html">3.7.17230.1</a> release.</p>
<h2>Display additional columns for the Apex Log timeline (Experimental)</h2>
<p>Let's say you just ran all the tests in your org in parallel and captured the debug logs for each transaction. Now you have a huge number of debug logs to peruse, but where to start?</p>
<p>Experimental features that I've only tried a few times to the rescue!</p>
<a href="https://3.bp.blogspot.com/-VkOVUrGW2jU/W0Vd9rL4jfI/AAAAAAAD7tA/urJ5ektlj1syGeUlIbPNGIjXGe0t60psQCLcBGAs/s1600/LogTimelines.PNG" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-VkOVUrGW2jU/W0Vd9rL4jfI/AAAAAAAD7tA/urJ5ektlj1syGeUlIbPNGIjXGe0t60psQCLcBGAs/s800/LogTimelines.PNG" width="760" data-original-width="1025" data-original-height="428" /></a>
<p>By double clicking in the Timeline column cells the full debug log will be pulled down and converted to a timeline. I'm still experimenting with this. Certainly having all the timelines shown at different scales is a bit problematic. It's hard to get a sense of how long the actual transaction took relative to the adjacent ones.</p>
<p>I might instead take a different approach here and extract some metadata from the debug log and just show the core details. I'd look for things like hitting limits, throwing exceptions, slow DML, etc...</p>
<h2 name="deleteAllLogs" id="deleteAllLogs">Button to delete ALL ApexLog's in an org.</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-cR7VBPRz17Q/W0ay3KlPcnI/AAAAAAAD7v0/q2D4Sh3mjmsuouTQTs4IrpC_rEvPqVj4wCLcBGAs/s1600/DeleteAllApexLogs.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-cR7VBPRz17Q/W0ay3KlPcnI/AAAAAAAD7v0/q2D4Sh3mjmsuouTQTs4IrpC_rEvPqVj4wCLcBGAs/s320/DeleteAllApexLogs.PNG" width="320" height="178" data-original-width="142" data-original-height="79" /></a></div>
<p>It's a small thing, but currently with Summer '18 it is all to easy to run into a message like:</p>
<blockquote>
The Developer Console didn't set the DEVELOPER_LOG trace flag on your user. Having an active trace flag triggers debug logging. You have 318 MB of the maximum 250 MB of debug logs. Before you can edit trace flags, delete some debug logs.
</blockquote>
<p>Then you get to play a game of whack-a-<strike>mole</strike>ApexLog with the Tooling API to clear them all out and then get on with your day. This button on the Apex Logs tab reduces it to a single click.</p>
<p>You might also like to vote for the idea: <a href="https://success.salesforce.com/ideaView?id=0873A000000TwBDQA0">Allow TraceFlags to indicate that Debug logs can be deleted automatically</a>. Adding a TTL and automatic purge of older logs would avoid most of these issues while still being flexible if you needed longer term logging.</p>
<h2 style="clear:both;">Show total counts per log category</h2>
<a href="https://3.bp.blogspot.com/-ALsn-abIOtg/W0xpIJ_HMtI/AAAAAAAD7x0/Lawz3vVgsrQSS4ePVWwUEUvFdwk4Ws7dQCLcBGAs/s1600/LogCategoryPercentages.PNG" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-ALsn-abIOtg/W0xpIJ_HMtI/AAAAAAAD7x0/Lawz3vVgsrQSS4ePVWwUEUvFdwk4Ws7dQCLcBGAs/s640/LogCategoryPercentages.PNG" width="760" data-original-width="859" data-original-height="25" /></a>
<p>The percentage that each log category contributes to the overall log size has been added to the footer. This can be useful for adjusting the log levels when the logs are getting too noisy or large.</p>
<h2 style="clear:both;">Skip Code Coverage</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-_MqQaeSOjXo/W0xsGTDBEuI/AAAAAAAD7yA/CP0u0IEjwEolqrYC_64mBE1QocOA8CeTACLcBGAs/s1600/SkipCodeCoverage.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-_MqQaeSOjXo/W0xsGTDBEuI/AAAAAAAD7yA/CP0u0IEjwEolqrYC_64mBE1QocOA8CeTACLcBGAs/s400/SkipCodeCoverage.PNG" width="400" height="122" data-original-width="268" data-original-height="82" /></a></div>
<p>With <a href="https://releasenotes.docs.salesforce.com/en-us/summer18/release-notes/rn_apex_opt_out_code_coverage.htm">Summer '18 it is possible to opt out of collecting code coverage with a test run</a>. This can save some time when running the test cases if you don't require the coverage data.</p>
<h2 style="clear:both;">Additional metadata deployment details</h2>
<a href="https://4.bp.blogspot.com/-eJew4Nu__34/W0-rZwJPIbI/AAAAAAAD8BA/nv8bH9GnHKske3gnysyp4Da1CNslRqvNACLcBGAs/s1600/MetadataDetails.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-eJew4Nu__34/W0-rZwJPIbI/AAAAAAAD8BA/nv8bH9GnHKske3gnysyp4Da1CNslRqvNACLcBGAs/s320/MetadataDetails.png" width="320" height="101" data-original-width="564" data-original-height="178" /></a>
<p>When selecting individual metadata files to deploy details such as path, local file system data modified, and CRC are displayed.</p>
<h2 style="clear:both;">Other changes 3.9</h2>
<ul>
<li>ApexLogContentControl - Format "SkippedBytesOfDetailedLog". Shortcut to line. Protected against parsing multiple log models at the same time.</li>
<li>Expose the Duration in the ApexLogTreeViewUserControl. Color code the Event column</li>
<li>ApexLogEntry - Prevent recursion in the model. New properties for Text, Message. Dedicated LimitUsageForNsApexLogEntry ApexLogEntry</li>
<li>Option to display timeline bookmarks over thumbs. Detect and highlight LimitUsage warnings and highlight in timeline. Display log running events (> 500ms) as lines in the timeline.</li>
<li>Make CSV field name checking case insensitive.</li>
<li>When searching for a term in a debug log, ensure that line scrolls into view.</li>
<li>When listing Apex classes include an "Uncovered Lines" column</li>
<li>Emphasize the Warning and Error entries with larger bars</li>
<li>ApexLogModel: Handle CODE_UNIT_STARTED for Validation. Reparent when transitioning from CUMULATIVE_LIMIT_USAGE_END to CODE_UNIT_STARTED without intermediate CODE_UNIT_FINISHED</li>
<li>PackageCreation - Include additional metadata when performing a HashDiff on a folder.</li>
<li>WSDL2Apex: Skip Complex Content Restrictions (with a warning) rather than throw an exception.</li>
<li>WSDL2Apex: Reset stored web service metadata between runs.</li>
<li>ApexLogService - Expand LogMessage enum with missing records</li>
<li>Update SalesforceSession to use Summer '18 v43.0 API version</li>
<li>T4 Code Generator - Allow for sObjects with no record types defined</li>
<li>SalesforceSession - Support for visualforce.com as a SOAP Partner URL</li>
<li>SalesforceServiceWrapper - Improve performance of ObjectTypeFromId(string id) for looking up the sObject type based on the keyprefix. EntityServiceGenerator.tt - Don't include a custom objects keyPrefix in RegisterKeyPrefixObjectType by default.</li>
<li>Include dedicated columns when listing Apex classes for Lines covered and Total lines</li>
<li>ApexClassService - Handling missing SOSL search results when looking for test classes.</li>
<li>MetadataServiceWrapper - include support for deployment of permissionsets</li>
<li>SalesforceException - Log direct web request with cookie exceptions.</li>
<li>Wsdl2Apex - handle services that require specific casing on the service URL query string</li>
<li>ToolingServiceWrapper - Check for API access to sObjects ApexOrgWideCoverage and ApexCodeCoverageAggregate before querying.</li>
</ul>
<h2>Other changes 3.8</h2>
<ul>
<li>Data Export Console: Support for connection strings using RefreshTokens rather than usernames and passwords.</li>
<li>Options to metadata deploy Aura components</li>
<li>Apex Test Results - Expand nodes to show test case failures initially.</li>
<li>UI - Improve menu overflow options</li>
<li>UI - Track selected log event as the DataGrid scrolls</li>
<li>Data Export Console - Improvements to parameter validation. Especially cases where the wrong number of parameters are provided.</li>
<li>Option to collapse/expand the log selection with the log viewer.</li>
<li>MetadataServiceWrapper: deploy aura components in packages.</li>
</ul>
<h2>Other changes 3.7.17251.2</h2>
<ul>
<li>FitDx: --filter option for user defined events. --summary option for count by event type.</li>
<li>Add optional allowExistingSObjectsWithoutId="true" to the binding configuration element to allow sObjects to be created with a null Id. Typically this isn't allowed as the ID is used to control insert/update operations and to identify relationship types. This setting can be used for more basic SOQL queries where the results won't be subsequently used for DML.</li>
</ul>
FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-71874908113875719992018-06-27T08:35:00.004+12:002018-06-27T08:39:38.951+12:00Speeding up Salesforce unit testing performance<div class="separator" style="clear: both; text-align: center;"><a href="https://salesforce.stackexchange.com/a/3697/102" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://i.stack.imgur.com/tmikt.png" width="400" height="349" data-original-width="413" data-original-height="360" /></a></div>
<p>For many years I've had this thorn in my side with Apex test cases. It goes by the name of "Disable Parallel Apex Testing" and since Spring `13 (v27.0) it has needed to be enabled constantly else I'd get a <a href="https://salesforce.stackexchange.com/a/8518/102">UNABLE_TO_LOCK_ROW error</a> due to the custom hierarchy settings that get updated in the test cases.</p>
<p>Leap forward to Summer/Winter `18 (v42.0/v43.0) and I'm still tangling with this. That's five years of waiting for all the test classes to run sequentially one after another. That's about the right amount of procrastination to finally fix this problem! So hold on, were going to raise a support case and keep at this until the bitter end<sup><a href="#foot1">1</a></sup>.</p>
<p><i>What follows is a true story. The namespaces and class names have been changed to protect the innocent, but the snippets of messages with support are real.</i></p>
<h2 style="clear: both;">Step 1 - Turning off "Disable Parallel Apex Testing"</h2>
<p>This was simple enough. Uncheck a checkbox, run all the tests, and... Uh oh.</p>
<blockquote><a href="https://salesforce.stackexchange.com/q/217551/102">Could not run tests on class 01p400000000001 because: connection was cancelled here</a></blockquote>
<p>With that setting checked all my tests were passing. Now some of my previously passing test classes are falling over when run in parallel. Worse still, the specific test classes to fail were intermittent. It wasn't just one specific class causing a problem. Any one of a dozen or so classes could fail in this manner and they would change from run to run.</p>
<h2 id="isParallel" name="isParallel">Step 2 - Winter `18 @isTest isParallel annotation to the rescue</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-kjd4CFF-G5g/WwPqJSdH2YI/AAAAAAAD5-g/lQLa5PSTT9ADq8RIZA27Xw_yERE6Z3vewCLcBGAs/s1600/the-istestisparallelfalse-attributes-they-do-nothing.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-kjd4CFF-G5g/WwPqJSdH2YI/AAAAAAAD5-g/lQLa5PSTT9ADq8RIZA27Xw_yERE6Z3vewCLcBGAs/s400/the-istestisparallelfalse-attributes-they-do-nothing.jpg" width="400" height="300" data-original-width="500" data-original-height="375" /></a></div>
<p><a href="https://releasenotes.docs.salesforce.com/en-us/winter18/release-notes/rn_apex_annotation_istest_isparallel.htm">Winter 18</a> added the new <code>@isTest(isParallel=true)</code> annotation to:</p>
<blockquote>indicate test classes that can run in parallel and aren’t restricted by the default limits on the number of concurrent tests.</blockquote>
<p>That's great, except I don't want to modify 90% of my test classes to deal with the 10% that are having issues. No, I'll just use <code>@isTest(isParallel=false)</code> and explicitly exclude the problem cases... except that doesn't work. At least not at the time of writing. Please vote for the idea <a href="https://success.salesforce.com/ideaView?id=0873A0000003VhKQAU">Parallel Tests Option (isParallel) on the @IsTest Annotation to exclude tests</a> to make this a viable approach.</p>
<h2 style="clear: both;">Step 3 - Once more unto the support case, dear friends, once more</h2>
<p><span class="supportDay">[Day 1 - 2018-05-09]</span> Desperate times. Let's raise a support case to see if they can isolate the underlying issue.</p>
<p><span class="supportDay">[Day 6 - 2018-05-14]</span> After a bit of back and forth with tier 2 to establish how to reproduce the problem (press the run all test cases in the org) the initial advice back was:</p>
<blockquote class="supportResponse">
The root cause of this issue?<br/>
Answer: - Parallel test execution is the root cause of this issue. <br/>
<br/>
Solution?<br/>
Answer: - Disable Parallel test execution. <br/>
<br/>
Question: - Why do you need to Disable Parallel Apex Testing? <br/>
Answer: - As per the salesforce document, Tests that are started from the Salesforce user interface (including the Developer Console) run in parallel. Parallel test execution can speed up test run time. Sometimes, parallel test execution results in data contention issues, and you can turn off parallel execution in those cases.
</blockquote>
<p>Huh. Well, yes I knew that already (as per the case description when I raised it). That would work, but it's been five years of this situation, so lets push a bit harder. <a href="https://www.xkcd.com/806/">SHIBBOLEET!</a></p>
<h2>Step 4 - How bad is the problem empirically?</h2>
<p>Before responding with a kiwi <a href="https://youtu.be/xWsQ5lwgpSw?t=3m1s">yeah-nah</a> to support I timed the total test run time for both parallel and synchronous test execution:</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Synchronous test run is ~12% slower, but I could easily lose that time needing to watch for the parallel failures and rerun them.</p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/996535323558473729?ref_src=twsrc%5Etfw">May 15, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>As it stood, the parallel test execution was marginally faster as long as I reran any initially failed test cases immediately after the first run completed. That was an interesting result, and I think I can explain the similar timing between the two <a href="#reason">later on</a>.</p>
<p>The <a href="https://twitter.com/FishOfPrey/status/996542962514382848">general consensus on twitter</a> was that updates to custom hierarchy settings were probably to blame for the contention and subsequent timeouts.</p>
<h2>Step 5 - Isolating Custom Hierarchy Settings using the Stubbing API</h2>
<p>I use a number of hierarchy custom settings to toggle various functions in the app. All interactions with those settings from Apex are done via a single class. This allows for sensible defaults etc...</p>
<p><i>Those with particular feelings on how mock testing should be performed or a sensitive testing disposition may want to look away now...</i></p>
<p>To prevent any potentially blocking DML operations on the custom hierarchy settings I've injected a <a href="https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_interface_System_StubProvider.htm">StubProvider</a> when in a testing context. The StubProvider prevents any DML occuring when altering the settings in Apex tests. This isn't the typical usage for a test mocking framework, but it serves my needs here to help avoid database locking issues.</p>
<p>Here is a shortened version of how it looks:</p>
<style type="text/css">
.gist-file
.gist-data {max-height: 400px;}
span.supportDay {color: #227777;}
pre.supportResponse { white-space: pre-wrap; background-color: #112222;}
blockquote.supportResponse { border-left-width: 6px; border-left-color: #112222; border-left-style: double; padding-left: 2px; }
pre.toSupport { border-left-width: 6px; border-left-color: #222211; border-left-style: double; }
</style>
<!--FishOfPreySettings-->
<script src="https://gist.github.com/FishOfPrey/9ef39b32948f23b604c8e8e0ab041c53.js"></script>
<!--FishOfPreySettingsMockProvider-->
<script src="https://gist.github.com/FishOfPrey/fdf33076e404ebc878b683f6383c67fc.js"></script>
<!--FishOfPreySettings_Test-->
<script src="https://gist.github.com/FishOfPrey/23ebd116f84b520b0db57e3c489a6c60.js"></script>
<p>And after all that the result on the asynchronous test execution was..... not much really. The tests were marginally faster, but the underlying problem with the test classes having the connection closed remains. At least I can rule out DML on the custom settings as being the problem.</p>
<h2>Step 6 - Back to the drawing board</h2>
<p>Blaming the problem on the custom hierarchy settings made sense from a historical perspective, but it doesn't appear to be the source of my challenges.</p>
<p>I went back and had a closer look at the debug logs from the tests in the run. That's when I saw it:</p>
<a href="https://3.bp.blogspot.com/-ztKcQhepCck/WzFr1A99xII/AAAAAAAD6Zw/ddTA2r5ycmEqpYojzgp9BgkBll94Og3mACLcBGAs/s1600/Timelines.png" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-ztKcQhepCck/WzFr1A99xII/AAAAAAAD6Zw/ddTA2r5ycmEqpYojzgp9BgkBll94Og3mACLcBGAs/s800/Timelines.png" width="760" data-original-width="896" data-original-height="388" /></a>
<p><b>"What?"</b>, you may ask, <b>"Am I currently looking at?"</b></p>
<p>That colorful image is a selection of debug log timelines from various parallel test runs. Ignoring the majority of the markings, the important thing is a <span style="color: #DA70D6">purple line representing a DML operation</span> that is taking more and more of each transaction. Right up until the point that Salesforce starts terminating it. If we drill into one of those logs we can see it is a DML operation to insert 6 records taking 3 minutes 52 seconds (for 6 records!):</p>
<a href="https://1.bp.blogspot.com/-XTzCmkDRH_A/WzGWadC5NFI/AAAAAAAD6Z8/-9yevKYAWT4124DjquaRg1NCNiS7XSQxACLcBGAs/s1600/LogEventZoom.png" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-XTzCmkDRH_A/WzGWadC5NFI/AAAAAAAD6Z8/-9yevKYAWT4124DjquaRg1NCNiS7XSQxACLcBGAs/s800/LogEventZoom.png" width="760" data-original-width="1009" data-original-height="135" /></a>
<pre style="white-space: pre-wrap;">
14:14:15.0 (325578801)|DML_BEGIN|[992]|Op:Insert|Type:DFB__OpportunityPriceType__c|Rows:6
14:18:07.376 (232376043557)|DML_END|[992]
</pre>
<h2>Step 7 - Returning to the support case with proof</h2>
<p><span class="supportDay">[Day 9 - 2018-05-17]</span> If that isn't a smoking gun then I don't know what is. Let's take this new evidence back to the support case. </p>
<p>Me to support:</p>
<blockquote class="toSupport">
I've just noticed something else odd in the debug logs for the running tests. Inserts to DFB__OpportunityPriceType__c for 6 rows are taking an excessive amount of time. <br/>
<br/>
I'm seeing times for the CommonTestSetup.createPriceTypeMappings() method of between 20 and 45 seconds to complete. <br/>
Attached Log1.txt - Look for the lines:<br/>
16:18:58.0 (150146460)|DML_BEGIN|[941]|Op:Insert|Type:DFB__OpportunityPriceType__c|Rows:6<br/>
16:19:19.214 (21214892238)|DML_END|[941]<br/>
<br/>
It isn't clear to me why the inserts for those records are taking so long. The fields on it are [reasonably] basic and it is only 6 rows. <br/>
The only thing I can think of is that all the test classes are trying to use the same records and they all have the same indexed values in the PriceTypeID__c field.
</blockquote>
<p>A bit more context as you dear reader can't see into the org like support can.</p>
<p>I have a custom object <code>DFB__OpportunityPriceType__c</code> with a unique external id field. That field is indexed. Just about every single test class I have requires data in this custom object as it contains configuration that gets linked to from Opportunity records. I thought I was being clever and used a <code>@TestSetup</code> method to insert 6 of these records once for all the other test methods to use. There are no triggers, workflow outbound messages or other automatons hanging off this custom object. None of the test classes use <code>SeeAllData=true</code>.</p>
<h2>Step 8 - More steps in the Support dance</h2>
<p><span class="supportDay">[Day 13 - 2018-05-21]</span> Support responds:</p>
<blockquote class="supportResponse">I reviewed your case and ran the test classes to troubleshoot the issue further. I noticed the error in the server log "RunningForTooLongException: connection was canceled here".<br/>
The error is thrown because - for asynchronous Apex tests, any test method execution has a timeout limit. Once an Apex test run exceeds 6 minutes, an internal process "kills" it. <br/>
<br/>
After reviewing the logs, it seems that this particular method is taking time to execute when all the tests classes are run asynchronously -
CommonTestSetup.minimalSetupWithSettings();<br/>
<br/>
Can you please try to refactor this method and see if it can be optimized?<br/>
I see there are many methods being called from this function. </blockquote>
<p>Another frustrating response. They pointed out the method that gets called directly in the <code>@testSetup</code> method. I'd already identified the inner method from that one and even the line in question. That it is a RunningForTooLongException is new information, but I'm no closer to what is blocking those records from inserting.</p>
<p><span class="supportDay">[Day 15 - 2018-05-23]</span> We try a GoToMeeting. I work through with Tier 2 support in ensuring the external ID's that get assigned to DFB__OpportunityPriceType__c records in the test cases are unique per <code>@testSetup</code>. It didn't seem to help. Beyond that, I think support has a good grasp of the problem now.</p>
<p><span class="supportDay">[Day 20 - 2018-05-28]</span> The support case has been escalated to Tier 3.</p>
<p><span class="supportDay">[Day 30 - 2018-06-07]</span> Tier 3 responds:</p>
<blockquote class="supportResponse">
- When 'Disable Parallel Apex Testing' checkbox is checked in Developer Console. <br/>
Result: Running Perfect <br/>
<br/>
- When 'Disable Parallel Apex Testing' checkbox is unchecked in Developer Console. <br/>
Result: Exception is thrown <br/>
<br/>
- In Serial Mode, it's working perfectly because it's running in Sync mode and no data contention issue. <br/>
However, in Parallel mode, multiple classes and trigger have been invoked in the same context/transaction where performing DML Operation and also making callouts and waiting for a response in Async mode. This may cause some data contention. Also since there is a lot of custom code involved it becomes very difficult for us to analyze the scenario. <br/>
<br/>
- This is WAD as per Salesforce documentation. If Apex Class will take more than 6 minutes of time to complete the transaction, the connection will be closed by Salesforce DB. <br/>
<br/>
- Test classes are run in serial mode during deployment. This error will not occur when you are trying to deploy your package. <br/>
<br/>
Possible workarounds - <br/>
<br/>
- Try to modify the code and remove that many dependencies.<br/>
- Else run in Serial Mode as a Workaround.
</blockquote>
<p>This seems distinctly like they put it in the too hard basket and are trying to avoid investigating the actual issue. I'll push back to see if I can get a better answer.</p>
<p>Also, I'm not sure what they mean by "making callouts and waiting for a response in Async mode". This is all happening in a test context. There are no callouts or waiting for async responses. I'm not going to pursue that as I don't want to get distracted.</p>
<p><span class="supportDay">[Day 35 - 2018-06-12]</span> Update from support.</p>
<blockquote class="supportResponse">My T3 has officially logged an investigation with our RnD team on this issue. My RnD team is currently reviewing the case. </blockquote>
<p><span class="supportDay">[Day 36 - 2018-06-13]</span> An update from R&amp;D, as communicated via Tier 2.</p>
<blockquote class="supportResponse">
I have received an update from the DB team on this case. <br/>
<br/>
As per the DB team, it seems that the parallel sessions are working on common recordsets and causing the wait events. <br/>
Because of this waitime <u title="Huh - What?">the DB team is high</u> and eventually the classes are getting timed out. <br/>
<br/>
May be this explains why the insertion to DFB__OpportunityPriceType__c is taking time. We cannot exactly comment on the common recordsets but DFB__OpportunityPriceType__c records is one of the potential candidate. <br/>
<br/>
Solution - <br/>
Have the parallel sessions work on different recordset/dependencies to avoid this situation.
</blockquote>
<p>So it appears that all the test cases are trying to work with the same record sets and they are all blocking until they can get exclusive access to those records. They either stack up and run one after another or wait so long they time out. This <a name="reason" id="reason">explains</a> why the parallel test performance is so similar to the serial performance.</p>
<p>I'm fairly certain the <code>DFB__OpportunityPriceType__c</code> records are unique between <code>@testSteup</code> executions, so it must be something else...</p>
<h2>Step 9 - Time to regroup</h2>
<p><span class="supportDay">[Day 42 - 2018-06-19]</span> Much to my shame I've closed the case out. I need time to regroup and revisit this from a new approach. I'm certainly not giving up on it yet and have a few ideas to try:</p>
<ul>
<li>Try adjusting the Logging levels used when capturing the logs. Particularly around the Database logging levels. Also try capturing no logs at all.</li>
<li>Push the entire packages metadata into a Scratch Org and and retry the tests. The dev org doesn't have much data loaded, but the scratch org can provide a completely empty environment to ensure it isn't a data siloing issue.</li>
<li>If it is reproducable in the scratch org, start hacking parts out until it becomes easier to reproduce.</li>
<li>Remove the creation of the DFB__OpportunityPriceType__c records from the @testSetup. Only create them when required in the individual test methods.</li>
<li>Simplify the test cases where possible. The challenge is almost all of them work with Opportunities and OpportunityLineItems at some level. Which means a large overhead to construct all the dependent records as well.</li>
</ul>
<hr/>
<h2>Footnotes</h2>
<ol>
<li name="foot1" id="foot1">The <i>bitter end</i> may or may not occur within the timeframe of this blog post.</li>
</ol>
<!--I was tempted to open with a click bait type title: How I quadruppled my test throughput with this one simple trick-->FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com7tag:blogger.com,1999:blog-7249827273468308081.post-11271973889490637922018-05-16T09:31:00.000+12:002018-05-16T09:31:03.920+12:00Fiddling with the SFDX CLI API calls<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-TT_OZVOKnpg/WvVpMYKpGCI/AAAAAAAD54I/S2QVVDFXq2kYqj0ZawZloF2vGdUWYCI1wCLcBGAs/s1600/MakesItTick_sml.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-TT_OZVOKnpg/WvVpMYKpGCI/AAAAAAAD54I/S2QVVDFXq2kYqj0ZawZloF2vGdUWYCI1wCLcBGAs/s320/MakesItTick_sml.png" width="320" height="320" data-original-width="601" data-original-height="601" /></a></div>
<p>What makes the <code>sfdx</code> CLI tick? Sometimes learning how something works can be as much fun as actually using it.</p>
<p>The goal here is to capture the raw API calls the <code>sfdx</code> CLI is sending to the Salesforce APIs. In addition to a better understanding of what it is doing you can use it to debug the CLI itself.</p>
<p>This post was inspired by a, ahem, very similar post by Christian Carter - <a href="http://cdcarter.github.io/sfdx/2018/05/22/charles-proxy-sfdx">SFDX With Charles Proxy</a>. The primary difference is that I'm using <a href="https://www.telerik.com/fiddler">Fiddler</a> on Windows rather than Charles Proxy on macOS.</p>
<p>Using the <a href="https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_cli_log_messages.htm">direct sfdx logging support</a> is another option to monitor what is going on. Or even browsing the source directly under <code>%LOCALAPPDATA%\sfdx</code> (seems they are going to some lengths to hide the source now). While something like fiddler is more complicated (some might even say "fiddly") to configure, it is harder to hide anything from it (intentionally or otherwise).</p>
<h2 style="clear: both;">Configure to intercept HTTPS traffic</h2>
<p>After installation the first this is to configure Fiddler and Windows to allow interception and decryption of HTTPS traffic.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-Pc9GQuYheok/WvV163O6pQI/AAAAAAAD54Y/h0jitbzCl48CIax_O5Zs9B9iClV8r7wqACLcBGAs/s1600/ScaryText.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-Pc9GQuYheok/WvV163O6pQI/AAAAAAAD54Y/h0jitbzCl48CIax_O5Zs9B9iClV8r7wqACLcBGAs/s1600/ScaryText.png" data-original-width="549" data-original-height="377" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-jNcCtWeLm0w/Wvo_gBgyJBI/AAAAAAAD564/FolFfCYGMQYDgsCCOHOlkz5mHtk-mDoNwCLcBGAs/s1600/PerformDecryption.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-jNcCtWeLm0w/Wvo_gBgyJBI/AAAAAAAD564/FolFfCYGMQYDgsCCOHOlkz5mHtk-mDoNwCLcBGAs/s1600/PerformDecryption.PNG" data-original-width="513" data-original-height="89" /></a></div>
<ol>
<li>Tools &gt; Options</li>
<li>HTTPS tab</li>
<li>Check <b>Decrypt HTTPS traffic</b></li>
<li>Click 'Yes' to reconfigure Windows' Trusted CA Certificate
<b>You might want to <a href="https://en.wikipedia.org/wiki/Root_certificate">read up</a> on what a Root Certificate is before doing so.</b></li>
<li>(Optional) change the drop down from "... from all processes" to "... from non-browsers only"</li>
<li>(Optional) Toggle the "Skip decryption for the following hosts" to "Perform decryption for the following hosts". Then add *.salesforce.com</li>
</ol>
<p style="clear: both;">Now the Fiddler is ready to intercept the traffic you need to configure <code>sfdx</code> to send it to the correct location. The default proxy port of 8888 is configured under <i>Options &gt; Connections &gt; Fiddler listens on port:...</i> </p>
<pre>
set http_proxy=http://localhost:8888
set https_proxy=https://localhost:8888
</pre>
<p>If you just stop there and try and call <code>sfdx force:org:list</code> you will find the CONNECTED STATUS comes back as "ECONNRESET". It would appear that node.js doesn't like our self signed root certificate. You can tell node to mind its own <!--damn--> business with:</p>
<pre>
set NODE_TLS_REJECT_UNAUTHORIZED=0
</pre>
<p>Again, you only want to do this for a single session and not configure that across all processes. It potentially opens you up to all sorts of man in the middle security attacks.</p>
<h2>Now what?</h2>
<p>Now my friend, now we can see each and every callout to the Salesforce APIs and the corresponding responses.</p>
<p>Lets look at what happens with the command <code>sfdx force:org:list</code>.</p>
<a href="https://1.bp.blogspot.com/-ATVrbt-oyIA/WvleH6DontI/AAAAAAAD56Y/n0_Pu2clWP0lkn2o69x3usyyysdfZfcSgCLcBGAs/s1600/forceOrgList.png" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-ATVrbt-oyIA/WvleH6DontI/AAAAAAAD56Y/n0_Pu2clWP0lkn2o69x3usyyysdfZfcSgCLcBGAs/s1600/forceOrgList.png" data-original-width="783" data-original-height="122" width="755" /></a>
<a href="https://4.bp.blogspot.com/-QV6_hiC4270/WvldjmDTGbI/AAAAAAAD56Q/X2anii2yUEQlDcrFJF6mKOxEDILuX6amgCLcBGAs/s1600/FiddlerOutput.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-QV6_hiC4270/WvldjmDTGbI/AAAAAAAD56Q/X2anii2yUEQlDcrFJF6mKOxEDILuX6amgCLcBGAs/s1600/FiddlerOutput.png" data-original-width="905" data-original-height="302" width="755" /></a>
<p>This reveals up to four API calls per valid Org. The exact calls will depend on the org types and if you have recently successfully authenticated to them. Generally, you get:</p>
<ol>
<li>A failed <b>GET /services/data/v42.0</b> with an invalid token</li>
<li>A <b>POST /services/oauth2/token</b> to refresh the access token</li>
<li>A successful <b>GET /services/data/v42.0</b></li>
<li>A <b>GET /services/data/v42.0/query?q=...</b> with a SOQL query over ScratchOrgInfo. Presumably this only works with Scratch Orgs at this time.</li>
</ol>
<p>So every org you register with SFDX needs 3 or 4 API calls with a force:org:list. There is certainly something to be said for dropping unused orgs.</p>
<p>Anyway... Happy Fiddling!</p>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-73790652061415084052018-05-09T12:28:00.000+12:002018-05-30T10:18:52.745+12:00Salesforce Log Categories and Events by Level - Revisited<p>Way back in June 2014 I posted a table to logging events by level and category - <a href="http://www.fishofprey.com/2014/06/salesforce-log-categories-and-events-by.html">Salesforce Log Categories and Events by Level.</a></p>
<p>I was never really happy with the table layout trying to squeeze that much data in. Also, new logging levels keep getting added and several have been shuffled around recently with respect to the level they occur at.</p>
<p>Here is a hopefully simpler revised attempt using lists. It is compiled directly from the Debug Log Levels detail page, so it should be eaiser to keep up to date.</p>
<p>Similar data can be found in the <a href="https://help.salesforce.com/articleView?id=code_setting_debug_log_levels.htm#DebugEventTypesSect">Debug Log Levels documentation</a>. I found at the time of publishing that I had several in my list that didn't appear on that page, such as CALLOUT_REQUEST_PREPARE, USER_DEBUG_WARN, and DUPLICATE_DETECTION_MATCH_INVOCATION_DETAILS.</p>
<p>The levels are cumulative. So everything that appears at the ERROR level will appear at all the lower levels as well. Everything at the WARN level will appear at INFO, DEBUG, FINE, ... and so on.</p>
<style>
.gist { color: inherit !important;}
.gist-data {background-color:inherit !important;}
</style>
<script src="https://gist.github.com/FishOfPrey/9c6b6d3a6d9e0147143b0c54d615a677.js"></script>
<ul>
<li>Updated for <a href="https://gist.github.com/FishOfPrey/9c6b6d3a6d9e0147143b0c54d615a677/revisions#diff-8a48cbc688e7dc15b4808c2343ae0092">Summer '18 - v43.0</a></li>
</ul>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-62882549519199898232018-04-10T12:06:00.000+12:002018-04-20T10:57:25.957+12:00The unofficial way to install Apps and Packages in Your Trailhead Playground <div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-Lg-7lA0yeO0/Wsv5Y8FIDlI/AAAAAAAD2CU/SgK_g_Lub0cq1D_i6nnxOY1FE_fNbPSKwCLcBGAs/s1600/DeploymentFishApproved.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-Lg-7lA0yeO0/Wsv5Y8FIDlI/AAAAAAAD2CU/SgK_g_Lub0cq1D_i6nnxOY1FE_fNbPSKwCLcBGAs/s1600/DeploymentFishApproved.png" data-original-width="200" data-original-height="200" /></a></div>
<p>There is a Trailhead module called <a href="https://trailhead.salesforce.com/modules/trailhead_playground_management">Trailhead Playground Management</a> that includes a unit on <a href="https://trailhead.salesforce.com/modules/trailhead_playground_management/units/install-apps-and-packages-in-your-trailhead-playground">Install[ing] Apps and Packages in Your Trailhead Playground</a>. This is an important step in many other modules and day to day Salesforce work. You need to be able to install an App/Package into a target org so you can use its features.</p>
<p>The unit goes into the steps in some detail and focuses on being accessible to those just starting out with Salesforce.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-m-zjDoEbfNg/WkniIDNuflI/AAAAAAADp4Q/R5PzuAcWtyk_tcPh-AsAyph5Hx12dwh7ACLcBGAs/s1600/InstallManagedPackage.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-m-zjDoEbfNg/WkniIDNuflI/AAAAAAADp4Q/R5PzuAcWtyk_tcPh-AsAyph5Hx12dwh7ACLcBGAs/s1600/InstallManagedPackage.PNG" width="700" data-original-width="902" data-original-height="390" /></a></div>
<p>Above is a quiz question from the <a href="https://trailhead.salesforce.com/modules/trailhead_playground_management/units/get-your-trailhead-playground-username-and-password">Trailhead Playground Management</a> module. It's technically correct as per the instructions in that module, but I don't think it is the best approach and from what I've seen is a common source of confusion.</p>
<p>I'd like to present an alternative approach. It might not be as accessible, but it does bypass a number of steps that can lead to further complications. So, without any more fanfare I present the Deployment Fish approved way of installing packages and apps into a Salesforce Org.</p>
<h2>The URL <strike>Hack</strike> Manipulation Maneuver</h2>
<p>The steps are as follows:</p>
<ol>
<li>Obtain the package installation URL.</li>
<li>Copy everything from that URL except for the domain</li>
<li>Log into the org where you want to install the package</li>
<li>Paste the content copied from Step 2 after the domain.</li>
<li>Follow the remaining prompts.</li>
</ol>
<p>Let us try this as a worked example using the <a href="https://login.salesforce.com/packaging/installPackage.apexp?p0=04tB00000009UeX">Install the DreamHouse app package</a> from the Trailhead unit</p>
<ol>
<li>Obtain the package installation URL:<br/>Just right click on the package link and Copy Link Address. The approach will vary based on the browser, but there should be a fairly simple way to extract the link.<br/>https://login.salesforce.com/packaging/installPackage.apexp?p0=04tB00000009UeX</li>
<li>Copy everything from that URL except for the domain:<br/><b>/packaging/installPackage.apexp?p0=04tB00000009UeX</b><br/>The ID with the <a href="http://www.fishofprey.com/2011/09/obscure-salesforce-object-key-prefixes.html#04t">04t keyprefix</a> is the important part here. That identifies what package/app you are installing.</li>
<li>Log into the org/playground where you want to install the package<br/><a href="https://2.bp.blogspot.com/-JUvgWtx4tYo/Wsv21yN7K8I/AAAAAAAD2CA/VMIDGgTKXgQAURj607nmF_OjudVXa6SqgCLcBGAs/s1600/LaunchPlaygrount.gif" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-JUvgWtx4tYo/Wsv21yN7K8I/AAAAAAAD2CA/VMIDGgTKXgQAURj607nmF_OjudVXa6SqgCLcBGAs/s1600/LaunchPlaygrount.gif" data-original-width="561" data-original-height="301" /></a></li>
<li id="step4">Paste the content copied from Step 2 after the domain.<br/>
<b>URL before:</b> <b>https://curious-raccoon-286917-dev-ed.lightning.force.com</b>/one/one.app#/home<br/>
<b>URL after:</b> <b>https://curious-raccoon-286917-dev-ed.lightning.force.com</b><u>/packaging/installPackage.apexp?p0=04tB00000009UeX</u><br/>
<a href="https://1.bp.blogspot.com/-VQipC_Aqmuw/Wsv3b-qCs4I/AAAAAAAD2CI/NX5TqDPyB0E9MqEO0PIcNnyfilHVD-I_QCLcBGAs/s1600/PackageURL.gif" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-VQipC_Aqmuw/Wsv3b-qCs4I/AAAAAAAD2CI/NX5TqDPyB0E9MqEO0PIcNnyfilHVD-I_QCLcBGAs/s1600/PackageURL.gif" data-original-width="720" data-original-height="301" width="720" /></a></li>
<li>Follow the remaining prompts.</li>
</ol>
<h2>AppExchange packages</h2>
<p>A similar technique can be used with the AppExchange. The only catch here is they make it a bit harder to get the package version id. Lets use the <a href="https://appexchange.salesforce.com/appxListingDetail?listingId=a0N30000004gHhLEAU">Salesforce Adoption Dashboards</a> as an example app.</p>
<ol>
<li>From the <a href="https://appexchange.salesforce.com/appxListingDetail?listingId=a0N30000004gHhLEAU">app listing</a>, press "Get It Now".</li>
<li>You may need to login to the AppExchange. It doesn't matter what account you log in with at this step.</li>
<li>On the "Where do you want to install this package?" select either the "Install in Production" or "Install in Sandbox" buttons. It shouldn't matter.
</li>
<li>Move past the Confirm Installation Details page and press "Confirm and Install"</li>
<li>You will end up at a login page. <b>DO NOT LOGIN HERE!</b><br/>Look closely at the URL. You will see the package version ID.<br/>
In my case it was https://login.salesforce.com/?startURL=%2Fpackaging%2FinstallPackage.apexp%3Fp0%3D<u><b>04t410000009jsfAAA</b></u>%26newUI%3D1%26src%3Du
<br/><a href="https://1.bp.blogspot.com/-eR-aJLxpv7k/Wsv899bon4I/AAAAAAAD2Cg/lmtk0kClNz08CmTx25wMhNEljETTLBBxACLcBGAs/s1600/PackageVersionId.png" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-eR-aJLxpv7k/Wsv899bon4I/AAAAAAAD2Cg/lmtk0kClNz08CmTx25wMhNEljETTLBBxACLcBGAs/s1600/PackageVersionId.png" data-original-width="1048" data-original-height="95" width=720 /></a></li>
<li>Append that ID to /packaging/installPackage.apexp?p0= on your actual target org as you did in <a href="#step4">step 4</a> of the URL Manipulation Maneuver.<br/>/packaging/installPackage.apexp?p0=04t410000009jsfAAA</li>
</ol>
<hr/>
<p>These steps may look complicated at first, but in reality it is a small cut-and-paste job to find the package version Id. Once you have the Id the rest of the process becomes much simpler. You could even look at <a href="https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_dev2gp_install_pkg_cli.htm">automating the process using the SalesforceDX CLI</a>.</p>
<p>The real benefit is for those that routinely work between multiple orgs. It provides more certainty that you are installing in the org you intended to. Plus you don't even need to figure out your Trailhead playgrounds authentication details.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">¯\_(ツ)_/¯<br>When you jump between multiple orgs all day directly changing the URL often seems like the easiest way to target the correct org. Way to often the auth process sends me into an unexpected org.</p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/983468022554812416?ref_src=twsrc%5Etfw">April 9, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-16318487462332766752018-04-01T21:59:00.000+12:002018-04-01T21:59:52.008+12:00Breeding your own deployment fish<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-n5700wnmSDo/WsChfRgfoUI/AAAAAAAD15I/Hzu1dhrZEKAMJrafTbMARuPzVT1voQbkACLcBGAs/s1600/DXjEuE_VQAAg6yM.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-n5700wnmSDo/WsChfRgfoUI/AAAAAAAD15I/Hzu1dhrZEKAMJrafTbMARuPzVT1voQbkACLcBGAs/s320/DXjEuE_VQAAg6yM.jpg" width="320" height="296" data-original-width="1600" data-original-height="1480" /></a></div>
<p>Sometimes it just isn't practical to head out into the ocean to catch your own deployment fish. Or the metadata gods don't favor your change set with a fresh catch.</p>
<p>What are you to do if the deployment fish aren't biting?</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">That is quite improper... good luck!<br>Is there a guide to making <a href="https://twitter.com/hashtag/deploymentfish?src=hash&amp;ref_src=twsrc%5Etfw">#deploymentfish</a> ? <a href="https://twitter.com/deploymentfish?ref_src=twsrc%5Etfw">@deploymentfish</a></p>&mdash; NA8 Pepper (@SForceBeWithYou) <a href="https://twitter.com/SForceBeWithYou/status/980166121432084480?ref_src=twsrc%5Etfw">March 31, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>JavaScript to the rescue!</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Experimenting with <a href="https://twitter.com/deploymentfish?ref_src=twsrc%5Etfw">@deploymentfish</a> anatomy. Too much Tail? <a href="https://t.co/VNp0imQ4I7">pic.twitter.com/VNp0imQ4I7</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/800484684165574656?ref_src=twsrc%5Etfw">November 20, 2016</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>A few moments of playing on the /changemgmt/monitorDeploymentsDetails.apexp page reveals that JSON data about the current deployment status flows through <code>SfdcApp.MonitorDeployment.InProgressComponent.refreshInProgressSection</code>. We can call the same function ourselves and manipulate the <code>totalComponentsCount</code> and <code>succeededComponentsCount</code> data as required:</p>
<p>
<code>
SfdcApp.MonitorDeployment.InProgressComponent.refreshInProgressSection(Sfdc.JSON.parse('{"hasErrors":false,"hasFatalError":false,"refreshInternalInMillis":3000,"hasCodeCoverageError":false,"totalTestsCount":20,"totalComponentsCount":4,"succeededComponentsCount":21,"isComponentSaveFailing":false,"isDeployComplete":true,"failedComponentsCount":0,"failedTestsCount":0,"isDeployCanceled":false,"completedDate":"4/1/2018 1:33 AM","hasTestRunStarted":false,"isCheckOnly":true,"isTestRunFailing":false,"isAbortRequested":false,"hasPayloadError":false,"isTestRunRequired":false,"stateDetail":"","succeededTestsCount":20,"deployStatus":"Succeeded"}'), !0);
</code>
</p>
<p>Better yet, we can wrap it in our own function to call as required.</p>
<pre>
function deploymentFish(a, b) {
var c = Sfdc.JSON.parse(document.getElementById(chartDataHiddenElementId).value);
c.succeededComponentsCount = a;
c.totalComponentsCount = b;
document.getElementById(chartDataHiddenElementId).value = Sfdc.JSON.stringify(c);
SfdcApp.MonitorDeployment.InProgressComponent.refreshInProgressSectionBasedOnServerData();
}
</pre>
<p>Or, if you prefer, in <a href="https://en.wikipedia.org/wiki/Bookmarklet">bookmarklet</a> form: (Installation link for <a href="javascript:(function(){
var d = prompt('Deployment fish size?', '17/14');
var a = parseInt(d.split('/')[0]);
var b = parseInt(d.split('/')[1]);
var c = Sfdc.JSON.parse(document.getElementById(chartDataHiddenElementId).value);
c.succeededComponentsCount = a;
c.totalComponentsCount = b;
document.getElementById(chartDataHiddenElementId).value = Sfdc.JSON.stringify(c);
SfdcApp.MonitorDeployment.InProgressComponent.refreshInProgressSectionBasedOnServerData();
})();">Deployment Fish</a>)</p>
<pre>
javascript:(function(){
var d = prompt('Deployment fish size?', '17/14');
var a = parseInt(d.split('/')[0]);
var b = parseInt(d.split('/')[1]);
var c = Sfdc.JSON.parse(document.getElementById(chartDataHiddenElementId).value);
c.succeededComponentsCount = a;
c.totalComponentsCount = b;
document.getElementById(chartDataHiddenElementId).value = Sfdc.JSON.stringify(c);
SfdcApp.MonitorDeployment.InProgressComponent.refreshInProgressSectionBasedOnServerData();
})();
</pre>
FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-30049511424790268692018-03-12T22:08:00.000+13:002018-03-23T22:10:37.824+13:00The Trailhead Electric Imp project<div class="separator" style="clear: both; text-align: center;"><a href="https://developer.salesforce.com/resource/images/trailhead/badges/projects/trailhead_project_build_an_iot_integration_with_electric_imp.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://developer.salesforce.com/resource/images/trailhead/badges/projects/trailhead_project_build_an_iot_integration_with_electric_imp.png" width="180" height="180" data-original-width="200" data-original-height="200" /></a></div>
<p>For some time now I've been meaning to complete the <a href="https://trailhead.salesforce.com/en/projects/workshop-electric-imp">Electric Imp Trailhead project</a>. Unfortunately there were a couple of things holding me back from completing it.</p>
<p>Firstly, I needed the physical <a href="https://store.electricimp.com/collections/featured-products/products/impexplorer-developer-kit-for-salesforce-trailhead?variant=31720746706">impExplorer Developer Kit hardware</a>.</p>
<p>The board itself is a pretty reasonable $25 USD. And then I got to the shipping options to New Zealand in the cart and it looked something like this: (circa 2017)</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-TZFi0CKSjBI/WpUXcq4OFNI/AAAAAAADzoM/f0ezJOWmX_wjMKaf8aWw0n97qeQ7qR7cACLcBGAs/s1600/ShippingMethod.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-TZFi0CKSjBI/WpUXcq4OFNI/AAAAAAADzoM/f0ezJOWmX_wjMKaf8aWw0n97qeQ7qR7cACLcBGAs/s1600/ShippingMethod.PNG" data-original-width="606" data-original-height="286" /></a></div>
<p>Ouch! A bit hard to justify $116.49 shipping on a $25 purchase. To be fair, I checked it again today while writing this post and they now have some much more reasonable options via USPS starting at $16.33 USD.</p>
<p>Anyway, as luck would have it I found some time to sit down in the Dreamforce '17 Developer Forest and start working through the project on site with the provided hardware. For some reason that I can't recall I opted to work through this module with an existing Developer edition org (lack of time to delete and then create a new trail org?). It was also my downfall, as the org had a namespace defined and the IoT Contexts didn't seem to support namespaces. There were GACKs all over the place. Lesson learned, next time I'll be tackling it against a clean org.</p>
<p>Fast forward to today, some months after Dreamforce and I've got some spare time to sit down and tackle this again. An no, I haven't forgotten the second thing that was holding me back. I'm just getting to that now.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-dHkLJWk-eIQ/WpfCnE_9w9I/AAAAAAADzqw/zBHmHVgWGIwlTEJGHidXREAE8JxeH_mdACLcBGAs/s1600/20180301_215033.jpg" imageanchor="1" style=" float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-dHkLJWk-eIQ/WpfCnE_9w9I/AAAAAAADzqw/zBHmHVgWGIwlTEJGHidXREAE8JxeH_mdACLcBGAs/s320/20180301_215033.jpg" width="180" height="320" data-original-width="900" data-original-height="1600" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://1.bp.blogspot.com/-4-zsJ-sKIV8/WpfCnFwbBHI/AAAAAAADzqs/pPKAtLurQL4yErvE2djnpJ873pKnpGN8wCLcBGAs/s1600/20180301_215024.jpg" imageanchor="1" style=" float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-4-zsJ-sKIV8/WpfCnFwbBHI/AAAAAAADzqs/pPKAtLurQL4yErvE2djnpJ873pKnpGN8wCLcBGAs/s320/20180301_215024.jpg" width="180" height="320" data-original-width="900" data-original-height="1600" /></a></div>
<p>I needed a fridge to use for the project. One that also got a good WiFi connection. I don't know about you, but my actual fridge is a bit of a WiFi dead zone. It's probably either the proximity of the microwave or the Faraday cage I wrapped it in to protect my ripe avocados from EMP attacks. This was an easy enough problem to solve. Like anyone else with a 3D printer, the solution was only several hours of plastic extrusion away. It turns out there was a purpose made fridge ready to go on Thingiverse- <a href="https://www.thingiverse.com/thing:2645340">Model Fridge for Salesforce IoT Electric Imp Developer Board</a>.</p>
<p>With my new mini fridge, electric imp board, and three AA-batteries-I-borrowed-from-my-sons-remote-control-car-but-will-replace-before-he-wakes-up I embarked on completing the project.</p>
<p>There appears to be two general ways to approach this trailhead project. You can either follow the instructions carefully and double check everything as you go (my chosen path), or you can just jump through pressing <button style="color: #fff;background: #3c9040;min-height: 25px;padding: 0 1rem;border-radius: 5px; border: 0 none;" onclick="alert('Winner, winner, chicken dinner!');">Verify Step</button> as fast as you can because each step ends with the statement:</p>
<blockquote>We won’t check any of your setup. Click <b>Verify Step</b> to go to the next step in the project.</blockquote>
<p><b>Boo!</b> This is actually a real pain, as there isn't anything to indicate if you've taken a misstep at any point along the way.</p>
<p>That said, it is all fairly straight forward if you are proficient at following instructions plus cutting and pasting.</p>
<p>The IoT orchestrations were new to me and hence the most challenging part of the project. For instance, I initially found it confusing that the conditions you define on a State are the exit criteria that transition to the other states. I guess this makes sense as a finite-state machine would only be concerned with the transitions it can make from the current state. E.g. If I'm in a door open state, I'm only interested in transitioning back to the default state when the door is closed again.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-l05Mcx9vf30/WqY40r0NPqI/AAAAAAAD1AI/IdtW3mHx5NUmeqn8V3T71FXmhYGdPy0tACLcBGAs/s1600/DoorOpen.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-l05Mcx9vf30/WqY40r0NPqI/AAAAAAAD1AI/IdtW3mHx5NUmeqn8V3T71FXmhYGdPy0tACLcBGAs/s400/DoorOpen.PNG" width="400" height="83" data-original-width="992" data-original-height="205" /></a></div>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-NYY56l5D5_c/WqY4iKMQvmI/AAAAAAAD1AE/anDFyk3FU8QWKUHqlIDtSraAfaCxHMeAACLcBGAs/s1600/OverTemperature.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-NYY56l5D5_c/WqY4iKMQvmI/AAAAAAAD1AE/anDFyk3FU8QWKUHqlIDtSraAfaCxHMeAACLcBGAs/s400/OverTemperature.PNG" width="400" height="178" data-original-width="878" data-original-height="390" /></a></div>
<p>As the Electric Imp is communicating to Salesforce via Platform Events it provided an easy mechanism to send in mock readings via Apex. This helped with testing when your child wanted the AA batteries back.</p>
<pre class='brush: java'>
Smart_Fridge_Reading__e mockReading = new Smart_Fridge_Reading__e();
mockReading.deviceId__c = '23733t1ed87bf1ee';
mockReading.door__c = 'Closed';
mockReading.humidity__c = 10.441;
mockReading.temperature__c = 8.6859;
mockReading.ts__c=DateTime.now();
EventBus.publish(mockReading);
</pre>
<p>Things I'd change? I'd probably take the light sensor LUX reading and relay that directly back to Salesforce rather than having a configured level to indicate that the door is open. I'm assuming it has the current form to show how values can be passed back from the Device to the Agent for additional processing. I'm also interested in the <a href="https://developer.electricimp.com/hardware/resources/reference-designs/explorerkit/#impexplorer-kit-sensors">accelerator and air pressure sensor</a> which are part of the imp001 hardware but aren't utilized in this badge. Maybe another state if the fridge door is closed too hard?</p>
<hr/>
<p>If you go away from the project for several days you might get the following error message from the Agent code:</p>
<blockquote>
[Agent] ERROR: [ { "message": "Session expired or invalid", "errorCode": "INVALID_SESSION_ID" } ]
</blockquote>
<p>In my case that was caused by the agent code persisting the access_token that it gets via the initial OAuth process. There isn't an automated mechanism to drop the expired session details or refresh it. Instead you need to use the Agent URL to complete the OAuth process again.</p>
<p>See <a href="https://github.com/electricimp/Salesforce/issues/13">Expired OAuth details being restored by getStoredCredentials()</a></p>
<hr/>
<p>Other related Trailhead modules:</p>
<ul>
<li><a href="https://trailhead.salesforce.com/en/modules/platform_events_basics">Platform Events Basics</a></li>
<li><a href="https://trailhead.salesforce.com/en/modules/iot_explorer_basics">Salesforce IoT Explorer Edition Basics</a></li>
</ul>
FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-56612378102138890812017-11-17T22:37:00.000+13:002017-11-28T08:36:05.171+13:00Dreamforce 2017 Round-up / Summary<div class="separator" style="clear: both; text-align: center;"></div>
<div class="separator" style="clear: both; text-align: center;">
<figure style="clear: right; float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://1.bp.blogspot.com/-oqhgeVdXQxA/WgqmB2Fi25I/AAAAAAADlfI/xiJG1wXHAq4jJpU_gRCIk7IKvNSYHe13gCLcBGAs/s1600/HoldingOntoAgenda.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-oqhgeVdXQxA/WgqmB2Fi25I/AAAAAAADlfI/xiJG1wXHAq4jJpU_gRCIk7IKvNSYHe13gCLcBGAs/s320/HoldingOntoAgenda.png" width="320" height="245" data-original-width="1120" data-original-height="858" /></a>
<figcaption>My carefully curated session agenda. It didn't last long.</figcaption>
</figure>
</div>
<p>In what is becoming a <a href="http://www.fishofprey.com/2016/10/dreamforce-2016-round-up-summary.html">tradition for me</a> I went into Dreamforce 2017 with a wildly optimistic 72 breakouts, theater sessions, and keynotes bookmarked that I wanted to attend. Like Codey desperately holding onto the clock on the Agenda page, I knew that was never going to work.</p>
<p>Instead I had a few key sessions and keynotes that I knew I wanted to attend. Beyond that, I just tried to go with the flow of the conference and to focus on doing things that I knew I could only do there in person.</p>
<p>It was certainly several whirlwind days that I'm only just now starting to piece back together from a trail of photos on my phone and tweets. Even seeing some of it coalescing in a single blog post is somewhat daunting assemblage.</p>
<!--Which reminds me - to the person who stopped me on the street who-->
<h2 style="clear: both;">Table of Contents</h2>
<ol>
<li><a href="#preconference">Preconference</a></li>
<li><a href="#astro">Self Driving Astro in the IoT Grove</a></li>
<li><a href="#day1">Day One</a>
<ul>
<li>Mini Hacks</li>
<li>Main Keynote</li>
</ul>
</li>
<li><a href="#day2">Day Two</a>
<ul>
<li>Mass Actions, Composite, and Bulk APIs</li>
<li>Salesforce Platform Limits</li>
<li>Developer Keynote</li>
<li>Meet the Apex Developers</li>
<li>Dreamfest</li>
</ul>
</li>
<li><a href="#day3">Day Three</a>
<ul>
<li>Advanced Logging Patterns With Platform Events</li>
<li>Build Custom Setup Apps & Config Tools With the All-New Apex Metadata API</li>
<li>The Future of Salesforce DX</li>
<li>True to the Core</li>
<li>Meet the Developers</li>
</ul>
</li>
<li><a href="#day4">Day Four</a>
<ul>
<li>Lightning Round Table</li>
<li>Salesforce Tower</li>
</ul>
</li>
<li><a href="#peanutButter">Walking Peanut butter distributor</a></li>
<li><a href="#random">Random Photos</a></li>
</ol>
<hr/>
<h2 style="clear: both;" id="preconference">Preconference</h2>
<div class="separator" style="text-align: center;"><a href="https://1.bp.blogspot.com/-bfRQLS3BSYo/Wgqru6YR1hI/AAAAAAADlfg/3qUGcIkAuPIn8-8qIgjyaTnsS8ffYyXSgCLcBGAs/s1600/20171105_131601.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-bfRQLS3BSYo/Wgqru6YR1hI/AAAAAAADlfg/3qUGcIkAuPIn8-8qIgjyaTnsS8ffYyXSgCLcBGAs/s320/20171105_131601.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://2.bp.blogspot.com/-P70atWjCFPY/WgqruSaHpaI/AAAAAAADlfY/y6E5ntfEhvYXmYOrTEJF7OYsQgZwx5tLwCLcBGAs/s1600/20171105_131608.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-P70atWjCFPY/WgqruSaHpaI/AAAAAAADlfY/y6E5ntfEhvYXmYOrTEJF7OYsQgZwx5tLwCLcBGAs/s320/20171105_131608.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://1.bp.blogspot.com/-6pXLU1CFwpU/Wgqt2ka0QFI/AAAAAAADlf0/gbnAvYdTMHIsPSgxjZa2b7fysiFcdkfcACLcBGAs/s1600/20171105_192313_tower.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-6pXLU1CFwpU/Wgqt2ka0QFI/AAAAAAADlf0/gbnAvYdTMHIsPSgxjZa2b7fysiFcdkfcACLcBGAs/s320/20171105_192313_tower.jpg" width="320" height="192" data-original-width="1600" data-original-height="962" /></a></div>
<p style="clear: right;">According to my phone the first on the ground Dreamforce thing I did after picking up my badge was head into Moscone West. Pretty much business as usually, <i>except it was the Sunday before the conference opened</i>.</p>
<p>I was there to setup my Self Driving Astro. More on that <a href="#astro">below</a>.</p>
<p>It was an interesting glimpse into what goes into setting up such a massive conference. There were <b>a lot</b> of people working hard and putting in some big hours to get everything setup and going in time of the opening the following morning.</p>
<p>On Sunday afternoon I also participated in the MVP volunteering event</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Perfect way to kick off <a href="https://twitter.com/Dreamforce?ref_src=twsrc%5Etfw">@Dreamforce</a>! Volunteering with the MVPs at E.R. Taylor Elementary School warms my ❤️ <a href="https://t.co/FPz6bWuksh">pic.twitter.com/FPz6bWuksh</a></p>&mdash; ericakuhl (@ericakuhl) <a href="https://twitter.com/ericakuhl/status/927414216578887682?ref_src=twsrc%5Etfw">November 6, 2017</a></blockquote>
<h2 id="astro" style="clear: both;">Self Driving Astro in the IoT Grove</h2>
<p>Way back in April this year after giving my <a href="http://www.fishofprey.com/2017/03/the-mad-catter-salesforce-predictive.html">automated Einstein powered cat sprinkler talk</a> to the Sydney Developer User Group I'd latched onto the idea to use the Einstein Vision Services to make a rudimentary self driving car. It was an itch that needed to be scratched. So I started working on it on and off since then and submitted it as a possible talk for Dreamforce.</p>
<p>Sadly it didn't get accepted as a talk this year. I suspect it was a little too far off the actual promoted use cases for the Einstein services. So while it was fun in concept, it could have lead to some confusion with customers about what the Salesforce offerings would be used for. Contrast it to something like the <a href="https://success.salesforce.com/Sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yuqDQAQ">Identify Protein Structures Using Einstein</a> talk, which sounds interesting in its own right and is on my list of sessions to catch up on.</p>
<p>It did leave me with a partially finished project that I'd put in all that effort and resources into. So rather than just mothball it I decided to carry on and make it a <a href="http://www.fishofprey.com/2017/10/teaching-salesforce-to-drive-with.html">blog post instead</a>. I'm glad I did, as it has been an excellent excuse to dive deep into the Metamind APIs and contribute to some GitHub projects along the way. It also formed part of an extended Einstien vision talk that I gave to the <a href="https://www.meetup.com/New-Zealand-Salesforce-Developer-Group/events/243903167/">Auckland Developer User Group</a>.</p>
<p>I published the blog post about it on the 20th of October. That was just over two weeks out from Dreamforce to avoid it getting lost in the noise of the conference. Then the next day Reid Carlberg asked if I was bringing it Dreamforce and if it could be setup on display. I jumped at the opportunity and set about reworking it from something that had only ever been run for 10 to 15 minutes at a time to something that would have to run continuously for 9+ hours a day. I had 11 days before I needed to be on my flight for Dreamforce.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-uDDt3V2L8tw/Wg1Cw4rCMMI/AAAAAAADljM/6jez0wSLbFMe1YLati5m_5OsXrEIpP7QACLcBGAs/s1600/AstroModel.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-uDDt3V2L8tw/Wg1Cw4rCMMI/AAAAAAADljM/6jez0wSLbFMe1YLati5m_5OsXrEIpP7QACLcBGAs/s320/AstroModel.PNG" width="320" height="314" data-original-width="709" data-original-height="695" /></a></div>
<p>Needless to say, it was a busy run up to departing for Dreamforce to:</p>
<ul>
<li>Print a new slightly larger seat,</li>
<li>Design and print a new steering column,</li>
<li>Upgrade to a larger metal gear servo,</li>
<li>Print a new steering wheel for the new servo,</li>
<li>Print a modified case for the Raspberry Pi to support the HAT and relay,</li>
<li>Design and print a case for the Larson Scanner board to prevent shorts,</li>
<li>Reinforce all the wiring to survive travel in checked luggage.</li>
</ul>
<p>Somehow it all came together in time along with a number of software updates to minimize the API counts and make the error handling more resilient. Many thanks to:</p>
<ul>
<li><a href="https://twitter.com/ReidCarlberg">Reid Carlberg</a> for the opportunity to set it up in the IoT Grove.</li>
<li><a href="https://twitter.com/MichaelEMachado">Michael Machado</a> for giving me a temporary bump on the Metamind API limits.</li>
<li><a href="https://twitter.com/joshbirk">Josh Birk</a> for helping to get it setup and running. Plus keeping an eye on it during the conference along with <a href="https://twitter.com/codefriar">@codefriar</a> and others in the IoT Grove.</li>
</ul>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Look who you can find driving away in the Moscone West IoT Grove at <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a>. Insert your hand for Larson scanning. <a href="https://t.co/e4rXSL51rm">pic.twitter.com/e4rXSL51rm</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/927911085130235904?ref_src=twsrc%5Etfw">November 7, 2017</a></blockquote>
<h2 id="day1">Day One</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-t6MkjUzq9AQ/WgvyLJ0XoEI/AAAAAAADlhM/h5gQbtiDkhcGxX8Tu7zX5d1hQZFOEsHVACLcBGAs/s1600/20171106_074410.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-t6MkjUzq9AQ/WgvyLJ0XoEI/AAAAAAADlhM/h5gQbtiDkhcGxX8Tu7zX5d1hQZFOEsHVACLcBGAs/s320/20171106_074410.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<p>The first day went past in a bit of a blur. I was in and about the Mini hacks area volunteering during the morning and then over to the main keynote in the afternoon. In between I did a bit of exploring around Moscone West.</p>
<p>There were a number of areas touched on in the main keynote. In particular:</p>
<ul>
<li>the customization of <a href="https://www.salesforce.com/blog/2017/11/mylightning-lightning-customization">myLightning</a> to change the color themes and visual designs of the UX.</li>
<li><a href="https://www.salesforce.com/blog/2017/11/mysalesforce-branded-mobile-apps.html">mySalesforce </a> for creating mobile apps in Lightning.</li>
<li><a href="https://www.salesforce.com/blog/2017/11/mytrailhead-reinventing-trailblazer-learning.html">myTrailhead</a> for Trailhead customized to other companies.</li>
</ul>
<p>Also confetti - lots and lots of confetti.</p>
<blockquote class="twitter-video" data-lang="en"><p lang="en" dir="ltr">Confetti at the <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> keynote <a href="https://t.co/9ME0C5q5dS">pic.twitter.com/9ME0C5q5dS</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/927700779992891392?ref_src=twsrc%5Etfw">November 7, 2017</a></blockquote>
<h2 id="day2">Day Two</h2>
<p>The day started off with finding confetti from the keynote just about everywhere in my bag.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Going into day two of <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> and taking handfuls of confetti out of my conference bag. <a href="https://twitter.com/hashtag/GoodConferenceProblems?src=hash&amp;ref_src=twsrc%5Etfw">#GoodConferenceProblems</a> <a href="https://t.co/3xlFwoN3bh">pic.twitter.com/3xlFwoN3bh</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/927904555924254720?ref_src=twsrc%5Etfw">November 7, 2017</a></blockquote>
<h3 style="clear: both;">Mass Actions, Composite, and Bulk APIs</h3>
<div class="separator" style="text-align: center;"><a href="https://2.bp.blogspot.com/-K5QLJIBvQTs/Wg01oMSuraI/AAAAAAADliE/9njnx61ueN0jY8eb017asvAb-pz7tYunwCLcBGAs/s1600/20171107_093648.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-K5QLJIBvQTs/Wg01oMSuraI/AAAAAAADliE/9njnx61ueN0jY8eb017asvAb-pz7tYunwCLcBGAs/s320/20171107_093648.jpg" width="320" height="206" data-original-width="1600" data-original-height="1029" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://1.bp.blogspot.com/-OuiTmJpe_Po/Wg01kbUU_aI/AAAAAAADlh4/oVA0zv911ncfGRngiDNKQ8EhdMfb1wiIgCLcBGAs/s1600/20171107_093710.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-OuiTmJpe_Po/Wg01kbUU_aI/AAAAAAADlh4/oVA0zv911ncfGRngiDNKQ8EhdMfb1wiIgCLcBGAs/s320/20171107_093710.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://3.bp.blogspot.com/-rZMc7Ctg4Oc/Wg01kdol3lI/AAAAAAADlh8/dUF6WupRRvkerEsycRqUYTk9uqvOopkNQCLcBGAs/s1600/20171107_095411.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-rZMc7Ctg4Oc/Wg01kdol3lI/AAAAAAADlh8/dUF6WupRRvkerEsycRqUYTk9uqvOopkNQCLcBGAs/s320/20171107_095411.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-swdZolWPjfU/Wg01nYV48XI/AAAAAAADliA/I63AvnHl_ps1pGKfVv7b0O2d_u1YCH1lgCLcBGAs/s1600/20171107_122928.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-swdZolWPjfU/Wg01nYV48XI/AAAAAAADliA/I63AvnHl_ps1pGKfVv7b0O2d_u1YCH1lgCLcBGAs/s320/20171107_122928.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<p style="clear: both;">I managed to catch my first full session today by <a href="https://twitter.com/thunderberry">@thunderberry</a> and <a href="https://twitter.com/abhinavchadda">@abhinavchadda</a>. While both the Bulk API v2 improvements and Open API support sound really useful, what intrigued me most was the proposed Mass Actions API. This could radically speed up a number of operations by defining a transform over a query. The exact details on how it will be split into transactions and how limits will be addressed is something to keep an eye on with this. </p>
<h3 style="clear: both;">Salesforce Platform Limits</h3>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-NV_mBmc0Yl4/Wg04bQUXlbI/AAAAAAADlig/qnSSSUAJ4m4WBE6IKn4bnIFLdRSKw5ZoACLcBGAs/s1600/20171107_122019.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-NV_mBmc0Yl4/Wg04bQUXlbI/AAAAAAADlig/qnSSSUAJ4m4WBE6IKn4bnIFLdRSKw5ZoACLcBGAs/s320/20171107_122019.jpg" width="320" height="181" data-original-width="1600" data-original-height="905" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://1.bp.blogspot.com/-VbZ1ebSXdQY/Wg04ciSrnuI/AAAAAAADliw/Q1XlC8ZjM7sdFisnLKsp33hxJVXSdXzewCLcBGAs/s1600/20171107_122154.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-VbZ1ebSXdQY/Wg04ciSrnuI/AAAAAAADliw/Q1XlC8ZjM7sdFisnLKsp33hxJVXSdXzewCLcBGAs/s320/20171107_122154.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://2.bp.blogspot.com/-uvCCQanTxws/Wg04cvRSjXI/AAAAAAADlis/Zl-r6e6EAuY-44-kGX5kGv_kAD0l7IRaQCLcBGAs/s1600/20171107_122248.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-uvCCQanTxws/Wg04cvRSjXI/AAAAAAADlis/Zl-r6e6EAuY-44-kGX5kGv_kAD0l7IRaQCLcBGAs/s320/20171107_122248.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-j55uQ0U1k5M/Wg04dE3Yb9I/AAAAAAADli0/RXywNlybb6UICIFr7paXSLFHb8NkVSwvACLcBGAs/s1600/20171107_122638.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-j55uQ0U1k5M/Wg04dE3Yb9I/AAAAAAADli0/RXywNlybb6UICIFr7paXSLFHb8NkVSwvACLcBGAs/s320/20171107_122638.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://1.bp.blogspot.com/-4PT2b5HTlLY/Wg04fEnqqhI/AAAAAAADli8/8eIFuZZCE2kAtSAd-Up60AUdI0_dQXVAwCLcBGAs/s1600/20171107_122855.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-4PT2b5HTlLY/Wg04fEnqqhI/AAAAAAADli8/8eIFuZZCE2kAtSAd-Up60AUdI0_dQXVAwCLcBGAs/s320/20171107_122855.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-x5Al_QTStqo/Wg04fAJ6Y0I/AAAAAAADli4/h3bw9coAR6Msj6e9w5QAkDTfzY9BG1h8wCLcBGAs/s1600/20171107_122928.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-x5Al_QTStqo/Wg04fAJ6Y0I/AAAAAAADli4/h3bw9coAR6Msj6e9w5QAkDTfzY9BG1h8wCLcBGAs/s320/20171107_122928.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<p style="clear: both;">This session was again with <a href="https://twitter.com/thunderberry">@thunderberry</a>. The proposal here is to move a number of existing limits to become "soft limits". Then depending on the pod heath you would be able to burst past the standard limit for short periods. There would still be an upper maximum burst limit. Consistently running past the soft limit will get you on a naughty list where Salesforce will encourage you to either upgrade or pin you to the existing limits. If the pod is particularly busy you would be more limited in what you could burst to. </p>
<p>This all sounds good in theory, but it will make debugging more difficult. Something that works within the soft limits one moment could fail the next due to the org load. This does seem a bit like the current Apex CPU limit.</p>
<p>A quick tip from this session:</p>
<blockquote>Don't poll the existing limits API to frequently. The calls to that still count against your API call count limit.</blockquote>
<h3 style="clear: both;">Developer Keynote</h3>
<p>The first thing that struck me about the Developer Keynote is what a fashionable bunch the developers are.</p>
<div class="separator" style="text-align: center;">
<figure style="clear: right; float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://3.bp.blogspot.com/-FsKW4Sw4E20/Wg1Feu47SYI/AAAAAAADljg/Z_Ce9_vkqssyOyHvYPIfkIbVX5c3JIT6gCLcBGAs/s1600/20171107_133639.jpg" imageanchor="1" style="margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-FsKW4Sw4E20/Wg1Feu47SYI/AAAAAAADljg/Z_Ce9_vkqssyOyHvYPIfkIbVX5c3JIT6gCLcBGAs/s320/20171107_133639.jpg" width="240" height="320" data-original-width="1200" data-original-height="1600" /></a>
<figcaption><a href="https://twitter.com/FishOfPrey/status/928013713935761408">Codey bear and Deployment Fish sweater.</a> <a href="https://twitter.com/WadeWegner/status/928034366810341376">Made by his dad.</a></figcaption>
</figure>
</div>
<div class="separator" style="text-align: center;">
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://1.bp.blogspot.com/-3JQLXkUeo0E/Wg1F_EdZCLI/AAAAAAADljo/TNhhwwfW0kcrdTR37VJmGgycweH80y4vACLcBGAs/s1600/20171107_135251.jpg" imageanchor="1" style=" margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-3JQLXkUeo0E/Wg1F_EdZCLI/AAAAAAADljo/TNhhwwfW0kcrdTR37VJmGgycweH80y4vACLcBGAs/s320/20171107_135251.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a>
<figcaption><a href="https://twitter.com/FishOfPrey/status/928018204525338625">df17__e - A Platform event shirt</a></figcaption>
</figure>
</div>
<div class="separator" style="text-align: center;">
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://1.bp.blogspot.com/-HemaGM8mzL0/Wg1H-faNywI/AAAAAAADlj4/V57CtR5NX5QQfTzQp_WQfQX-j4xIRTADwCLcBGAs/s1600/SuspiciousLookingPlant.gif" imageanchor="1" style="margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-HemaGM8mzL0/Wg1H-faNywI/AAAAAAADlj4/V57CtR5NX5QQfTzQp_WQfQX-j4xIRTADwCLcBGAs/s320/SuspiciousLookingPlant.gif" width="240" height="320" data-original-width="390" data-original-height="520" /></a>
<figcaption>A "Suspicious Looking Plant" and some Aloe vera.</figcaption>
</figure>
</div>
<p style="clear: both;">If time permits I'll come back to this section to recap some of the content.</p>
<h3 style="clear: both;">Meet the Apex Developers</h3>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">. <a href="https://twitter.com/slettehaugh?ref_src=twsrc%5Etfw">@slettehaugh</a> assures us at the <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> Meet the Apex developers session that switch statement support is at the top of the list now the new Apex compiler is out.</p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928058959059025920?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<p>I asked a question in this session about ENTERING_MANAGED_PKG appearing in debug logs and how it can frequently consume 80% plus of a 2MB log in a dev packaging org. They had been forewarned that this question was coming and thankfully the response was they are onto it and will be fixing the excess logging in the future. 🎉</p>
<h3 style="clear: both;">Dreamfest</h3>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-z4vQp2zbEgY/Wg6kzIgPaxI/AAAAAAADlkc/Nuf5EZuXNGcXPMGyvvOfQZWB28ytw55JQCLcBGAs/s1600/20171107_205543.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-z4vQp2zbEgY/Wg6kzIgPaxI/AAAAAAADlkc/Nuf5EZuXNGcXPMGyvvOfQZWB28ytw55JQCLcBGAs/s320/20171107_205543.jpg" width="320" height="110" data-original-width="1600" data-original-height="550" /></a></div>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">From the top of <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> Dreamfest <a href="https://t.co/Wws5WRe6R6">pic.twitter.com/Wws5WRe6R6</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928120220878749696?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<p>AT&amp;T park made an excellent venue in terms of proximity to most of the hotels, visibility of the stage, and facilities for dealing with that many attendees.</p>
<h2 style="clear: both;" id="day3">Day Three</h2>
<h3><a href="https://success.salesforce.com/myagenda?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000000GUyaQAG">Advanced Logging Patterns With Platform Events</a></h3>
<p>Using Platform Events to push logging events out and then monitoring them using a Utility Bar app.</p>
<h3><a href="https://success.salesforce.com/myagenda?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytx3QAA">Build Custom Setup Apps & Config Tools With the All-New Apex Metadata API</a></h3>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Watch your namespaces when using the Apex metadata API <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a><a href="https://twitter.com/slettehaugh?ref_src=twsrc%5Etfw">@slettehaugh</a> <a href="https://t.co/zusb39Qolk">pic.twitter.com/zusb39Qolk</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928341363988373505?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="in" dir="ltr"><a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> Roadmap for Apex metadata API <a href="https://t.co/EW87hCDYo3">pic.twitter.com/EW87hCDYo3</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928342214421372928?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<h3><a href="https://success.salesforce.com/myagenda?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytBHQAY">The Future of Salesforce DX</a></h3>
<p>This session is well worth a repeat watch. Very dense on upcoming changes with DX. Mostly forward looking, but still interesting.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">New Locked and Unlocked flavors of managed packages for distribution. <br>Future of Salesforce DX <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> by <a href="https://twitter.com/WadeWegner?ref_src=twsrc%5Etfw">@WadeWegner</a> <a href="https://t.co/gQThxX0Jij">pic.twitter.com/gQThxX0Jij</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928355231275954176?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Better support coming to sandboxes to track changes and automated change set creation from that data. <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> <a href="https://twitter.com/hashtag/ForwardLooking?src=hash&amp;ref_src=twsrc%5Etfw">#ForwardLooking</a> <a href="https://t.co/CyyGRJCbQ3">pic.twitter.com/CyyGRJCbQ3</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928357406651719680?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">New Monaco and Language Services based web editor to use instead of Developer Console. <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> <a href="https://twitter.com/hashtag/FowardLooking?src=hash&amp;ref_src=twsrc%5Etfw">#FowardLooking</a> <a href="https://t.co/WGhYl0oDHM">pic.twitter.com/WGhYl0oDHM</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928359421821849600?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">UI for the CLI. Now accepting nominations for a better name. <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> <a href="https://t.co/LDbFSFTqym">pic.twitter.com/LDbFSFTqym</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928359713607114752?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Improvements coming for Apex debug logs. Including streaming to get past the 2mb log size limit. <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> <a href="https://t.co/Hn4RVXIBA3">pic.twitter.com/Hn4RVXIBA3</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928360459194966017?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<h3><a href="https://success.salesforce.com/myagenda?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000000GUwyQAG">Change Data Capture: Data Synchronization in the Cloud</a></h3>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Schema ID exposed via the change data capture events. Can be used to check metadata that existed when event occured. <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> <a href="https://t.co/KEtwUrOMgi">pic.twitter.com/KEtwUrOMgi</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928369635388399616?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Change Data Capture roadmap <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> <a href="https://t.co/jTASYb3muj">pic.twitter.com/jTASYb3muj</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928374840137887745?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<h3>True to the Core</h3>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Field to field filters coming to reports. I can haz in SOQL as well? <a href="https://t.co/ute5are8nu">pic.twitter.com/ute5are8nu</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928403172594352128?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Apex switch statement stated for <a href="https://twitter.com/hashtag/Summer18?src=hash&amp;ref_src=twsrc%5Etfw">#Summer18</a> <a href="https://twitter.com/hashtag/TrueToTheCore?src=hash&amp;ref_src=twsrc%5Etfw">#TrueToTheCore</a> <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> <a href="https://t.co/0hlAgWHuUs">pic.twitter.com/0hlAgWHuUs</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928403493349552128?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<h3>Meet the Developers</h3>
<div class="separator" style="clear: both; text-align: center;">
<figure style="clear: right; float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://2.bp.blogspot.com/-dlz8fPtHozE/Wg6lernCgrI/AAAAAAADlko/FYMs2iAguFIovx91XCauQm70iSJJKUo5QCLcBGAs/s1600/20171108_164241.jpg" imageanchor="1" style="margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-dlz8fPtHozE/Wg6lernCgrI/AAAAAAADlko/FYMs2iAguFIovx91XCauQm70iSJJKUo5QCLcBGAs/s320/20171108_164241.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a>
<figcaption>Steve Tamm rallying the troops prior to the Meet the Developers session</figcaption>
</figure>
</div>
<p>I asked a question in this session about raising support cases for GACKs and those without Premier support being turned away to the developer forums. If you have specific examples of this happening that you want to share please see <a href="https://salesforce.meta.stackexchange.com/q/2657/102">Examples of being bounced by Salesforce Support to forums when encountering a GACK</a></p>
<h2 style="clear: both;" id="day4">Day Four</h2>
<p>Today was a bit interrupted from attending sessions, but in a good way!</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Raw, real, and candid feedback on <a href="https://twitter.com/salesforce?ref_src=twsrc%5Etfw">@salesforce</a> Lighting happening in this room. The brain power is overwhelming! <a href="https://t.co/zODXrQJTq3">pic.twitter.com/zODXrQJTq3</a></p>&mdash; ericakuhl (@ericakuhl) <a href="https://twitter.com/ericakuhl/status/928689555519049728?ref_src=twsrc%5Etfw">November 9, 2017</a></blockquote>
<h3>Salesforce Tower - Ending Dreamforce on a high</h3>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Just going for a stroll down Mission to Fremont. Any chance you can hook us up <a href="https://twitter.com/hollygfirestone?ref_src=twsrc%5Etfw">@hollygfirestone</a>? <a href="https://t.co/aYFXwlskMD">pic.twitter.com/aYFXwlskMD</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928738640456433664?ref_src=twsrc%5Etfw">November 9, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Awesome experience visiting the <a href="https://twitter.com/salesforce?ref_src=twsrc%5Etfw">@Salesforce</a> tower with some of our amazing Salesforce MVPs!! <a href="https://twitter.com/hashtag/df17?src=hash&amp;ref_src=twsrc%5Etfw">#df17</a> <a href="https://t.co/lzTwtYRAtK">pic.twitter.com/lzTwtYRAtK</a></p>&mdash; Holly Firestone (@hollygfirestone) <a href="https://twitter.com/hollygfirestone/status/928755030999445504?ref_src=twsrc%5Etfw">November 9, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Thanks so much <a href="https://twitter.com/hollygfirestone?ref_src=twsrc%5Etfw">@hollygfirestone</a> for taking my crude plan constructed in a paint program and doing all the hard work to turn it into reality. It meant a lot to visit the SFO skyline from such an awesome building. <a href="https://t.co/xdzNLeQE28">pic.twitter.com/xdzNLeQE28</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/928775500624928769?ref_src=twsrc%5Etfw">November 10, 2017</a></blockquote>
<iframe width="560" height="315" src="https://www.youtube.com/embed/zElTTAVfuK4" frameborder="0" allowfullscreen></iframe>
<h2 style="clear: both;" id="peanutButter">Walking Peanut butter distributor</h2>
<p>This year I brought 3kg/6.6lb of <a href="https://www.picspeanutbutter.com">Pic's peanut butter</a> with me to give out to people. It's made right here in Nelson, New Zealand and I figured I'd be the only person giving out peanut butter at Dreamforce. At the very least it made for an interesting conversation starter. Not too many people thought I was the nutty one in the transaction (at least out loud).<br/> And to those with peanut allergies - no hard feelings, can we still be friends? I wasn't really trying to kill you.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-pZURIErNKU0/Wg1FEYk6M3I/AAAAAAADljc/Dpg9HYWNlO83Q_oxGfrE-jMc8VAJvcn_ACKgBGAs/s1600/20171026_082812.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-pZURIErNKU0/Wg1FEYk6M3I/AAAAAAADljc/Dpg9HYWNlO83Q_oxGfrE-jMc8VAJvcn_ACKgBGAs/s320/20171026_082812.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">I think I just get the best goodie of this <a href="https://twitter.com/Dreamforce?ref_src=twsrc%5Etfw">@Dreamforce</a>, thanks <a href="https://twitter.com/FishOfPrey?ref_src=twsrc%5Etfw">@FishOfPrey</a> <a href="https://twitter.com/FuseInfoTech?ref_src=twsrc%5Etfw">@FuseInfoTech</a> ! <a href="https://t.co/2dDjHARHc9">pic.twitter.com/2dDjHARHc9</a></p>&mdash; Fabien Taillon (@FabienTaillon) <a href="https://twitter.com/FabienTaillon/status/928055038668300288?ref_src=twsrc%5Etfw">November 8, 2017</a></blockquote>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/FishOfPrey?ref_src=twsrc%5Etfw">@FishOfPrey</a> knew we were hungry, so he brought us yummy 🥜 butter <a href="https://twitter.com/hashtag/snacks?src=hash&amp;ref_src=twsrc%5Etfw">#snacks</a>! Find him to avoid getting <a href="https://twitter.com/hashtag/hangry?src=hash&amp;ref_src=twsrc%5Etfw">#hangry</a> at <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a>! <a href="https://t.co/tDwf51waaO">pic.twitter.com/tDwf51waaO</a></p>&mdash; Gillian Bruce (@gilliankbruce) <a href="https://twitter.com/gilliankbruce/status/927973511062102016?ref_src=twsrc%5Etfw">November 7, 2017</a></blockquote>
<h2>Sessions I mean to catch up on</h2>
<iframe width="560" height="315" src="https://www.youtube.com/embed/videoseries?list=PL2v7tlhuOlyaWgrY4D_iHOHfyCMlDPgLM" frameborder="0" gesture="media" allowfullscreen></iframe>
<h2 style="clear: both;" id="random">Random Photos</h2>
<div class="separator" style=" text-align: center;"><a href="https://4.bp.blogspot.com/-IPqGdK3jdlw/Wg03ImTMckI/AAAAAAADliU/TaMBcFwa4lwYGaECqLgHn6w-3ad6R_gYwCLcBGAs/s1600/20171107_113101.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-IPqGdK3jdlw/Wg03ImTMckI/AAAAAAADliU/TaMBcFwa4lwYGaECqLgHn6w-3ad6R_gYwCLcBGAs/s640/20171107_113101.jpg" width="640" height="317" data-original-width="1600" data-original-height="792" /></a></div>
<a href="https://2.bp.blogspot.com/-jlOyDL5aHZU/Wg6kNPgA1zI/AAAAAAADlkQ/MRf3d5B4NJ4kjhwBteNVdzmQ2bnY-5ViACLcBGAs/s1600/20171106_121401.jpg" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-jlOyDL5aHZU/Wg6kNPgA1zI/AAAAAAADlkQ/MRf3d5B4NJ4kjhwBteNVdzmQ2bnY-5ViACLcBGAs/s320/20171106_121401.jpg" width="240" height="320" data-original-width="1200" data-original-height="1600" /></a>
<div class="separator" style=" text-align: center;"><a href="https://1.bp.blogspot.com/-vrgst6fWfsc/Wg6kVlpUHeI/AAAAAAADlkU/3I_aOvP_3hgxSkFLiqjweGseB6OQlHdmQCLcBGAs/s1600/20171106_194258.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-vrgst6fWfsc/Wg6kVlpUHeI/AAAAAAADlkU/3I_aOvP_3hgxSkFLiqjweGseB6OQlHdmQCLcBGAs/s320/20171106_194258.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style=" text-align: center;"><a href="https://3.bp.blogspot.com/-Yx4QFEtyktw/Wg6lNJWKvEI/AAAAAAADlkg/hgX2bTxD41ckPxsSrUnzq_yADrNLjd93gCLcBGAs/s1600/20171108_080926.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-Yx4QFEtyktw/Wg6lNJWKvEI/AAAAAAADlkg/hgX2bTxD41ckPxsSrUnzq_yADrNLjd93gCLcBGAs/s320/20171108_080926.jpg" width="320" height="163" data-original-width="1600" data-original-height="817" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-SIqFJaXLeik/Wg6liQdjX9I/AAAAAAADlks/1Nyj79GB4Gg3jBKDsk_YyR7XWRtAC4YoACLcBGAs/s1600/20171108_223555.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-SIqFJaXLeik/Wg6liQdjX9I/AAAAAAADlks/1Nyj79GB4Gg3jBKDsk_YyR7XWRtAC4YoACLcBGAs/s320/20171108_223555.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://1.bp.blogspot.com/-QrO2GuMt_Ek/Wg6lnnaq-tI/AAAAAAADlkw/QlkrEP6125ssBwUlNFG5rnvYfocu9VmPQCLcBGAs/s1600/20171109_083023.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-QrO2GuMt_Ek/Wg6lnnaq-tI/AAAAAAADlkw/QlkrEP6125ssBwUlNFG5rnvYfocu9VmPQCLcBGAs/s320/20171109_083023.jpg" width="240" height="320" data-original-width="1200" data-original-height="1600" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://3.bp.blogspot.com/-QlXYO4gPZIQ/Wg6lzp6c4eI/AAAAAAADlk0/J6IWXwUkGA4BJKGKAOED14Jv2mvMnN5bwCLcBGAs/s1600/20171109_124548.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-QlXYO4gPZIQ/Wg6lzp6c4eI/AAAAAAADlk0/J6IWXwUkGA4BJKGKAOED14Jv2mvMnN5bwCLcBGAs/s320/20171109_124548.jpg" width="320" height="181" data-original-width="1600" data-original-height="904" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-BC5OXcm2nxs/Wg6mRAOKvdI/AAAAAAADlk8/SiAJdxErA4sSsACVsunT3CWPK6GxCuAgwCLcBGAs/s1600/20171110_103120.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-BC5OXcm2nxs/Wg6mRAOKvdI/AAAAAAADlk8/SiAJdxErA4sSsACVsunT3CWPK6GxCuAgwCLcBGAs/s320/20171110_103120.jpg" width="320" height="62" data-original-width="1600" data-original-height="311" /></a></div>
<div style="clear: both;">
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Mental note. Next Dreamforce - vacuum bags! <a href="https://t.co/nAaA83VDXI">pic.twitter.com/nAaA83VDXI</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/935006640981516288?ref_src=twsrc%5Etfw">November 27, 2017</a></blockquote>
</div>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-18511524085526326282017-10-20T10:33:00.000+13:002017-10-20T13:34:00.793+13:00Teaching Salesforce to drive with Einstein Vision Services<p style="float:right;">tl;dr. <a href="#results">Skip to the results</a>.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Technology is not good or bad. It is what we do with it that matters. AI is a gift! May we use it for the betterment of all mankind! <a href="https://t.co/E09tuBYCnZ">pic.twitter.com/E09tuBYCnZ</a></p>&mdash; Marc Benioff (@Benioff) <a href="https://twitter.com/Benioff/status/909269849171402754">September 17, 2017</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<!--div class="separator" style="clear: both; text-align: center;"><div style="margin-bottom: 1em; margin-left: 1em;"><a href="https://4.bp.blogspot.com/-VvmBv5VPMT0/WbuSgkJ53-I/AAAAAAADgtY/80w-gc3dEJERtluRwxqKs9-Vn7v1smjTACLcBGAs/s1600/TonyStarkCatWeapons.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-VvmBv5VPMT0/WbuSgkJ53-I/AAAAAAADgtY/80w-gc3dEJERtluRwxqKs9-Vn7v1smjTACLcBGAs/s320/TonyStarkCatWeapons.png" width="320" height="195" data-original-width="460" data-original-height="280" /></a></div></div-->
<div class="separator" style="clear: both; text-align: center;">
<figure style="clear: right; float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<img border="0" src="https://4.bp.blogspot.com/-VvmBv5VPMT0/WbuSgkJ53-I/AAAAAAADgtY/80w-gc3dEJERtluRwxqKs9-Vn7v1smjTACLcBGAs/s320/TonyStarkCatWeapons.png" width="320" height="207" data-original-width="622" data-original-height="403" />
<figcaption>For your consideration, the <a href="http://www.fishofprey.com/2017/03/the-mad-catter-salesforce-predictive.html">Mad Catter</a></figcaption>
</figure>
</div>
<p>There's something in that tweet that speaks to me. Maybe it's time to pull a Tony Stark and take a new direction with my creations. Moving away from the world of <strike>weapons</strike> <a href="http://www.fishofprey.com/2016/08/integrating-uavdrone-remote-data.html">questionable</a> <a href="http://www.fishofprey.com/2017/03/the-mad-catter-salesforce-predictive.html">development</a> and more towards something for the public good. Something topical.</p>
<p>Tesla, Apple, Uber, Google, Toyota, BMW, Ford, these are just a few of the companies working on creating self-driving cars. Maybe I should dabble with that too.</p>
<p id="limitedResources">My first challenge was that my R&amp;D budget doesn't stretch quite as far as the aforementioned companies. While they all have market values measured in the billions my resources are a little more modest. So the kiwi <a href="https://www.freshtakepublishers.com/blogs/news/new-zealand-colloquialisms-and-kiwi-slang-today-number-eight-wire-mentality">Number Eight Wire Mentality</a> will come into play when <a href="https://en.wikipedia.org/wiki/Lidar">LIDAR</a> isn't an option.</p>
<div class="separator" style="clear: both; text-align: center;"><div style="margin-bottom: 1em; margin-left: 1em;"><blockquote>
Steve Rogers: Big man in a suit of armor. Take that off, what are you?</br>
Tony Stark: Genius billionaire playboy philanthropist.
</blockquote></div></div>
<!--div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-miqgtA9ftME/Wb-Fj536E7I/AAAAAAADgwk/qshIUOq79osQ_TkCOxFFrXOoPgg18K-OgCLcBGAs/s1600/CaveStark.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-miqgtA9ftME/Wb-Fj536E7I/AAAAAAADgwk/qshIUOq79osQ_TkCOxFFrXOoPgg18K-OgCLcBGAs/s320/CaveStark.jpg" width="320" height="213" data-original-width="600" data-original-height="399" /></a></div-->
<div class="separator" style="clear: both; text-align: center;">
<figure style="clear: right; float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<img border="0" src="https://1.bp.blogspot.com/-miqgtA9ftME/Wb-Fj536E7I/AAAAAAADgwk/qshIUOq79osQ_TkCOxFFrXOoPgg18K-OgCLcBGAs/s1600/CaveStark.jpg" width="320" height="207" data-original-width="622" data-original-height="403" />
<figcaption>* Actual project did not require hammering an anvil... yet</figcaption>
</figure>
</div>
<p>It's another way I fancy myself a bit like Tony Stark. Less like the genius, billionaire, playboy Stark and more like trapped in a cave with only the resources at hand Stark. I'll make do with whatever I've got on hand. Some servos, a Raspberry Pi, an old web cam, a massive amount of cloud computing resources.</p>
<p>That last point is important, while Stark has J.A.R.V.I.S. I've got <a href="https://developer.salesforce.com/einstein#section_2">Einstein Platform Services</a>.</p>
<p>I don't have a background in AI as a Data Scientist. My AI exposure has been more from copious amounts of science fiction and that <a href="http://homepages.mcs.vuw.ac.nz/~elvis/db/Publications.shtml#COMP307">one trimester back in university</a> where I did a single paper. <!--But isn't that what university is for? Experimenting with new things to help "find yourself'?--> I'm assuming the depiction of AI in the media is <a href="https://media.giphy.com/media/yUlFNRDWVfxCM/giphy.gif">about as accurate as the depiction of computing in general</a>. So while it's fun to take inspiration from an 80's documentary on self driving cars the most useful learning from something like Knight Rider is that Larson scanners make AI's look cool (more on that later). </p>
<!--h2>Knight Rider - Astro Rider AstroLarson</h2-->
<!--p>Other parallels include:</p>
<ul>
<li>Moving away from the world of <a href="http://www.fishofprey.com/2017/03/the-mad-catter-salesforce-predictive.html">weapons development</a> </li>
<li>One other parallel I can draw that will particularly helpful for this process is access to a J.A.R.V.I.S. like AI that helps with problem solving. Only less sassy and more Einstien like. </li>
</ul-->
<h2>Elephant in the room</h2>
<p>This is probably a good point to pause for a minute and note that, yes, I <i><b>know</b></i> that a solely cloud based AI isn't the best option, or even a very good option, for an autonomous vehicle. An image classification AI would only be part of a larger collection of sensor input used to make a self driving car. Connectivity and latency issues will pretty much rule out any full scale testing or moving at significant speed. That, and, I don't think my vehicle insurance would cover crashing the family car while it was trying to navigate a tricky intersection by itself. But I want to see how far I can push it anyway with a purely optical solution using a single camera.</p>
<p>I know that a number of the components I'm using on the prototype are less than ideal. I.e. using continuous rotation servos rather than stepper motors which would have given a more precise measure of the distance traveled. See my earlier point <a href="#limitedResources">above</a> that I'm mostly working with what I've got on hand. Feel free to contact me if you have a spare sum of cash burning a whole in your pocket that you want to invest in something like this.</p>
<p>Remember, this is just supposed to be a Mark 1 prototype. If it starts to look promising it can be refined in the future.</p>
<h2>The Overly Simplistic Plan</h2>
<p>The general idea for a minimalistic vehicle is:</p>
<ol>
<li>A single forward facing web cam to capture the current position on the road and what is immediately ahead.</li>
<li>A Raspberry Pi to capture the image and process it.</li>
<li>Einstien Predictive Vision Services to identify what is in the current image and the implied probable next course of action to take.</li>
<li>Some basic servo based motor functions to perform the required movements.</li>
<li>Repeat from step (2)</li>
</ol>
<p>I say overly simplistic here in that it doesn't really localize the position of the vehicle in the environment or provide it with an accurate course to follow. Instead it just makes a best guess prediction based on what is currently seen immediately ahead. There is no feedback mechanism based on where the car was previously or how well it is following the required trajectory. </p>
<!--p>There are three important parts to this project.</p>
<ol>
<li>Identifying the current position of the vehicle with respect to the oncoming road. <a href="#positioning">Positioning</a></li>
<li>Identifying the next action to take. <a href="#predicting">Predicting</a></li>
<li>Putting together a hardware platform that can gather the required data for (1) and then move as required by (2).</li>
</ol-->
<!--h2 id="positioning">Positioning</h2>
<p>Given a single input, the current web cam image, can I identify what is immediately ahead of me on the "road".</p>
<h2 id="positioning">Positioning</h2-->
<h3>Hardware - Some assembly required</h3>
<div class="separator" style="clear: both; text-align: center;">
<figure style="clear: right; float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<img border="0" src="https://3.bp.blogspot.com/-7rQCpE5cs1w/WbuDCn9gPnI/AAAAAAADgtI/i9ggX9zFhlYPnyqFStQVsth1Ry78AWzVQCLcBGAs/s320/knightrider.gif" width="320" height="207" data-original-width="622" data-original-height="403" />
<figcaption>The model for the Mk2 Prototype</figcaption>
</figure>
</div>
<p>In the ideal world I'd have repurposed an RC toy car that had full steering and drive capabilities already. Or even just 3D printed a basic car. This would have given a realistic steering behavior via the front wheels. Instead I've started out with skid steering and a dolly wheel on a lego frame. This was quick and easy to put together and drive with some servos from a <a href="http://www.fishofprey.com/2017/06/controlling-analog-pwm-servos-from.html">previous project</a>.</p>
<p>While simple is good for v1, it was still not without it's challenges.</p>
<p>The first problem was that I couldn't just go from zero to full power on the drive wheel servos. It tended to either damage the lego gears or send the whole vehicle rolling over backwards. I needed to accelerate smoothly to get predictable movement.</p>
<div class="separator" style="clear: both; text-align: center;">
<figure style="clear: right; float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<img border="0" src="https://3.bp.blogspot.com/-eOqA1QMsJY4/Wdc_IX82r-I/AAAAAAADiHM/0ISRBfcdQrYaGsg8YJ2LrD1ySK_ghjQLQCLcBGAs/s320/20171004_210118.jpg" width="320" height="207" data-original-width="622" data-original-height="403" />
<figcaption>Custom LiPo based high drain power supply</figcaption>
</figure>
</div>
<p>Another challenge was providing a portable power source. Originally I was powering the whole setup off a fairly stock standard portable cellphone charger to power both the Pi and the servovs. However, when I powered up the servos to move forward the Raspberry Pi would reset. The voltage sag when driving the motors was enough to reset the Pi. I was able to work around this by using one of the LiPo (Lithium polymer) batteries from my quadcopters with a high C rating (they can deliver a lot of power quickly if required) and a pair of step down transformers. </p>
<h3 style="clear: both;">Software</h3>
<p>The standard software control loop is:</p>
<ol>
<li>Capture webcam image</li>
<li>Send image off to the Einstein Image service for the steering Model.</li>
<li>Based on the result with the highest probability, activate servos to move the vehicle</li>
<li>Repeat</li>
</ol>
<p>Capturing the image is fairly straight forward.</p>
<p>Since latency was already going to make fast movement impractical I opted to relay the calls via Salesforce Apex services. This made the authentication easier as I only needed to establish and maintain a valid session with the Salesforce instance to then access the Predictive Vision API via Apex. It also meant I didn't need to reimplement the metamind API and could instead use René Winkelmeyer's <a href="https://github.com/muenzpraeger/salesforce-einstein-platform-apex">salesforce-einstein-platform-apex Github project</a>. This already included workarounds for <a href="http://www.fishofprey.com/2017/04/steps-required-to-support-posting.html">posting multipart requests</a> required by the API.</p>
<h3>Training</h3>
<p>This is something that has evolved greatly since my last project using Einstein Vision Services on the <a href="http://www.fishofprey.com/2017/03/the-mad-catter-salesforce-predictive.html">automated sprinkler</a>. Previously you created a dataset from a pre-collected set of examples and then created your model from that to make the predictions with. If it needed refining you repeated the entire process. Now you can <a href="https://releasenotes.docs.salesforce.com/en-us/winter18/release-notes/rn_einstein_vision_feedback.htm">add feedback directly to a dataset</a> and retrain the model in-place with v2 of the Metamind API. No more adding all the same examples again.</p>
<p>I found it really useful to have four operating modes for the control software to facilitate the different training scenarios.</p>
<p>The first mode would kick in if no DataSet could be found. In these case the car would wait for input at each step. Using a wireless keyboard I could indicate the correct course of action to take. The image visible at the time would be captured and filed away with the required label. This made creating the initial DataSet much easier.</p>
<p>The second mode was used if the DataSet existed but didn't have a Model created yet. In this case the input I gave could be used to directly create examples. This mode wasn't so useful and I'd tend to skip it.</p>
<p>The third option applied if the Model could be found. In this case I could still indicate the correct course of action to take. If the model returned the expected prediction nothing further needed to occur. However, if the highest prediction differed from the expected result I could directly add the required feedback into the model.</p>
<p>The final forth mode was to use the model to make predictions and take action solely off the highest prediction returned.</p>
<h3>Control theory - PID control</h3>
<p>The earlier statement about the software action "Based on the result, activate servos to move the vehicle" is a gross simplification of what is required to have a vehicle follow a desired trajectory. In the real world you don't merely turn your vehicle full lock left or right to follow the desired course (what is referred to as bang-bang steering). Instead you make adjustments proportional to how far off the desired trajectory you are,
how rapidly you are approaching the desired trajectory, and finally, if there are any environmental factors that are affecting the steering of the vehicle.</p>
<p>This should be familiar to anyone who makes and flys quadcopters (<a href="https://www.youtube.com/watch?v=wzopjkW8y40">another interest of mine</a>). It's referred to as <a href="https://en.wikipedia.org/wiki/PID_controller">PID (proportional–integral–derivative) Control</a> and provides the automated mechanism to keep the vehicle/quadcopter at the desired state where the output needs to adapt to meet the desired input. The following video does an excellent job of explaining how it works.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/4Y7zG48uHRo?start=8" frameborder="0" allowfullscreen></iframe>
<p>With my v1 prototype the control loop speed is so slow and the positioning so basic that I'm practically going with bang-bang style steering. It is less than ideal as most scenarios require fine motor control to accurately follow the curve of the road ahead. Definitely something to look at improving in v2</p>
<h2 id="results">The prototype results</h2>
<h3>Training example</h3>
<iframe width="560" height="315" src="https://www.youtube.com/embed/IbnjkG0UrM4" frameborder="0" allowfullscreen></iframe>
<p>This video shows the feedback training process for an existing model. The initial model was created off a fairly small number of examples in the dataset. I then refined it with feedback whenever the top prediction didn't match what I indicated it should be.</p>
<p>I've added some basic voice prompts to indicate what is occurring as it mostly runs headless during normal operation.</p>
<h3>Self steering example</h3>
<p>In this video I'm using an entirely different dataset to the training example above. Getting single laps out of this setup was a bit of a challenge. The real world is far less forgiving than software when something goes wrong. Through trial an error I found it was really important to get the camera angle right to give the best prediction for the current position. It was also important to adjust the amount of movement per action. Trying to make large corrective movements, particularly around corners, would often result in getting irrecoverably off course. This comes back to the "bang-bang" style steering mentioned above.</p>
<p>I've edited out some of the pauses to make it more watchable. So don't consider this a real time representation of how fast it actually goes.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/xw0qMdQI5wU" frameborder="0" allowfullscreen></iframe>
<p>This process was also not without its share of failures. The following video shows a more realistic speed for the device and what happens when it misjudges the cornering.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/0Yfsh1_U2kA" frameborder="0" allowfullscreen></iframe>
<h2>Conclusions</h2>
<p>Lets start with the Million Dollar Question -</p>
<blockquote>Can I use this today to convert my car into an AI powered automated driving machine and then live a life of leisure as it plays taxi for the all the trips the kids need to make?</blockquote>
<p>Umm, no. Even if you're happy with it only being around 50% confident that it should go in a straight line rather than swerve off to the right on occasion there are still a number of things holding it back.</p>
<p>For example, while it could be extended to recognize a stop sign there is currently no way to judge how far away that stop sign is<a href="#1"><sup>1</sup></a>.</p>
<p>That's not even taking into account other things that real roads have, like traffic, pedestrians, roadworks, ... At least I'm still someway off before I need to start worrying about things like the <a href="https://en.wikipedia.org/wiki/Trolley_problem">Trolley car problem</a></p>
<h2>Future Improvements</h2>
<p id="1" name="1">A couple of days before this post came out Salesforce released the Einstein Object Detection API. It's very similar to the Image recognition API I used above except in addition to the identified label in the image it will also give you a bounding box indicating where it was detected. With something like this I could then guess the relative position of the object in question to the current camera direction and maybe even distance based on a know size and how large the bounding box is coming back as. Definitely something to investigate for the MK2.</p>
<!--Future improvements when pilot API's for object detection with position are added. Important things like, is there a stop sign ahead and how far away is it (based on size).</p>
<p>A bit ahead of my time. Still waiting on the required functionality to judge distance and object position. How far away is that stop sign? Is that a car up ahead in front of me or to the left? </p-->
<!--p>Refinements - How much of the input image does it need to be trained on and reacting to for basic locomotion? If only the bottom half of the screen shows road then I could crop the top half off for the purpose of predictions. Assumes driving on a level road. No hills. Signs up high.</p-->
<h2>See also</h2>
<ul>
<li>Youtube - <a href="https://www.youtube.com/watch?v=OFhRT5Gz3lk">Build Smarter Apps with Einstein Platform Services</a></li>
<li><a href="https://developer.salesforce.com/blogs/2017/10/use-einstein-object-detection.html">How to use Einstein Object Detection</a></li>
<li>Dreamforce 2017 Session: <a href="https://success.salesforce.com/Sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yuqDQAQ">Identify Protein Structures using Einstein</a> </li>
<li><a href="http://www.fishofprey.com/2017/03/the-mad-catter-salesforce-predictive.html">The Mad Catter</a> - Using Einstein Image Services to identify and spray cats with a sprinkler.</li>
<li>Trailmix - <a href="https://trailhead.salesforce.com/users/005500000060Y7FAAU/trailmixes/einstein-platform-services">Einstein Platform Services - 🤖</a></li-->
</ul>
<h2>Gallery</h2>
<div class="separator" style="clear: both; text-align: center;">
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://1.bp.blogspot.com/-l6_ACOL5u_U/WecT-Ou6zJI/AAAAAAADi8U/PJ6tN13fG4kWeKQx06IQyNdJDe59zWgCQCLcBGAs/s1600/20170629_204319.jpg" imageanchor="1" style="margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-l6_ACOL5u_U/WecT-Ou6zJI/AAAAAAADi8U/PJ6tN13fG4kWeKQx06IQyNdJDe59zWgCQCLcBGAs/s320/20170629_204319.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a>
<figcaption>Putting together a dedicated Pi Screen</figcaption>
</figure>
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://2.bp.blogspot.com/-jMzrNy8WwhQ/WecT_l36tcI/AAAAAAADi8Y/W09s2ZKH2bEczbRnzWbpF7jvH1U213izgCLcBGAs/s1600/20170917_110740.jpg" imageanchor="1" style=margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-jMzrNy8WwhQ/WecT_l36tcI/AAAAAAADi8Y/W09s2ZKH2bEczbRnzWbpF7jvH1U213izgCLcBGAs/s320/20170917_110740.jpg" width="240" height="320" data-original-width="1200" data-original-height="1600" /></a>
<a href="https://1.bp.blogspot.com/-YVFcU9ZGRDE/WecT9vWuQNI/AAAAAAADi8M/2IALuOcrKDkJX-OyWpaRZa2AJ8s8nc9rwCLcBGAs/s1600/20170917_062006.jpg" imageanchor="1" style="margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-YVFcU9ZGRDE/WecT9vWuQNI/AAAAAAADi8M/2IALuOcrKDkJX-OyWpaRZa2AJ8s8nc9rwCLcBGAs/s320/20170917_062006.jpg" width="240" height="320" data-original-width="1200" data-original-height="1600" /></a>
<figcaption>Astro sized 3D printed racing seat</figcaption>
</figure>
</div>
<div class="separator" style="text-align: center;">
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://2.bp.blogspot.com/-0St9-jV0moo/WecUAcfGxRI/AAAAAAADi8c/5OjrGVGNvjgZV7GIMaV2esOysqLnmLlXQCLcBGAs/s1600/20170919_210914.jpg" imageanchor="1" style="margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-0St9-jV0moo/WecUAcfGxRI/AAAAAAADi8c/5OjrGVGNvjgZV7GIMaV2esOysqLnmLlXQCLcBGAs/s320/20170919_210914.jpg" width="240" height="320" data-original-width="1200" data-original-height="1600" /></a>
<a href="https://4.bp.blogspot.com/-SzNVJkkg7vs/WecUA_Po38I/AAAAAAADi8g/gdxG4Ge7nIo59LbiYfg1otqKYf2BuF5SwCLcBGAs/s1600/20170919_211515.jpg" imageanchor="1" style="margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-SzNVJkkg7vs/WecUA_Po38I/AAAAAAADi8g/gdxG4Ge7nIo59LbiYfg1otqKYf2BuF5SwCLcBGAs/s320/20170919_211515.jpg" width="240" height="320" data-original-width="1200" data-original-height="1600" /></a>
<figcaption>Original clunky power setup. Result of clunky power setup</figcaption>
</figure>
</div>
<div class="separator" style="text-align: center;">
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://2.bp.blogspot.com/-7XFkz-EP35k/WecUBLpDCsI/AAAAAAADi8k/uwmrQX2rQkc4s4Jq0f3Tv80xZeDalvBgQCLcBGAs/s1600/20171002_211719.jpg" imageanchor="1" style="margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-7XFkz-EP35k/WecUBLpDCsI/AAAAAAADi8k/uwmrQX2rQkc4s4Jq0f3Tv80xZeDalvBgQCLcBGAs/s320/20171002_211719.jpg" width="240" height="320" data-original-width="1200" data-original-height="1600" /></a>
<figcaption>Creating the blue Larson scanner</figcaption>
</figure>
</div>
<div class="separator" style="text-align: center;">
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://3.bp.blogspot.com/-lbYXldvQ_V4/WecT-G72I0I/AAAAAAADi8Q/Y100kSLIeKgAMCy2cAaY2s_caDDhm8ELQCLcBGAs/s1600/20170628_215734_1.gif" imageanchor="1" style="margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-lbYXldvQ_V4/WecT-G72I0I/AAAAAAADi8Q/Y100kSLIeKgAMCy2cAaY2s_caDDhm8ELQCLcBGAs/s320/20170628_215734_1.gif" width="180" height="320" data-original-width="369" data-original-height="656" /></a>
<figcaption>Red Astro Larson Scanner</figcaption>
</figure>
</div>
<div class="separator" style="text-align: center;">
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://2.bp.blogspot.com/-60NDjt5n2CI/WecUCxqEVCI/AAAAAAADi8s/UEzuDI_W-QknGoLjqvR6e1glJwL5iAgggCLcBGAs/s1600/20171010_214658.jpg" imageanchor="1" style="margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-60NDjt5n2CI/WecUCxqEVCI/AAAAAAADi8s/UEzuDI_W-QknGoLjqvR6e1glJwL5iAgggCLcBGAs/s320/20171010_214658.jpg" width="240" height="320" data-original-width="1200" data-original-height="1600" /></a><figcaption>Training setup for driving from Google Street View</figcaption>
</figure>
</div>
<div class="separator" style="text-align: center;">
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://4.bp.blogspot.com/-VAVYpmKFECU/WecUCw8-pJI/AAAAAAADi8w/m0vZyjGR2oMS7PSUEyWU14rBKc4bKJbmgCLcBGAs/s1600/20171015_122640.jpg" imageanchor="1" style="margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-VAVYpmKFECU/WecUCw8-pJI/AAAAAAADi8w/m0vZyjGR2oMS7PSUEyWU14rBKc4bKJbmgCLcBGAs/s320/20171015_122640.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a>
<figcaption>Basic servo setup</figcaption>
</figure>
</div>
<div class="separator" style="text-align: center;">
<figure style="float: right; background: #222222; -webkit-margin-start: 0px; -webkit-margin-end: 0px;">
<a href="https://2.bp.blogspot.com/-H5aH8j0vUg0/WecUD06cPtI/AAAAAAADi80/-m-H4raRUN8H6wpzR59QP5lZ5biYFomZACLcBGAs/s1600/20171015_135157.jpg" imageanchor="1" style="margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-H5aH8j0vUg0/WecUD06cPtI/AAAAAAADi80/-m-H4raRUN8H6wpzR59QP5lZ5biYFomZACLcBGAs/s320/20171015_135157.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a>
<figcaption>Lipo battery loaded onto the frame with a mess of wiring</figcaption>
</figure>
</div>
<iframe width="560" height="315" src="https://www.youtube.com/embed/-ck-sMzvTAY" frameborder="0" allowfullscreen></iframe>
<!-- Tweet with #EinsteinAI -->FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-22609136179727907492017-09-30T06:00:00.000+13:002017-10-27T09:45:52.134+13:00Dreamforce 2017 Session picks<p>Here are some of my current picks for Dreamforce 2017 sessions. I'm aiming for a mix of developer related topics in areas I want to learn more about plus anything that sounds informative. There are other sessions that I think are interesting as well, but I don't know how to access my bookmarked sessions.</p>
<h2>Keynotes</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yvlEQAQ">Salesforce for Developers Keynote</a></li>
</ul>
<h2>Meet The *'s</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yswOQAQ">Meet the Developers</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yvl6QAA">Parker Harris' True to the Core: What's Next for Our Core Products</a> - Wednesday at 3 (probably) <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Of course! Currently scheduled for Wednesday at 3.</p>&mdash; Shawna Wolverton (@shawnawol) <a href="https://twitter.com/shawnawol/status/910984625220067328?ref_src=twsrc%5Etfw">September 21, 2017</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script></li>
<li><a href="https://success.salesforce.com/Sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ysy0QAA">Meet the Apex Engineering and Product Team</a><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Got burning Q&#39;s about where the SWITCH statement is in <a href="https://twitter.com/hashtag/Apex?src=hash&amp;ref_src=twsrc%5Etfw">#Apex</a> or why your log is 90% ENTERING_MANAGED_PKG. This is the <a href="https://twitter.com/hashtag/DF17?src=hash&amp;ref_src=twsrc%5Etfw">#DF17</a> session for you! <a href="https://t.co/BDfVyZOUF4">https://t.co/BDfVyZOUF4</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/923625160199958528?ref_src=twsrc%5Etfw">October 26, 2017</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</li>
</ul>
<h2>APIs</h2>
<ul>
<li><a href="https://success.salesforce.com/Sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000000GV6SQAW">Exploring Bulk API V2</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000000GUx8QAG">Salesforce Platform API Soft Limits</a>
<blockquote>As more and more applications are interconnected with the Salesforce APIs, daily limits become a growing concern. Join us as we go over the existing mechanisms to help you manage your API limits, and discuss how we are approaching a new generation of "soft" limits and what this will mean for your developers.</blockquote>
</li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yulNQAQ">Synchronize Salesforce Data to Postgres with Change Data Capture</a></li>
<li><a href="https://success.salesforce.com/Sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yto6QAA">Connect Any External Datasource with Custom APEX Connector</a></li>
</ul>
<h2>Salesforce DX</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yvA8QAI">Salesforce DX and Fully Automated Deployments</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000000GUzsQAG">Salesforce DX Packaging for ISV Partners</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yswdQAA">Test Automation and continuous integration in SDLC using Salesforce DX</a></li>
</ul>
<h2>IDE</h2>
<ul>
<li><a href="https://success.salesforce.com/Sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytfTQAQ">
Salesforce Development in IntelliJ IDEA - What's New in Illuminated Cloud</a></li>
</ul>
<h2>Peeking under the hood</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytGEQAY">New Apex Compiler Rollout</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001pwnDQAQ">How We Do the Magic We Do - A Peek into Multi-tenancy</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yuLjQAI">SOQL: Performance Explained</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytGCQAY">How Salesforce does Incremental Code Deploys for internal orgs</a></li>
</ul>
<h2>Platform Events</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytswQAA">Rethinking Your Architecture With Platform Events</a></li>
<li><a href="https://success.salesforce.com/Sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000000GUyaQAG">Advanced Logging Patterns with Platform Events</a></li>
</ul>
<h2>Platform</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytB6QAI">The Salesforce Platform Roadmap - Part I</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytB0QAI">The Salesforce Platform Roadmap - Part II</a></li>
</ul>
<p>It needs two roadmap sessions? That's hardcore!</p>
<h2>Einstein / AI</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytyLQAQ">Simplify Data-Driven Processes Using the Einstein Analytics Apex SDK</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ysyHQAQ">Auto-Machine Learning: The Magic Behind Einstein</a></li>
</ul>
<h2>Security</h2>
<ul>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001pwWmQAI">Avoiding Common Security Mistakes</a></li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yvnBQAQ">Common Web Security Vulnerabilities and their Fixes</a></li>
</ul>
<h2>Fun</h2>
<ul>
<li><a href="https://success.salesforce.com/Sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001ytadQAA">New Life for a Commodore 64: An Old-Is-New-Again Salesforce Client</a><br/>
You had me at Commodore 64
</li>
<li><a href="https://soundcloud.com/blazingtrailspodcast/the-road-to-dreamforce-developers-at-dreamforce-update-2#t=7:10">Bear Labs in the developer forest</a> - <a href="https://twitter.com/bear_labs">@bear_labs</a></li>
<li>Codey Escape Room
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Dreamforce will have it&#39;s own Codey themed <a href="https://twitter.com/hashtag/escaperoom?src=hash&amp;ref_src=twsrc%5Etfw">#escaperoom</a><a href="https://t.co/UHMbhhMqKg">https://t.co/UHMbhhMqKg</a> <a href="https://t.co/1ObwHZUz54">pic.twitter.com/1ObwHZUz54</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/913496561522315264?ref_src=twsrc%5Etfw">September 28, 2017</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script></li>
<li>Lego Salesforce Tower
<blockquote class="twitter-tweet" data-lang="en"><p lang="es" dir="ltr">. | ■ ■ |<br> | ■ ■ |<br> | ■ ■ |<br> | ■ ■ ■ |<br> | ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> | ■ ■ ■ ■ |<br> ========<a href="https://twitter.com/Dreamforce?ref_src=twsrc%5Etfw">@Dreamforce</a> lego <a href="https://twitter.com/SalesforceTower?ref_src=twsrc%5Etfw">@SalesforceTower</a> <a href="https://t.co/ZFILb2dl0S">https://t.co/ZFILb2dl0S</a></p>&mdash; Daniel Ballinger 🦈 (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/913490789036023809?ref_src=twsrc%5Etfw">September 28, 2017</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
</li>
</ul>
<h2>Lightning</h2>
<ul>
<li>TODO - Need something to fill out this area to be a well rounded developer. Lightning Roadmap maybe?</li>
<li><a href="https://success.salesforce.com/sessions?eventId=a1Q3A00000stRRuUAM#/session/a2q3A000001yuX1QAI">Callbacks, Events and More - Bring Sanity to Asynchronous Lightning Code</a></li>
</ul>
<!--René Winkelmeyer-->
<h2>See also</h2>
<ul>
<li>My <a href="http://www.fishofprey.com/2016/08/dreamforce-2016-session-picks-and.html#tips">General Dreamforce Tips</a> from last year</li>
<li><a href="https://www.salesforce.com/blog/2017/09/advice-dreamforce-first-timers.html?eid=ss-blog-nl">30 Really Good Pieces of Advice for Dreamforce First-Timers (Like Appy!)</a></li>
</ul>
FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-37877356366514772472017-09-22T07:46:00.000+12:002017-09-22T07:46:09.160+12:00Teaching an Old Log Parser New Tricks - FIT sfdx CLI plugin<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-aaANSwPbyX8/WcOVPnvdwnI/AAAAAAADgzg/JV_JFA2vK2cfrpQjysrr3Ek5iQPXhmOxACLcBGAs/s1600/New-Or-Improved.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-aaANSwPbyX8/WcOVPnvdwnI/AAAAAAADgzg/JV_JFA2vK2cfrpQjysrr3Ek5iQPXhmOxACLcBGAs/s320/New-Or-Improved.jpg" width="176" height="176" data-original-width="176" data-original-height="176" title="Some parts are new, some parts are improved" /></a></div>
<p>I've currently got two problems with the prototype <a href="http://www.fishofprey.com/2017/09/back-to-command-line-fitdx-apex-debug.html">FitDX Apex debug log parser</a> I released a couple of weeks ago.</p>
<ol>
<li>It comes from the FuseIT SFDC Explorer source and is excessively large for what it does. It's got a lot of baggage for features it doesn't currently expose.</li>
<li>To be a native SFDX CLI plugin I need to be working from node.js.</li>
</ol>
<p>Enter <a href="https://github.com/tjanczuk/edge">Edge.js</a> and it's quote from <a href="https://github.com/tjanczuk/edge#what-problems-does-edgejs-solve">"What problems does Edge.js solve?"</a></p>
<blockquote>
<p>Ah, whatever problem you have. If you have this problem, this solves it.</p>
</blockquote>
<p><em><a href="https://twitter.com/shanselman/status/461532471037677568">--Scott Hanselman (@shanselman)</a></em></p>
<p>As promised, it does what it says on the box and helps solve both problems by connecting the .NET world with node.js - <i>in process</i>.</p>
<p>In my case I've got a .NET library that has <i>many</i> years of work put into it to parse Apex debug logs from a raw string. It's also still actively developed and utilized in the FuseIT SFDC Explorer UI. Between that and my lack of experience with node.js I wasn't in a hurry to rewrite it all so it could be used as a native SFDX CLI plugin. While I still needed to reduce the .NET DLL down to just the required code for log parsing I don't need to rewrite it all in JavaScript. Bonus!</p>
<p>This seems like a good place for a quick acknowledgement to <a href="https://twitter.com/FishOfPrey/status/910253448544665600">@Amorelandra</a> for pointing me in the right direction with node.js. Thanks again!</p>
<p>With the edge module the node.js code becomes the plumbing to pass off input from the heroku plugins to my happy place (the .NET code). And it is all transparent to the user (assuming the OS specific <a href="https://github.com/tjanczuk/edge#what-you-need">edge.js requirements</a> are meet). A big bonus being a plugin is that it gives me a prebuilt framework for gathering all the inputs in a consistent way.</p>
<p>I used <a href="https://github.com/wadewegner/sfdx-waw-plugin">Wade Wegner's SFDX plugin</a> as a boiler plate for integrating with the cli, chopped most of it out, and kept the command to pull the latest debug log from an org. This debug log is then feed directly into the .NET library to use the existing FuseIT parser.</p>
<p>Enough patting myself of the back for getting this all working. How does it work in practice?</p>
<h2>Summary</h2>
<p>Like with fitdx, this will give you a count for each event type that appears in the log.</p>
<pre style="clear: both;white-space: pre-wrap;">&gt;sfdx fit:apex:log:latest -u &lt;targetusername&gt; --summary
41.0 : 41.0 APEX_CODE,DEBUG;APEX_PROFILING,NONE;CALLOUT,NONE;DB,INFO;SYSTEM,INFO;VALIDATION,INFO;VISUALFORCE,NONE;WAVE,INFO;WORKFLOW,INFO
INFO;WORKFLOW,INFO
CODE_UNIT_FINISHED : 2
CODE_UNIT_STARTED : 11
CUMULATIVE_LIMIT_USAGE : 18
CUMULATIVE_LIMIT_USAGE_END : 18
DML_BEGIN : 22
DML_END : 22
ENTERING_MANAGED_PKG : 768
LIMIT_USAGE_FOR_NS : 54
SOQL_EXECUTE_BEGIN : 25
SOQL_EXECUTE_END : 25
USER_DEBUG : 2
USER_INFO : 1
</pre>
<p>768 ENTERING_MANAGED_PKG events! Story of my life.</p>
<h2>Debug only</h2>
<pre style="clear: both;white-space: pre-wrap;">&gt;sfdx fit:apex:log:latest -u <targetusername> --debugOnly
28.0 APEX_CODE,DEBUG;APEX_PROFILING,INFO;CALLOUT,INFO;DB,INFO;SYSTEM,DEBUG;VALIDATION,INFO;VISUALFORCE,INFO;WAVE,INFO;WORKFLOW,INFO
18:56:59.180 (1181147014)|USER_DEBUG|[4]|ERROR|UserInfo.getSessionId(): 00D700000000001!ApexTestSession
18:56:59.416 (1416651819)|USER_DEBUG|[4]|ERROR|UserInfo.getSessionId(): 00D700000000001!ApexTestSession</pre>
<h2>Filtering</h2>
<pre style="clear: both;white-space: pre-wrap;">&gt;sfdx fit:apex:log:latest -u <targetusername> --filter USER_INFO,CODE_UNIT_STARTED
28.0 APEX_CODE,DEBUG;APEX_PROFILING,INFO;CALLOUT,INFO;DB,INFO;SYSTEM,DEBUG;VALIDATION,INFO;VISUALFORCE,INFO;WAVE,INFO;WORKFLOW,INFO
18:56:58.400 (4013418)|USER_INFO|[EXTERNAL]|005700000000001|daniel.ballinger@example.com|Pacific Standard Time|GMT-07:00
18:56:58.979 (979316201)|CODE_UNIT_STARTED|[EXTERNAL]|01q70000000HFV6|DFB.TestAccountBeforeUpdate1 on Account trigger event BeforeInsert for [new]
18:56:58.982 (982430091)|CODE_UNIT_STARTED|[EXTERNAL]|01q70000000HFVB|DFB.TestAccountBeforeUpdate2 on Account trigger event BeforeInsert for [new]
18:56:58.985 (985467868)|CODE_UNIT_STARTED|[EXTERNAL]|01q70000000blnO|DFB.AccountTrigger on Account trigger event BeforeInsert for [new]
18:56:59.108 (1108633716)|CODE_UNIT_STARTED|[EXTERNAL]|01q70000000Ti5b|DFB.ToolingTest on Account trigger event BeforeUpdate for [0010g00001YxxQ5]
18:56:59.180 (1180950117)|CODE_UNIT_STARTED|[EXTERNAL]|01q70000000H1vs|DFB.RestrictContactByName on Contact trigger event BeforeInsert for [new]</pre>
<h2>Format Shifting</h2>
<pre>&gt;sfdx fit:apex:log:latest -u <targetusername> --json</pre>
<pre>&gt;sfdx fit:apex:log:latest -u <targetusername> --csv</pre>
<pre>&gt;sfdx fit:apex:log:latest -u <targetusername> --human</pre>
<p>The first two should still be fairly self explanatory and the latter is still intended to make reading the log easier for those that run on digesting food rather than electricity.</p>
<h2>Installation</h2>
<p>Most of the steps are covered in the <a href="https://github.com/FuseInfoTech/FitCLI#install-from-source">Github repo</a>. I found the most challenging part here to be piggybacking off the existing install of node.js and npm that lives under sfdx.</p>
<p>Your results may vary. It is likely safer to install your own instance of node.js. The <a href="http://www.fishofprey.com/2009/10/blog-disclaimer.html">standard disclaimer</a> applies. </p>
<h2>One more thing</h2>
<a href="https://2.bp.blogspot.com/-CAkfnbd5OxM/WcObIavPteI/AAAAAAADgzw/8v2EcDidL8U06ohoykMXmRMjvyiHnLd2wCLcBGAs/s1600/fit_deploymentfish.gif" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-CAkfnbd5OxM/WcObIavPteI/AAAAAAADgzw/8v2EcDidL8U06ohoykMXmRMjvyiHnLd2wCLcBGAs/s1600/fit_deploymentfish.gif" data-original-width="438" data-original-height="600" /></a>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com3tag:blogger.com,1999:blog-7249827273468308081.post-55237357490772965632017-09-08T13:13:00.000+12:002017-09-15T21:46:50.241+12:00Back to the command line - FitDx Apex debug log parser<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-ApqJ7uQm2is/WayV6K4nOQI/AAAAAAADdUs/mw4wb_zKzBkvoWmxmpsryyuYKfqbT0FYwCLcBGAs/s1600/BackToTheCommandLine.gif" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-ApqJ7uQm2is/WayV6K4nOQI/AAAAAAADdUs/mw4wb_zKzBkvoWmxmpsryyuYKfqbT0FYwCLcBGAs/s1600/BackToTheCommandLine.gif" data-original-width="424" data-original-height="237" class="GiffferMe" /></a></div>
<p>Love it or loathe it, there are times in a developers day where you are going to find yourself at the command line. I'll leave the GUI vs command line <a href="https://www.hanselman.com/blog/CommentView.aspx?guid=ab6d9e19-af5a-4196-b500-ab2852263bfb#commentstart">debate</a> to others to sort out.</p>
<p>My general position is they both have their strengths and weaknesses and we as developers should try and use the right tool for the job. Very much like clicks and code with Salesforce. Use them together where possible to get the most done in the least amount of effort.</p>
<p>Tooling shouldn't be a zero sum game.</p>
<h2 style="clear: both;">Debug logs</h2>
<p>Salesforce Debug logs have been a pet project area for me for some time now <sup>E.g. <a href="http://www.fishofprey.com/2011/02/using-notepad-to-make-apex-test-runner.html">2011</a>, <a href="http://www.fishofprey.com/2014/06/salesforce-log-categories-and-events-by.html">2014</a>, <a href="http://www.fishofprey.com/2017/02/fuseit-sfdc-explorer-35170233-more.html">2017</a></sup>. They form such a fundamental part of the development process but don't get a lot of love. Let me paint you a picture for a typical day in development ...</p>
<blockquote>Earlier in the day something broke badly in production and you’ve only just been provided a raw 2MB text version of the debug log to diagnose the problem from. What do you do?<br/><br/>
You can’t use any of the tooling that the Developer Console provides for log inspection on a text file. There is no way of opening it. So there will be no stack tree or execution overview to help you process a file approximately the same size as a <a href="https://www.quora.com/What-is-the-average-file-size-of-an-e-book">300 page ebook</a>. <sup>I say approximately here because there are <i>so many</i> factors that could affect the size in bytes for a book. The general point is that a 2MB text file contains <b>a lot</b> of information that you aren't going to read in a few minutes.</sup> <br/><br/>
The interactive debugger isn't any help for a production fault and you don't even know the steps to reproduce the problem let alone if the same thing would happen in a sandbox.<br/><br/>
You could start combing the log file with a variety of text processing tools, or, or...</blockquote>
<p>It's normally about this point that I'd direct you to the <a href="http://www.fishofprey.com/2017/08/fuseit-sfdc-explorer-36171841-and.html">FuseIT SFDC Explorer</a> debug log tooling which gives you:</p>
<ul>
<li>(somewhat retro) GUI tooling for grouping and filtering events by category.</li>
<li>Helps highlight important events such as fatal errors and failed validations.</li>
<li>And, my current favorite, gives you a timeline view to make more sense of execution time measured in nanoseconds.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-J65HZXhVYsA/WbEbq-MthpI/AAAAAAADfYU/iW0kLBnt8FY3OmUDm8r_IQWRe7z5ssxRgCLcBGAs/s1600/BeforeBut.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-J65HZXhVYsA/WbEbq-MthpI/AAAAAAADfYU/iW0kLBnt8FY3OmUDm8r_IQWRe7z5ssxRgCLcBGAs/s320/BeforeBut.png" width="320" height="176" data-original-width="599" data-original-height="330" /></a></div>
<p>But I'm not going to cover any of that in this post. Instead we're going to cover something more experimental than the software that's been in perpetual beta for who knows how many years.</p>
<h2 style="clear: both;">Fit DX</h2>
<p>The FuseIT DX CLI, or FitDx in usage, is a proof of concept that I could take the debug log parser from the explorer product mentioned above and apply it directly to debug logs at the command line. I'm just putting it out there to see if there is any interest in such a thing.</p>
<p>You can get it right now as part of the zip download for the <a href="http://www.fuseit.com/explorer">FuseIT SFDC Explorer</a>.</p>
<p>There are certainly areas for improvement. First and foremost is the executable size. It's swollen up a fair bit from features that it doesn't expose. I'll look at whipping those out in a future release and should result in a significantly smaller package.</p>
<p>But enough about how big FitDx is. What's more important is how you use it.</p>
<h2>Summary Command</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-yYbTmDH42Dg/WbGnLctSWGI/AAAAAAADfYk/pbI5PWeFxacbMg-lXnrk83WXvtl33L-mACLcBGAs/s1600/FullLog.gif" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-yYbTmDH42Dg/WbGnLctSWGI/AAAAAAADfYk/pbI5PWeFxacbMg-lXnrk83WXvtl33L-mACLcBGAs/s1600/FullLog.gif" data-original-width="506" data-original-height="255" class="GiffferMe" /></a></div>
<p>If you ask sfdx for a debug log, that's exactly what you'll get. The complete, raw, unabridged debug log dumped to the command line. An experienced command line person would at this stage type out some grep regex wizardry to just show them what they wanted to see. Such is the power of the command line, but it isn't always clear where to start.</p>
<p>I wanted something simpler that would give you a very quick overview of what is in the log.</p>
<pre style="clear: both;white-space: pre-wrap;">
&gt;sfdx force:apex:log:get -i 07L7000004G0U8kEAF -u DeveloperOrg | fitdx --summary
28.0 : 28.0 APEX_CODE,FINE;APEX_PROFILING,FINEST;CALLOUT,ERROR;DB,FINEST;SYSTEM,FINE;VALIDATION,DEBUG;VISUALFORCE,FINER;WORKFLOW,INFO
CODE_UNIT_FINISHED : 14
CODE_UNIT_STARTED : 14
CONSTRUCTOR_ENTRY : 14
CONSTRUCTOR_EXIT : 14
CUMULATIVE_LIMIT_USAGE : 14
CUMULATIVE_LIMIT_USAGE_END : 14
CUMULATIVE_PROFILING : 4
CUMULATIVE_PROFILING_BEGIN : 1
CUMULATIVE_PROFILING_END : 1
ENTERING_MANAGED_PKG : 184
EXCEPTION_THROWN : 2
EXECUTION_FINISHED : 14
EXECUTION_STARTED : 14
FATAL_ERROR : 4
LIMIT_USAGE_FOR_NS : 14
METHOD_ENTRY : 73
METHOD_EXIT : 73
SYSTEM_CONSTRUCTOR_ENTRY : 33
SYSTEM_CONSTRUCTOR_EXIT : 33
SYSTEM_METHOD_ENTRY : 165
SYSTEM_METHOD_EXIT : 165
TOTAL_EMAIL_RECIPIENTS_QUEUED : 14
USER_INFO : 14
</pre>
<p>The summary will currently give you a count for the events that are covered in the piped input. In this case I can see that there were 4 FATAL_ERROR events</p>
<h2>Filtering</h2>
<p>Now that I know I what I'm looking for I want to see it just those events of interest. The <code>--filter</code> command accepts a comma separated list of event types that you want to see. Everything else gets dropped. There is also the <code>--debugOnly</code> option which is a preset to filter for USER_DEBUG only.</p>
<pre style="clear: both;white-space: pre-wrap;">
&gt;sfdx force:apex:log:get -i 07L7000004G0U8kEAF -u DeveloperOrg | fitdx --filter FATAL_ERROR
28.0 APEX_CODE,FINE;APEX_PROFILING,FINEST;CALLOUT,ERROR;DB,FINEST;SYSTEM,FINE;VALIDATION,DEBUG;VISUALFORCE,FINER;WORKFLOW,INFO
20:25:09.695 (698828931)|FATAL_ERROR|System.AssertException: Assertion Failed: Expected: {"filters":{"footer":{"settings":{"enable":"1","text/plain":"You can haz footers!"}}}}, Actual: {"filters":{"footer":{"settings":{"text/plain":"You can haz footers!","enable":"1"}}}}
Class.DFB.SmtpapiTest.setSetFilters: line 116, column 1
20:25:09.695 (698839161)|FATAL_ERROR|System.AssertException: Assertion Failed: Expected: {"filters":{"footer":{"settings":{"enable":"1","text/plain":"You can haz footers!"}}}}, Actual: {"filters":{"footer":{"settings":{"text/plain":"You can haz footers!","enable":"1"}}}}
Class.DFB.SmtpapiTest.setSetFilters: line 116, column 1
20:25:10.160 (1162726841)|FATAL_ERROR|System.AssertException: Assertion Failed: Expected: {"section":{"set_section_key":"set_section_value","set_section_key_2":"set_section_value_2"}}, Actual: {"section":{"set_section_key_2":"set_section_value_2","set_section_key":"set_section_value"}}
Class.DFB.SmtpapiTest.testAddSection: line 80, column 1
20:25:10.160 (1162743640)|FATAL_ERROR|System.AssertException: Assertion Failed: Expected: {"section":{"set_section_key":"set_section_value","set_section_key_2":"set_section_value_2"}}, Actual: {"section":{"set_section_key_2":"set_section_value_2","set_section_key":"set_section_value"}}
Class.DFB.SmtpapiTest.testAddSection: line 80, column 1
</pre>
<h2>Format Shifting</h2>
<p>The final set of alpha features are around changing how the debug log is presented. Options include <code>--json</code>, <code>--csv</code>, and <code>--human</code>. The first two should be fairly self explanatory on what the output will be. The human option was just an idea to use more column like alignment rather than vertical bars (|) to separate the elements. </p>
<p>In hindsight I should probably allow for multiple options at once so you can specify both the required filters and output format in one command. For the time being you can just pipe the filtered output back in. </p>
<pre style="clear: both;white-space: pre-wrap;">&gt;sfdx force:apex:log:get -i 07L7000004G0U8kEAF -u DeveloperOrg | fitdx --filter EXCEPTION_THROWN | fitdx --human
28.0 APEX_CODE,FINE;APEX_PROFILING,FINEST;CALLOUT,ERROR;DB,FINEST;SYSTEM,FINE;VALIDATION,DEBUG;VISUALFORCE,FINER;WORKFLOW,INFO
20:25:09.695 EXCEPTION_THROWN [116]System.AssertException: Assertion Failed: Expected: {"filters":{"footer":{"settings":{"enable":"1","text/plain":"You can haz footers!"}}}}, Actual: {"filters":{"footer":{"settings":{"text/plain":"You can haz footers!","enable":"1"}}}}
20:25:10.160 EXCEPTION_THROWN [80]System.AssertException: Assertion Failed: Expected: {"section":{"set_section_key":"set_section_value","set_section_key_2":"set_section_value_2"}}, Actual: {"section":{"set_section_key_2":"set_section_value_2","set_section_key":"set_section_value"}}</pre>
<h2>The Forward Looking Statements</h2>
<p>This is the part where I promise you the world with all the cool things I could potentially do in the future. Things like:</p>
<ul>
<li>Direct integration as a SFDX plugin via a <a href="https://twitter.com/FishOfPrey/status/895031407721893888">node.js to .NET library</a></li>
<li>Expose other features from the FuseIT SFDC Explorer at the command line, such as the alternative version of WSDL2Apex.</li>
<li>I've got a crazy idea for extra things like pulling apex class and trigger definitions directly from a Salesforce StackExchange question or answer and then load them directly into a newly created <strike>scratch</strike> <a href="https://twitter.com/FishOfPrey/status/900518756039905280">burner org</a>. The process could then also be reversed with the contents of the scratch org created into the required markdown. This could greatly speed up the process of asking and answering questions on the SFSE.</li>
</ul>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com4tag:blogger.com,1999:blog-7249827273468308081.post-2871569438932068772017-08-18T14:42:00.000+12:002017-08-18T14:42:27.290+12:00FuseIT SFDC Explorer 3.6.17184.1 and 3.7.17230.1 - Summer '17<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-dIqYjCH_5iQ/WJKQRN35nbI/AAAAAAAC2ec/BVFCBwIw31sgMuI-rOGxsO-X2TYZCcPTACLcB/s1600/FuseITSFDCTree.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-dIqYjCH_5iQ/WJKQRN35nbI/AAAAAAAC2ec/BVFCBwIw31sgMuI-rOGxsO-X2TYZCcPTACLcB/s640/FuseITSFDCTree.PNG" width="320" height="229" /></a></div>
<p>The latest v3.6 release of the <a href="http://www.fuseit.com/explorer">FuseIT SFDC Explorer</a> is out and contains a couple of new features around Apex Debug logs.</p>
<p>Aaaaaannnd... because I got distracted before publishing this post, 3.7 is also out. So it's now a twofer release post.</p>
<h2 style="clear: both;">Improvements to log parsing</h2>
<p>Carrying on from the <a href="http://www.fishofprey.com/2017/02/fuseit-sfdc-explorer-35170233-more.html">previous release</a>, I've made a number of modifications to the debug log viewer.</p>
<p>All the calculations for duration's are now done of the underlying nanosecond ticks. This helps prevent compounding errors due to the lack of definition in C# Timespans.</p>
<p>The improved duration calculation is particularly important for the new grouping functionality. Take the two examples below. With the standard log you can get 200 CODE_UNIT_STARTED events for the validation of each record in a transaction (plus the corresponding 200 CODE_UNIT_FINISHED events). With the new "Group Recurring Events" button in the tool bar any repeating events of the same type will be collapsed into a single node with a summary.</p>
<p><b>Before</b></p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ROjdscZZDKc/WVmr1PFFXtI/AAAAAAAC_98/yK1swEyv78UmQcedIFqhmutRrZl881-owCLcBGAs/s1600/MultipleValidation.PNG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-ROjdscZZDKc/WVmr1PFFXtI/AAAAAAAC_98/yK1swEyv78UmQcedIFqhmutRrZl881-owCLcBGAs/s320/MultipleValidation.PNG" width="320" height="99" data-original-width="674" data-original-height="209" /></a></div>
<p style="clear: both;"><b>After</b></p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-tij2aQZboZA/WVmsgzU1FyI/AAAAAAAC_-A/DGDDbWRaTsA-BcOL7AfR88YfGvrQ08UrwCLcBGAs/s1600/GroupedMultiValidation.PNG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-tij2aQZboZA/WVmsgzU1FyI/AAAAAAAC_-A/DGDDbWRaTsA-BcOL7AfR88YfGvrQ08UrwCLcBGAs/s320/GroupedMultiValidation.PNG" width="320" height="83" data-original-width="732" data-original-height="191" /></a></div>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-SOils_m8V-o/WZZLU2-Jp5I/AAAAAAADAY8/ktZPmsKTcJMTW8N37PQ8vrowTP4jvtHHgCLcBGAs/s1600/DmlIcons.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-SOils_m8V-o/WZZLU2-Jp5I/AAAAAAADAY8/ktZPmsKTcJMTW8N37PQ8vrowTP4jvtHHgCLcBGAs/s320/DmlIcons.PNG" width="320" height="149" data-original-width="264" data-original-height="123" /></a></div>
<p >While still on the topic of the Apex Log Tree View, DML operations now have a basic icons to indicate which variant of Insert, Update, Upsert, or Delete operation they are. Trigger, Validation, and Workflow have consistent color coding between the Tree View and the Timeline. </p>
<h2 style="clear: both;">Auto build a Metadata package</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-XLXGLewIVD0/WZZRlfrdXMI/AAAAAAADAZY/p5JXu0R0J5gsMaG2li9oJSIeRYK8KfzuACLcBGAs/s1600/PackageCrc.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-XLXGLewIVD0/WZZRlfrdXMI/AAAAAAADAZY/p5JXu0R0J5gsMaG2li9oJSIeRYK8KfzuACLcBGAs/s320/PackageCrc.png" width="320" height="251" data-original-width="446" data-original-height="350" /></a></div>
<p>There is a new option on the Metadata Deploy tab under "Deploy Selected Files" called "Compare File Hashes". As per the name, it will do a CRC32 diff of the body of Apex classes, triggers and visualforce pages between the local file system and the connected Salesforce org. If anything is different it gets added to the current package ready for deployment. </p>
<p>If it turns out the local file is the one that should be updated there is a context menu option to update the local file.</p>
<h2 style="clear: both;">Colored logging</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-KYMUKcNtOBU/WZZS7z7tPVI/AAAAAAADAZk/KuFt6awOFNUxYyNdq9h3Pjt7ycNKSnXcACLcBGAs/s1600/MetadataMonitorColour.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-KYMUKcNtOBU/WZZS7z7tPVI/AAAAAAADAZk/KuFt6awOFNUxYyNdq9h3Pjt7ycNKSnXcACLcBGAs/s320/MetadataMonitorColour.png" width="320" height="116" data-original-width="648" data-original-height="234" /></a></div>
<p>Both the Metadata Deployment results and test method error results now include some basic color.</p>
<h2 style="clear: both;">Salesforce DX CLI integration</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-sVNj_oqhR_o/WZZN_hBnESI/AAAAAAADAZM/60uEOWGG9xsAvJdpz3hPdVwPJq5M7JQPwCLcBGAs/s1600/SalesforceDXCli.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-sVNj_oqhR_o/WZZN_hBnESI/AAAAAAADAZM/60uEOWGG9xsAvJdpz3hPdVwPJq5M7JQPwCLcBGAs/s320/SalesforceDXCli.png" width="254" height="320" data-original-width="486" data-original-height="613" /></a></div>
<p>If you have the <a href="https://developer.salesforce.com/tools/sfdxcli">Salesforce DX CLI</a> installed (and in the path) you can use it to manage org logins. It might take a moment after pressing "Refresh Orgs" for the list to come back. Once it does, just select an org and press login.</p>
<h2 style="clear: both;">Other changes 3.6</h2>
<ul>
<li>Log Tree view. Color code Trigger, Validation, and Workflow events. Open debug log from File. Toggle Timeline and Treeview display.</li>
<li>Highlight FLOW_ELEMENT_ERROR messages in the log.</li>
<li>Handle log highlighting for Fine, Finer, and Finest logging levels. Provide links for ids for the active org.</li>
<li>Optionally display scale on the log. Highlight VALIDATION_FAIL events. </li>
<li>Highlight Exception and skipped/maxlog events in the Log Timeline.</li>
<li>When parsing a log, identify the log levels applied (if present) in the footer.</li>
<li>Attempt to determine if an ApexClass contains tests. Use this to drive direct running tests cases from the class.</li>
<li>Improve performance of building the EntityTree from sObject describe results. Especially for orgs with a large number of custom objects and settings.</li>
</ul>
<h2>Other changes 3.7</h2>
<ul>
<li>Add optional allowExistingSObjectsWithoutId="true" to the binding configuration element to allow sObjects to be created with a null Id. Typically this isn't allowed as the ID is used to control insert/update operations and to identify relationship types. This setting can be used for more basic SOQL queries where the results won't be subsequently used for DML.</li>
<li>Highlight CALLOUT_REQUESTs in the timeline as they often take up large chunks of time.</li>
<li>Fix for updating available metadata types with Session change.</li>
<li>Improve SessionId validation pre login </li>
<li>When exporting results, give the user the choice of folder to export to.</li>
<li>Handle SOQL Aggregate Results that return only a single value</li>
<li>Improve IDEWorkspace integration. Allow for switching of workspaces.</li>
<li>Wsdl2Apex: Handle case where the Schema TargetNamespace is missing</li>
<li>Perform Metadata Deploy Hash on Apex Pages. Option to refresh local metadata from Server.</li>
</ul>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com5tag:blogger.com,1999:blog-7249827273468308081.post-91733613393065258822017-06-13T09:10:00.001+12:002017-06-13T09:12:10.462+12:00Importing the Salesforce Summer 17 (v40.0) Tooling and Metadata APIs to .NET<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-wggtTmI670M/WT8DMRQdb4I/AAAAAAAC-30/w5m2bcmkP9cy8MJbqTMw6rmrRp6OUxEhgCLcB/s1600/AngrySummer17.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-wggtTmI670M/WT8DMRQdb4I/AAAAAAAC-30/w5m2bcmkP9cy8MJbqTMw6rmrRp6OUxEhgCLcB/s320/AngrySummer17.png" width="320" height="147" data-original-width="522" data-original-height="240" /></a></div>
<p>The Salesforce Summer '17 release includes the return of an old friend in both the Tooling and Metadata APIs. After refreshing to the latest WSDLs and making sure the project builds it starts producing errors like the following with calling the API:</p>
<blockquote style="clear: both;">error CS0029: Cannot implicitly convert type 'XYZ.SalesforceConnector.SalesforceTooling.NavigationMenuItem' to 'XYZ.SalesforceConnector.SalesforceTooling.NavigationMenuItem[]'</blockquote>
<p>I say it's an old friend, because <a href="http://www.fishofprey.com/2014/10/importing-salesforce-winter-15-partner.html">I've seen this before</a> with the Partner API when it transitioned to Winter '15 (v32.0) for the QuickActionLayoutItem. It's also <a href="http://www.fishofprey.com/2013/10/importing-salesforce-winter-13-metadata.html">cropped up with Winter '13</a> (v29.0). Thankfully the same fix that was applied in those cases will also work here.</p>
<p>The problem is the <code>navigationLinkSet</code> property that gets generated with a multidimensional array return type. The type for the corresponding XmlArrayItemAttribute is incorrect.</p>
<h2>Before Fix</h2>
<pre class='brush: csharp'>
/// <remarks/>
[System.Xml.Serialization.XmlArrayItemAttribute("navigationMenuItem", typeof(NavigationMenuItem), IsNullable=false)]
public NavigationMenuItem[][] navigationLinkSet {
get {
return this.navigationLinkSetField;
}
set {
this.navigationLinkSetField = value;
}
}
</pre>
<h2>After Fix</h2>
<pre class='brush: csharp'>
/// <remarks/>
[System.Xml.Serialization.XmlArrayItemAttribute("navigationMenuItem", typeof(NavigationMenuItem[]), IsNullable=false)]
public NavigationMenuItem[][] navigationLinkSet {
get {
return this.navigationLinkSetField;
}
set {
this.navigationLinkSetField = value;
}
}
</pre>
<p>The same fix needs to be applied to both the Tooling API and the Metadata API.</p>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-30761440853487603102017-06-12T21:11:00.001+12:002017-11-20T21:37:54.779+13:00Controlling analog PWM servos from a Raspberry Pi<p>As part of a project I'm working on I needed a way to generate motion from my Raspberry Pi. It needed to be a precise variable motion rather than the all or nothing type motion of a straight DC motor. I opted to go with PWM (<a href="https://learn.adafruit.com/adafruit-raspberry-pi-lesson-9-controlling-a-dc-motor/pwm">Pulse Width Modulation</a>) based servos as I'm familiar with them and have some on hand for <a href="http://www.fishofprey.com/2016/10/experiments-in-3d-printing.html#arrow">RC planes</a>. Stepper motors or a DC motor with a rotary encoder were other options, but I have neither in my spare parts bin.</p>
<p>So PWM servos it is. Now, how to control them from the Pi? Getting the Pi to feed the hard real-time response requirements of a servo motor directly can be <a href="https://raspberrypi.stackexchange.com/q/298/37004">difficult</a>, so I opted instead to offload the task to an <a href="https://www.adafruit.com/product/2327">Adafruit 16-Channel PWM / Servo HAT</a>.</p>
<h2>Some assembly required</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-MCW3CmUFLc8/WT5P5PgH71I/AAAAAAAC-2U/lIp3sEoEsGsrkhxWaSKS7Lr6GjemFLr8wCLcB/s1600/20170609_200303.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-MCW3CmUFLc8/WT5P5PgH71I/AAAAAAAC-2U/lIp3sEoEsGsrkhxWaSKS7Lr6GjemFLr8wCLcB/s320/20170609_200303.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;">
<a href="https://2.bp.blogspot.com/-AGdbuE8wXPM/WT5P49wQB8I/AAAAAAAC-2Q/zDeH_fjcF5ogdFUShkZbBhAZYDcrJrFqgCLcB/s1600/20170609_200256.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-AGdbuE8wXPM/WT5P49wQB8I/AAAAAAAC-2Q/zDeH_fjcF5ogdFUShkZbBhAZYDcrJrFqgCLcB/s320/20170609_200256.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<div class="separator" style="text-align: center;">
<a href="https://3.bp.blogspot.com/-89dTxeIOhCg/WT5P6YSvRZI/AAAAAAAC-2c/9etPeuHYyvUh419GOv-KstWboj7eNiUfgCLcB/s1600/20170609_201500.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-89dTxeIOhCg/WT5P6YSvRZI/AAAAAAAC-2c/9etPeuHYyvUh419GOv-KstWboj7eNiUfgCLcB/s320/20170609_201500.jpg" width="320" height="240" data-original-width="1600" data-original-height="1200" /></a></div>
<h2 style="clear: both;">Software</h2>
<p>Because I'm a .NET person I used Windows IoT to program the Pi. Adafruit even provide the <a href="https://learn.adafruit.com/adafruit-class-library-for-windows-iot-core/overview">AdafruitClassLibrary</a> for Windows IoT. The rough code, less exception handling.</p>
<pre class='brush: csharp'>
var i2cSettings = new Windows.Devices.I2c.I2cConnectionSettings(0x40);
var gpio = Windows.Devices.Gpio.GpioController.GetDefault();
Pca9685 hat = new Pca9685(0x40);
await hat.InitPCA9685Async();
hat.SetPWMFrequency(50);
ushort servoMinPulseLength = 150;
ushort servoMaxPulseLength = 600;
hat.SetAllPWM(0, servoMinPulseLength);
while (true)
{
hat.SetAllPWM(0, servoMinPulseLength);
await System.Threading.Tasks.Task.Delay(500);
hat.SetAllPWM(0, servoMaxPulseLength);
await System.Threading.Tasks.Task.Delay(500);
}
</pre>
<p>The exception handling turned out to be one of the harder parts of getting this to work. I kept getting debug messages like <code>Value cannot be null.:I2C Write Exception: {0}</code>. Not very helpful. Once I found the corresponding <a href="https://github.com/adafruit/AdafruitClassLibrary">Github project</a> and worked directly from the source the <a href="https://github.com/adafruit/AdafruitClassLibrary/issues/6">problem was easier to isolate</a>. They are catching the majority of the exceptions in their code and dumping them out to the debug log and then carrying on. This was causing all sorts of problems, with the first occuring in the `InitPCA9685Async` call. The ultimate resolution was to up the Microsoft.NETCore.UniversalWindowsPlatform NuGet package to at least 5.2.2.</p>
<p>After that I just needed to set sensible values for the PWMFrequency and minimum and maximum pulse lengths. Most of the <a href="https://learn.adafruit.com/adafruit-class-library-for-windows-iot-core/pca9685-class">demo code</a> I've seen has a default frequency of 1000. That seemed way to high and the servos became a jittery mess. The general consensus for an analog RC servo is somewhere in the 30Hz to 60Hz range. Some good reading on <a href="https://electronics.stackexchange.com/a/129963">the difference between PPM and PWM</a>.</p>
<h2>The Result</h2>
<iframe width="560" height="315" src="https://www.youtube.com/embed/F9Lg-6h_rmw" frameborder="0" allowfullscreen></iframe>
<p>All going well I'll have more posts soon to bring this together with the other parts I'm working on.</p>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-72105051458108233812017-04-07T11:09:00.000+12:002017-04-07T11:09:16.806+12:00Steps required to support POSTing multipart/form-data Content-Type from Apex<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-DG8CPDzYlEo/WOYPYGw0vrI/AAAAAAAC5iU/PhYowSKLmW07G8RHf9u2r6iey2okhzN2QCLcB/s1600/multipartOR2.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;" title="More that you ever wanted to know about Base64 encoding and the hoops you need to jump through to get multipart/form-data working in Apex."><img border="0" src="https://4.bp.blogspot.com/-DG8CPDzYlEo/WOYPYGw0vrI/AAAAAAAC5iU/PhYowSKLmW07G8RHf9u2r6iey2okhzN2QCLcB/s400/multipartOR2.png" width="286" height="400" /></a></div>
<!--div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-cWLjGg9q_Mk/WORRcZ1zWLI/AAAAAAAC5gI/KqUS0difyxAx7_Srxyj3bTuJRO429qymgCLcB/s1600/multipartOR.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;" title="More that you ever wanted to know about Base64 encoding and the hoops you need to jump through to get multipart/form-data working in Apex."><img border="0" src="https://1.bp.blogspot.com/-cWLjGg9q_Mk/WORRcZ1zWLI/AAAAAAAC5gI/KqUS0difyxAx7_Srxyj3bTuJRO429qymgCLcB/s400/multipartOR.png" width="286" height="400" /></a></div> -->
<p>I came across an interesting challenge when transferring example images to the <a href="https://metamind.readme.io/v1/docs">Einstein Predictive Vision Service</a> (PVS). As part of PVS you can upload example images to train an image classifier. As a gross generalization - here is a picture of a cat. If you get another image like that tell me it is probably a cat.</p>
<p>What follows are the lengths I had to go to in order to be able to call this web service method from Apex.</p>
<p style="clear: both;">In <a href="https://metamind.readme.io/docs/create-an-example">their documentation</a> MetaMind were kind enough to provide an example cURL command to submit the example image. </p>
<pre>
curl -X POST -H "Authorization: Bearer &lt;TOKEN&gt;" -H "Cache-Control: no-cache"
-H "Content-Type: multipart/form-data"
-F "name=77880132.jpg" -F "labelId=614" -F "data=@C:\Mountains vs Beach\Beaches\77880132.jpg"
https://api.metamind.io/v1/vision/datasets/57/examples
</pre>
<div>
<p>It always struck me as odd that specifying the API usage via a specific command line tool has somehow become the de facto standard. It's like describing how to use a REST API is so difficult that it is easiest to just rely on the conventions of a known command line tool. Nothing against cURL, but it is hiding some of the mechanics of how that API call is working. In particular with this example is how the <a href="https://curl.haxx.se/docs/manpage.html#-F">-F</a> is composing the multipart/form-data and how the <code>@</code> in the command will be substituted with the actual file from that path.</p>
</div>
<p>Back to uploading example images to PVS. Performing this API call from Apex proves to be a bit of a challenge as there isn't currently direct native support for a multipart form callouts from Apex. Please pause your blog reading at this point and consider voting for the idea - <a href="https://success.salesforce.com/ideaView?id=08730000000Kr80AAC">Image upload using multipart/form-data</a>.</p>
<p>My first naive attempt at replicating the cURL call from Apex using the supplied <a href="https://github.com/MetaMind/apex-utils/blob/master/HttpFormBuilder.apex">HttpFormBuilder</a> class failed miserably. The binary blob data that I was extracting from an image stored in a ContentVersion record wasn't being accepted by MetaMind.</p>
<pre class='brush: java'>
public ExampleResponse addExample(string access_token, string datasetId, string labelId, string filename,
blob exampleImageData) {
string postUrl = CREATE + '/' + datasetId + '/examples';
string contentType = HttpFormBuilder.GetContentType();
System.assertEquals('multipart/form-data', contentType);
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('labelId', EncodingUtil.urlEncode(labelId, 'UTF-8'));
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('name', EncodingUtil.urlEncode(filename, 'UTF-8'));
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('data', EncodingUtil.base64Encode(exampleImageData));
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
// Compose the http request
HttpRequest httpRequest = new HttpRequest();
httpRequest.setBodyAsBlob(formBlob);
httpRequest.setHeader('Connection', 'keep-alive');
httpRequest.setHeader('Content-Length', contentLength);
httpRequest.setHeader('Content-Type', contentType);
httpRequest.setMethod('POST');
httpRequest.setTimeout(120000);
httpRequest.setHeader('Authorization','Bearer ' + access_token);
httpRequest.setEndpoint(postUrl);
//...
}
</pre>
<p>Where did I go wrong? Well, unlike a prediction there isn't a variant of this call API that accepts the Base64 encoded image. Using <a href="https://github.com/MetaMind/apex-utils/blob/master/HttpFormBuilder.apex#L77"><code>HttpFormBuilder.WriteBodyParameter</code></a> with the Base64 encoded blob to write the data wasn't going to work. MetaMind want the actual bytes and associated Content-Type header. Here is how the working request appears in <a href="https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en">Postman</a>:</p>
<pre>
POST /v1/vision/datasets/1001419/examples HTTP/1.1
Host: api.metamind.io
Authorization: Bearer 12346539689dae622cbd91d9d4880b3314bfb747
Cache-Control: no-cache
----WebKitFormBoundaryE19zNvXGzXaLvS5C
Content-Disposition: form-data; name="data"; filename="ItsAFrog.png"
Content-Type: image/png
&lt;bytes of file go here&gt;
----WebKitFormBoundaryE19zNvXGzXaLvS5C
Content-Disposition: form-data; name="labelId"
8588
----WebKitFormBoundaryE19zNvXGzXaLvS5C
Content-Disposition: form-data; name="name"
itafrog.png
----WebKitFormBoundaryE19zNvXGzXaLvS5C
</pre>
<p>An alternative method was needed to <code>HttpFormBuilder.WriteBodyParameter</code>. One that would set the correct Content-Disposition and Content-Type headers for the binary data and then also correctly append the bytes from the image. That last point is really important. Correcting the headers wasn't very difficult, but with the binary file data I found the last few bytes were getting corrupted on the file.</p>
<p>Thankfully Enrico Murru had already covered a lot of the details around how to get proper multipart form POSTs working in Apex, so full credit to him for the blog post <a href="http://blog.enree.co/2013/01/salesforce-apex-post-mutipartform-data.html#finalCode_multipart">POST Mutipart/form-data with HttpRequest</a>. This was then later refined by Grant Wickman in an answer to <a href="http://salesforce.stackexchange.com/a/33326/102">Post multipart without Base64 Encoding the body</a>. The more I researched why HttpFormBuilder wasn't working the more I found MetaMind had based HttpFormBuilder partially on Enrico and Grant's work but hadn't taking it through to completion. For instance, the <a href="https://github.com/MetaMind/apex-utils/blob/master/HttpFormBuilder.apex#L95"><code>EndingType</code> enum</a> has the following comment indicating they were clearly thinking about it, just never applied it.</p>
<blockquote>
Helper enum indicating how a file's base64 padding was replaced.
</blockquote>
<p>To get it working I needed to finish applying Enrico and Grant's solutions.</p>
<h2>It's all about the Base(64 encoding)</h2>
<p>The smallest unit of storage on a file is a (octet) byte, which is composed of 8 bits. In contrast, <a href="https://en.wikipedia.org/wiki/Base64">Base64 encoding</a> represents 6-bits with each character. So 4 Base64 characters can represent 3 bytes of input data. That ratio of 4:3 is very important with Base64 encoding. If the number of input bytes to be encoded is divisible by 3 then everything is fine as all 4 Base64 characters will represent meaningful input.</p>
<p>The problem occurs if the input byte length isn't divisible by 3. In that case the Base64 encoding process would normally append a padding symbol (=) or two to the end of the encoding to indicate which were the valid bytes in the last 3 and which were padding to get to the correct grouping of 3 bytes to 4 characters.</p>
<p>The following table shows how the padding needs to be applied if there are only two or one bytes to be encoded. You can see how some of the bits in the 6-Bit groupings no longer represent actual data when encoding only one or two input bytes.</p>
<style>
<!--
table.base64 {
background:#FFFFFF;
color: #000000;
}
.xl154254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
white-space:nowrap;}
.xl634254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
white-space:nowrap;}
.xl644254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:.5pt solid windowtext;
white-space:nowrap;}
.xl654254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:none;
white-space:nowrap;}
.xl664254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:none;
white-space:nowrap;}
.xl674254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:.5pt solid windowtext;
border-bottom:.5pt solid windowtext;
border-left:none;
white-space:nowrap;}
.xl684254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:.5pt solid windowtext;
white-space:nowrap;}
.xl694254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:none;
border-left:.5pt solid windowtext;
white-space:nowrap;}
.xl704254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:none;
border-left:none;
white-space:nowrap;}
.xl714254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:none;
border-left:none;
white-space:nowrap;}
.xl724254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:.5pt solid windowtext;
border-bottom:none;
border-left:none;
white-space:nowrap;}
.xl734254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:none;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:.5pt solid windowtext;
white-space:nowrap;}
.xl744254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:none;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:none;
white-space:nowrap;}
.xl754254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:none;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:none;
white-space:nowrap;}
.xl764254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:none;
border-right:.5pt solid windowtext;
border-bottom:.5pt solid windowtext;
border-left:none;
white-space:nowrap;}
.xl774254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:none;
border-left:.5pt solid windowtext;
white-space:nowrap;}
.xl784254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:none;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:.5pt solid windowtext;
white-space:nowrap;}
.xl794254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:none;
border-left:.5pt solid windowtext;
background:#E7E6E6;
white-space:nowrap;}
.xl804254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:none;
border-left:none;
background:#E7E6E6;
white-space:nowrap;}
.xl814254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:.5pt solid windowtext;
border-bottom:none;
border-left:none;
background:#E7E6E6;
white-space:nowrap;}
.xl824254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:.5pt solid windowtext;
background:#E7E6E6;
white-space:nowrap;}
.xl834254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:.5pt solid windowtext;
border-left:none;
background:#E7E6E6;
white-space:nowrap;}
.xl844254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:.5pt solid windowtext;
border-bottom:.5pt solid windowtext;
border-left:none;
background:#E7E6E6;
white-space:nowrap;}
.xl854254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:none;
border-bottom:none;
border-left:none;
background:#FCE4D6;
white-space:nowrap;}
.xl864254
{padding-top:1px;
padding-right:1px;
padding-left:1px;
text-decoration:none;
text-align:general;
vertical-align:bottom;
border-top:.5pt solid windowtext;
border-right:.5pt solid windowtext;
border-bottom:none;
border-left:none;
background:#FCE4D6;
white-space:nowrap;}
-->
</style>
<table border="0" cellpadding="0" cellspacing="0" width="621" style="border-collapse:collapse;table-layout:fixed;width:466pt" class="base64">
<colgroup><col width="237" style="mso-width-source:userset;mso-width-alt:8667;width:178pt">
<col class="xl634254" width="16" span="3" style="mso-width-source:userset;
mso-width-alt:585;width:12pt">
<col width="16" span="21" style="mso-width-source:userset;mso-width-alt:585;
width:12pt">
</colgroup><tbody><tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" width="237" style="height:15.0pt;width:178pt">Data
Bytes In Binary Form</td>
<td class="xl644254" width="16" style="width:12pt">1</td>
<td class="xl654254" width="16" style="width:12pt">1</td>
<td class="xl654254" width="16" style="width:12pt">0</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">0</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">0</td>
<td class="xl664254" width="16" style="width:12pt">0</td>
<td class="xl684254" width="16" style="width:12pt">0</td>
<td class="xl664254" width="16" style="width:12pt">0</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">0</td>
<td class="xl664254" width="16" style="width:12pt">0</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl674254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">0</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl664254" width="16" style="width:12pt">1</td>
<td class="xl674254" width="16" style="width:12pt">1</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">Data Rearranged Into 6-Bit
Groups</td>
<td class="xl694254" style="border-top:none">1</td>
<td class="xl704254" style="border-top:none">1</td>
<td class="xl704254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl724254" style="border-top:none">1</td>
<td class="xl774254" style="border-top:none;border-left:none">0</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl724254" style="border-top:none">0</td>
<td class="xl774254" style="border-top:none;border-left:none">0</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl724254" style="border-top:none">1</td>
<td class="xl774254" style="border-top:none;border-left:none">1</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl724254" style="border-top:none">1</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">6-Bit Groups in Decimal
Form</td>
<td class="xl644254">&nbsp;</td>
<td class="xl654254">&nbsp;</td>
<td class="xl654254">5</td>
<td class="xl664254" >3</td>
<td class="xl664254">&nbsp;</td>
<td class="xl674254">&nbsp;</td>
<td class="xl684254" style="border-left:none">&nbsp;</td>
<td class="xl664254">&nbsp;</td>
<td class="xl664254">&nbsp;</td>
<td class="xl664254" >2</td>
<td class="xl664254">&nbsp;</td>
<td class="xl674254">&nbsp;</td>
<td class="xl684254" style="border-left:none">&nbsp;</td>
<td class="xl664254">&nbsp;</td>
<td class="xl664254" >3</td>
<td class="xl664254" >1</td>
<td class="xl664254">&nbsp;</td>
<td class="xl674254">&nbsp;</td>
<td class="xl684254" style="border-left:none">&nbsp;</td>
<td class="xl664254">&nbsp;</td>
<td class="xl664254" >5</td>
<td class="xl664254" >5</td>
<td class="xl664254">&nbsp;</td>
<td class="xl674254">&nbsp;</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">Groups Converted to ASCII
Characters</td>
<td class="xl734254">&nbsp;</td>
<td class="xl744254">&nbsp;</td>
<td class="xl744254">&nbsp;</td>
<td class="xl754254" >1</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
<td class="xl784254" style="border-left:none">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">C</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
<td class="xl784254" style="border-left:none">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">f</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
<td class="xl784254" style="border-left:none">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254" >3</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt"></td>
<td class="xl634254"></td>
<td class="xl634254"></td>
<td class="xl634254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">Data Bytes In Binary Form</td>
<td class="xl644254">1</td>
<td class="xl654254">1</td>
<td class="xl654254">0</td>
<td class="xl664254" >1</td>
<td class="xl664254" >0</td>
<td class="xl664254" >1</td>
<td class="xl664254" >0</td>
<td class="xl664254" >0</td>
<td class="xl684254" >0</td>
<td class="xl664254" >0</td>
<td class="xl664254" >1</td>
<td class="xl664254" >0</td>
<td class="xl664254" >0</td>
<td class="xl664254" >1</td>
<td class="xl664254" >1</td>
<td class="xl674254" >1</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl844254">&nbsp;</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">Data Rearranged Into 6-Bit
Groups</td>
<td class="xl694254" style="border-top:none">1</td>
<td class="xl704254" style="border-top:none">1</td>
<td class="xl704254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl724254" style="border-top:none">1</td>
<td class="xl774254" style="border-top:none;border-left:none">0</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl724254" style="border-top:none">0</td>
<td class="xl774254" style="border-top:none;border-left:none">0</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl854254" style="border-top:none">0</td>
<td class="xl864254" style="border-top:none">0</td>
<td class="xl794254" style="border-top:none;border-left:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl814254" style="border-top:none">&nbsp;</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">6-Bit Groups in Decimal
Form</td>
<td class="xl644254">&nbsp;</td>
<td class="xl654254">&nbsp;</td>
<td class="xl654254">5</td>
<td class="xl664254" >3</td>
<td class="xl664254">&nbsp;</td>
<td class="xl674254">&nbsp;</td>
<td class="xl684254" style="border-left:none">&nbsp;</td>
<td class="xl664254">&nbsp;</td>
<td class="xl664254">&nbsp;</td>
<td class="xl664254" >2</td>
<td class="xl664254">&nbsp;</td>
<td class="xl674254">&nbsp;</td>
<td class="xl684254" style="border-left:none">&nbsp;</td>
<td class="xl664254">&nbsp;</td>
<td class="xl664254" >2</td>
<td class="xl664254" >8</td>
<td class="xl664254">&nbsp;</td>
<td class="xl674254">&nbsp;</td>
<td class="xl824254" style="border-left:none">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl844254">&nbsp;</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">Groups Converted to ASCII
Characters</td>
<td class="xl734254">&nbsp;</td>
<td class="xl744254">&nbsp;</td>
<td class="xl744254">&nbsp;</td>
<td class="xl754254" >1</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
<td class="xl784254" style="border-left:none">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">C</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
<td class="xl784254" style="border-left:none">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">c</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
<td class="xl784254" style="border-left:none">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">=</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt"></td>
<td class="xl634254"></td>
<td class="xl634254"></td>
<td class="xl634254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
<td class="xl154254"></td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">Data Bytes In Binary Form</td>
<td class="xl644254">1</td>
<td class="xl654254">1</td>
<td class="xl654254">0</td>
<td class="xl664254">1</td>
<td class="xl664254">0</td>
<td class="xl664254">1</td>
<td class="xl664254">0</td>
<td class="xl664254">0</td>
<td class="xl824254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl844254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl844254">&nbsp;</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">Data Rearranged Into 6-Bit
Groups</td>
<td class="xl694254" style="border-top:none">1</td>
<td class="xl704254" style="border-top:none">1</td>
<td class="xl704254" style="border-top:none">0</td>
<td class="xl714254" style="border-top:none">1</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl724254" style="border-top:none">1</td>
<td class="xl774254" style="border-top:none;border-left:none">0</td>
<td class="xl714254" style="border-top:none">0</td>
<td class="xl854254" style="border-top:none">0</td>
<td class="xl854254" style="border-top:none">0</td>
<td class="xl854254" style="border-top:none">0</td>
<td class="xl864254" style="border-top:none">0</td>
<td class="xl794254" style="border-top:none;border-left:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl814254" style="border-top:none">&nbsp;</td>
<td class="xl794254" style="border-top:none;border-left:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl804254" style="border-top:none">&nbsp;</td>
<td class="xl814254" style="border-top:none">&nbsp;</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">6-Bit Groups in Decimal
Form</td>
<td class="xl644254">&nbsp;</td>
<td class="xl654254">&nbsp;</td>
<td class="xl654254">5</td>
<td class="xl664254" >3</td>
<td class="xl664254">&nbsp;</td>
<td class="xl674254">&nbsp;</td>
<td class="xl684254" style="border-left:none">&nbsp;</td>
<td class="xl664254">&nbsp;</td>
<td class="xl664254">&nbsp;</td>
<td class="xl664254" >0</td>
<td class="xl664254">&nbsp;</td>
<td class="xl674254">&nbsp;</td>
<td class="xl824254" style="border-left:none">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl844254">&nbsp;</td>
<td class="xl824254" style="border-left:none">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl834254">&nbsp;</td>
<td class="xl844254">&nbsp;</td>
</tr>
<tr height="20" style="height:15.0pt">
<td height="20" class="xl154254" style="height:15.0pt">Groups Converted to ASCII
Characters</td>
<td class="xl734254">&nbsp;</td>
<td class="xl744254">&nbsp;</td>
<td class="xl744254">&nbsp;</td>
<td class="xl754254" >1</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
<td class="xl784254" style="border-left:none">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">A</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
<td class="xl784254" style="border-left:none">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">=</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
<td class="xl784254" style="border-left:none">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">&nbsp;</td>
<td class="xl754254">=</td>
<td class="xl754254">&nbsp;</td>
<td class="xl764254">&nbsp;</td>
</tr>
<!--[if supportMisalignedColumns]-->
<tr height="0" style="display:none">
<td width="237" style="width:178pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
<td width="16" style="width:12pt"></td>
</tr>
<!--[endif]-->
</tbody></table>
<p>Enrico's method to build up the full multipart form submission from Apex is to first build each part individually using Base64 encoding and then concatenate those parts together. That final Base64 string is then converted back to a Blob to become the body of the HTTP POST request.</p>
<p>The challenge with this method is you can't have intermediate padding characters. The Apex Base64 decoding process is going to ignore those intermediate padding characters. This in turn causes an incorrect mapping between the 4 Base64 characters and the 3 bytes they are supposed to represent. Here is an example using some anonymous Apex. Note how the decoding the concatenated values gives the same result with or without the internal padding: </p>
<pre class='brush: java'>
String encodedA = EncodingUtil.base64Encode(Blob.valueOf('A'));
String encodedB = EncodingUtil.base64Encode(Blob.valueOf('BC'));
System.debug(encodedA);
System.debug(encodedB);
System.debug(EncodingUtil.base64Decode(encodedA).toString());
System.debug(EncodingUtil.base64Decode(encodedB).toString());
System.debug(EncodingUtil.base64Decode(encodedA + encodedB).toString());
System.debug(EncodingUtil.base64Decode('QQ' + 'QkM=').toString());
</pre>
<p>Output</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ftZYWETvjkU/WOSpUK-65TI/AAAAAAAC5go/jPZ9VMkQLI8zfYykM0FtxWbfYxXvVbkbQCLcB/s1600/DebugLog.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-ftZYWETvjkU/WOSpUK-65TI/AAAAAAAC5go/jPZ9VMkQLI8zfYykM0FtxWbfYxXvVbkbQCLcB/s400/DebugLog.PNG" width="400" height="163" /></a></div>
<pre>
20:17:50.65 (69059707)|USER_DEBUG|[3]|DEBUG|QQ==
20:17:50.65 (69127131)|USER_DEBUG|[4]|DEBUG|QkM=
20:17:50.65 (69373165)|USER_DEBUG|[5]|DEBUG|A
20:17:50.65 (69500491)|USER_DEBUG|[6]|DEBUG|BC
20:17:50.65 (69636544)|USER_DEBUG|[7]|DEBUG|A$
20:17:50.65 (69759126)|USER_DEBUG|[8]|DEBUG|A$
</pre>
<p> For encoding text this problem is bypassed by appending additional whitespace characters to the base text until the Base64 encoded representation no longer requires padding. Without the padding it is then possible to concatenate the two base 64 encoded values and then later decode the complete string back to the Blob sans additional bytes. This however doesn't work with a binary file like an image. You can't just go on appending additional bytes on the end without corrupting the file.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-30Opw6qHhcU/WOQuHf76PfI/AAAAAAAC5f4/U9Zs711hlKE5vJsaK_WxPgvo88vHv7lrACLcB/s1600/Padding.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-30Opw6qHhcU/WOQuHf76PfI/AAAAAAAC5f4/U9Zs711hlKE5vJsaK_WxPgvo88vHv7lrACLcB/s400/Padding.png" width="400" height="207" /></a></div>
<p>Instead the solution presented by <a href="http://salesforce.stackexchange.com/a/33326/102">Grant Wickman</a> is to borrow one or both of the CR (\r) or LF (\n) characters that separate the end of the file bytes from the multipart boundary.</p>
<p>By shuffling these legitimate characters onto the end of the files bytes the need for padding can be removed. The padding on the boundary can then be adjusted as required using the white-space technique where it won't affect the integrity of the data.</p>
<p>There is a certain elegance to it as a solution. Note how the first 4 bits of the CR align with the last 6 bits of the second Base64 character so they don't need to be changed. And then again, if only switching out one padding = for a LF the first two bits again align. </p>
<a href="https://2.bp.blogspot.com/-YT4w63I9qO8/WOS_U6_2BlI/AAAAAAAC5hI/eVb2kc45BSY7Doq9dw6mNE94NYdoSOnGQCLcB/s1600/Padding.PNG" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-YT4w63I9qO8/WOS_U6_2BlI/AAAAAAAC5hI/eVb2kc45BSY7Doq9dw6mNE94NYdoSOnGQCLcB/s1600/Padding.PNG" /></a>
<p>I find that table very satisfying. I think I might print it out and keep it at my desk like one of those inspirational posters.</p>
<h2>Example of additional HttpFormBuilder method</h2>
<script src="https://gist.github.com/FishOfPrey/fe7f5ac7cdf7f423c003598e7d6af4df.js"></script>
<h2>What did we learn?</h2>
<ol>
<li>That rogue = padding characters in the middle of a Base64 string will corrupt the data.</li>
<li>If the Base64 encoding for the file ends with "==", crop those characters off and replace with <code>0K</code> to represent a CRLF. Then don't prepend the following boundary footer with those characters.</li>
<li>If the Base64 encoding for the file ends with "=", crop that character off and replace with <code>N</code> to represent a CR. Then only prepend the following boundary footer a LF.</li>
<li>That <a href="https://twitter.com/muenzpraeger">@muenzpraeger</a> has <a href="https://github.com/muenzpraeger/salesforce-einstein-vision-apex">a more refined Apex project</a> for working with the Einstein Predictive Vision Service. I've <a href="https://github.com/muenzpraeger/salesforce-einstein-vision-apex/pull/7">submitted my changes</a> there.</li>
<li><div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-UuZPKyPSNtY/WOYHm9bxK9I/AAAAAAAC5ho/77DN9uzp1rsb2Rr5_idyiKYBG7V7Qt2DQCLcB/s1600/hedcut.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-UuZPKyPSNtY/WOYHm9bxK9I/AAAAAAAC5ho/77DN9uzp1rsb2Rr5_idyiKYBG7V7Qt2DQCLcB/s200/hedcut.png" width="200" height="200" /></a></div>The O'Reilly Book covers featuring pictures of animals are done in a <a href="https://en.wikipedia.org/wiki/Woodcut">woodcut</a>/<a href="https://en.wikipedia.org/wiki/Hedcut">hedcut</a> style. <!-- http://www.squaregear.net/gimptips/hedcut.shtml --> </li>
<li style="clear: both;">
<div class="separator" style="clear: both; text-align: center;"><a style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;" href="https://4.bp.blogspot.com/-hWZuBEjUI34/WOX0Nv3fBkI/AAAAAAAC5hY/hRHhzSDqBwQULDpmOmti_rC1zcRpOJY7ACLcB/s1600/HangInThere.jpg" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-hWZuBEjUI34/WOX0Nv3fBkI/AAAAAAAC5hY/hRHhzSDqBwQULDpmOmti_rC1zcRpOJY7ACLcB/s320/HangInThere.jpg" width="320" height="180" /></a></div>
Base64 encoding examples can be inspirational and look great at your desk.
</li>
</ol>
FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-36618861974716177632017-03-24T21:35:00.000+13:002017-03-25T10:31:04.768+13:00The Mad Catter - Salesforce Predictive Vision Services<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ixGjDrkFP2s/WJwfL3DjxVI/AAAAAAAC234/-v0chvDlbkMqchhCu9HfHgtNeTOr4XpiwCLcB/s1600/MadCatter.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-ixGjDrkFP2s/WJwfL3DjxVI/AAAAAAAC234/-v0chvDlbkMqchhCu9HfHgtNeTOr4XpiwCLcB/s1600/MadCatter.png" /></a></div>
<div style="float: right; width: 200px; border-style: inset; border-width: 3px; margin: 10px">
<h2>Disclaimer</h2>
<p>No animals were harmed in the creation of this blog post or the associated presentation. The standard <a href="http://www.fishofprey.com/2009/10/blog-disclaimer.html">disclaimer</a> applies.
</p>
</div>
<p>I've got a problem. A cat problem to be precise. While I'm more of a dog person, I don't particularly mind cats. However, recently there has been a bit of an influx of them with the neighbors. There are at least a half dozen of them that roam freely around the neighborhood. Not the end of the world, but they have a nasty habit of leaving presents on the lawn for the kids to find. Things like the following:</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-BDQcreYd9As/WJwbb3kfcKI/AAAAAAAC23k/Pt0hxsn62n4ZpctSFLaaO9Www_dOsLkwgCLcB/s1600/WP_20170209_15_28_05_Pro.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-BDQcreYd9As/WJwbb3kfcKI/AAAAAAAC23k/Pt0hxsn62n4ZpctSFLaaO9Www_dOsLkwgCLcB/s320/WP_20170209_15_28_05_Pro.jpg" width="320" height="180" /></a></div><div class="separator" style=" text-align: center;"><a href="https://1.bp.blogspot.com/-PUkygHT4b9k/WJwbb-miNxI/AAAAAAAC23s/d3UnVlVTO3kk97rOua3LKfmn_OAOF9yrACLcB/s1600/WP_20170209_15_28_27_Pro.jpg" imageanchor="1" style="float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-PUkygHT4b9k/WJwbb-miNxI/AAAAAAAC23s/d3UnVlVTO3kk97rOua3LKfmn_OAOF9yrACLcB/s320/WP_20170209_15_28_27_Pro.jpg" width="320" height="180" /></a></div><div class="separator" style="text-align: center;"><a href="https://2.bp.blogspot.com/-crDcujaobtU/WJwbb6EGq-I/AAAAAAAC23o/TA6DWI1-5CsITc01ecscejPy1d1_f2CAwCLcB/s1600/WP_20170209_15_30_06_Pro.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-crDcujaobtU/WJwbb6EGq-I/AAAAAAAC23o/TA6DWI1-5CsITc01ecscejPy1d1_f2CAwCLcB/s320/WP_20170209_15_30_06_Pro.jpg" width="320" height="180" /></a></div><div class="separator" style="text-align: center;"><a href="https://2.bp.blogspot.com/-OkQGPZYrM34/WJwbdxZa-6I/AAAAAAAC23w/k7DWYVp5Z5srOV1OnCKZkPw_Ro4trAsgQCLcB/s1600/WP_20170209_15_34_04_Pro.jpg" imageanchor="1" style="float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-OkQGPZYrM34/WJwbdxZa-6I/AAAAAAAC23w/k7DWYVp5Z5srOV1OnCKZkPw_Ro4trAsgQCLcB/s320/WP_20170209_15_34_04_Pro.jpg" width="320" height="180" /></a></div>
<p style="clear: both">In short, they poop everywhere and leave the remains of birds lying around. Things that aren't so great to step on or for the kids to find when playing in the garden.</p>
<p>I spoke with the immediate neighbor who owns two of the cats, and he suggested spraying them with water to deter them away. While that did indeed prove to be a very effective, amusing, and satisfying approach to move them on it required 24 hour vigilance as they just kept coming back.</p>
<h2>Get a Dog</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://www.youtube.com/embed/jqZueLp_3yU" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;" target="_blank"><img border="0" src="https://1.bp.blogspot.com/-g3hdXGemj38/WNMluowoR4I/AAAAAAAC45Q/u3N9gSB9M0wqKooRiLkzCGM3vhlSEGTRwCLcB/s320/dogSpray.jpg" width="320" height="180" /></a></div>
<p>My wife keeps saying we should just get a dog. I guess getting a dog to chase the cats away is an option. But it seems like training a dog to use the hose might be more trouble in the long run.</p>
<h2 style="clear:both">Technology to the Rescue</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-zMU5UkcYw4s/WJwhXYiQMnI/AAAAAAAC24A/4HLb6kffyDY1WscJMtKEEj5rEGk6nHhXQCLcB/s1600/MotionSensitiveSprayer.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-zMU5UkcYw4s/WJwhXYiQMnI/AAAAAAAC24A/4HLb6kffyDY1WscJMtKEEj5rEGk6nHhXQCLcB/s320/MotionSensitiveSprayer.jpg" width="320" height="320" /></a></div>
<p>Thankfully I found a handy device the attaches to the end of the house and activates a sprinkler head when a built in motion detector is set off.</p>
<p>Perfect! Great! Cat comes into range, then a sudden noise and spray of water sends them off towards someone else's lawn to do their cat business. Problem solved and I can go back to doing more fun activities.</p>
<p>Except there was one small problem. The PIR motion sensor didn't particularly care what was moving in front of it. Cats, birds, the kids on their way to school, a courier with a parcel for me, a tree in the wind, the mother in law. It would spray them all regardless of whether I wanted it to or not.</p>
<h2 style="clear: both">Salesforce Predictive Vision Service</h2>
<p>Technology wasn't solving my problem. I needed to use more of it!</p>
<p>I recalled a recent presentation by the Salesforce Developers team - <a href="https://developer.salesforce.com/events/webinars/predictivevisionservice">Build Smarter Apps with New Predictive Vision Service</a>. The short version of that presentation is you can train a deep learning image classifier with a set of training images. Then when you give it a new image it will give you probabilities about what is likely in the picture. I've create a <a href="http://www.fishofprey.com/2017/02/visualforce-quick-start-with-salesforce.html">quick start unmanaged package</a> for it to save you going through most of the install steps.</p>
<p>To make this work I needed a large collection of cat images to train the dataset from a create a model. Luckily for me, providing pictures of cats is something that the internet excels at.</p>
<p>The second challenge with using the predictive vision services is managing how many images I am going to send through to the service. If I just point a web camera out the window it could be capturing 30+ frames per second. Not really practical to send off each frame to the service for identification when there might be nothing of interest happening 99% of the time.</p>
<h2>Motion detection</h2>
<p>I had a few options here.</p>
<p>Option One would to to stick with the basic <span title="passive infrared sensor">PIR</span> motion sensor, but it would still be generating a ton of false positives that would need to pass through the image recognition. A simple cool down timer would help, but the image captured immediately after the first motion is detected would likely get something as it is just entering the frame.</p>
<p>I figure since I'm going to need a camera to capture the initial image I might as well get some usage out of it in detecting the motion. Because of the initial processing step I can exclude motion from certain areas, such as a driveway or tree that often moves in the wind. There can also be a slight delay after the motion is detected and before the prediction image is captured. This gives the subject time to move into the image.</p>
<p>The prototype solution looks like this:</p>
<ol>
<li><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-BMqLUQDgtfA/WNN9wdo_vOI/AAAAAAAC45o/gYKKJvHG1gUgrrekJN8Gy6byKj7-LzdWgCLcB/s1600/c920.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-BMqLUQDgtfA/WNN9wdo_vOI/AAAAAAAC45o/gYKKJvHG1gUgrrekJN8Gy6byKj7-LzdWgCLcB/s200/c920.png" width="200" height="172" /></a></div>
A webcam that can be pointed at the area to be monitored. </li>
<li><div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-o1AcqMVdy_o/WNN9403_yqI/AAAAAAAC45s/shTTH7QZGi4nOXvAtKQmLEfmyahLanFtQCLcB/s1600/motionDetection.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-o1AcqMVdy_o/WNN9403_yqI/AAAAAAAC45s/shTTH7QZGi4nOXvAtKQmLEfmyahLanFtQCLcB/s200/motionDetection.jpg" width="200" height="164" /></a></div>
Motion Detection software to process the video feed and determine where the movement is and the magnitude. The magnitude is useful, as particularly small subjects like birds can be completely ignored.</li>
<li><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-S_BjmPQeGsI/WNN-Z4ZOymI/AAAAAAAC450/IVITM5vMNnE5COdjbLf3vcnXMTXDthH3wCLcB/s1600/MetaMind.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-S_BjmPQeGsI/WNN-Z4ZOymI/AAAAAAAC450/IVITM5vMNnE5COdjbLf3vcnXMTXDthH3wCLcB/s200/MetaMind.png" width="200" height="200" /></a></div>The ability for that software to pass frames of interest off to the Salesforce Predictive Vision Service. This is a <a href="https://metamind.readme.io/docs/prediction-with-image-base64-string">simple REST POST request</a> using an access token.</li>
<li><div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-0A4nwRm4DAY/WNN-HZsF71I/AAAAAAAC45w/dzgZ-UUDCcoD5qi4ZBCfE8Pc9Xs06CYQwCLcB/s1600/RP2_Pinout.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://3.bp.blogspot.com/-0A4nwRm4DAY/WNN-HZsF71I/AAAAAAAC45w/dzgZ-UUDCcoD5qi4ZBCfE8Pc9Xs06CYQwCLcB/s200/RP2_Pinout.png" width="200" height="137" /></a></div>If the probability from the frame indicates a Cat is present, send a signal to the Raspberry Pi.</li>
<li>On the signal, the Raspberry Pi activates the GPIO pin connected to a relay.</li>
<li><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-SQL_XmAqNwQ/WNN-jRSVzoI/AAAAAAAC454/MWue7MmD8q8XNkVftqMtvh4XFeykZ8SrgCLcB/s1600/relay3.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-SQL_XmAqNwQ/WNN-jRSVzoI/AAAAAAAC454/MWue7MmD8q8XNkVftqMtvh4XFeykZ8SrgCLcB/s200/relay3.jpg" width="200" height="108" /></a></div>When activated, the relay supplies power to the existing automated sprinkler, which activates on initial power on when the sensitivity is set to maximum. Another option here is directly connecting a solenoid water value to the hose line.</li>
</ol>
<p style="clear: both;">When all put together the end result looks something like this:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/oFD9H9Mfgi0" frameborder="0" allowfullscreen></iframe>
<p>The Einstein bust with terminator-esque glowing red eyes was part of the <a href="#ug">presentation</a> I gave on this topic.</p>
<p>While filming that video I inadvertently live tested it on myself as well. An aging fitting on the hose connector to the sprinkler had come loose outside at the tap. So I went out to fix that, restored the water pressure to the sprinkler, then walked back to the laptop to resume the test. Only when I checked the motion detection screen did I realize it had captured my image passing in front of the sprinkler. Thankfully the predictive vision services came back indicating I didn't resemble a cat and the sprinkler didn't activate. Success!</p>
<h2>Refinements</h2>
<p>It occured to me that there were further improvements that could be made.</p>
<p>The first and easiest change I made is to activate on things other than cats. It can be equally selective to activate on a wandering neighbors dog, squirrels, general wildlife, etc...</p>
<p>I needed a way to deal with unfortunate false positives, such as a person wearing something with a picture of a cat on it. These can partially be avoided by looking at all the probabilities that Einstein is returning and having thresholds against each label. I.e. Activate on any Cat prediction above 50% unless there is any prediction indicating a person in the field of view. Images kept from the activations could also be used to further refine the examples in the dataset.</p>
<p>These first two refinements are actually present in the video above. When using the <a href="https://metamind.readme.io/v1/page/general-image-model-class-list">general image classifier</a> it typically identifies the stuffed cat as a teddy bear. So in the small section to the bottom right of the app you can mark labels to activate on and labels that will override and prevent the activation.</p>
<p>Other changes I might consider making:</p>
<p>The motion sensor could be maintained and introduced as the step prior to activating the video feed. This would increase the time between the target entering the area and the sprinkler activating, but would save a lot of endless processing loops looking at an unchanging image.</p>
<p>If I forgo some of the more processing intensive motion tracking the whole solution could be moved onto the Raspberry Pi. This would make it a much more economical solution.</p>
<p>However, another option with the motion detection still in place would be to crop the frame image to just the area where the motion was detected. This should lead to much higher prediction accuracy as it is only focusing on the moving subject.</p>
<p>When real world testing commences with live subjects I'll need to add a video capture option to cover the time from just before the sprinkler is activated till just after it switches off. I think the results will be worth the extra effort.</p>
<p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-odIM7sjtUz0/WNTPaD4CQLI/AAAAAAAC46w/5FlV_MxRbcwdIJjUst3reHshrJ4eGEG6QCLcB/s1600/Ultrasonic.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-odIM7sjtUz0/WNTPaD4CQLI/AAAAAAAC46w/5FlV_MxRbcwdIJjUst3reHshrJ4eGEG6QCLcB/s320/Ultrasonic.png" width="84" height="320" /></a></div>I have a range of other devices that could easily be activated via the relay attached to the Raspberry Pi. One such device is an ultrasonic pest repeller. Perhaps combined with a temperature sensor as a slightly kinder deterrent on cold nights.</p>
<h2 id="ug" style="clear: both">User Group Presentation</h2>
<p>I gave a talk to the <a href="https://www.meetup.com/Sydney-Salesforce-Platform-Developer-Users-Group/events/237886182/">Sydney developer user group</a> on this project. The slides, as they were:</p>
<iframe id="iframe_container" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen="" width="700" height="510" src="https://prezi.com/embed/wcw3_otnypad/?bgcolor=ffffff&amp;lock_to_path=0&amp;autoplay=0&amp;autohide_ctrls=0&amp;landing_data=bHVZZmNaNDBIWnNjdEVENDRhZDFNZGNIUE43MHdLNWpsdFJLb2ZHanI0eW9iekpCZHdtazlOa2tuczluRFJvaS9BPT0&amp;landing_sign=NHy36W5BYbN2rsJ3scL1My7ak2P8z491ayU5bLyPjeg"></iframe>
<!--h2 style="clear: both">Other possible applications</h2>
<!--p>Other applications could be far broader. Looking for fires from Richmond hills. Monitoring boats entering the port.</p-->
<hr/>
<p>I still feel the need to settle on a name for the project. Options include:</p>
<ul>
<li>The Mad Catter (after the elusive Catter Trailhead badge)</li>
<li>The Einstein Cannon</li>
<li>The Cattinator (After the general them of the <a href="#ug">presentation</a>.)</li>
</ul>
<hr/>
<p>See also:</p>
<ul>
<li><a href="https://einstein-vision.herokuapp.com/">Einstein Vision Demo of the General Image Classifier</a></li>
<li>Trailhead <a href="https://trailhead.salesforce.com/projects/predictive_vision_apex">Quick Start: Predictive Vision Service</a> </li>
<li><a href="https://www.salesforce.com/blog/2017/03/introducing-einstein-vision.html">Introducing Einstein Vision: Your Apps, Now Powered by Salesforce Einstein</a></li>
<li>Java based wrapper for using the Predictive Vision Services - <a href="https://github.com/muenzpraeger/salesforce-einstein-vision-java">salesforce-einstein-vision-java</a></li>
<li><a href="https://www.youtube.com/embed/uIbkLjjlMV8">Inspirational video</a> dealing with cats spraying cars</li>
</ul>
<h2>Gallery</h2>
<div class="separator" style="text-align: center;"><a href="https://2.bp.blogspot.com/-lQBqh1K0GjQ/WNWMTi5fkMI/AAAAAAAC5DI/C4bRKT2HLSAE8Sv5tS_HBzF697-xtA6mgCLcB/s1600/raspberrypiCircuit2.png" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-lQBqh1K0GjQ/WNWMTi5fkMI/AAAAAAAC5DI/C4bRKT2HLSAE8Sv5tS_HBzF697-xtA6mgCLcB/s320/raspberrypiCircuit2.png" width="320" height="213" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-eFNR9ddlaYg/WNWMS5MprUI/AAAAAAAC5DA/tmyFsHVFiaUx4IbtV3LRSSv49C7jG3cdwCLcB/s1600/Relay.png" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-eFNR9ddlaYg/WNWMS5MprUI/AAAAAAAC5DA/tmyFsHVFiaUx4IbtV3LRSSv49C7jG3cdwCLcB/s320/Relay.png" width="320" height="213" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://1.bp.blogspot.com/-5bLFv9coGzA/WNWMTOADoEI/AAAAAAAC5DE/r6QtuF-h3hk1FsQzCkhHv_jvEMku7btSACLcB/s1600/Sprinkler.png" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-5bLFv9coGzA/WNWMTOADoEI/AAAAAAAC5DE/r6QtuF-h3hk1FsQzCkhHv_jvEMku7btSACLcB/s320/Sprinkler.png" width="213" height="320" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://2.bp.blogspot.com/-sk4w05x8DZw/WNWMPgg8U7I/AAAAAAAC5Cw/jSwRtvIUj6UR_qGSio9CI-yCC1Plo7Y4gCLcB/s1600/P3250082.JPG" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-sk4w05x8DZw/WNWMPgg8U7I/AAAAAAAC5Cw/jSwRtvIUj6UR_qGSio9CI-yCC1Plo7Y4gCLcB/s320/P3250082.JPG" width="320" height="213" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-tq_Pu_lBgNM/WNWPnRWGfiI/AAAAAAAC5Es/NrglXO4dyQMxjFAkJ0sRP2qCz6vzKYq4ACLcB/s1600/WP_20170320_15_22_13_Pro%2B%25282%2529.jpg" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-tq_Pu_lBgNM/WNWPnRWGfiI/AAAAAAAC5Es/NrglXO4dyQMxjFAkJ0sRP2qCz6vzKYq4ACLcB/s320/WP_20170320_15_22_13_Pro%2B%25282%2529.jpg" width="320" height="193" /></a></div>
<div class="separator" style="clear: both;text-align: center;"><a href="https://1.bp.blogspot.com/-AH1uKqOmP-4/WNWMPoUC55I/AAAAAAAC5C0/7Q1xAtnbk7UrMb96Q1PcJCtiv_K1TmXVwCLcB/s1600/P3250088.JPG" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-AH1uKqOmP-4/WNWMPoUC55I/AAAAAAAC5C0/7Q1xAtnbk7UrMb96Q1PcJCtiv_K1TmXVwCLcB/s320/P3250088.JPG" width="213" height="320" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-rLd07ITZ24M/WNWMP7K9C7I/AAAAAAAC5C4/B7wWrMKGyTAHvMB6Etcbu-uIc0cBLRKjQCLcB/s1600/P3250103.JPG" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-rLd07ITZ24M/WNWMP7K9C7I/AAAAAAAC5C4/B7wWrMKGyTAHvMB6Etcbu-uIc0cBLRKjQCLcB/s320/P3250103.JPG" width="320" height="213" /></a></div>
<div class="separator" style="text-align: center;"><a href="https://4.bp.blogspot.com/-ys-XHhR8EMw/WNWMQf2KZvI/AAAAAAAC5C8/XKASkAMm6Ak5nSfE8N8vA0xoiFx82HndwCLcB/s1600/P3250104.JPG" imageanchor="1" style="float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-ys-XHhR8EMw/WNWMQf2KZvI/AAAAAAAC5C8/XKASkAMm6Ak5nSfE8N8vA0xoiFx82HndwCLcB/s320/P3250104.JPG" width="320" height="213" /></a></div>
FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-42918228064053743422017-03-02T09:09:00.000+13:002017-06-15T19:40:21.934+12:00Salesforce SOAP Callout debugging trickery <p>Here's a handy practice when making SOAP callouts from Salesforce and handling errors.</p>
<p>When a Callout goes pear-shaped and you get an exception, keep track of the request parameters by doing a JSON serialize and keeping the result in a custom object.</p>
<p>Then in the dev packaging org you can rehydrate the same request by deserializing the JSON from the custom object and making the same callout. Because you are now in a dev org you can see the raw SOAP message in the CALLOUT_REQUEST logging.</p>
<a href="https://1.bp.blogspot.com/-x-kjG7gsA-Q/WLcqGEIleRI/AAAAAAAC4NQ/NKP6EoqOt3k5P18yi7Dw7XD-Ttdx2YPowCLcB/s1600/ErrorQuery.PNG" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-x-kjG7gsA-Q/WLcqGEIleRI/AAAAAAAC4NQ/NKP6EoqOt3k5P18yi7Dw7XD-Ttdx2YPowCLcB/s800/ErrorQuery.PNG" width="760" /></a>
<pre class='brush: java'>
string jsonData = [Select ReferenceData__c from Error_Details__c where ID = 'a084000000w6ReO'].ReferenceData__c;
SoapWebService.Order order = (SoapWebService.Order)JSON.deserialize(jsonData, SoapWebService.Order.class);
SoapWebService.ServiceCredential credential = new SoapWebService.ServiceCredential();
SoapWebService.BasicHttpBinding_IConnectNS service = new SoapWebService.BasicHttpBinding_IConnectNS();
service.UpdateOrder(credential, order);
</pre>
<a href="https://4.bp.blogspot.com/-iM16Y-cIbeI/WLcoo-QBtPI/AAAAAAAC4NE/3O2O93fbcPYciDha8DiQArw5H2Gtz-zsQCLcB/s1600/CalloutRequest.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-iM16Y-cIbeI/WLcoo-QBtPI/AAAAAAAC4NE/3O2O93fbcPYciDha8DiQArw5H2Gtz-zsQCLcB/s1600/CalloutRequest.png" width="760" /></a>
<p>From there you can take the raw SOAP request over to something like SOAP UI to debug it further.</p>
<p>See also:</p>
<ul>
<li>Ideas: <a href="https://success.salesforce.com/ideaView?id=08730000000BqG9AAK">SOAPFault Information for Apex</a></li>
</ul>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-24828589826799751002017-02-10T22:07:00.000+13:002017-02-10T22:07:32.537+13:00Visualforce Quick Start with the Salesforce Predictive Vision Services<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-U2p-KcCSkhY/WJ2Bq17cxRI/AAAAAAAC25Y/ivz8hj3oDcM2_puhoRW2gHqwM1CowIggQCLcB/s1600/ItsAFrog.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-U2p-KcCSkhY/WJ2Bq17cxRI/AAAAAAAC25Y/ivz8hj3oDcM2_puhoRW2gHqwM1CowIggQCLcB/s320/ItsAFrog.png" width="318" height="320" /></a></div>
<p>Salesforce recently released the <a href="https://metamind.readme.io/v1/docs">Salesforce Predictive Vision Services Pilot</a>. You can watch the corresponding <a href="https://developer.salesforce.com/events/webinars/predictivevisionservice">webinar</a>.</p>
<p>I went through the <a href="https://metamind.readme.io/docs/apex_qs_scenario">Apex Quick Start</a> steps and thought I could simplify them a bit. At the end of the process you should have the basic Apex and a visualforce page to test image predictions against.</p>
<h2 style="clear:both">Steps</h2>
<ol>
<li>Sign up for a predictive services account using a Developer Org. The instructions here are fairly straight forward.<br/> Go to <a href="https://metamind.io/">https://metamind.io/</a> and use the <i>Free Sign Up</i> link. OAuth to your dev org. Download the resulting <b>predictive_services.pem</b> file that contains your private key and make note of the "you've signed up with" email address. You will need the file later and the email address if your org users email address differs.<br/>
<a href="https://3.bp.blogspot.com/-DONafdClvZ0/WJ1v44IMlhI/AAAAAAAC24w/Az5922nWI_4PKoag80NANdQAAiAQ576LgCLcB/s1600/Key.png" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-DONafdClvZ0/WJ1v44IMlhI/AAAAAAAC24w/Az5922nWI_4PKoag80NANdQAAiAQ576LgCLcB/s320/Key.png" width="320" height="252" /></a>
<ul>
<li>Note: the signup is associated with the Users Email address, not the username. So you might get conflicts between multiple dev orgs sharing the same email address.</li>
</ul>
</li>
<li>Upload your predictive_services.pem private key to the same developer org into Files and title it 'predictive_services'. This title is used to get the details of the private key by the Apex Code.<br/>
<a href="https://3.bp.blogspot.com/-0XUJFfCzUC8/WJ1wovYKnLI/AAAAAAAC244/i3MDG8qZqn0f9rjhWbUHEoiUGkm8Bw5cgCLcB/s1600/Files.PNG" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-0XUJFfCzUC8/WJ1wovYKnLI/AAAAAAAC244/i3MDG8qZqn0f9rjhWbUHEoiUGkm8Bw5cgCLcB/s320/Files.PNG" width="320" height="219" /></a>
<ul>
<li>The Spring '17 release has the <a href="https://releasenotes.docs.salesforce.com/en-us/spring17/release-notes/rn_forcecom_development_custom_metadata_long_text_areas.htm?edition=&impact=">Long Text Areas in Custom Metadata Types (Pilot)</a> which would allow a custom metadata setting to store the 2KB private key. Until then Content seems to be the easiest place to keep it.</li>
</ul>
</li>
<li>Install the <a href="https://login.salesforce.com/packaging/installPackage.apexp?p0=04to00000004ROz">unmanaged package</a> that I created (Requires Spring '17).<br/>
I've pulled the required parts together from <a href="https://github.com/salesforceidentity/jwt">https://github.com/salesforceidentity/jwt</a> and <a href="https://github.com/MetaMind/apex-utils">https://github.com/MetaMind/apex-utils</a>. I've also made some modifications to the Visualforce page and corresponding controller to give more flexibility defining the image URL.</li>
<li>Browse to the PredictService Visualforce Tab.</li>
<li>Press the [Vision From Url] button.</li>
<li>Examine the predictions against the <a href="https://metamind.readme.io/v1/page/general-image-model-class-list">General Image Model Class List</a>.
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-u6fqliDpk0Q/WJ12sXrpCrI/AAAAAAAC25E/HNw4_elTWKUhVpv1UboIOqcm-htlFT9aQCLcB/s1600/Example.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-u6fqliDpk0Q/WJ12sXrpCrI/AAAAAAAC25E/HNw4_elTWKUhVpv1UboIOqcm-htlFT9aQCLcB/s320/Example.PNG" width="307" height="320" /></a></div></li>
<li>Change the <i>Image URL</i> to something publicly accessible and repeat the previous couple of steps as required.</li>
</ol>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-60826878670512038652017-02-02T22:03:00.000+13:002017-02-03T07:28:33.846+13:00FuseIT SFDC Explorer 3.5.17023.3 - The more logging edition<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-dIqYjCH_5iQ/WJKQRN35nbI/AAAAAAAC2ec/BVFCBwIw31sgMuI-rOGxsO-X2TYZCcPTACLcB/s1600/FuseITSFDCTree.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://4.bp.blogspot.com/-dIqYjCH_5iQ/WJKQRN35nbI/AAAAAAAC2ec/BVFCBwIw31sgMuI-rOGxsO-X2TYZCcPTACLcB/s640/FuseITSFDCTree.PNG" width="320" height="229" /></a></div>
<p>The latest v3.5 release of the <a href="http://www.fuseit.com/explorer">FuseIT SFDC Explorer</a> is out and contains a couple of new features around Apex Debug logs.</p>
<h2 style="clear: both">The Challenge and Premise</h2>
<p>Understanding an Apex log can require understanding events occurring at vastly different time scales.</p>
<p>At the very fine end, each event timestamp is supplemented with the elapsed time in nanoseconds since the start of the request. At the other end is the duration of the entire log itself, which can span seconds or even minutes of execution time.</p>
<p>By my figuring that is <a href="https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)#Low_order_of_magnitude_-_measures_by_the_unit_second_.28s.29">3 orders of magnitude</a> difference.</p>
<p>To try and put that into perspective...</p>
<table>
<tr>
<tr><th>Duration</th><th>Example (using something very fast)</th></tr>
<tr><td>1 nanosecond</td><td>Light travels 30 centimeters (12 inches)</td></tr>
<!--tr><td>20–40 nanoseconds</td><td>Time of fusion reaction in a hydrogen bomb</td></tr-->
<tr><td>1 minute</td><td>Light travels 17,990,000 kilometers (11,180,000 miles)</td></tr>
</tr>
</table>
<!--p>1 second = 1,000,000,000 nanoseconds</p-->
<p>So while in a nanosecond reflected light could travel from your hand to your eyes. In a minute it could travel between the earth and moon 46 times. Or around the circumference of the earth almost 450 times. Yes, I'm playing a bit fast and loose using the speed of light in a vacuum, but you get the general gist of how vastly different a nanosecond duration is to seconds or minutes. It takes one billion nanoseconds to make a second, and that is a very big number when you are dealing with log events.</p>
<p>That's enough of a detour trying to make the point that we are dealing with periods of time at vastly different scales. I'll now take a similarly cavalier approach to how I'm going to address this challenge.</p>
<blockquote>The human brain processes visual data 60,000 times faster than text</blockquote>
<p>That's an awesome quote for what I'm trying to demonstrate, except it doesn't seem to be backed up by any <a href="https://enveritasgroup.com/campfire/you-cant-trust-the-internet/">actual research</a>. Let's roll with it anyway.</p>
<p>When looking at a log it is useful to see how an events timing relates to those events immediately around it and where it sits in the overall transaction. To that end, I've started plotting the debug log events out in a timeline view under the core log.</p>
<p>"But Daniel" you say, "The Developer Console <a href="https://help.salesforce.com/articleView?id=code_dev_console_view_system_log.htm&type=0&language=en_US&release=206.6">Log Inspector</a> has had the Execution Overview for yonks, why do we need another log viewer?"<br/>
To which I reply, "Are you British? Because yonks sounds like a unit of time you would hear about when watching something from the BBC." and "How on earth did you include a hyperlink in speech? That's some next level DOM injection right there."</p>
<p>My primary reason for making a log parser has always been that the Developer Console is of no use to you if you can't load the log of interest into it. Logs don't just come from directly in the console. They get emailed to you in the "Developer script exception" emails, or from a well meaning admin. They get saved to disk and then examined days after the fact. In cases like these the Developer Console can't help you at all.</p>
<p>While the FuseIT SFDC Explorer will happily load logs captured directly in the org, it can also have them pasted straight in and parse them all the same.</p>
<h2 style="clear:both">Debug log Timeline view</h2>
<p>I've deliberately tried to avoid making a carbon copy of the existing Developer Console functionality. What would be the point? Instead I've looked for a way to visualize all the events in one timeline view. Of course, with some things occurring so closely together the finer details get lost. Where I've found it useful is:</p>
<ul>
<li>in identifying clumps of events,</li>
<li>where an event sits in relation to the rest of the log, and</li>
<li>to jump to events of importance quickly.</li>
</ul>
<p>Let's look at the timeline the came out of a test class run. The log had reached the 2MB limit and covered 13,000 events over 39,500 lines. One of the test methods failed, and we want to hone in on that in the log.</p>
<a href="https://3.bp.blogspot.com/-1qJAPeQzhQ8/WJLhp5Sx__I/AAAAAAAC2e4/5zGC8t3_wEMtZzewkuE1jCSlv2uGVwOWgCLcB/s1600/Timeline.PNG" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-1qJAPeQzhQ8/WJLhp5Sx__I/AAAAAAAC2e4/5zGC8t3_wEMtZzewkuE1jCSlv2uGVwOWgCLcB/s800/Timeline.PNG" width="760px" /></a>
<p>Note the bang icon in the middle of the timeline. Clicking on that takes use straight to the FATAL_ERROR in question.</p>
<a href="https://2.bp.blogspot.com/-8mbc2Y_-UBs/WJLkmZOBoqI/AAAAAAAC2fI/hDAuDAolOtoTyaEtx9lrCqemlkUQWqV2gCLcB/s1600/FatalError.png" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-8mbc2Y_-UBs/WJLkmZOBoqI/AAAAAAAC2fI/hDAuDAolOtoTyaEtx9lrCqemlkUQWqV2gCLcB/s800/FatalError.png" width="760px" /></a>
<h2>Debug log Tree view</h2>
<p>The Developer Console provides both the Stack Tree and Execution Stack for the currently selected event. I've always found these a little odd to be honest in the slight disconnect with the actual log events. E.g. USER_DEBUG becomes "debug".</p>
<p>Let's start with something simple. Execute anonymous for a for loop that does 8 iterations of a debug statement.</p>
<pre>
for(integer i = 0; i < 8; i++) {
System.debug(i);
}
</pre>
<p>The Developer Console shows the 8 debug statements. All with a duration of 0.01 ms with the exception of one that took 0.06 ms. The Execution Stack shows similar details for the currently selected event.</p>
<a href="https://1.bp.blogspot.com/-e3nOtptxUCM/WJKP2DO_NMI/AAAAAAAC2eY/Fz_n0cc-Bnox77zMeLa-mmxzjR28tTNIgCLcB/s1600/DeveloperConsoleTraces.PNG" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-e3nOtptxUCM/WJKP2DO_NMI/AAAAAAAC2eY/Fz_n0cc-Bnox77zMeLa-mmxzjR28tTNIgCLcB/s800/DeveloperConsoleTraces.PNG" width="760" /></a>
<p>What can we see from the same code in the FuseIT SFDC Explorer? That depends on how you filter the log.</p>
<p>If you keep the default settings with opening the log with [Prefilter Log] enabled that various events like SYSTEM_METHOD_ENTRY and SYSTEM_METHOD_EXIT will be completely omitted. This makes the log easier to work with, but mucks with the event durations. With logs you can easily tell when something happened, but to get an accurate duration you need a BEGIN/END or ENTRY/EXIT pair of events. Hence the duration of the first USER_DEBUG seems excessively long as it was measured from the prior event.</p>
<a href="https://3.bp.blogspot.com/-Y5mzwO9eZWA/WJKROI9lvNI/AAAAAAAC2ek/corV-RzpUqI_-yB9d4TGTUVcP1vmxANuACLcB/s1600/FuseITSFDCTreeSmall.png" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-Y5mzwO9eZWA/WJKROI9lvNI/AAAAAAAC2ek/corV-RzpUqI_-yB9d4TGTUVcP1vmxANuACLcB/s1600/FuseITSFDCTreeSmall.png" /></a>
<p>If you keep all the log events in then you get a tree with very similar figures. The main difference being that you and see the ENTRY/EXIT pairs.</p>
<a href="https://4.bp.blogspot.com/-dIqYjCH_5iQ/WJKQRN35nbI/AAAAAAAC2ec/BVFCBwIw31sgMuI-rOGxsO-X2TYZCcPTACLcB/s1600/FuseITSFDCTree.PNG" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-dIqYjCH_5iQ/WJKQRN35nbI/AAAAAAAC2ec/BVFCBwIw31sgMuI-rOGxsO-X2TYZCcPTACLcB/s640/FuseITSFDCTree.PNG" width="760px" /></a>
<h2>Real World example</h2>
<p>Have a look at the <a href="http://salesforce.stackexchange.com/q/158342/102">Apex CPU time limit exceeded in tidy trigger pattern</a> question on the Salesforce StackExchange <b>without skipping down to the answer</b> (NO CHEATING). Grab the <a href="https://www.dropbox.com/s/zqv9h5k3p6qexgc/apex-07Lp000000BAcuBEAT.log?dl=0">apex log they attached</a> and try and figure out what the likely cause of the CPU limit exception is.</p>
<hr/>
<p>Read on when you've figured it out...</p>
<hr/>
<a href="https://i.stack.imgur.com/h4D1L.png" imageanchor="1" ><img border="0" src="https://i.stack.imgur.com/h4D1L.png" width="760" /></a>
<p>Here's what I can tell you from the log timeline.<p>
<p>Notice the recurring pattern of red (before update triggers), green (validation), orange (after update triggers), and purple (workflow). As per the question they are updating 2956 Account records. So the records are processed in batches of 200. You can also see where the skipped log section is (exclamation mark about 3/4 away along) and the FATAL_ERRORs at the end of the log.</p>
<p>If you then look at one of those batches in the tree view I can see that the triggers themselves are relatively quick and the longest duration from any of the code units is for the workflow. Definitely the smoking gun to investigate first.</p>
<a href="https://i.stack.imgur.com/k5qE5.png" imageanchor="1" ><img border="0" src="https://i.stack.imgur.com/k5qE5.png" width="760" /></a>
<p>I like to think that the combination of the timeline and treeview made isolating the problem much easier. Especially considering the Developer Console wasn't available in this case.</p>
<h2>The forward looking statements</h2>
<p>It's still very much a work in progress.</p>
<p>The biggest thing that stands out to me at the moment is the color coding for events. I want similar events to have similar colors. Important events to stand out and less important events to fade away. The CODE_UNIT color categories not to conflict with the event colors. This is a tricky thing to do when you struggle to name more than the standard 16 colors supported with <a href="https://en.wikipedia.org/wiki/List_of_8-bit_computer_hardware_palettes#CGA">the Windows VGA palette</a></p>
<p>The accuracy of the duration measurements is important. In the current 3.5 release the elapsed times were all converted to C# Timespans, which lacked the nanosecond accuracy. In the next release I'll do all the calculations from the raw nanoseconds and convert to Timespans only when needed for display.</p>
<!--h2>Other changes</h2>
<ul>
<li>Improved Tooling API Metadata exploring by caching the describesObject results.</li>
</ul-->FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0tag:blogger.com,1999:blog-7249827273468308081.post-7235545552985399572017-01-20T20:00:00.000+13:002017-12-19T08:01:26.158+13:00Choose Your Own Adventure - Dirty Dozen showdown with the REST API vs SOAP API vs BULK API<style>
.gameover{background-color: red;}
.youwin{background-color: green;}
.ambiguous{background-color: yellow; color: black;}
.gameover span{float: right; padding-right: 10px;}
h2.pathheading{margin-top: 200px !important;}
</style>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-3j3bXmHceKA/WIFdqSKQhbI/AAAAAAAC2Bo/3ZTCna2qEEUhrXAzr05lRfEv2c3g65UsQCLcB/s1600/choose-your-own-adventure.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://2.bp.blogspot.com/-3j3bXmHceKA/WIFdqSKQhbI/AAAAAAAC2Bo/3ZTCna2qEEUhrXAzr05lRfEv2c3g65UsQCLcB/s320/choose-your-own-adventure.jpg" width="187" height="320" /></a></div>
<p>You're an external system to Salesforce. Stuff happened and now there are a dozen dirty records that need to be updated in Salesforce to reflect the changes. An active Salesforce Session ID (a.k.a access token) that can be used to make API calls with is available. All the records have the corresponding Salesforce Ids, so a direct update can be performed. <i>Ignore for the moment that the records might also be deleted and in the recycle bin or fully deleted (really truly gone).</i></p>
<p>To further complicate matters, there is a quagmire of triggers, workflow, and validation on the objects in Salesforce. This is a subscriber org for a managed package, so you can't just fix those.</p>
<p>Which API do you use to update those records in Salesforce?<br/>
<b>Pick a path:</b></p>
<ol id="startover">
<li>You use REST API PATCH requests to update records. <i>Turn to <a href="#RestAPI">page 666</a></i></li>
<li>You use the REST API composite batch resource to update records. <i>Turn to <a href="#RestAPIComposite">page 78</a></i></li>
<li>You use the REST API composite tree resource to update the records. <i>Turn to <a href="#RestAPICompositeTree">page √–1</a></i></li>
<li>You use the SOAP API update() call. <i>Turn to <a href="#SoapAPI">page 42</a></i></li>
<li>You use the Bulk API to update them. <i>Turn to page <a href="#BulkAPI">page 299792458</a></i></li>
<li>You hand craft an Apex REST web service to do the processing. <i>Turn to page <a href="#ApexRestService">0</a></i></li>
<li><b>UPDATE</b> Use the sObject Collections REST API resource. <i>Turn to <a href="https://releasenotes.docs.salesforce.com/en-us/spring18/release-notes/rn_api_rest.htm#rn_api_rest_collections">The Spring `18 release notes.</a></i></li>
</ol>
<hr/>
<h2 id="RestAPI" class="pathheading">REST API PATCH requests</h2>
<p>There are 12 records and the API will only allow you to <a href="https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/dome_update_fields.htm">PATCH one at a time</a>. So that's 12 API calls.</p>
<p class="gameover"><b>You die a slow and painful death. <span>GAME OVER</span></b></p>
<p><a href="#startover">Try Again?</a></p>
<h3>Postmortem:</h3>
<p>Each request round trips to Salesforce, processes all the triggers,workflow, validation on each individual record, and returns the result. Individually each request is only a couple of seconds, but collectively they take way too long for the waiting user.</p>
<p>Request</p>
<pre>
POST /services/data/v38.0/sobjects/OpportunityLineItem/00k7000000eaaZBAAY HTTP/1.1
Host: na5.salesforce.com
Authorization: Bearer 00D700000000001!AQ0AQOzUlrjD_NotARealSession_x61fsbSS6GGWJ123456789mKjmhS0myiYYK_sW_zba
Content-Type: application/json
</pre>
<p>Request Body</p>
<pre class='brush: javascript'>
{"End_Date__c": "2017-01-19"}
</pre>
<p>204 Response: Time (2,018 to 2,758 ms) multiplied by twelve records gives 24,216 to 33,096 ms</p>
<a href="https://2.bp.blogspot.com/-cLyDRveZTzQ/WIE4glYm5jI/AAAAAAAC2BU/AoLcj0TZXegpp_GoWZhU4yA3F9VAt1GIACLcB/s1600/SingleUpdate.PNG" imageanchor="1" ><img border="0" style="max-width: 734px;" src="https://2.bp.blogspot.com/-cLyDRveZTzQ/WIE4glYm5jI/AAAAAAAC2BU/AoLcj0TZXegpp_GoWZhU4yA3F9VAt1GIACLcB/s1600/SingleUpdate.PNG" /></a>
<a href="https://1.bp.blogspot.com/-88aigBSYrNw/WIE4nwonLQI/AAAAAAAC2BY/a8WTi056-Qw8OSM9Yw6-K50qj2Ntq7UCgCLcB/s1600/SingleUpdate1.PNG" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-88aigBSYrNw/WIE4nwonLQI/AAAAAAAC2BY/a8WTi056-Qw8OSM9Yw6-K50qj2Ntq7UCgCLcB/s1600/SingleUpdate1.PNG" /></a>
<h2 id="RestAPIComposite" class="pathheading">REST API Composite batch</h2>
<p>You learnt your lesson with the individual REST API calls (or maybe you came straight here), so switch to a single composite batch call. This will give you one round trip to the server.</p>
<p class="gameover"><b>You die a (slightly less, but still very much) slow and painful death. <span>GAME OVER</span></b></p>
<p><a href="#startover">Try Again?</a></p>
<h3>Postmortem:</h3>
<p>You're down to one API request, which is good. But less than desirable things are happening in Salesforce. Each sub request in the batch is splitting into a separate transaction.</p>
<p>There is still a big penalty to pay for running the accumulation of triggers and other gunk one record at a time. The trigger bulkification can't help you is they are all separate transactions.</p>
<p>Also, don't forget that you can only do 25 records per batch. Not such a problem with 12 records, but it has limited scaling potential.</p>
<p>Request</p>
<pre>
POST /services/data/v38.0/composite/batch HTTP/1.1
Host: na5.salesforce.com
Authorization: Bearer 00D700000000001!AQ0AQOzUlrjD_StillNotARealSession_x61fsbSS6GGWJ123456789mKjmhS0myiYYK
Content-Type: application/json
</pre>
<p id="RestAPICompositeRequestBody">Request Body</p>
<pre class='brush: javascript'>
{
"batchRequests": [{
"method": "PATCH",
"url": "v38.0/sobjects/OpportunityLineItem/00k7000000eaaZBAAY",
"richInput": {
"End_Date__c": "2017-01-19"
}
}, {
"method": "PATCH",
"url": "v38.0/sobjects/OpportunityLineItem/00k7000000eaaZCAAY",
"richInput": {
"End_Date__c": "2017-01-19"
}
}, {
"method": "PATCH",
"url": "v38.0/sobjects/OpportunityLineItem/00k7000000eaaZDAAY",
"richInput": {
"End_Date__c": "2017-01-19"
}
}, {
"method": "PATCH",
"url": "v38.0/sobjects/OpportunityLineItem/00k7000000eaaZEAAY",
"richInput": {
"End_Date__c": "2017-01-19"
}
}, {
"method": "PATCH",
"url": "v38.0/sobjects/OpportunityLineItem/00k7000000eaaZFAAY",
"richInput": {
"End_Date__c": "2017-01-19"
}
},
//...
]
}
</pre>
<p>Response: Time (20,053 ms)</p>
<pre class='brush: javascript'>
{
"hasErrors": false,
"results": [
{
"statusCode": 204,
"result": null
},
{
"statusCode": 204,
"result": null
},
{
"statusCode": 204,
"result": null
},
{
"statusCode": 204,
"result": null
},
{
"statusCode": 204,
"result": null
},
//...
]
}
</pre>
<a href="https://4.bp.blogspot.com/-ACvBbXQ2bM8/WIE1ocG9g6I/AAAAAAAC2BA/w9NZlMDPaV8N1XGVHF9Z2DZ9WaO3zqc2ACLcB/s1600/CompositeBatch.png" imageanchor="1" ><img border="0" style="max-width: 758px;" src="https://4.bp.blogspot.com/-ACvBbXQ2bM8/WIE1ocG9g6I/AAAAAAAC2BA/w9NZlMDPaV8N1XGVHF9Z2DZ9WaO3zqc2ACLcB/s1600/CompositeBatch.png" /></a>
<a href="https://2.bp.blogspot.com/-g7U7ekdcrX4/WIE1_oQT45I/AAAAAAAC2BE/GQpPFCkyURwbuGJjJGo1wX2NeUnnCpYYACLcB/s1600/CompositeBatch2.PNG" imageanchor="1" ><img border="0" style="max-width: 758px;" src="https://2.bp.blogspot.com/-g7U7ekdcrX4/WIE1_oQT45I/AAAAAAAC2BE/GQpPFCkyURwbuGJjJGo1wX2NeUnnCpYYACLcB/s1600/CompositeBatch2.PNG" /></a>
<!--a href="https://1.bp.blogspot.com/-3XF6N1W3_QM/WH6f7WgdV-I/AAAAAAAC16c/rhrxrT6DhlEe4OUVfSFdAup3exo4LZnogCLcB/s1600/Composite.PNG" imageanchor="1" ><img border="0" style="max-width: 758px;" src="https://1.bp.blogspot.com/-3XF6N1W3_QM/WH6f7WgdV-I/AAAAAAAC16c/rhrxrT6DhlEe4OUVfSFdAup3exo4LZnogCLcB/s320/Composite.PNG" width="320" height="130" /></a-->
<h3>Bonus</h3>
<p>Look at the log duration for each sub request. They appear to be the accumulation of time for the entire API request rather than each individual sub transaction. It certainly confused me for a bit. </p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/FishOfPrey">@FishOfPrey</a> Looking at the StartTime&#39;s versus DurationSeconds it looks more like some type of logging error. Duration since composite start.</p>&mdash; Daniel Ballinger (@FishOfPrey) <a href="https://twitter.com/FishOfPrey/status/820813147908894720">January 16, 2017</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="RestAPICompositeTree" class="pathheading">REST API Composite tree</h2>
<p>Currently (as at Spring '17) it can work with up to 200 records, which is a good start. However, the composite tree resource is only for creating records, not updating them. </p>
<p class="gameover"><b>You die of embarrassment from trying to use an incompatible API. <span>GAME OVER</span></b></p>
<p><a href="#startover">Try Again?</a></p>
<h3>Postmortem:</h3>
<p>Always check the <a href="https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/resources_composite_sobject_tree.htm">documentation</a> first.</p>
<h2 id="SoapAPI" class="pathheading">SOAP API update call</h2>
<p>SOAP, are you sure? That API's been <a href="https://developer.salesforce.com/page/Earlier_Reference_Documentation#SOAP_API_Developer.27s_Guide">rattling around</a> since 2004 in <a href="http://www.salesforce.com/us/developer/docs/sforce50/wwhelp/wwhimpl/js/html/wwhelp.htm">API v5.0</a>.</p>
<p class="youwin"><b>Success, the records are all updated in a <i title="relative to the hot mess of triggers etc... in this org.">reasonable</i> timeframe.</b></p>
<p><a href="#startover">Try something else?</a></p>
<h3>Review:</h3>
<p>One POST request, and 4262 ms later you have a response. Processing time does increase with each record added, but nowhere near the overhead of the previous REST API's.</p>
<p>POST Request to https://na5.salesforce.com/services/Soap/u/38.0</p>
<pre class="brush: xml">
&lt;soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:partner.soap.sforce.com" xmlns:urn1="urn:sobject.partner.soap.sforce.com"&gt;
&lt;soapenv:Header&gt;
&lt;urn:SessionHeader&gt;
&lt;urn:sessionId&gt;00D700000000001!AQ0AQOzUlrjD_SessionIdCleanedWithSoap_x61fsbSS6GGWJ123456789mKjmhS0myiYYK_sW_zba&lt;/urn:sessionId&gt;
&lt;/urn:SessionHeader&gt;
&lt;/soapenv:Header&gt;
&lt;soapenv:Body&gt;
&lt;urn:update&gt;
&lt;urn:sObjects&gt;
&lt;urn1:type&gt;OpportunityLineItem&lt;/urn1:type&gt;
&lt;urn1:fieldsToNull&gt;&lt;/urn1:fieldsToNull&gt;
&lt;urn1:Id&gt;00k7000000eaaZBAAY&lt;/urn1:Id&gt;
&lt;urn1:End_Date__c&gt;2017-01-19&lt;/urn1:End_Date__c&gt;
&lt;/urn:sObjects&gt;
&lt;urn:sObjects&gt;
&lt;urn1:type&gt;OpportunityLineItem&lt;/urn1:type&gt;
&lt;urn1:fieldsToNull&gt;&lt;/urn1:fieldsToNull&gt;
&lt;urn1:Id&gt;00k7000000eaaZCAAY&lt;/urn1:Id&gt;
&lt;urn1:End_Date__c&gt;2017-01-19&lt;/urn1:End_Date__c&gt;
&lt;/urn:sObjects&gt;
&lt;urn:sObjects&gt;
&lt;urn1:type&gt;OpportunityLineItem&lt;/urn1:type&gt;
&lt;urn1:fieldsToNull&gt;&lt;/urn1:fieldsToNull&gt;
&lt;urn1:Id&gt;00k7000000eaaZDAAY&lt;/urn1:Id&gt;
&lt;urn1:End_Date__c&gt;2017-01-19&lt;/urn1:End_Date__c&gt;
&lt;/urn:sObjects&gt;
&lt;urn:sObjects&gt;
&lt;urn1:type&gt;OpportunityLineItem&lt;/urn1:type&gt;
&lt;urn1:fieldsToNull&gt;&lt;/urn1:fieldsToNull&gt;
&lt;urn1:Id&gt;00k7000000eaaZEAAY&lt;/urn1:Id&gt;
&lt;urn1:End_Date__c&gt;2017-01-19&lt;/urn1:End_Date__c&gt;
&lt;/urn:sObjects&gt;
&lt!-- ... -- &gt;
&lt;/urn:update&gt;
&lt;/soapenv:Body&gt;
&lt;/soapenv:Envelope&gt;
</pre>
<p>Response</p>
<pre class="brush: xml">
&lt;soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns="urn:partner.soap.sforce.com"&gt;
&lt;soapenv:Header&gt;
&lt;LimitInfoHeader&gt;
&lt;limitInfo&gt;
&lt;current&gt;465849&lt;/current&gt;
&lt;limit&gt;6700000&lt;/limit&gt;
&lt;type&gt;API REQUESTS&lt;/type&gt;
&lt;/limitInfo&gt;
&lt;/LimitInfoHeader&gt;
&lt;/soapenv:Header&gt;
&lt;soapenv:Body&gt;
&lt;updateResponse&gt;
&lt;result&gt;
&lt;id&gt;00k7000000eaaZBAAY&lt;/id&gt;
&lt;success&gt;true&lt;/success&gt;
&lt;/result&gt;
&lt;result&gt;
&lt;id&gt;00k7000000eaaZCAAY&lt;/id&gt;
&lt;success&gt;true&lt;/success&gt;
&lt;/result&gt;
&lt;result&gt;
&lt;id&gt;00k7000000eaaZDAAY&lt;/id&gt;
&lt;success&gt;true&lt;/success&gt;
&lt;/result&gt;
&lt;!-- ... --&gt;
&lt;/updateResponse&gt;
&lt;/soapenv:Body&gt;
&lt;/soapenv:Envelope&gt;
</pre>
<h2 id="BulkAPI" class="pathheading">Bulk API</h2>
<p>It's <a href="https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/asynch_api_intro.htm">primarily billed</a> as a way to asynchronously load <b>large sets</b> of data into Salesforce. Let's see how we go with only 12...</p>
<p class="youwin"><b>You have a harrowing brush with death by API ceremony. If the asynchronous gods favor you it is a timely update. Otherwise disgruntled users tear you limb from limb as they get fed up of waiting for the results to come back.</b></p>
<p><a href="#startover">Try something else?</a></p>
<h3>Results:</h3>
<p>There are five API calls to be made to complete this operation on a good day. If things go bad then you might be waiting longer than expected. You need to keep polling the API for the job to complete before you can get the results back. You're also burning five API calls where you could be using one to complete the entire operation.</p>
<h4>Create Job</h4>
<p>Request</p>
<pre>
POST /services/async/38.0/job HTTP/1.1
Host: na5.salesforce.com
X-SFDC-Session: Bearer 00D700000000001!AQ0AQOzUlrjD_NothingToSeeHere_x61fsbSS6GGWJ123456789mKjmhS0my
Content-Type: application/xml
</pre>
<p>Request Body</p>
<pre class='brush: xml'>
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload"&gt;
&lt;operation&gt;update&lt;/operation&gt;
&lt;object&gt;OpportunityLineItem&lt;/object&gt;
&lt;contentType&gt;CSV&lt;/contentType&gt;
&lt;/jobInfo&gt;
</pre>
<p>Response Time (617 ms)</p>
<pre class='brush: xml'>
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;jobInfo
xmlns="http://www.force.com/2009/06/asyncapi/dataload"&gt;
&lt;id&gt;75070000003qVrHAAU&lt;/id&gt;
&lt;operation&gt;update&lt;/operation&gt;
&lt;object&gt;OpportunityLineItem&lt;/object&gt;
&lt;createdById&gt;00570000004uCVJAA2&lt;/createdById&gt;
&lt;createdDate&gt;2017-01-19T23:08:06.000Z&lt;/createdDate&gt;
&lt;systemModstamp&gt;2017-01-19T23:08:06.000Z&lt;/systemModstamp&gt;
&lt;state&gt;Open&lt;/state&gt;
&lt;concurrencyMode&gt;Parallel&lt;/concurrencyMode&gt;
&lt;contentType&gt;CSV&lt;/contentType&gt;
&lt;numberBatchesQueued&gt;0&lt;/numberBatchesQueued&gt;
&lt;numberBatchesInProgress&gt;0&lt;/numberBatchesInProgress&gt;
&lt;numberBatchesCompleted&gt;0&lt;/numberBatchesCompleted&gt;
&lt;numberBatchesFailed&gt;0&lt;/numberBatchesFailed&gt;
&lt;numberBatchesTotal&gt;0&lt;/numberBatchesTotal&gt;
&lt;numberRecordsProcessed&gt;0&lt;/numberRecordsProcessed&gt;
&lt;numberRetries&gt;0&lt;/numberRetries&gt;
&lt;apiVersion&gt;38.0&lt;/apiVersion&gt;
&lt;numberRecordsFailed&gt;0&lt;/numberRecordsFailed&gt;
&lt;totalProcessingTime&gt;0&lt;/totalProcessingTime&gt;
&lt;apiActiveProcessingTime&gt;0&lt;/apiActiveProcessingTime&gt;
&lt;apexProcessingTime&gt;0&lt;/apexProcessingTime&gt;
&lt;/jobInfo&gt;
</pre>
<h4>Add a Batch to the Job</h4>
<p>Request</p>
<pre>
POST /services/async/38.0/job/75070000003qVrHAAU/batch HTTP/1.1
Host: na5.salesforce.com
X-SFDC-Session: Bearer 00D700000000001!AQ0AQOzUlrjD_HereIsSomeWorkToDo_x61fsbSS6GGWJ123456mKjmhS0myiYYK_sW_zba
Content-Type: text/csv
</pre>
<p>Request Body</p>
<pre>
Id,End_Date__c
"00k7000000eaaZBAAY","2017-01-19"
"00k7000000eaaZCAAY","2017-01-19"
"00k7000000eaaZDAAY","2017-01-19"
"00k7000000eaaZEAAY","2017-01-19"
"00k7000000eaaZFAAY","2017-01-19"
"00k7000000eaaYDAAY","2017-01-19"
"00k7000000eaaZQAAY","2017-01-19"
"00k7000000eaaZpAAI","2017-01-19"
"00k7000000eaaa4AAA","2017-01-19"
"00k7000000eaaZkAAI","2017-01-19"
"00k7000000eaaZlAAI","2017-01-19"
"00k7000000eaaXKAAY","2017-01-19"
</pre>
<p>Response time: 964 ms</p>
<pre class='brush: xml'>
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;batchInfo
xmlns="http://www.force.com/2009/06/asyncapi/dataload"&gt;
&lt;id&gt;75170000005cAFMAA2&lt;/id&gt;
&lt;jobId&gt;75070000003qVrHAAU&lt;/jobId&gt;
&lt;state&gt;Queued&lt;/state&gt;
&lt;createdDate&gt;2017-01-19T23:15:21.000Z&lt;/createdDate&gt;
&lt;systemModstamp&gt;2017-01-19T23:15:21.000Z&lt;/systemModstamp&gt;
&lt;numberRecordsProcessed&gt;0&lt;/numberRecordsProcessed&gt;
&lt;numberRecordsFailed&gt;0&lt;/numberRecordsFailed&gt;
&lt;totalProcessingTime&gt;0&lt;/totalProcessingTime&gt;
&lt;apiActiveProcessingTime&gt;0&lt;/apiActiveProcessingTime&gt;
&lt;apexProcessingTime&gt;0&lt;/apexProcessingTime&gt;
&lt;/batchInfo&gt;
</pre>
<h4>Close the Job</h4>
<p>Request</p>
<pre>
POST /services/async/38.0/job/75070000003qVrHAAU HTTP/1.1
Host: na5.salesforce.com
X-SFDC-Session: Bearer 00D700000000001!AQ0AQOzUlrjD_AnotherApiCall_ReallyQ_x61fsbSS6GGWJ56789mKjmhS0myiYYK_sW_zba
Content-Type: application/xml; charset-UTF-8
</pre>
<p>Request Body</p>
<pre class='brush: xml'>
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload"&gt;
&lt;state&gt;Closed&lt;/state&gt;
&lt;/jobInfo&gt;
</pre>
<p>Response time: 1291 ms</p>
<pre class='brush: xml'>
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;jobInfo
xmlns="http://www.force.com/2009/06/asyncapi/dataload"&gt;
&lt;id&gt;75070000003qVrHAAU&lt;/id&gt;
&lt;operation&gt;update&lt;/operation&gt;
&lt;object&gt;OpportunityLineItem&lt;/object&gt;
&lt;createdById&gt;00570000004uCVJAA2&lt;/createdById&gt;
&lt;createdDate&gt;2017-01-19T23:08:06.000Z&lt;/createdDate&gt;
&lt;systemModstamp&gt;2017-01-19T23:08:06.000Z&lt;/systemModstamp&gt;
&lt;state&gt;Closed&lt;/state&gt;
&lt;concurrencyMode&gt;Parallel&lt;/concurrencyMode&gt;
&lt;contentType&gt;CSV&lt;/contentType&gt;
&lt;numberBatchesQueued&gt;0&lt;/numberBatchesQueued&gt;
&lt;numberBatchesInProgress&gt;0&lt;/numberBatchesInProgress&gt;
&lt;numberBatchesCompleted&gt;0&lt;/numberBatchesCompleted&gt;
&lt;numberBatchesFailed&gt;1&lt;/numberBatchesFailed&gt;
&lt;numberBatchesTotal&gt;1&lt;/numberBatchesTotal&gt;
&lt;numberRecordsProcessed&gt;0&lt;/numberRecordsProcessed&gt;
&lt;numberRetries&gt;0&lt;/numberRetries&gt;
&lt;apiVersion&gt;38.0&lt;/apiVersion&gt;
&lt;numberRecordsFailed&gt;0&lt;/numberRecordsFailed&gt;
&lt;totalProcessingTime&gt;0&lt;/totalProcessingTime&gt;
&lt;apiActiveProcessingTime&gt;0&lt;/apiActiveProcessingTime&gt;
&lt;apexProcessingTime&gt;0&lt;/apexProcessingTime&gt;
&lt;/jobInfo&gt;
</pre>
<h4>Check the Batch Status</h4>
<p>Request</p>
<pre>
GET /services/async/38.0/job/75070000003qVrHAAU/batch/75170000005cAFMAA2 HTTP/1.1
Host: na5.salesforce.com
X-SFDC-Session: Bearer 00D700000000001!AQ0AQOzUlrjD_LosingTheWillToLive_x61fsbSS6GGWJ126789mKjmhS0myiYYK_sW_zba
</pre>
<p>Response time: 242 ms</p>
<pre class='brush: xml'>
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;batchInfo
xmlns="http://www.force.com/2009/06/asyncapi/dataload"&gt;
&lt;id&gt;75170000005cAFMAA2&lt;/id&gt;
&lt;jobId&gt;75070000003qVrHAAU&lt;/jobId&gt;
&lt;state&gt;Completed&lt;/state&gt;
&lt;createdDate&gt;2017-01-19T23:27:54.000Z&lt;/createdDate&gt;
&lt;systemModstamp&gt;2017-01-19T23:27:56.000Z&lt;/systemModstamp&gt;
&lt;numberRecordsProcessed&gt;12&lt;/numberRecordsProcessed&gt;
&lt;numberRecordsFailed&gt;1&lt;/numberRecordsFailed&gt;
&lt;totalProcessingTime&gt;1889&lt;/totalProcessingTime&gt;
&lt;apiActiveProcessingTime&gt;1741&lt;/apiActiveProcessingTime&gt;
&lt;apexProcessingTime&gt;1555&lt;/apexProcessingTime&gt;
&lt;/batchInfo&gt;
</pre>
<h4>Retrieve the Batch Results</h4>
<p>Request</p>
<pre>
GET /services/async/38.0/job/75070000003qVrHAAU/batch/75170000005cAFMAA2/result HTTP/1.1
Host: na5.salesforce.com
X-SFDC-Session: Bearer 00D700000000001!AQ0AQOzUlrjD_AreWeThereYet_x61fsbSS6GGWJ123456789mKjmhS0myiYYK_sW_zba
</pre>
<p>Response time: 236 ms</p>
<pre>
"Id","Success","Created","Error"
"00k7000000eaaZBAAY","true","false",""
"00k7000000eaaZCAAY","true","false",""
"00k7000000eaaZDAAY","true","false",""
"00k7000000eaaZEAAY","true","false",""
"00k7000000eaaZFAAY","true","false",""
"00k7000000eaaYDAAY","true","false",""
"00k7000000eaaZQAAY","true","false",""
"00k7000000eaaZpAAI","true","false",""
"00k7000000eaaa4AAA","true","false",""
"00k7000000eaaZkAAI","true","false",""
"00k7000000eaaZlAAI","true","false",""
"00k7000000eaaXKAAY","true","false",""
</pre>
<h3>Review:</h3>
<p>With only a single call to check the batch status it came back at a respectable 3350 ms total for all the API calls. That doesn't include any of the overhead on the client side. There could be some variance here while waiting for they Async job to complete.</p>
<h2 id="ApexRestService" class="pathheading">Apex REST Web Service</h2>
<p>OK, I'll be honest, after all those BULK API calls I'm exhausted. Also, I can't just deploy an Apex Web Service to the production org where I was bench marking against.</p>
<p class="ambiguous"><b>Your fate is ambiguous because the narrator was to lazy to test it.</b> Go to page <a href="#ApexRestService">0</a>.</p>
<p><a href="#startover">Try something else?</a> or <a href="#ApexRestServiceRnd2">Try again?</a></p>
<h3>Review:</h3>
<p>Performance is probably "pretty good"™ with only one API call and one transaction that can use the bulkification in the triggers. However, you'll need to define the interface, maintain the code, create tests and mocks.</p>
<h3>Revised Results</h3>
<p>I had some time to revisit this, create an Apex REST web service in the sandbox, and test it.</p>
<p class="youwin"><b>It takes a bit more effort to create the Apex class with the associated test methods and then deploy them to production. The end result is a timely response.</b></p>
<h3 id="ApexRestServiceRnd2">Revised Review:</h3>
<p>In the ideal world the Apex REST web service would be streamlined to the operation being performed. I sort of cheated a bit and created it to have the same signature as the composite batch API. It also bypasses any sort of error checking or handling.</p>
<pre class='brush: java'>
@RestResource(urlMapping='/compositebatch/*')
global class TestRestResource {
@HttpPatch
global static BatchRequestResult updateOlis() {
RestRequest req = RestContext.request;
BatchRequest input = (BatchRequest)JSON.deserialize(req.requestBody.toString(), BatchRequest.class);
BatchRequestResult result = new BatchRequestResult();
result.hasErrors = false;
result.results = new List&lt;BatchResult&gt;();
List&lt;OpportunityLineItem&gt; olisToUpdate = new List&lt;OpportunityLineItem&gt;();
for(BatchRequests br : input.batchRequests) {
olisToUpdate.add(br.richInput);
Id oliId = br.url.substringAfterLast('/');
br.richInput.Id = oliId;
result.results.add(new BatchResult(204));
}
System.debug('Updating: ' + olisToUpdate.size() + ' records');
// Should be using Database.update so any errors could be split out.
update olisToUpdate;
return result;
}
global class BatchRequest {
public List&lt;BatchRequests&gt; batchRequests;
}
global class BatchRequests {
public String method;
public String url;
public OpportunityLineItem richInput;
}
global class BatchRequestResult {
boolean hasErrors;
List&lt;BatchResult&gt; results;
}
global class BatchResult {
public integer statusCode;
public string result;
public BatchResult(integer status) {
this.statusCode = status;
}
}
}
</pre>
<p>This can then use exactly the same request that the <a href="#RestAPICompositeRequestBody">composite batch</a> did.</p>
<p>Response: Time (3,362 ms) <i>against a sandbox Org</i></p>
<p>To give a relative benchmark in the same sandbox, the SOAP API took 3,172 ms. That gives a time of around 4,500 ms in "production time".</p>
<h2 class="pathheading">Summary</h2>
<p>Lets recap how long it took to update our dozen dirty records:</p>
<ul>
<li>REST API PATCH requests — 24,216 to 33,096 ms</li>
<li>REST API Composite batch — 20,053 ms</li>
<li>REST API Composite tree — n/a for updates</li>
<li>SOAP API update call — 4262 ms</li>
<li>Bulk API — 3350 ms = 617 ms + 964 ms + 1291 ms + n*242 ms + 236 ms</li>
<li>Apex REST Web Service — 4,517 ms (extrapolated from sandbox)</li>
</ul>
<p>I was expecting the SOAP API to fare better against the Bulk API with such a small set of records and one API call versus five. But they came out pretty comparable.</p>
<p>Certainly as the number of records increases the Bulk API should leave the SOAP API in the dust. Especially with the SOAP API needing to start batching ever 200 records.</p>
<p>The other flavors of the REST API are pretty awful when updating multiple records of the same type as they get processed in individual transactions. To be fair, that's not what they are intended for.</p>
<p>Your results will vary significantly as the subscriber org I was testing against had some pretty funky triggers going on. Those triggers were magnifying the impact of sub request transaction splitting by the composite batch processing. I wouldn't usually classify 4 second responses as "timely". It's all relative.</p>
<p>Also, I could have been more rigorous in how the timing measurements were made. E.g. trying multiple times, etc... It's pretty difficult to get consistent times when there are so many variables in a multi-tenanted environment. Repeated calls could easily create ± 500 ms variance between calls.</p>
<p>The idea did occur to me to <a href="https://success.salesforce.com/ideaView?id=0873A000000COZYQA4">Allow REST API composite batch subrequests to be processed in one transaction</a>. That would overcome the gap in the REST API where a small number of related records could be updated in one API call.</p>
<hr/>
<p>See Also:</p>
<ul>
<li>This came out a couple of days after this post from Salesforce Developers - <a href="https://developer.salesforce.com/blogs/tech-pubs/2017/01/simplify-your-api-code-with-new-composite-resources.html">Simplify Your API Code with New Composite Resources</a></li>
<li>Summer '17 <a href="http://docs.releasenotes.salesforce.com/en-us/summer17/release-notes/rn_api_bulk.htm#rn_api_bulk_v2_pilot">Simplified Job Process for Bulk API 2.0 (Pilot)</a> and Collections in <a href="https://youtu.be/wHqp6laTnio?t=18m52s">Next Generation APIs</a></li>
</ul>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com4Nelson, New Zealand-41.27083972239646 173.28357696533203-41.282773222396457 173.26340696533202 -41.258906222396462 173.30374696533204tag:blogger.com,1999:blog-7249827273468308081.post-50439957197064970882017-01-10T21:17:00.000+13:002017-01-23T08:05:54.529+13:00JavaScript Security for Visualforce<!--div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ViKBMdBNzak/WHNRnS86TqI/AAAAAAAC1IM/c-hrTaFsDK4s2v-Rmaond6Rt28y3mUHfwCLcB/s1600/Anyone_notice_the_javascript.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-ViKBMdBNzak/WHNRnS86TqI/AAAAAAAC1IM/c-hrTaFsDK4s2v-Rmaond6Rt28y3mUHfwCLcB/s320/Anyone_notice_the_javascript.png" width="291" height="320" /></a></div-->
<div class="separator" style="clear: both; text-align: center;"><a href="https://xkcd.com/292/" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-DafzxwNP4KA/WHSXeeJsZwI/AAAAAAAC1Jg/51rcydDV6FUkiaNJvY3e5gOdgn1kWePuQCLcB/s1600/xkcdDeploy.png" title="The app exchange security review team take their job _very_ seriously" /></a></div>
<p>I thought I'd touch on some of the security considerations that should be made when working with JavaScript from Visualforce.</p>
<h2>Cross Site Scripting (XSS)</h2>
<p>The risk of cross site scripting is always something to be aware of when developing web applications and needs to be considered when using JavaScript in Visualforce as well. Generally speaking, you want to prevent untrusted user input from being reflected back into JavaScript code.</p>
<p>As an example - say you were trying to read a page parameter info JavaScript with the following (Example only - <b>DON'T DO THIS</b>):</p>
<pre name="code" lang="HTML">
&lt;apex:page&gt;
&lt;script&gt;var foo = '{!$CurrentPage.parameters.userparam}';&lt;/script&gt;
&lt;/apex:page&gt;
</pre>
<p>If you load this in the browser and include a <code>&amp;userParam=bar</code> in the query string then the resulting HTML is:</p>
<a href="https://3.bp.blogspot.com/-0sVrLD0k_Bw/WHQQ-dTxqPI/AAAAAAAC1Is/cl6RtW2uJBcyrJAzmDviA4Llav4FR0n1wCLcB/s1600/FooBar.PNG" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-0sVrLD0k_Bw/WHQQ-dTxqPI/AAAAAAAC1Is/cl6RtW2uJBcyrJAzmDviA4Llav4FR0n1wCLcB/s1600/FooBar.PNG" /></a>
<p>Great! But what if someone puts something malicious into the URL? Something like <code>1';alert('All%20your%20Salesforce%20are%20belong%20to%20us');var%20foo='2</code>. Here is the result in the browser:</p>
<a href="https://2.bp.blogspot.com/-r29N7mEskvo/WHQSwkbzXOI/AAAAAAAC1I0/F0hEZREAuEIuSF8Dz3eXOEbOqZQhLDC9ACLcB/s1600/AllYourSalesforce.PNG" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-r29N7mEskvo/WHQSwkbzXOI/AAAAAAAC1I0/F0hEZREAuEIuSF8Dz3eXOEbOqZQhLDC9ACLcB/s1600/AllYourSalesforce.PNG" /></a>
<p>And again from the page source:</p>
<a href="https://3.bp.blogspot.com/-qtIwkrHOzKs/WHQTEmidzdI/AAAAAAAC1I8/8kmZnOC37HEVPdU-QL6hz1lylpfJEL2yACLcB/s1600/AllYourSalesforceSource.PNG" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-qtIwkrHOzKs/WHQTEmidzdI/AAAAAAAC1I8/8kmZnOC37HEVPdU-QL6hz1lylpfJEL2yACLcB/s1600/AllYourSalesforceSource.PNG" /></a>
<p>So definitely not what we wanted. Notice how the apostrophe characters in the user input allow the code to take on an entirely different meaning. It would be very easy to extend the example to submit the current session cookies to an external resource - compromising you Session Id and pretty much everything else from there.</p>
<p>The solution here is to use <a href="https://developer.salesforce.com/docs/atlas.en-us.pages.meta/pages/pages_variables_functions.htm#d380125e1348">JSENCODE</a> to encode any text before reflection into JavaScript. This will use a backslash to escape any unsafe JavaScript characters, such as apostrophe (').</p>
<pre name="code" lang="HTML">
&lt;apex:page&gt;
&lt;script&gt;var foo = '{!JSENCODE($CurrentPage.parameters.userparam)}';&lt;/script&gt;
&lt;/apex:page&gt;
</pre>
<p>See also:</p>
<ul>
<li><a href="https://developer.salesforce.com/docs/atlas.en-us.pages.meta/pages/pages_security_tips_xss.htm">Cross Site Scripting (XSS)</a></li>
<li>Trailhead <a href="https://trailhead.salesforce.com/en/secdev_injection_vulnerabilities/secdev_inject_cross_site_scripting">Understand Cross-Site Scripting (XSS)</a>
<li>Trailhead <a href="https://trailhead.salesforce.com/en/secdev_injection_vulnerabilities/secdev_inject_preventing_xss_inforce">Prevent XSS in Force.com Applications</a></li>
</ul>
<h2>Javascript Remoting with escape: false</h2>
<p>When using <a href="https://developer.salesforce.com/docs/atlas.en-us.pages.meta/pages/pages_js_remoting.htm">Javascript Remoting</a> beware the using <code>{escape:false}</code> in the <a href="https://developer.salesforce.com/docs/atlas.en-us.pages.meta/pages/pages_js_remoting_configuring_request.htm">configuration </a>and loading the result into the DOM. Much like the basic XSS example above, it can be used to inject executing JavaScript into the page.</p>
<p>See also:</p>
<ul>
<li><a href="http://salesforce.stackexchange.com/a/9434/102">Is there a security risk with JavaScript Remoting, escape = false</a></li>
<li><a href="http://salesforce.stackexchange.com/q/155019/102">What is the major benefit of using Remote Objects</a></li>
</ul>
<h2>Avoid the urge to hack the Salesforce DOM</h2>
<p>Not so long ago developers would put JavaScript in the sidebar so it would load with every page. This could then be used to manipulate the standard Salesforce DOM to do all sorts of things, such as showing/hiding components, adding additional validation, or changing their presentation.</p>
<p>Salesforce took umbrage with this approach as it opened up all sorts of consistency and security problems. As such, they shut the practice down in the Summer '15 release. Needless to say, those who had used the sidebar to manipulate the page found their <strike>workaround</strike> hack solution no longer working. Best to avoid these sorts of unsupported shenanigans if you can as they will be closed off sooner or later.</p>
<p>See also:</p>
<ul>
<li><a href="http://salesforce.stackexchange.com/q/38918/102">End of javascript sidebar workarounds?</a></li>
<li><a href="http://salesforce.stackexchange.com/q/44971/102">Why do we still need to hack the Sidebar? Usecases - Workarounds - Alternatives</a></li>
</ul>
<h2>Protect the Session Id / AccessToken</h2>
<p>The Session ID is the key to the kingdom. If someone can get hold of yours they can interact with Salesforce like you would<a href="#restrict">*</a>. If you can avoid exposing it in Visualforce then do so. E.g. Use JavaScript Remoting rather than interacting with the APIs directly.</p>
<p>Depending on the context where you request it, you can get a "second class session id" that doesn't grant you the same level of access as a full "first class" UI session type.</p>
<p>For instance, a Session ID in Visualforce from <code><apex:outputText value="{!$Api.Session_ID}"/></code> can be used to make API calls, but can't be used to access the full web UI (Such as via frontdoor.jsp).</p>
<p>Checking the User Session Information page can show the different Session Types that get created. In the example below note the TempVisualforceExchange Session that was created from the Parent UI Session.</p>
<a href="https://1.bp.blogspot.com/-XMYdVs2UIxA/WHSR6s0ppvI/AAAAAAAC1JQ/KuUeAqA3_ignbwfYNy1HVZXfFFS3JsC1ACLcB/s1600/UserSessionInformation.png" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-XMYdVs2UIxA/WHSR6s0ppvI/AAAAAAAC1JQ/KuUeAqA3_ignbwfYNy1HVZXfFFS3JsC1ACLcB/s1600/UserSessionInformation.png" /></a>
<p>See also:</p>
<ul>
<li><a href="http://salesforce.stackexchange.com/q/23277/102">Different Session Ids in Different contexts</a></li>
<li><a href="http://salesforce.stackexchange.com/q/42403/102">Get a FIRST-CLASS SessionID for API Calls</a></li>
</ul>
<p><a name="restrict">*</a> There are some exceptions on if they can use it if "Lock sessions to the IP address from which they originated" or "Enforce login IP ranges on every request" are enabled.</p>
<h2>Static Resource rather than CDN</h2>
<p>Using a CDN such as <a href="https://developers.google.com/speed/libraries/">Google Hosted Libraries</a> or the <a href="https://www.asp.net/ajax/cdn">Microsoft Ajax Content Delivery Network</a> to bring in something like jQuery is appealing, but can open you up to <a href="http://security.stackexchange.com/q/16797">security problems</a> and overall make the app exchange security review more troublesome than it needs to be. Instead consider using a zip file in a static resource and referencing the contents of the zip using <a href="https://developer.salesforce.com/docs/atlas.en-us.pages.meta/pages/pages_variables_functions.htm#d380125e2224">URLFOR</a>. </p>
<p>See also:</p>
<ul>
<li>Trailhead - <a href="https://trailhead.salesforce.com/en/visualforce_fundamentals/visualforce_static_resources">Use Static Resources</a></li>
<li><a href="https://developer.salesforce.com/page/Developing_Apps_with_jQuery">Developing Apps with jQuery</a></li>
</ul>
<!--actionFunctions
<h2>Side effects in page GET</h2>
Events between Visualforce Components-->
<hr/>
<p>Further reading:</p>
<ul>
<li>Trailhead - <a href="https://trailhead.salesforce.com/en/project/salesforce_developer_workshop/using_javascript_in_visualforce">Using JavaScript in Visualforce Pages</a></li>
<li><a href="https://developer.salesforce.com/dev-center-security">App Cloud Security Center</a></li>
</ul>FishOfPreyhttp://www.blogger.com/profile/14095711104515084650noreply@blogger.com0