Jul 06 23:54:10.442353 ip-10-0-141-86 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 23:54:10.442498 ip-10-0-141-86 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 23:54:10.442590 ip-10-0-141-86 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 23:54:10.443047 ip-10-0-141-86 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 23:54:20.555541 ip-10-0-141-86 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 23:54:20.555559 ip-10-0-141-86 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot eb80baaf692942f8b25527756f395fde -- Jul 06 23:56:18.672772 ip-10-0-141-86 systemd[1]: Starting Kubernetes Kubelet... Jul 06 23:56:19.050406 ip-10-0-141-86 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:56:19.050406 ip-10-0-141-86 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 23:56:19.050406 ip-10-0-141-86 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:56:19.050406 ip-10-0-141-86 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 23:56:19.050406 ip-10-0-141-86 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:56:19.052035 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.051945 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 23:56:19.056165 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056146 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:19.056165 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056165 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:19.056165 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056168 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056171 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056174 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056178 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056181 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056184 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056187 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056189 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056192 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056194 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056200 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056204 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056207 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056210 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056213 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056215 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056219 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056222 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056224 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:19.056264 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056227 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056229 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056232 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056234 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056237 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056239 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056242 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056244 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056247 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056249 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056253 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056258 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056261 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056264 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056267 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056270 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056273 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056276 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056279 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056281 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:19.056740 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056284 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056286 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056289 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056291 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056293 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056296 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056298 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056301 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056303 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056306 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056308 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056311 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056313 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056315 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056318 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056320 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056323 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056326 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056328 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:19.057219 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056331 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056333 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056335 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056338 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056341 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056343 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056346 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056349 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056352 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056355 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056357 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056360 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056363 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056365 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056368 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056370 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056373 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056375 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056377 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056380 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:19.057693 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056382 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056384 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056387 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056389 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056391 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056394 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056397 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056401 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056404 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056407 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056410 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056412 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056415 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056417 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056419 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056423 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056425 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056428 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.056447 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:19.058193 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057205 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057213 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057216 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057219 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057223 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057226 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057230 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057233 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057236 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057239 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057243 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057247 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057250 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057253 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057256 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057258 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057261 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057264 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057267 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057270 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:19.058686 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057273 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057275 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057278 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057280 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057283 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057285 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057288 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057290 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057293 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057296 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057298 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057300 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057303 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057306 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057309 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057311 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057314 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057316 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057319 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057322 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:19.059184 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057324 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057327 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057329 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057331 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057334 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057336 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057339 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057341 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057343 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057346 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057349 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057351 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057356 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057359 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057362 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057365 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057368 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057371 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057373 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:19.059689 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057376 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057379 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057382 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057384 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057387 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057390 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057392 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057395 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057398 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057401 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057404 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057406 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057409 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057412 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057414 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057417 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057420 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057422 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057425 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:19.060146 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057427 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057446 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057449 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057452 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057454 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057456 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057459 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057462 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057464 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057467 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057470 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057473 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057475 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057478 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057481 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057483 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057486 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057489 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057491 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057494 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:19.060621 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057496 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057885 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057889 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057892 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057894 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057900 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057903 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057907 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057910 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057913 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057915 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057918 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057920 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057922 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057925 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057928 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057930 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057932 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057935 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057938 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:19.061119 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057940 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057942 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057946 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057949 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057951 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057954 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057957 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057959 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057961 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057964 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057966 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057969 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057971 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057973 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057976 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057978 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057980 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057983 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057986 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057988 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:19.061632 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057990 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057993 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057995 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.057997 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058000 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058002 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058005 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058008 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058010 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058012 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058015 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058018 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058022 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058025 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058028 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058030 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058034 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058036 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058039 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:19.062124 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058041 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058043 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058046 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058049 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058051 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058054 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058056 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058059 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058061 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058063 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058065 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058068 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058071 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058073 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058075 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058078 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058080 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058083 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058086 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:19.062600 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058090 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058093 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058095 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058098 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058100 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058103 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058105 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058108 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058111 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058113 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058116 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058118 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058121 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058123 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058126 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058128 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058131 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058133 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058135 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058138 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:19.063121 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058140 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058142 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058221 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058229 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058234 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058238 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058244 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058247 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058251 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058255 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058259 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058261 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058265 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058268 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058272 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058275 2568 flags.go:64] FLAG: --cgroup-root="" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058278 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058281 2568 flags.go:64] FLAG: --client-ca-file="" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058284 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058287 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058292 2568 flags.go:64] FLAG: --cluster-domain="" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058295 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058298 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058301 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 23:56:19.063625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058304 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058308 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058311 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058314 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058317 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058320 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058323 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058326 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058329 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058332 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058336 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058339 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058342 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058344 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058347 2568 flags.go:64] FLAG: --enable-server="true" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058351 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058359 2568 flags.go:64] FLAG: --event-burst="100" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058362 2568 flags.go:64] FLAG: --event-qps="50" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058365 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058368 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058371 2568 flags.go:64] FLAG: --eviction-hard="" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058374 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058377 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058380 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058383 2568 flags.go:64] FLAG: --eviction-soft="" Jul 06 23:56:19.064226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058386 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058389 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058393 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058396 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058399 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058402 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058404 2568 flags.go:64] FLAG: --feature-gates="" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058412 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058416 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058419 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058421 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058424 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058427 2568 flags.go:64] FLAG: --help="false" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058445 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058448 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058451 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058454 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058457 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058461 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058463 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058466 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058469 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058472 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 23:56:19.064837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058480 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058484 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058486 2568 flags.go:64] FLAG: --kube-reserved="" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058489 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058492 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058495 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058498 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058500 2568 flags.go:64] FLAG: --lock-file="" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058503 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058506 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058509 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058516 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058518 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058521 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058524 2568 flags.go:64] FLAG: --logging-format="text" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058526 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058530 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058532 2568 flags.go:64] FLAG: --manifest-url="" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058535 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058539 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058542 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058547 2568 flags.go:64] FLAG: --max-pods="110" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058549 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058552 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058555 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 23:56:19.065469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058558 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058561 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058564 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058567 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058593 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058597 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058600 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058604 2568 flags.go:64] FLAG: --pod-cidr="" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058611 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058617 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058623 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058626 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058629 2568 flags.go:64] FLAG: --port="10250" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058632 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058635 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08971e08ad44be979" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058638 2568 flags.go:64] FLAG: --qos-reserved="" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058641 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058644 2568 flags.go:64] FLAG: --register-node="true" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058648 2568 flags.go:64] FLAG: --register-with-taints="" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058651 2568 flags.go:64] FLAG: --registry-burst="10" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058654 2568 flags.go:64] FLAG: --registry-qps="5" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058656 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058659 2568 flags.go:64] FLAG: --reserved-memory="" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058663 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058665 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 23:56:19.066091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058668 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058671 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058674 2568 flags.go:64] FLAG: --runonce="false" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058677 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058680 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058682 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058686 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058688 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058691 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058694 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058697 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058700 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058703 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058706 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058709 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058713 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058717 2568 flags.go:64] FLAG: --system-cgroups="" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058719 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058725 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058727 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058730 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058734 2568 flags.go:64] FLAG: --tls-min-version="" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058737 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058740 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058742 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 23:56:19.066707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058746 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058749 2568 flags.go:64] FLAG: --v="2" Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058753 2568 flags.go:64] FLAG: --version="false" Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058757 2568 flags.go:64] FLAG: --vmodule="" Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058761 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.058764 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058884 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058888 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058892 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058895 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058898 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058901 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058903 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058905 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058908 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058910 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058913 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058915 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058918 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058920 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058923 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:19.067409 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058925 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058929 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058932 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058935 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058937 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058940 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058943 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058945 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058947 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058950 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058953 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058957 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058959 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058962 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058964 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058967 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058969 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058971 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058974 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058977 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:19.067981 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058979 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058983 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058987 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058990 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058992 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058995 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058997 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.058999 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059002 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059004 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059006 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059009 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059011 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059015 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059017 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059020 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059023 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059025 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059028 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059030 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:19.068482 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059032 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059035 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059037 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059041 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059043 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059046 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059048 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059050 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059053 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059055 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059057 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059061 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059064 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059067 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059070 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059073 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059075 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059078 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059080 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059083 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:19.069320 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059085 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059088 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059090 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059093 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059095 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059099 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059102 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059104 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059107 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059109 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059112 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059114 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059117 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059119 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059121 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059125 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059128 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059132 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059135 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:19.070012 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059138 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059141 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059143 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059146 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.059148 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.059623 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.066954 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.066974 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.066991 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.067000 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067061 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067067 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067071 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067076 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067079 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:19.070529 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067082 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067085 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067087 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067090 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067092 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067095 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067098 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067100 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067103 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067105 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067107 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067110 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067112 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067115 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067118 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067120 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067122 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067125 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067128 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067130 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:19.070919 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067133 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067135 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067138 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067140 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067149 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067152 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067155 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067157 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067160 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067162 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067165 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067167 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067170 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067172 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067174 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067177 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067179 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067182 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067184 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:19.071401 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067187 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067190 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067192 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067195 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067197 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067200 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067202 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067205 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067207 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067210 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067212 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067215 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067217 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067220 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067222 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067225 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067227 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067231 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067242 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067245 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067247 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:19.071880 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067250 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067252 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067255 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067257 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067260 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067264 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067268 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067271 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067273 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067276 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067278 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067281 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067283 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067285 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067288 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067290 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067293 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067295 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067297 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:19.072382 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067300 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067302 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067304 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067307 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067309 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067311 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067314 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067316 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067319 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067322 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067324 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067333 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067337 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067340 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:19.072865 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067343 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.067348 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067512 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067522 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067525 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067528 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067531 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067534 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067536 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067539 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067542 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067545 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067547 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067550 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:19.073199 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067552 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067555 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067557 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067560 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067562 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067564 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067567 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067570 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067572 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067574 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067577 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067579 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067581 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067584 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067587 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067590 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067597 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067600 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067602 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:19.073568 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067605 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067607 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067609 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067612 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067614 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067616 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067619 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067621 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067624 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067626 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067628 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067630 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067638 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067642 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067645 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067647 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067650 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067652 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067654 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067656 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:19.074026 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067659 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067661 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067664 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067666 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067669 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067671 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067674 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067677 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067679 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067681 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067689 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067692 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067694 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067696 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067699 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067701 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067703 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067706 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067708 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067710 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:19.074520 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067713 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067715 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067717 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067720 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067722 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067725 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067727 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067729 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067732 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067734 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067736 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067739 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067741 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067743 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067746 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067749 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067751 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067753 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067756 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067759 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:19.075010 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067762 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067764 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067766 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067775 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067779 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067782 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067784 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:19.067787 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.067792 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.067922 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.072929 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.074233 2568 server.go:1019] "Starting client certificate rotation" Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.074331 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:56:19.075668 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.074369 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:56:19.077029 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.077013 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:56:19.081563 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.081548 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:56:19.090493 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.090473 2568 log.go:25] "Validated CRI v1 runtime API" Jul 06 23:56:19.092403 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.092386 2568 log.go:25] "Validated CRI v1 image API" Jul 06 23:56:19.093925 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.093901 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 23:56:19.098304 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.098283 2568 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 93ecf58d-4569-425a-b9d4-5aeb117dd9dd:/dev/nvme0n1p3 fab0049b-1d68-4753-a4a0-19ab998f553a:/dev/nvme0n1p4] Jul 06 23:56:19.098349 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.098305 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 23:56:19.102701 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.102685 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:56:19.104893 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.104788 2568 manager.go:216] Machine: {Timestamp:2026-07-06 23:56:19.103351484 +0000 UTC m=+0.316383218 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3163465 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20e54b6e192c6e8d51a6069df05834 SystemUUID:ec20e54b-6e19-2c6e-8d51-a6069df05834 BootID:eb80baaf-6929-42f8-b255-27756f395fde Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:7c:d2:8f:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:7c:d2:8f:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:dc:d9:a8:b0:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 23:56:19.104893 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.104888 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 23:56:19.105014 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.104980 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 23:56:19.109345 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.109146 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 23:56:19.109503 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.109348 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-86.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 23:56:19.110041 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.110031 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 23:56:19.110078 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.110044 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 23:56:19.110078 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.110051 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:56:19.110626 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.110615 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:56:19.111807 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.111797 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 23:56:19.114544 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.114530 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:56:19.114678 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.114668 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 23:56:19.114709 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.114703 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 06 23:56:19.114736 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.114719 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 23:56:19.114763 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.114744 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 23:56:19.114763 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.114757 2568 kubelet.go:398] "Adding apiserver pod source" Jul 06 23:56:19.114820 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.114777 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 23:56:19.115243 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.115232 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:56:19.115279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.115252 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:56:19.117223 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.117209 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 23:56:19.118941 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.118925 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 23:56:19.119027 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.118963 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 23:56:19.120072 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120054 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120082 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120090 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120095 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120104 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120110 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120117 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120123 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120130 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120135 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120143 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 23:56:19.120145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.120149 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 23:56:19.121649 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.121637 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 23:56:19.121649 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.121648 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 23:56:19.125003 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.124985 2568 server.go:1268] "Started kubelet" Jul 06 23:56:19.125081 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.125051 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 23:56:19.125151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.125080 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 23:56:19.125234 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.125220 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 23:56:19.125745 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.125718 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zncjk" Jul 06 23:56:19.125947 ip-10-0-141-86 systemd[1]: Started Kubernetes Kubelet. Jul 06 23:56:19.127566 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.126728 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-86.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 23:56:19.127566 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.126757 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 23:56:19.127566 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.126768 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 23:56:19.127566 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.127009 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 23:56:19.127761 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.127750 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 06 23:56:19.131062 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.130161 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-86.ec2.internal.18bfd8a4a5814f2a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-86.ec2.internal,UID:ip-10-0-141-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-86.ec2.internal,},FirstTimestamp:2026-07-06 23:56:19.12495697 +0000 UTC m=+0.337988704,LastTimestamp:2026-07-06 23:56:19.12495697 +0000 UTC m=+0.337988704,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-86.ec2.internal,}" Jul 06 23:56:19.132547 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.132525 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 23:56:19.133017 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.132999 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 23:56:19.133109 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.133018 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 23:56:19.133171 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.133129 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.133694 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.133622 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 23:56:19.133694 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.133642 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 23:56:19.133816 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.133624 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 23:56:19.133816 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.133776 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 06 23:56:19.133816 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.133783 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 06 23:56:19.133816 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.133792 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 23:56:19.133816 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.133806 2568 factory.go:55] Registering systemd factory Jul 06 23:56:19.133816 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.133816 2568 factory.go:223] Registration of the systemd container factory successfully Jul 06 23:56:19.134083 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.134069 2568 factory.go:153] Registering CRI-O factory Jul 06 23:56:19.134083 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.134083 2568 factory.go:223] Registration of the crio container factory successfully Jul 06 23:56:19.134218 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.134101 2568 factory.go:103] Registering Raw factory Jul 06 23:56:19.134218 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.134110 2568 manager.go:1196] Started watching for new ooms in manager Jul 06 23:56:19.134643 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.134629 2568 manager.go:349] Starting recovery of all containers Jul 06 23:56:19.137316 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.137291 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 23:56:19.137561 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.137539 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 23:56:19.137932 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.137908 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zncjk" Jul 06 23:56:19.146891 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.146728 2568 manager.go:354] Recovery completed Jul 06 23:56:19.150957 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.150941 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:19.152962 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.152940 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:19.153065 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.152979 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:19.153065 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.152990 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:19.153569 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.153553 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 23:56:19.153641 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.153570 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 23:56:19.153641 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.153598 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:56:19.153641 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.153638 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:19.154314 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.154299 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:19.154366 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.154326 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:19.154366 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.154338 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:19.154742 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.154732 2568 policy_none.go:49] "None policy: Start" Jul 06 23:56:19.154786 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.154748 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 23:56:19.154786 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.154763 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 23:56:19.155723 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.155712 2568 policy_none.go:47] "Start" Jul 06 23:56:19.202192 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.202174 2568 manager.go:337] "Starting Device Plugin manager" Jul 06 23:56:19.202309 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.202221 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 23:56:19.202309 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.202234 2568 server.go:87] "Starting device plugin registration server" Jul 06 23:56:19.202596 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.202582 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 23:56:19.202645 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.202598 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 23:56:19.202796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.202773 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 23:56:19.203228 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.203050 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 23:56:19.203228 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.203065 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 23:56:19.203557 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.203538 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 23:56:19.203623 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.203596 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.303001 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.302906 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:19.304411 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.303932 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:19.304411 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.303961 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:19.304411 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.303973 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:19.304411 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.304023 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.307621 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.307591 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 23:56:19.308976 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.308958 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 23:56:19.309070 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.309002 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 23:56:19.309070 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.309032 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 23:56:19.309150 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.309073 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 23:56:19.311524 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.311497 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:19.315136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.315118 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.315217 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.315144 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-86.ec2.internal\": node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.334026 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.334005 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.409514 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.409472 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal"] Jul 06 23:56:19.409592 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.409575 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:19.411174 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.411158 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:19.411269 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.411204 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:19.411269 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.411218 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:19.412738 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.412725 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:19.412892 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.412876 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.412930 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.412915 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:19.413622 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.413604 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:19.414303 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.413893 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:19.414303 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.413980 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:19.414303 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.414010 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:19.414303 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.414030 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:19.414303 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.414057 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:19.415396 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.415372 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.415516 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.415418 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:19.416821 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.416798 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:19.416911 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.416835 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:19.416911 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.416851 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:19.434664 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.434635 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.435808 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.435789 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c937bb41e34a0bdc42a4d5667923deb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal\" (UID: \"8c937bb41e34a0bdc42a4d5667923deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.435895 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.435817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c937bb41e34a0bdc42a4d5667923deb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal\" (UID: \"8c937bb41e34a0bdc42a4d5667923deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.435895 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.435837 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4c4d400fcc6b91a31ccccc5f9a95b03-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.ec2.internal\" (UID: \"f4c4d400fcc6b91a31ccccc5f9a95b03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.441283 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.441262 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-86.ec2.internal\" not found" node="ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.445711 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.445692 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-86.ec2.internal\" not found" node="ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.535153 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.535117 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.536251 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.536227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c937bb41e34a0bdc42a4d5667923deb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal\" (UID: \"8c937bb41e34a0bdc42a4d5667923deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.536338 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.536268 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c937bb41e34a0bdc42a4d5667923deb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal\" (UID: \"8c937bb41e34a0bdc42a4d5667923deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.536338 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.536296 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4c4d400fcc6b91a31ccccc5f9a95b03-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.ec2.internal\" (UID: \"f4c4d400fcc6b91a31ccccc5f9a95b03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.536421 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.536344 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c937bb41e34a0bdc42a4d5667923deb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal\" (UID: \"8c937bb41e34a0bdc42a4d5667923deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.536421 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.536349 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4c4d400fcc6b91a31ccccc5f9a95b03-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.ec2.internal\" (UID: \"f4c4d400fcc6b91a31ccccc5f9a95b03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.536421 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.536348 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c937bb41e34a0bdc42a4d5667923deb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal\" (UID: \"8c937bb41e34a0bdc42a4d5667923deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.635895 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.635831 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.736267 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.736235 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.744488 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.744470 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.749156 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.749141 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal" Jul 06 23:56:19.836986 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.836943 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.937525 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:19.937416 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:19.956338 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:19.956313 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:20.037717 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:20.037677 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:20.074859 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.074817 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 23:56:20.075517 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.074971 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:56:20.075517 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.075027 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:56:20.138718 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:20.138683 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:20.139828 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.139791 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 23:51:20 +0000 UTC" deadline="2028-02-19 17:08:15.438822895 +0000 UTC" Jul 06 23:56:20.139920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.139829 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14225h11m55.298997325s" Jul 06 23:56:20.238993 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:20.238961 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-86.ec2.internal\" not found" Jul 06 23:56:20.328153 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.328129 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:20.333584 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.333561 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" Jul 06 23:56:20.338599 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:20.338565 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c4d400fcc6b91a31ccccc5f9a95b03.slice/crio-1b4bfaaed927a04070c7c3ae3cd8bd957042f41c3bf6aa19fd5befb40c9cd8a5 WatchSource:0}: Error finding container 1b4bfaaed927a04070c7c3ae3cd8bd957042f41c3bf6aa19fd5befb40c9cd8a5: Status 404 returned error can't find the container with id 1b4bfaaed927a04070c7c3ae3cd8bd957042f41c3bf6aa19fd5befb40c9cd8a5 Jul 06 23:56:20.339157 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:20.339124 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c937bb41e34a0bdc42a4d5667923deb.slice/crio-1a0303dcbded381bb73ddfb51d3e6f3953fddb490c670a6fc26153f7d532432c WatchSource:0}: Error finding container 1a0303dcbded381bb73ddfb51d3e6f3953fddb490c670a6fc26153f7d532432c: Status 404 returned error can't find the container with id 1a0303dcbded381bb73ddfb51d3e6f3953fddb490c670a6fc26153f7d532432c Jul 06 23:56:20.345999 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.345984 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:56:20.356311 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.356287 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:56:20.358295 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.358278 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal" Jul 06 23:56:20.371036 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.371009 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:56:20.459081 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.459058 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:20.990808 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:20.990770 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:21.117679 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.117648 2568 apiserver.go:52] "Watching apiserver" Jul 06 23:56:21.127154 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.127123 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 23:56:21.129192 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.129156 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-v592j","kube-system/konnectivity-agent-z6ql9","kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt","openshift-cluster-node-tuning-operator/tuned-vnzqk","openshift-dns/node-resolver-f8h8b","openshift-multus/multus-p9zd2","openshift-multus/network-metrics-daemon-gdjjf","openshift-image-registry/node-ca-xl5vt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal","openshift-multus/multus-additional-cni-plugins-ncwhk","openshift-network-diagnostics/network-check-target-tdphw","openshift-network-operator/iptables-alerter-6bvgd","openshift-ovn-kubernetes/ovnkube-node-lvndj"] Jul 06 23:56:21.132176 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.132149 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.134057 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.134032 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 23:56:21.134629 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.134610 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.135189 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.135124 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 23:56:21.135189 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.135135 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 23:56:21.136012 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.135989 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w6v94\"" Jul 06 23:56:21.136290 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.136234 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 23:56:21.137353 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.137326 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 23:56:21.137652 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.137621 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:56:21.138013 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.137994 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 23:56:21.138300 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.138285 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6b7w4\"" Jul 06 23:56:21.138383 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.138328 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 23:56:21.139566 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.139544 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.142285 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.142265 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 23:56:21.142692 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.142666 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.142790 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.142759 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 23:56:21.142977 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.142956 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 23:56:21.143199 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.143176 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wjt67\"" Jul 06 23:56:21.143870 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.143852 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 23:56:21.144047 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144025 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 23:56:21.144227 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-var-lib-cni-multus\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144321 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144242 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55h9r\" (UniqueName: \"kubernetes.io/projected/36d56419-1664-4eff-aaac-26a7440264bd-kube-api-access-55h9r\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144321 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-multus-cni-dir\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144321 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-os-release\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144321 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144315 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-var-lib-cni-bin\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144334 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84de0174-13d9-43cd-93e5-9a0f244c672b-host-slash\") pod \"iptables-alerter-6bvgd\" (UID: \"84de0174-13d9-43cd-93e5-9a0f244c672b\") " pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.143887 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfwnq\" (UniqueName: \"kubernetes.io/projected/84de0174-13d9-43cd-93e5-9a0f244c672b-kube-api-access-kfwnq\") pod \"iptables-alerter-6bvgd\" (UID: \"84de0174-13d9-43cd-93e5-9a0f244c672b\") " pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36d56419-1664-4eff-aaac-26a7440264bd-cni-binary-copy\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144393 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-var-lib-kubelet\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144420 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-hostroot\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144435 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-run-multus-certs\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144457 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-etc-kubernetes\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144475 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-system-cni-dir\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144490 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-cnibin\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144535 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-run-netns\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144550 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-multus-conf-dir\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.144561 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144566 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36d56419-1664-4eff-aaac-26a7440264bd-multus-daemon-config\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.145085 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/84de0174-13d9-43cd-93e5-9a0f244c672b-iptables-alerter-script\") pod \"iptables-alerter-6bvgd\" (UID: \"84de0174-13d9-43cd-93e5-9a0f244c672b\") " pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.145085 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-multus-socket-dir-parent\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.145085 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144623 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-run-k8s-cni-cncf-io\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.145085 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.144908 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 23:56:21.145644 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.145592 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:56:21.145920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.145739 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 23:56:21.145920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.145597 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 23:56:21.145920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.145745 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9h5rp\"" Jul 06 23:56:21.146136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.145990 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.146982 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.146963 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.149345 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.149315 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 23:56:21.149790 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.149603 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8s2lb\"" Jul 06 23:56:21.149790 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.149629 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.149790 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.149688 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:56:21.150184 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.150150 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 23:56:21.150406 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.150386 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8g88m\"" Jul 06 23:56:21.150525 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.150512 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 23:56:21.152238 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.152219 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 23:56:21.152447 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.152428 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.152576 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.152540 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:21.152660 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.152646 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l2m9d\"" Jul 06 23:56:21.152723 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.152668 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 23:56:21.152887 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.152869 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 23:56:21.155011 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.154482 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:21.156839 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.156821 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 23:56:21.157242 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.157224 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:21.157321 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.157298 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:21.157762 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.157742 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 23:56:21.157870 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.157848 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z8kw2\"" Jul 06 23:56:21.160206 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.160185 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.162833 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.162815 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 23:56:21.163104 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.162894 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 23:56:21.163214 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.162920 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mtjlm\"" Jul 06 23:56:21.163689 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.163610 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:21.163689 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.163664 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:21.174354 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.174328 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7tf8r" Jul 06 23:56:21.182868 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.182844 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7tf8r" Jul 06 23:56:21.235599 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.235574 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 23:56:21.245697 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245636 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:21.245697 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245671 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b854p\" (UniqueName: \"kubernetes.io/projected/daaa7108-9765-49af-a9cf-187b26bcba6f-kube-api-access-b854p\") pod \"node-ca-xl5vt\" (UID: \"daaa7108-9765-49af-a9cf-187b26bcba6f\") " pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.245697 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-registration-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.245947 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-device-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.245947 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245748 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-cnibin\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.245947 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245779 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-modprobe-d\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.245947 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245808 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-sys\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.245947 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245852 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-run-ovn\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.245947 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245881 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-os-release\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.245947 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-dbus\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.245947 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245922 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbe077f3-897c-4c32-b14a-98a8d3d04ffa-hosts-file\") pod \"node-resolver-f8h8b\" (UID: \"bbe077f3-897c-4c32-b14a-98a8d3d04ffa\") " pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245951 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-cnibin\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-multus-conf-dir\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.245988 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-log-socket\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbe077f3-897c-4c32-b14a-98a8d3d04ffa-tmp-dir\") pod \"node-resolver-f8h8b\" (UID: \"bbe077f3-897c-4c32-b14a-98a8d3d04ffa\") " pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246071 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-multus-conf-dir\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63306a16-c22f-423d-bb78-aef7a70665d2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246141 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-cnibin\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246201 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvdh\" (UniqueName: \"kubernetes.io/projected/63306a16-c22f-423d-bb78-aef7a70665d2-kube-api-access-nvvdh\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246233 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-kubernetes\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246261 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-sysctl-d\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.246287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-kubelet-config\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246313 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-systemd-units\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246338 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-var-lib-openvswitch\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246370 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-var-lib-cni-bin\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246396 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-system-cni-dir\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246420 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-run\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246432 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-var-lib-cni-bin\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246472 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85ffbaa2-9a9a-4794-8431-f56685f531de-tmp\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246499 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-cni-bin\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246530 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8v5\" (UniqueName: \"kubernetes.io/projected/bbe077f3-897c-4c32-b14a-98a8d3d04ffa-kube-api-access-7z8v5\") pod \"node-resolver-f8h8b\" (UID: \"bbe077f3-897c-4c32-b14a-98a8d3d04ffa\") " pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246560 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-hostroot\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246599 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-hostroot\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-run-multus-certs\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-etc-kubernetes\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246661 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-run-multus-certs\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-host\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.246792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246700 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-etc-kubernetes\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/84de0174-13d9-43cd-93e5-9a0f244c672b-iptables-alerter-script\") pod \"iptables-alerter-6bvgd\" (UID: \"84de0174-13d9-43cd-93e5-9a0f244c672b\") " pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4kgj\" (UniqueName: \"kubernetes.io/projected/d67760ac-cf43-445f-979a-ef2d648355eb-kube-api-access-j4kgj\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246752 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggh9\" (UniqueName: \"kubernetes.io/projected/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-kube-api-access-7ggh9\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246768 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-kubelet\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246782 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-run-openvswitch\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-node-log\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246846 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-os-release\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84de0174-13d9-43cd-93e5-9a0f244c672b-host-slash\") pod \"iptables-alerter-6bvgd\" (UID: \"84de0174-13d9-43cd-93e5-9a0f244c672b\") " pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/49f94231-7b6c-4397-8d6f-6b31880621c5-ovnkube-config\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246895 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84de0174-13d9-43cd-93e5-9a0f244c672b-host-slash\") pod \"iptables-alerter-6bvgd\" (UID: \"84de0174-13d9-43cd-93e5-9a0f244c672b\") " pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246903 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63306a16-c22f-423d-bb78-aef7a70665d2-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246907 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-os-release\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246924 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9jz\" (UniqueName: \"kubernetes.io/projected/85ffbaa2-9a9a-4794-8431-f56685f531de-kube-api-access-xr9jz\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246962 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.246992 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-sys-fs\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247013 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-system-cni-dir\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.247640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247028 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36d56419-1664-4eff-aaac-26a7440264bd-multus-daemon-config\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247046 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-systemd\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247061 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-tuned\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-socket-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63306a16-c22f-423d-bb78-aef7a70665d2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247126 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-slash\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-run-netns\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247145 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-system-cni-dir\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-etc-openvswitch\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247184 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/49f94231-7b6c-4397-8d6f-6b31880621c5-env-overrides\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247199 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daaa7108-9765-49af-a9cf-187b26bcba6f-host\") pod \"node-ca-xl5vt\" (UID: \"daaa7108-9765-49af-a9cf-187b26bcba6f\") " pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247220 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-multus-cni-dir\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247247 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfwnq\" (UniqueName: \"kubernetes.io/projected/84de0174-13d9-43cd-93e5-9a0f244c672b-kube-api-access-kfwnq\") pod \"iptables-alerter-6bvgd\" (UID: \"84de0174-13d9-43cd-93e5-9a0f244c672b\") " pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247266 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-lib-modules\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247284 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-cni-netd\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.248564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247327 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247336 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/84de0174-13d9-43cd-93e5-9a0f244c672b-iptables-alerter-script\") pod \"iptables-alerter-6bvgd\" (UID: \"84de0174-13d9-43cd-93e5-9a0f244c672b\") " pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247364 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8lqf\" (UniqueName: \"kubernetes.io/projected/49f94231-7b6c-4397-8d6f-6b31880621c5-kube-api-access-d8lqf\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247384 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-multus-cni-dir\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36d56419-1664-4eff-aaac-26a7440264bd-cni-binary-copy\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-var-lib-kubelet\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b237712a-3bf7-4ae1-9eae-849409680965-konnectivity-ca\") pod \"konnectivity-agent-z6ql9\" (UID: \"b237712a-3bf7-4ae1-9eae-849409680965\") " pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247616 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247628 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36d56419-1664-4eff-aaac-26a7440264bd-multus-daemon-config\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247643 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-sysconfig\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247663 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-var-lib-kubelet\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-sysctl-conf\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247714 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-var-lib-kubelet\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247739 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-run-systemd\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b237712a-3bf7-4ae1-9eae-849409680965-agent-certs\") pod \"konnectivity-agent-z6ql9\" (UID: \"b237712a-3bf7-4ae1-9eae-849409680965\") " pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247792 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-run-netns\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daaa7108-9765-49af-a9cf-187b26bcba6f-serviceca\") pod \"node-ca-xl5vt\" (UID: \"daaa7108-9765-49af-a9cf-187b26bcba6f\") " pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.249077 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247837 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36d56419-1664-4eff-aaac-26a7440264bd-cni-binary-copy\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247842 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/49f94231-7b6c-4397-8d6f-6b31880621c5-ovn-node-metrics-cert\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247840 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-run-netns\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/49f94231-7b6c-4397-8d6f-6b31880621c5-ovnkube-script-lib\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247895 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247920 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247946 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-multus-socket-dir-parent\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.247975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-run-k8s-cni-cncf-io\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.248001 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-var-lib-cni-multus\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.248027 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-multus-socket-dir-parent\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.248029 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55h9r\" (UniqueName: \"kubernetes.io/projected/36d56419-1664-4eff-aaac-26a7440264bd-kube-api-access-55h9r\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.248070 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-var-lib-cni-multus\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.249713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.248069 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36d56419-1664-4eff-aaac-26a7440264bd-host-run-k8s-cni-cncf-io\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.255905 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.255873 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 23:56:21.259780 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.259753 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55h9r\" (UniqueName: \"kubernetes.io/projected/36d56419-1664-4eff-aaac-26a7440264bd-kube-api-access-55h9r\") pod \"multus-p9zd2\" (UID: \"36d56419-1664-4eff-aaac-26a7440264bd\") " pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.259906 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.259761 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfwnq\" (UniqueName: \"kubernetes.io/projected/84de0174-13d9-43cd-93e5-9a0f244c672b-kube-api-access-kfwnq\") pod \"iptables-alerter-6bvgd\" (UID: \"84de0174-13d9-43cd-93e5-9a0f244c672b\") " pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.314047 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.313990 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" event={"ID":"8c937bb41e34a0bdc42a4d5667923deb","Type":"ContainerStarted","Data":"1a0303dcbded381bb73ddfb51d3e6f3953fddb490c670a6fc26153f7d532432c"} Jul 06 23:56:21.315305 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.315275 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal" event={"ID":"f4c4d400fcc6b91a31ccccc5f9a95b03","Type":"ContainerStarted","Data":"1b4bfaaed927a04070c7c3ae3cd8bd957042f41c3bf6aa19fd5befb40c9cd8a5"} Jul 06 23:56:21.348542 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggh9\" (UniqueName: \"kubernetes.io/projected/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-kube-api-access-7ggh9\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:21.348542 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-kubelet\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.348796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-run-openvswitch\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.348796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-node-log\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.348796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/49f94231-7b6c-4397-8d6f-6b31880621c5-ovnkube-config\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.348796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63306a16-c22f-423d-bb78-aef7a70665d2-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.348796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9jz\" (UniqueName: \"kubernetes.io/projected/85ffbaa2-9a9a-4794-8431-f56685f531de-kube-api-access-xr9jz\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.348796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.348796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348736 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-sys-fs\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.348796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-systemd\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.348796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348780 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-node-log\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-tuned\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-socket-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348898 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63306a16-c22f-423d-bb78-aef7a70665d2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348926 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-slash\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-run-netns\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-etc-openvswitch\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349006 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/49f94231-7b6c-4397-8d6f-6b31880621c5-env-overrides\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349018 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-socket-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daaa7108-9765-49af-a9cf-187b26bcba6f-host\") pod \"node-ca-xl5vt\" (UID: \"daaa7108-9765-49af-a9cf-187b26bcba6f\") " pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349063 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-lib-modules\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349096 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-run-netns\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349094 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349132 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-cni-netd\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349139 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349178 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-etc-openvswitch\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.349279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349185 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8lqf\" (UniqueName: \"kubernetes.io/projected/49f94231-7b6c-4397-8d6f-6b31880621c5-kube-api-access-d8lqf\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349218 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b237712a-3bf7-4ae1-9eae-849409680965-konnectivity-ca\") pod \"konnectivity-agent-z6ql9\" (UID: \"b237712a-3bf7-4ae1-9eae-849409680965\") " pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349250 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349274 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-sysconfig\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349299 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-sysctl-conf\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-var-lib-kubelet\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-run-systemd\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b237712a-3bf7-4ae1-9eae-849409680965-agent-certs\") pod \"konnectivity-agent-z6ql9\" (UID: \"b237712a-3bf7-4ae1-9eae-849409680965\") " pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daaa7108-9765-49af-a9cf-187b26bcba6f-serviceca\") pod \"node-ca-xl5vt\" (UID: \"daaa7108-9765-49af-a9cf-187b26bcba6f\") " pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349418 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/49f94231-7b6c-4397-8d6f-6b31880621c5-ovn-node-metrics-cert\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/49f94231-7b6c-4397-8d6f-6b31880621c5-ovnkube-script-lib\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349474 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349692 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/49f94231-7b6c-4397-8d6f-6b31880621c5-env-overrides\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349737 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349759 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daaa7108-9765-49af-a9cf-187b26bcba6f-host\") pod \"node-ca-xl5vt\" (UID: \"daaa7108-9765-49af-a9cf-187b26bcba6f\") " pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349762 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b854p\" (UniqueName: \"kubernetes.io/projected/daaa7108-9765-49af-a9cf-187b26bcba6f-kube-api-access-b854p\") pod \"node-ca-xl5vt\" (UID: \"daaa7108-9765-49af-a9cf-187b26bcba6f\") " pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.350128 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349803 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-registration-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-device-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-cnibin\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-modprobe-d\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-sys\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349943 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-run-ovn\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349969 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-os-release\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-dbus\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350021 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbe077f3-897c-4c32-b14a-98a8d3d04ffa-hosts-file\") pod \"node-resolver-f8h8b\" (UID: \"bbe077f3-897c-4c32-b14a-98a8d3d04ffa\") " pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-log-socket\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350089 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbe077f3-897c-4c32-b14a-98a8d3d04ffa-tmp-dir\") pod \"node-resolver-f8h8b\" (UID: \"bbe077f3-897c-4c32-b14a-98a8d3d04ffa\") " pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63306a16-c22f-423d-bb78-aef7a70665d2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvdh\" (UniqueName: \"kubernetes.io/projected/63306a16-c22f-423d-bb78-aef7a70665d2-kube-api-access-nvvdh\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350172 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-kubernetes\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350199 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-sysctl-d\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-kubelet-config\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350255 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-systemd-units\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.350812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-var-lib-openvswitch\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350315 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-system-cni-dir\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350343 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350368 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-run\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85ffbaa2-9a9a-4794-8431-f56685f531de-tmp\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350429 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-cni-bin\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350457 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8v5\" (UniqueName: \"kubernetes.io/projected/bbe077f3-897c-4c32-b14a-98a8d3d04ffa-kube-api-access-7z8v5\") pod \"node-resolver-f8h8b\" (UID: \"bbe077f3-897c-4c32-b14a-98a8d3d04ffa\") " pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-host\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4kgj\" (UniqueName: \"kubernetes.io/projected/d67760ac-cf43-445f-979a-ef2d648355eb-kube-api-access-j4kgj\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350587 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63306a16-c22f-423d-bb78-aef7a70665d2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.348895 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-kubelet\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350668 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-slash\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350709 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-cni-netd\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350745 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-lib-modules\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350753 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.349066 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-run-openvswitch\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350813 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-registration-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.351495 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350862 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-device-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350911 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-cnibin\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.350997 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-modprobe-d\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351054 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-sys\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351099 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-run-ovn\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351162 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-os-release\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.351256 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.351362 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs podName:5faeb3a9-7522-4d7a-a47f-8e65ad2809b1 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:21.851324593 +0000 UTC m=+3.064356341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs") pod "network-metrics-daemon-gdjjf" (UID: "5faeb3a9-7522-4d7a-a47f-8e65ad2809b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351601 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-sysconfig\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351715 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b237712a-3bf7-4ae1-9eae-849409680965-konnectivity-ca\") pod \"konnectivity-agent-z6ql9\" (UID: \"b237712a-3bf7-4ae1-9eae-849409680965\") " pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351743 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-sysctl-conf\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351790 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-var-lib-kubelet\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351792 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbe077f3-897c-4c32-b14a-98a8d3d04ffa-hosts-file\") pod \"node-resolver-f8h8b\" (UID: \"bbe077f3-897c-4c32-b14a-98a8d3d04ffa\") " pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351838 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-log-socket\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.351871 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-run-systemd\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.352241 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.352050 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbe077f3-897c-4c32-b14a-98a8d3d04ffa-tmp-dir\") pod \"node-resolver-f8h8b\" (UID: \"bbe077f3-897c-4c32-b14a-98a8d3d04ffa\") " pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.353060 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.352496 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.353060 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.352525 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63306a16-c22f-423d-bb78-aef7a70665d2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.353060 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.352925 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daaa7108-9765-49af-a9cf-187b26bcba6f-serviceca\") pod \"node-ca-xl5vt\" (UID: \"daaa7108-9765-49af-a9cf-187b26bcba6f\") " pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.353060 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.352989 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/49f94231-7b6c-4397-8d6f-6b31880621c5-ovnkube-script-lib\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.353060 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.353056 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.353493 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.353468 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63306a16-c22f-423d-bb78-aef7a70665d2-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.353894 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d67760ac-cf43-445f-979a-ef2d648355eb-sys-fs\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.353991 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.354063 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret podName:f7dcfee5-cbb6-40da-9c87-6eb081c7d885 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:21.854044281 +0000 UTC m=+3.067076021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret") pod "global-pull-secret-syncer-v592j" (UID: "f7dcfee5-cbb6-40da-9c87-6eb081c7d885") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354157 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/49f94231-7b6c-4397-8d6f-6b31880621c5-ovnkube-config\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354232 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-systemd\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354234 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-system-cni-dir\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354306 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-systemd-units\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354361 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-kubelet-config\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354406 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-var-lib-openvswitch\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354408 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-kubernetes\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354363 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-sysctl-d\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354404 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63306a16-c22f-423d-bb78-aef7a70665d2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354454 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-run\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354526 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49f94231-7b6c-4397-8d6f-6b31880621c5-host-cni-bin\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354582 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85ffbaa2-9a9a-4794-8431-f56685f531de-host\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.354716 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354630 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-dbus\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.355573 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.354777 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85ffbaa2-9a9a-4794-8431-f56685f531de-etc-tuned\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.356514 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.356342 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/49f94231-7b6c-4397-8d6f-6b31880621c5-ovn-node-metrics-cert\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.357155 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.357118 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b237712a-3bf7-4ae1-9eae-849409680965-agent-certs\") pod \"konnectivity-agent-z6ql9\" (UID: \"b237712a-3bf7-4ae1-9eae-849409680965\") " pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:21.357155 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.357146 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85ffbaa2-9a9a-4794-8431-f56685f531de-tmp\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.358926 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.358874 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggh9\" (UniqueName: \"kubernetes.io/projected/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-kube-api-access-7ggh9\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:21.359627 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.359605 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:21.359719 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.359633 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:21.359719 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.359649 2568 projected.go:196] Error preparing data for projected volume kube-api-access-zdzt6 for pod openshift-network-diagnostics/network-check-target-tdphw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:21.359823 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.359752 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6 podName:9aafc90b-4ee0-4eda-afa3-ecb0faa2110c nodeName:}" failed. No retries permitted until 2026-07-06 23:56:21.859733492 +0000 UTC m=+3.072765242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zdzt6" (UniqueName: "kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6") pod "network-check-target-tdphw" (UID: "9aafc90b-4ee0-4eda-afa3-ecb0faa2110c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:21.360245 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.360218 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8lqf\" (UniqueName: \"kubernetes.io/projected/49f94231-7b6c-4397-8d6f-6b31880621c5-kube-api-access-d8lqf\") pod \"ovnkube-node-lvndj\" (UID: \"49f94231-7b6c-4397-8d6f-6b31880621c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.360354 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.360330 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b854p\" (UniqueName: \"kubernetes.io/projected/daaa7108-9765-49af-a9cf-187b26bcba6f-kube-api-access-b854p\") pod \"node-ca-xl5vt\" (UID: \"daaa7108-9765-49af-a9cf-187b26bcba6f\") " pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.361474 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.361438 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9jz\" (UniqueName: \"kubernetes.io/projected/85ffbaa2-9a9a-4794-8431-f56685f531de-kube-api-access-xr9jz\") pod \"tuned-vnzqk\" (UID: \"85ffbaa2-9a9a-4794-8431-f56685f531de\") " pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.361564 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.361436 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4kgj\" (UniqueName: \"kubernetes.io/projected/d67760ac-cf43-445f-979a-ef2d648355eb-kube-api-access-j4kgj\") pod \"aws-ebs-csi-driver-node-krdmt\" (UID: \"d67760ac-cf43-445f-979a-ef2d648355eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.362282 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.362259 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvdh\" (UniqueName: \"kubernetes.io/projected/63306a16-c22f-423d-bb78-aef7a70665d2-kube-api-access-nvvdh\") pod \"multus-additional-cni-plugins-ncwhk\" (UID: \"63306a16-c22f-423d-bb78-aef7a70665d2\") " pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.362711 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.362687 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8v5\" (UniqueName: \"kubernetes.io/projected/bbe077f3-897c-4c32-b14a-98a8d3d04ffa-kube-api-access-7z8v5\") pod \"node-resolver-f8h8b\" (UID: \"bbe077f3-897c-4c32-b14a-98a8d3d04ffa\") " pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.449799 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.449761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p9zd2" Jul 06 23:56:21.459233 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.459209 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6bvgd" Jul 06 23:56:21.469369 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.469343 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:21.476416 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.476395 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" Jul 06 23:56:21.483480 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.483458 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" Jul 06 23:56:21.491542 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.491518 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f8h8b" Jul 06 23:56:21.507539 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.507471 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xl5vt" Jul 06 23:56:21.514697 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.514674 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:21.521784 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.521762 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" Jul 06 23:56:21.855114 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.855021 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:21.855114 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.855077 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:21.855311 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.855191 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:21.855311 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.855191 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:21.855311 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.855261 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs podName:5faeb3a9-7522-4d7a-a47f-8e65ad2809b1 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:22.855242279 +0000 UTC m=+4.068274018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs") pod "network-metrics-daemon-gdjjf" (UID: "5faeb3a9-7522-4d7a-a47f-8e65ad2809b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:21.855311 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.855279 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret podName:f7dcfee5-cbb6-40da-9c87-6eb081c7d885 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:22.855270054 +0000 UTC m=+4.068301776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret") pod "global-pull-secret-syncer-v592j" (UID: "f7dcfee5-cbb6-40da-9c87-6eb081c7d885") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:21.956343 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:21.956305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:21.956559 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.956501 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:21.956559 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.956518 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:21.956559 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.956530 2568 projected.go:196] Error preparing data for projected volume kube-api-access-zdzt6 for pod openshift-network-diagnostics/network-check-target-tdphw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:21.956731 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:21.956591 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6 podName:9aafc90b-4ee0-4eda-afa3-ecb0faa2110c nodeName:}" failed. No retries permitted until 2026-07-06 23:56:22.956573263 +0000 UTC m=+4.169605005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdzt6" (UniqueName: "kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6") pod "network-check-target-tdphw" (UID: "9aafc90b-4ee0-4eda-afa3-ecb0faa2110c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:22.187150 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:22.186999 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:51:22 +0000 UTC" deadline="2027-12-16 20:14:50.519885964 +0000 UTC" Jul 06 23:56:22.187150 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:22.187109 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12668h18m28.332799848s" Jul 06 23:56:22.313569 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:22.313539 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ffbaa2_9a9a_4794_8431_f56685f531de.slice/crio-8877197da39e68cb572d5aba7903a0cee1e9ac085846e23e8b43914776e390ca WatchSource:0}: Error finding container 8877197da39e68cb572d5aba7903a0cee1e9ac085846e23e8b43914776e390ca: Status 404 returned error can't find the container with id 8877197da39e68cb572d5aba7903a0cee1e9ac085846e23e8b43914776e390ca Jul 06 23:56:22.315369 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:22.315337 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaaa7108_9765_49af_a9cf_187b26bcba6f.slice/crio-dfa17cd9a80a05df153464572216a41ad5ab60f89fcc0e202a4db68aa88d5043 WatchSource:0}: Error finding container dfa17cd9a80a05df153464572216a41ad5ab60f89fcc0e202a4db68aa88d5043: Status 404 returned error can't find the container with id dfa17cd9a80a05df153464572216a41ad5ab60f89fcc0e202a4db68aa88d5043 Jul 06 23:56:22.317322 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:22.317294 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d56419_1664_4eff_aaac_26a7440264bd.slice/crio-331c7cc4cf72fc092f79aa181cf48a45f98d2b93a5e4d59f62d7c89218911566 WatchSource:0}: Error finding container 331c7cc4cf72fc092f79aa181cf48a45f98d2b93a5e4d59f62d7c89218911566: Status 404 returned error can't find the container with id 331c7cc4cf72fc092f79aa181cf48a45f98d2b93a5e4d59f62d7c89218911566 Jul 06 23:56:22.318193 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:22.318159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xl5vt" event={"ID":"daaa7108-9765-49af-a9cf-187b26bcba6f","Type":"ContainerStarted","Data":"dfa17cd9a80a05df153464572216a41ad5ab60f89fcc0e202a4db68aa88d5043"} Jul 06 23:56:22.321338 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:22.321305 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" event={"ID":"85ffbaa2-9a9a-4794-8431-f56685f531de","Type":"ContainerStarted","Data":"8877197da39e68cb572d5aba7903a0cee1e9ac085846e23e8b43914776e390ca"} Jul 06 23:56:22.321760 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:22.321733 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84de0174_13d9_43cd_93e5_9a0f244c672b.slice/crio-8b0c6514309fc74a34f1abbe447340d6625b96e016134d963badaf4cf29a017b WatchSource:0}: Error finding container 8b0c6514309fc74a34f1abbe447340d6625b96e016134d963badaf4cf29a017b: Status 404 returned error can't find the container with id 8b0c6514309fc74a34f1abbe447340d6625b96e016134d963badaf4cf29a017b Jul 06 23:56:22.331911 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:22.329741 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe077f3_897c_4c32_b14a_98a8d3d04ffa.slice/crio-64ddb9aa88f34fb06c75d2ad6e49effa5d09e8a1c14a748b49ac2faeb7039f10 WatchSource:0}: Error finding container 64ddb9aa88f34fb06c75d2ad6e49effa5d09e8a1c14a748b49ac2faeb7039f10: Status 404 returned error can't find the container with id 64ddb9aa88f34fb06c75d2ad6e49effa5d09e8a1c14a748b49ac2faeb7039f10 Jul 06 23:56:22.671789 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:22.671572 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:22.864702 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:22.864605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:22.864889 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:22.864745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:22.864962 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:22.864900 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:22.865020 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:22.864969 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret podName:f7dcfee5-cbb6-40da-9c87-6eb081c7d885 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:24.864949429 +0000 UTC m=+6.077981168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret") pod "global-pull-secret-syncer-v592j" (UID: "f7dcfee5-cbb6-40da-9c87-6eb081c7d885") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:22.865440 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:22.865414 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:22.865554 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:22.865492 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs podName:5faeb3a9-7522-4d7a-a47f-8e65ad2809b1 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:24.865463671 +0000 UTC m=+6.078495420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs") pod "network-metrics-daemon-gdjjf" (UID: "5faeb3a9-7522-4d7a-a47f-8e65ad2809b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:22.966017 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:22.965856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:22.966017 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:22.966024 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:22.966263 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:22.966042 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:22.966263 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:22.966056 2568 projected.go:196] Error preparing data for projected volume kube-api-access-zdzt6 for pod openshift-network-diagnostics/network-check-target-tdphw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:22.966263 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:22.966121 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6 podName:9aafc90b-4ee0-4eda-afa3-ecb0faa2110c nodeName:}" failed. No retries permitted until 2026-07-06 23:56:24.96610064 +0000 UTC m=+6.179132370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdzt6" (UniqueName: "kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6") pod "network-check-target-tdphw" (UID: "9aafc90b-4ee0-4eda-afa3-ecb0faa2110c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:23.187842 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.187727 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:51:22 +0000 UTC" deadline="2028-04-06 19:15:30.174463536 +0000 UTC" Jul 06 23:56:23.187842 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.187790 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15355h19m6.986677695s" Jul 06 23:56:23.310096 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.309603 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:23.310096 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:23.309740 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:23.310096 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.309761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:23.310096 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.309785 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:23.310096 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:23.309856 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:23.310096 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:23.309939 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:23.357283 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.357241 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6bvgd" event={"ID":"84de0174-13d9-43cd-93e5-9a0f244c672b","Type":"ContainerStarted","Data":"8b0c6514309fc74a34f1abbe447340d6625b96e016134d963badaf4cf29a017b"} Jul 06 23:56:23.375680 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.375565 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f8h8b" event={"ID":"bbe077f3-897c-4c32-b14a-98a8d3d04ffa","Type":"ContainerStarted","Data":"64ddb9aa88f34fb06c75d2ad6e49effa5d09e8a1c14a748b49ac2faeb7039f10"} Jul 06 23:56:23.386396 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.386293 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z6ql9" event={"ID":"b237712a-3bf7-4ae1-9eae-849409680965","Type":"ContainerStarted","Data":"3b40c13776ccc4595fc5718152a75129d8b2299a1dbf5ea9b020f85e5ff47b1d"} Jul 06 23:56:23.398726 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.398687 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" event={"ID":"d67760ac-cf43-445f-979a-ef2d648355eb","Type":"ContainerStarted","Data":"ea3035a7d741cfe202df3bae5223ad61b92557133637ac419eb61b2ceb455b92"} Jul 06 23:56:23.405116 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.405077 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p9zd2" event={"ID":"36d56419-1664-4eff-aaac-26a7440264bd","Type":"ContainerStarted","Data":"331c7cc4cf72fc092f79aa181cf48a45f98d2b93a5e4d59f62d7c89218911566"} Jul 06 23:56:23.440522 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.440428 2568 generic.go:356] "Generic (PLEG): container finished" podID="8c937bb41e34a0bdc42a4d5667923deb" containerID="4a461cb6eaaf1ebb5f65f741ed0ee5bc4e234fcde42357152cef51bc7a1d19db" exitCode=0 Jul 06 23:56:23.441116 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.441039 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" event={"ID":"8c937bb41e34a0bdc42a4d5667923deb","Type":"ContainerDied","Data":"4a461cb6eaaf1ebb5f65f741ed0ee5bc4e234fcde42357152cef51bc7a1d19db"} Jul 06 23:56:23.470713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.470650 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal" event={"ID":"f4c4d400fcc6b91a31ccccc5f9a95b03","Type":"ContainerStarted","Data":"7bc1381300b23b42033d94078aa23d08673c98387cbf7556c8de928561a078c1"} Jul 06 23:56:23.495177 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.495129 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" event={"ID":"63306a16-c22f-423d-bb78-aef7a70665d2","Type":"ContainerStarted","Data":"c041242d65ac2d15f43f6fd897c994b9ef760a546cc5b6a4ea9891e4f711c6f1"} Jul 06 23:56:23.503180 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:23.503127 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerStarted","Data":"5c477a2e3646c9782292fde430c39a03ce627026c1f5f8f0dec45735e8a221ef"} Jul 06 23:56:24.543686 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:24.543573 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" event={"ID":"8c937bb41e34a0bdc42a4d5667923deb","Type":"ContainerStarted","Data":"2fbbb97b797020ac3dbcf331af6fe6741f12818c578ae5e4a255e1017edcd8e7"} Jul 06 23:56:24.562498 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:24.561533 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.ec2.internal" podStartSLOduration=3.561511339 podStartE2EDuration="3.561511339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:23.489696338 +0000 UTC m=+4.702728087" watchObservedRunningTime="2026-07-06 23:56:24.561511339 +0000 UTC m=+5.774543085" Jul 06 23:56:24.884758 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:24.883837 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:24.884758 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:24.883917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:24.884758 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:24.884107 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:24.884758 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:24.884180 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs podName:5faeb3a9-7522-4d7a-a47f-8e65ad2809b1 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:28.884159551 +0000 UTC m=+10.097191300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs") pod "network-metrics-daemon-gdjjf" (UID: "5faeb3a9-7522-4d7a-a47f-8e65ad2809b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:24.884758 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:24.884258 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:24.884758 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:24.884300 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret podName:f7dcfee5-cbb6-40da-9c87-6eb081c7d885 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:28.88428712 +0000 UTC m=+10.097318848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret") pod "global-pull-secret-syncer-v592j" (UID: "f7dcfee5-cbb6-40da-9c87-6eb081c7d885") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:24.985897 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:24.985235 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:24.985897 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:24.985423 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:24.985897 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:24.985442 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:24.985897 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:24.985456 2568 projected.go:196] Error preparing data for projected volume kube-api-access-zdzt6 for pod openshift-network-diagnostics/network-check-target-tdphw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:24.985897 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:24.985540 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6 podName:9aafc90b-4ee0-4eda-afa3-ecb0faa2110c nodeName:}" failed. No retries permitted until 2026-07-06 23:56:28.985520515 +0000 UTC m=+10.198552260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdzt6" (UniqueName: "kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6") pod "network-check-target-tdphw" (UID: "9aafc90b-4ee0-4eda-afa3-ecb0faa2110c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:25.311547 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:25.310651 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:25.311547 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:25.310693 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:25.311547 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:25.310796 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:25.311547 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:25.311242 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:25.311547 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:25.311323 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:25.311547 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:25.311424 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:27.309993 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:27.309960 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:27.310466 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:27.310122 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:27.310560 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:27.310546 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:27.310655 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:27.310617 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:27.310782 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:27.310755 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:27.310880 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:27.310860 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:28.918078 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:28.918033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:28.918624 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:28.918104 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:28.918624 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:28.918221 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:28.918624 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:28.918260 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:28.918624 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:28.918303 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret podName:f7dcfee5-cbb6-40da-9c87-6eb081c7d885 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:36.918279983 +0000 UTC m=+18.131311723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret") pod "global-pull-secret-syncer-v592j" (UID: "f7dcfee5-cbb6-40da-9c87-6eb081c7d885") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:28.918624 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:28.918324 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs podName:5faeb3a9-7522-4d7a-a47f-8e65ad2809b1 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:36.918312487 +0000 UTC m=+18.131344220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs") pod "network-metrics-daemon-gdjjf" (UID: "5faeb3a9-7522-4d7a-a47f-8e65ad2809b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:29.018594 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:29.018548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:29.018787 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:29.018757 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:29.018787 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:29.018783 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:29.018920 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:29.018797 2568 projected.go:196] Error preparing data for projected volume kube-api-access-zdzt6 for pod openshift-network-diagnostics/network-check-target-tdphw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:29.018920 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:29.018873 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6 podName:9aafc90b-4ee0-4eda-afa3-ecb0faa2110c nodeName:}" failed. No retries permitted until 2026-07-06 23:56:37.018858134 +0000 UTC m=+18.231889871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdzt6" (UniqueName: "kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6") pod "network-check-target-tdphw" (UID: "9aafc90b-4ee0-4eda-afa3-ecb0faa2110c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:29.311540 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:29.311160 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:29.311540 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:29.311269 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:29.311540 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:29.311293 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:29.311540 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:29.311372 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:29.311540 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:29.311429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:29.311540 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:29.311503 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:31.312148 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:31.312058 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:31.312717 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:31.312197 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:31.313136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:31.313113 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:31.313256 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:31.313231 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:31.313325 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:31.313287 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:31.313382 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:31.313362 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:33.309677 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:33.309637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:33.310122 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:33.309691 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:33.310122 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:33.309695 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:33.310122 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:33.309768 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:33.310122 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:33.309869 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:33.310122 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:33.309978 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:35.309593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:35.309563 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:35.310070 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:35.309563 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:35.310070 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:35.309695 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:35.310070 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:35.309563 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:35.310070 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:35.309776 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:35.310070 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:35.309923 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:36.983657 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:36.983614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:36.983657 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:36.983665 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:36.984161 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:36.983762 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:36.984161 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:36.983764 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:36.984161 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:36.983826 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs podName:5faeb3a9-7522-4d7a-a47f-8e65ad2809b1 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:52.983809213 +0000 UTC m=+34.196840939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs") pod "network-metrics-daemon-gdjjf" (UID: "5faeb3a9-7522-4d7a-a47f-8e65ad2809b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:36.984161 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:36.983840 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret podName:f7dcfee5-cbb6-40da-9c87-6eb081c7d885 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:52.983834146 +0000 UTC m=+34.196865868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret") pod "global-pull-secret-syncer-v592j" (UID: "f7dcfee5-cbb6-40da-9c87-6eb081c7d885") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:37.084812 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:37.084777 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:37.084964 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:37.084941 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:37.084964 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:37.084964 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:37.085041 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:37.084974 2568 projected.go:196] Error preparing data for projected volume kube-api-access-zdzt6 for pod openshift-network-diagnostics/network-check-target-tdphw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:37.085041 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:37.085034 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6 podName:9aafc90b-4ee0-4eda-afa3-ecb0faa2110c nodeName:}" failed. No retries permitted until 2026-07-06 23:56:53.08501598 +0000 UTC m=+34.298047706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdzt6" (UniqueName: "kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6") pod "network-check-target-tdphw" (UID: "9aafc90b-4ee0-4eda-afa3-ecb0faa2110c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:37.309342 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:37.309269 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:37.309506 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:37.309270 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:37.309506 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:37.309389 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:37.309506 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:37.309399 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:37.309622 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:37.309531 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:37.309663 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:37.309634 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:39.310813 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:39.310772 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:39.311283 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:39.310922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:39.311283 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:39.310944 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:39.311283 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:39.311003 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:39.311283 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:39.311071 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:39.311283 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:39.311069 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:41.310092 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:41.310054 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:41.310702 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:41.310132 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:41.310702 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:41.310126 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:41.310702 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:41.310257 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:41.310864 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:41.310746 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:41.310864 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:41.310824 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:42.580916 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.580886 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xl5vt" event={"ID":"daaa7108-9765-49af-a9cf-187b26bcba6f","Type":"ContainerStarted","Data":"49220c5a75db8f62dbdb047320018e26693d86b282289a23ade762e7e03253e3"} Jul 06 23:56:42.582456 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.582411 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f8h8b" event={"ID":"bbe077f3-897c-4c32-b14a-98a8d3d04ffa","Type":"ContainerStarted","Data":"ab180fe49cb3f2d0f79d1fd6eec7b6ffdad12e03114bcee90deec63f057b3aa5"} Jul 06 23:56:42.584469 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.584427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z6ql9" event={"ID":"b237712a-3bf7-4ae1-9eae-849409680965","Type":"ContainerStarted","Data":"87b98b2d8c05c0849eb8f127745bb5dcb06886ad54da49531db9ed31fc6d7cb5"} Jul 06 23:56:42.586252 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.586130 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" event={"ID":"d67760ac-cf43-445f-979a-ef2d648355eb","Type":"ContainerStarted","Data":"7a000493af49c691c9eaa397fdfba8d7d4c9104a8683e2f09b6645778dab53e6"} Jul 06 23:56:42.590287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.590119 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p9zd2" event={"ID":"36d56419-1664-4eff-aaac-26a7440264bd","Type":"ContainerStarted","Data":"fcfcc08aad0acb2b237b59579ceefd7fdfc4725c4d70e7a33ceba00ea1d0292f"} Jul 06 23:56:42.591764 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.591740 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" event={"ID":"85ffbaa2-9a9a-4794-8431-f56685f531de","Type":"ContainerStarted","Data":"5c93ad98081db7854d4c4b5e7fad2a8da963adae639b8b7234c595a4e07f8b3f"} Jul 06 23:56:42.598833 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.598786 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f8h8b" podStartSLOduration=2.920606518 podStartE2EDuration="22.598769231s" totalImagesPullingTime="19.678162713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:42.598623598 +0000 UTC m=+23.811655358" watchObservedRunningTime="2026-07-06 23:56:42.598769231 +0000 UTC m=+23.811800976" Jul 06 23:56:42.599064 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.599033 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.ec2.internal" podStartSLOduration=21.599023351 podStartE2EDuration="21.599023351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:24.562453225 +0000 UTC m=+5.775484974" watchObservedRunningTime="2026-07-06 23:56:42.599023351 +0000 UTC m=+23.812055096" Jul 06 23:56:42.599286 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.599264 2568 generic.go:356] "Generic (PLEG): container finished" podID="63306a16-c22f-423d-bb78-aef7a70665d2" containerID="daf2e8cb9383495a753f0e56c84d670311ee6cbaac44f3f7108b32e0c2d82671" exitCode=0 Jul 06 23:56:42.599367 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.599346 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" event={"ID":"63306a16-c22f-423d-bb78-aef7a70665d2","Type":"ContainerDied","Data":"daf2e8cb9383495a753f0e56c84d670311ee6cbaac44f3f7108b32e0c2d82671"} Jul 06 23:56:42.601836 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.601814 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 06 23:56:42.603250 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.603079 2568 generic.go:356] "Generic (PLEG): container finished" podID="49f94231-7b6c-4397-8d6f-6b31880621c5" containerID="b0495400ac76f4ada4e0a5f036f65b49ed12f295ed0546c7c51c8e6181de3bf4" exitCode=1 Jul 06 23:56:42.603250 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.603118 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerStarted","Data":"41ef6a8e1d3c585d3c11ed70df2c5cf9b1a3c81a518d1ffda515fae0c657599c"} Jul 06 23:56:42.603250 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.603167 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerDied","Data":"b0495400ac76f4ada4e0a5f036f65b49ed12f295ed0546c7c51c8e6181de3bf4"} Jul 06 23:56:42.603250 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.603205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerStarted","Data":"5ec386e66f63c725b1eed3e8fa31a67ffe46961a89b0e8846bc89352d21cc09f"} Jul 06 23:56:42.616143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.616099 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xl5vt" podStartSLOduration=2.891960701 podStartE2EDuration="22.616085175s" totalImagesPullingTime="19.724124474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:42.61590604 +0000 UTC m=+23.828937787" watchObservedRunningTime="2026-07-06 23:56:42.616085175 +0000 UTC m=+23.829116918" Jul 06 23:56:42.657425 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.657384 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vnzqk" podStartSLOduration=2.929352912 podStartE2EDuration="22.657367155s" totalImagesPullingTime="19.728014243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:42.657326937 +0000 UTC m=+23.870358695" watchObservedRunningTime="2026-07-06 23:56:42.657367155 +0000 UTC m=+23.870398896" Jul 06 23:56:42.675076 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.675027 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z6ql9" podStartSLOduration=11.567262752 podStartE2EDuration="22.675015284s" totalImagesPullingTime="11.107752532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:42.67467098 +0000 UTC m=+23.887702725" watchObservedRunningTime="2026-07-06 23:56:42.675015284 +0000 UTC m=+23.888047027" Jul 06 23:56:42.697484 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:42.697398 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p9zd2" podStartSLOduration=2.687532618 podStartE2EDuration="22.697374872s" totalImagesPullingTime="20.009842254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:42.697051894 +0000 UTC m=+23.910083643" watchObservedRunningTime="2026-07-06 23:56:42.697374872 +0000 UTC m=+23.910406618" Jul 06 23:56:43.313224 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.313194 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:43.313412 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.313295 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:43.313412 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.313311 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:43.313412 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:43.313299 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:43.313582 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:43.313401 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:43.313582 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:43.313506 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:43.539976 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.539946 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 23:56:43.609095 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.609063 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 06 23:56:43.609540 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.609472 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerStarted","Data":"ae36a4c5b6987559e24eaff0edf8bb5cbab5ac8827733de070ae98bf37eeb298"} Jul 06 23:56:43.609606 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.609563 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerStarted","Data":"3e93262b65f636b5b28e01f046a4ba6b850d47d4243408a41cb2380fe656cdfd"} Jul 06 23:56:43.609667 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.609610 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerStarted","Data":"f8675eb1a65aa91e7e99d67ca15194c9d9858b3c0cc12729ff4f43d6e4de3bd9"} Jul 06 23:56:43.611079 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.611052 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6bvgd" event={"ID":"84de0174-13d9-43cd-93e5-9a0f244c672b","Type":"ContainerStarted","Data":"d46f1696013d6210ab22fcc0a102e236b0f91227c2f3ea5b78c68abd5737c82d"} Jul 06 23:56:43.613175 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.613146 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" event={"ID":"d67760ac-cf43-445f-979a-ef2d648355eb","Type":"ContainerStarted","Data":"71b988bbb86253e4179ab47711ab65ea4e25d5d3761c594d796c4e5eb6672991"} Jul 06 23:56:43.631705 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:43.631648 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6bvgd" podStartSLOduration=12.520460593 podStartE2EDuration="23.631625514s" totalImagesPullingTime="11.111164921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:43.631578356 +0000 UTC m=+24.844610103" watchObservedRunningTime="2026-07-06 23:56:43.631625514 +0000 UTC m=+24.844657259" Jul 06 23:56:44.216668 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:44.216489 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T23:56:43.539967917Z","UUID":"9ee39fcf-ee52-47d3-bf81-90e024a7dc16","Handler":null,"Name":"","Endpoint":""} Jul 06 23:56:44.218200 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:44.218173 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 23:56:44.218334 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:44.218238 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 23:56:44.617976 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:44.617891 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" event={"ID":"d67760ac-cf43-445f-979a-ef2d648355eb","Type":"ContainerStarted","Data":"5939f8b3f262c3b3e399f1f8f75e779e03de21a3d610ea2a88d8fc71e4ea9389"} Jul 06 23:56:44.635144 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:44.635085 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krdmt" podStartSLOduration=3.042927089 podStartE2EDuration="24.635063241s" totalImagesPullingTime="21.592136152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:44.634945422 +0000 UTC m=+25.847977168" watchObservedRunningTime="2026-07-06 23:56:44.635063241 +0000 UTC m=+25.848094982" Jul 06 23:56:44.679225 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:44.679185 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:44.679901 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:44.679878 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:45.310262 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:45.310223 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:45.310453 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:45.310223 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:45.310453 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:45.310364 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:45.310453 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:45.310423 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:45.310648 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:45.310549 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:45.310705 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:45.310638 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:45.624079 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:45.624010 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 06 23:56:45.624503 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:45.624467 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerStarted","Data":"5cc6966505bb2a953b630dfd65c3b95f31d2289afad7b0b1016c420af9235d8a"} Jul 06 23:56:46.362414 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:46.362198 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:46.363052 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:46.362819 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z6ql9" Jul 06 23:56:47.310248 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:47.310159 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:47.310962 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:47.310159 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:47.310962 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:47.310277 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:47.310962 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:47.310159 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:47.310962 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:47.310357 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:47.310962 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:47.310425 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:47.631131 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:47.631048 2568 generic.go:356] "Generic (PLEG): container finished" podID="63306a16-c22f-423d-bb78-aef7a70665d2" containerID="b49adb350da16999ecaab4c9b2107a295f8d0aaf11d9727b69e6ebcac4acecb5" exitCode=0 Jul 06 23:56:47.631348 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:47.631139 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" event={"ID":"63306a16-c22f-423d-bb78-aef7a70665d2","Type":"ContainerDied","Data":"b49adb350da16999ecaab4c9b2107a295f8d0aaf11d9727b69e6ebcac4acecb5"} Jul 06 23:56:47.634775 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:47.634751 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 06 23:56:47.635201 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:47.635173 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerStarted","Data":"24dd982dbbe00c014476e6f47e46800c0e092e75074f1d5c77af8e53ac6ada4d"} Jul 06 23:56:47.635592 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:47.635578 2568 scope.go:117] "RemoveContainer" containerID="b0495400ac76f4ada4e0a5f036f65b49ed12f295ed0546c7c51c8e6181de3bf4" Jul 06 23:56:48.640301 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.640069 2568 generic.go:356] "Generic (PLEG): container finished" podID="63306a16-c22f-423d-bb78-aef7a70665d2" containerID="c5e036701c63ab6663859f7a68369aa64b7f2775e3dfd87511611a2eedd93794" exitCode=0 Jul 06 23:56:48.640712 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.640142 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" event={"ID":"63306a16-c22f-423d-bb78-aef7a70665d2","Type":"ContainerDied","Data":"c5e036701c63ab6663859f7a68369aa64b7f2775e3dfd87511611a2eedd93794"} Jul 06 23:56:48.644635 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.644614 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 06 23:56:48.645010 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.644976 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" event={"ID":"49f94231-7b6c-4397-8d6f-6b31880621c5","Type":"ContainerStarted","Data":"ae6010eb2f96b8d23bef26fff453460bcefddd6755fad28da185fcdb690bc86e"} Jul 06 23:56:48.645223 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.645206 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:48.645289 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.645266 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:48.645328 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.645298 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:48.661131 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.661106 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:48.664963 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.664937 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:56:48.712658 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:48.712605 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" podStartSLOduration=8.950599223 podStartE2EDuration="28.712588849s" totalImagesPullingTime="19.761989626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:48.712301233 +0000 UTC m=+29.925332977" watchObservedRunningTime="2026-07-06 23:56:48.712588849 +0000 UTC m=+29.925620605" Jul 06 23:56:49.222324 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:49.222292 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tdphw"] Jul 06 23:56:49.222483 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:49.222426 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:49.222596 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:49.222575 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:49.227623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:49.227595 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v592j"] Jul 06 23:56:49.227746 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:49.227717 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:49.227841 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:49.227819 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:49.231017 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:49.230998 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gdjjf"] Jul 06 23:56:49.231157 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:49.231132 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:49.231280 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:49.231259 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:49.649914 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:49.649805 2568 generic.go:356] "Generic (PLEG): container finished" podID="63306a16-c22f-423d-bb78-aef7a70665d2" containerID="b7c150c2af54736e009321f77ee0e278c9adc3e8cf77bbff7651f01c32898c1c" exitCode=0 Jul 06 23:56:49.649914 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:49.649891 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" event={"ID":"63306a16-c22f-423d-bb78-aef7a70665d2","Type":"ContainerDied","Data":"b7c150c2af54736e009321f77ee0e278c9adc3e8cf77bbff7651f01c32898c1c"} Jul 06 23:56:50.309368 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:50.309328 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:50.309564 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:50.309481 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:51.313606 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:51.313577 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:51.314018 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:51.313577 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:51.314018 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:51.313694 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:51.314018 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:51.313790 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:52.310015 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:52.309798 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:52.310208 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:52.310104 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:53.011303 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:53.011254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:53.011793 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:53.011357 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:53.011793 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.011455 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:53.011793 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.011495 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:53.011793 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.011535 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs podName:5faeb3a9-7522-4d7a-a47f-8e65ad2809b1 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:25.011514565 +0000 UTC m=+66.224546302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs") pod "network-metrics-daemon-gdjjf" (UID: "5faeb3a9-7522-4d7a-a47f-8e65ad2809b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:53.011793 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.011557 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret podName:f7dcfee5-cbb6-40da-9c87-6eb081c7d885 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:25.011545261 +0000 UTC m=+66.224576989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret") pod "global-pull-secret-syncer-v592j" (UID: "f7dcfee5-cbb6-40da-9c87-6eb081c7d885") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:53.112570 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:53.112526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:53.112754 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.112725 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:53.112754 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.112749 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:53.112867 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.112762 2568 projected.go:196] Error preparing data for projected volume kube-api-access-zdzt6 for pod openshift-network-diagnostics/network-check-target-tdphw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:53.112867 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.112831 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6 podName:9aafc90b-4ee0-4eda-afa3-ecb0faa2110c nodeName:}" failed. No retries permitted until 2026-07-06 23:57:25.112813602 +0000 UTC m=+66.325845340 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdzt6" (UniqueName: "kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6") pod "network-check-target-tdphw" (UID: "9aafc90b-4ee0-4eda-afa3-ecb0faa2110c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:53.309994 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:53.309421 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:53.309994 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:53.309479 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:53.309994 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.309584 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v592j" podUID="f7dcfee5-cbb6-40da-9c87-6eb081c7d885" Jul 06 23:56:53.309994 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:53.309737 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdjjf" podUID="5faeb3a9-7522-4d7a-a47f-8e65ad2809b1" Jul 06 23:56:54.309821 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:54.309730 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:54.310333 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:54.309835 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdphw" podUID="9aafc90b-4ee0-4eda-afa3-ecb0faa2110c" Jul 06 23:56:54.665005 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:54.664968 2568 generic.go:356] "Generic (PLEG): container finished" podID="63306a16-c22f-423d-bb78-aef7a70665d2" containerID="305ef54cd109b5f57859d22d51335f179b9a0e7682d9a16d4dc6c9a3510f9278" exitCode=0 Jul 06 23:56:54.665163 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:54.665037 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" event={"ID":"63306a16-c22f-423d-bb78-aef7a70665d2","Type":"ContainerDied","Data":"305ef54cd109b5f57859d22d51335f179b9a0e7682d9a16d4dc6c9a3510f9278"} Jul 06 23:56:55.076266 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.076191 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-86.ec2.internal" event="NodeReady" Jul 06 23:56:55.076401 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.076319 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 23:56:55.157330 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.157298 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9tv2d"] Jul 06 23:56:55.171982 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.171940 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qf9l5"] Jul 06 23:56:55.172121 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.172076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tv2d" Jul 06 23:56:55.178893 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.178864 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 23:56:55.179012 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.178864 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 23:56:55.179012 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.178875 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 23:56:55.186387 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.185517 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wljwx\"" Jul 06 23:56:55.188070 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.188047 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qf9l5"] Jul 06 23:56:55.188197 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.188177 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.188280 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.188260 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tv2d"] Jul 06 23:56:55.193588 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.193564 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lhsfz\"" Jul 06 23:56:55.193735 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.193651 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 23:56:55.193735 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.193723 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 23:56:55.220530 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.220492 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pcz8g"] Jul 06 23:56:55.238076 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.238041 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.241078 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.241050 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 23:56:55.241230 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.241160 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-274zg\"" Jul 06 23:56:55.241230 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.241160 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 23:56:55.241340 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.241311 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 23:56:55.241766 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.241752 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 23:56:55.248221 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.248192 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pcz8g"] Jul 06 23:56:55.309739 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.309701 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:56:55.309908 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.309701 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:56:55.314187 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.314161 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:56:55.314332 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.314294 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dplr9\"" Jul 06 23:56:55.316793 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.316771 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:56:55.329011 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.328955 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-config-volume\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.329011 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.328986 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-data-volume\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.329138 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.329024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm89s\" (UniqueName: \"kubernetes.io/projected/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-kube-api-access-mm89s\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.329138 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.329078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-tmp-dir\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.329138 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.329101 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/556a6fe3-9d48-45db-b666-d8ca7f5bface-cert\") pod \"ingress-canary-9tv2d\" (UID: \"556a6fe3-9d48-45db-b666-d8ca7f5bface\") " pod="openshift-ingress-canary/ingress-canary-9tv2d" Jul 06 23:56:55.329138 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.329119 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.329255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.329147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmw9m\" (UniqueName: \"kubernetes.io/projected/556a6fe3-9d48-45db-b666-d8ca7f5bface-kube-api-access-cmw9m\") pod \"ingress-canary-9tv2d\" (UID: \"556a6fe3-9d48-45db-b666-d8ca7f5bface\") " pod="openshift-ingress-canary/ingress-canary-9tv2d" Jul 06 23:56:55.329255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.329164 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-metrics-tls\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.329255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.329184 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2kq\" (UniqueName: \"kubernetes.io/projected/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-kube-api-access-9c2kq\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.329255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.329210 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.329255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.329228 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-crio-socket\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.430360 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm89s\" (UniqueName: \"kubernetes.io/projected/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-kube-api-access-mm89s\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.430502 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-tmp-dir\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.430502 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/556a6fe3-9d48-45db-b666-d8ca7f5bface-cert\") pod \"ingress-canary-9tv2d\" (UID: \"556a6fe3-9d48-45db-b666-d8ca7f5bface\") " pod="openshift-ingress-canary/ingress-canary-9tv2d" Jul 06 23:56:55.430502 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430453 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430497 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmw9m\" (UniqueName: \"kubernetes.io/projected/556a6fe3-9d48-45db-b666-d8ca7f5bface-kube-api-access-cmw9m\") pod \"ingress-canary-9tv2d\" (UID: \"556a6fe3-9d48-45db-b666-d8ca7f5bface\") " pod="openshift-ingress-canary/ingress-canary-9tv2d" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-metrics-tls\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430573 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2kq\" (UniqueName: \"kubernetes.io/projected/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-kube-api-access-9c2kq\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430642 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-crio-socket\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-config-volume\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430691 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-data-volume\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.430816 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-tmp-dir\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.431008 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-crio-socket\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.431217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.431041 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-data-volume\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.431676 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.431283 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.431676 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.431517 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-config-volume\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.434646 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.434623 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-metrics-tls\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.434724 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.434702 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.434820 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.434803 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/556a6fe3-9d48-45db-b666-d8ca7f5bface-cert\") pod \"ingress-canary-9tv2d\" (UID: \"556a6fe3-9d48-45db-b666-d8ca7f5bface\") " pod="openshift-ingress-canary/ingress-canary-9tv2d" Jul 06 23:56:55.451471 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.451427 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2kq\" (UniqueName: \"kubernetes.io/projected/3d0f8f96-cc8e-43f2-9325-bd89a90b61bf-kube-api-access-9c2kq\") pod \"insights-runtime-extractor-pcz8g\" (UID: \"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf\") " pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.453301 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.453251 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm89s\" (UniqueName: \"kubernetes.io/projected/4e1e414a-eea7-41d9-89a6-a7ca5825b7d3-kube-api-access-mm89s\") pod \"dns-default-qf9l5\" (UID: \"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3\") " pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.455731 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.455709 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmw9m\" (UniqueName: \"kubernetes.io/projected/556a6fe3-9d48-45db-b666-d8ca7f5bface-kube-api-access-cmw9m\") pod \"ingress-canary-9tv2d\" (UID: \"556a6fe3-9d48-45db-b666-d8ca7f5bface\") " pod="openshift-ingress-canary/ingress-canary-9tv2d" Jul 06 23:56:55.480569 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.480529 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tv2d" Jul 06 23:56:55.496804 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.496768 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qf9l5" Jul 06 23:56:55.547383 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.547349 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pcz8g" Jul 06 23:56:55.675285 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.673741 2568 generic.go:356] "Generic (PLEG): container finished" podID="63306a16-c22f-423d-bb78-aef7a70665d2" containerID="82fa3010d1eb113d9a2e3fc98cc8cfc7f62dc42a1a5fe067ab75d8bbd5bc38ad" exitCode=0 Jul 06 23:56:55.675285 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.674022 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" event={"ID":"63306a16-c22f-423d-bb78-aef7a70665d2","Type":"ContainerDied","Data":"82fa3010d1eb113d9a2e3fc98cc8cfc7f62dc42a1a5fe067ab75d8bbd5bc38ad"} Jul 06 23:56:55.719354 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.719324 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tv2d"] Jul 06 23:56:55.729828 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.729802 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qf9l5"] Jul 06 23:56:55.771277 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:55.771235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pcz8g"] Jul 06 23:56:55.773306 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:56:55.773279 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0f8f96_cc8e_43f2_9325_bd89a90b61bf.slice/crio-a073a1fd7be50c7056cf93b51a8968b727de866f34a8a463d7527c357e65c755 WatchSource:0}: Error finding container a073a1fd7be50c7056cf93b51a8968b727de866f34a8a463d7527c357e65c755: Status 404 returned error can't find the container with id a073a1fd7be50c7056cf93b51a8968b727de866f34a8a463d7527c357e65c755 Jul 06 23:56:56.310140 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:56.310109 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:56:56.313713 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:56.313683 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:56:56.314854 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:56.314832 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:56:56.314994 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:56.314833 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-czj4k\"" Jul 06 23:56:56.688979 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:56.688272 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" event={"ID":"63306a16-c22f-423d-bb78-aef7a70665d2","Type":"ContainerStarted","Data":"1bedce57991156358e4b2164ce84f2e8eeb8f959bdfd8dfddfcba212329a01f4"} Jul 06 23:56:56.691126 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:56.691047 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pcz8g" event={"ID":"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf","Type":"ContainerStarted","Data":"a073a1fd7be50c7056cf93b51a8968b727de866f34a8a463d7527c357e65c755"} Jul 06 23:56:56.692602 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:56.692523 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qf9l5" event={"ID":"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3","Type":"ContainerStarted","Data":"4f4fc138c438588fd29960fa42d2c8fbcde72610a8bd42058fa9ae4d7de3c647"} Jul 06 23:56:56.695595 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:56.695425 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tv2d" event={"ID":"556a6fe3-9d48-45db-b666-d8ca7f5bface","Type":"ContainerStarted","Data":"2603d43929ec6fa138987be4b041dd24d5bf6958c3bf85fc2d41ef81ed6fe738"} Jul 06 23:56:56.743616 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:56.743556 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ncwhk" podStartSLOduration=7.572054006 podStartE2EDuration="36.743532791s" totalImagesPullingTime="29.171478785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:56.741469161 +0000 UTC m=+37.954500909" watchObservedRunningTime="2026-07-06 23:56:56.743532791 +0000 UTC m=+37.956564535" Jul 06 23:56:57.329590 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.329551 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79854b6fd8-6mlnx"] Jul 06 23:56:57.344463 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.344300 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.349050 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.348729 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 23:56:57.349050 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.348827 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 23:56:57.349050 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.348728 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 23:56:57.351090 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.349389 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 23:56:57.351090 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.349615 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 23:56:57.351090 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.349825 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mswz6\"" Jul 06 23:56:57.351090 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.350031 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 23:56:57.355354 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.353533 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 23:56:57.360166 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.360101 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79854b6fd8-6mlnx"] Jul 06 23:56:57.368501 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.368476 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 23:56:57.449317 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.449278 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-oauth-serving-cert\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.449504 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.449333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-config\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.449504 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.449358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-trusted-ca-bundle\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.449504 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.449385 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5k9z\" (UniqueName: \"kubernetes.io/projected/05edae08-e6c7-4ac9-b6e9-87a64a78376a-kube-api-access-l5k9z\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.449504 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.449424 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-serving-cert\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.449504 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.449470 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-service-ca\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.449504 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.449502 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-oauth-config\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.550692 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.550650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-oauth-serving-cert\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.550692 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.550706 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-config\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.551091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.550736 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-trusted-ca-bundle\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.551091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.550761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k9z\" (UniqueName: \"kubernetes.io/projected/05edae08-e6c7-4ac9-b6e9-87a64a78376a-kube-api-access-l5k9z\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.551091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.550801 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-serving-cert\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.551091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.550827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-service-ca\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.551091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.550858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-oauth-config\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.552952 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.552873 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr"] Jul 06 23:56:57.559240 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.559201 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-oauth-serving-cert\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.559476 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.559248 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-config\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.559844 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.559822 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-service-ca\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.560210 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.560167 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-trusted-ca-bundle\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.564051 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.563996 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5k9z\" (UniqueName: \"kubernetes.io/projected/05edae08-e6c7-4ac9-b6e9-87a64a78376a-kube-api-access-l5k9z\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.567244 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.567215 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-oauth-config\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.571511 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.571424 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr"] Jul 06 23:56:57.571675 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.571614 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.575854 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.575037 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 23:56:57.575854 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.575312 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gsbsj\"" Jul 06 23:56:57.575854 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.575584 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 23:56:57.575854 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.575716 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 23:56:57.579461 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.579326 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-serving-cert\") pod \"console-79854b6fd8-6mlnx\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.579672 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.579526 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 23:56:57.579749 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.579668 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 23:56:57.651774 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.651736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec257cdb-b4fa-417a-b3fb-c6712391b321-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.651969 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.651784 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec257cdb-b4fa-417a-b3fb-c6712391b321-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.651969 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.651945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec257cdb-b4fa-417a-b3fb-c6712391b321-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.652074 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.651978 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-557cr\" (UniqueName: \"kubernetes.io/projected/ec257cdb-b4fa-417a-b3fb-c6712391b321-kube-api-access-557cr\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.659718 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.659685 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:56:57.753127 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.752783 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec257cdb-b4fa-417a-b3fb-c6712391b321-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.753575 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.753551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-557cr\" (UniqueName: \"kubernetes.io/projected/ec257cdb-b4fa-417a-b3fb-c6712391b321-kube-api-access-557cr\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.753717 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.753623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec257cdb-b4fa-417a-b3fb-c6712391b321-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.753717 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.753631 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec257cdb-b4fa-417a-b3fb-c6712391b321-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.753717 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.753651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec257cdb-b4fa-417a-b3fb-c6712391b321-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.753883 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:57.753784 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 06 23:56:57.753883 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:56:57.753842 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec257cdb-b4fa-417a-b3fb-c6712391b321-prometheus-operator-tls podName:ec257cdb-b4fa-417a-b3fb-c6712391b321 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:58.25382441 +0000 UTC m=+39.466856135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ec257cdb-b4fa-417a-b3fb-c6712391b321-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-h7pdr" (UID: "ec257cdb-b4fa-417a-b3fb-c6712391b321") : secret "prometheus-operator-tls" not found Jul 06 23:56:57.757494 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.757464 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec257cdb-b4fa-417a-b3fb-c6712391b321-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:57.765778 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:57.765730 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-557cr\" (UniqueName: \"kubernetes.io/projected/ec257cdb-b4fa-417a-b3fb-c6712391b321-kube-api-access-557cr\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:58.259303 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:58.259251 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec257cdb-b4fa-417a-b3fb-c6712391b321-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:58.262422 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:58.262392 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec257cdb-b4fa-417a-b3fb-c6712391b321-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-h7pdr\" (UID: \"ec257cdb-b4fa-417a-b3fb-c6712391b321\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:56:58.492933 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:56:58.492882 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" Jul 06 23:57:01.756670 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:01.756581 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr"] Jul 06 23:57:02.055597 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:02.055563 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec257cdb_b4fa_417a_b3fb_c6712391b321.slice/crio-c3fd9f4a24d924faab954c08900f7dbff0ac09c8038e5461d676903e959fc4ee WatchSource:0}: Error finding container c3fd9f4a24d924faab954c08900f7dbff0ac09c8038e5461d676903e959fc4ee: Status 404 returned error can't find the container with id c3fd9f4a24d924faab954c08900f7dbff0ac09c8038e5461d676903e959fc4ee Jul 06 23:57:02.202104 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:02.202074 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79854b6fd8-6mlnx"] Jul 06 23:57:02.205738 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:02.205708 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05edae08_e6c7_4ac9_b6e9_87a64a78376a.slice/crio-c4d764ef0953dd5b9b0992d8c2ae86318e422d87e600c42cdb1ef6e99c9a3fdd WatchSource:0}: Error finding container c4d764ef0953dd5b9b0992d8c2ae86318e422d87e600c42cdb1ef6e99c9a3fdd: Status 404 returned error can't find the container with id c4d764ef0953dd5b9b0992d8c2ae86318e422d87e600c42cdb1ef6e99c9a3fdd Jul 06 23:57:02.715635 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:02.715594 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qf9l5" event={"ID":"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3","Type":"ContainerStarted","Data":"b866e4657a3c5bc08b25a86d5fb1b89cc17199f523b1f3022f52b14add45034c"} Jul 06 23:57:02.715842 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:02.715663 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qf9l5" event={"ID":"4e1e414a-eea7-41d9-89a6-a7ca5825b7d3","Type":"ContainerStarted","Data":"1e1f4651c863d1dc274aac931f3a141cd4ce6519f2cb21b4d51b3769373e4504"} Jul 06 23:57:02.715842 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:02.715714 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qf9l5" Jul 06 23:57:02.717474 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:02.717421 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pcz8g" event={"ID":"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf","Type":"ContainerStarted","Data":"bfd0a03f8fc1514bf8d7145bf57d02af86ac165bc6c41a268630ec8441471427"} Jul 06 23:57:02.718819 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:02.718781 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79854b6fd8-6mlnx" event={"ID":"05edae08-e6c7-4ac9-b6e9-87a64a78376a","Type":"ContainerStarted","Data":"c4d764ef0953dd5b9b0992d8c2ae86318e422d87e600c42cdb1ef6e99c9a3fdd"} Jul 06 23:57:02.720378 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:02.720334 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tv2d" event={"ID":"556a6fe3-9d48-45db-b666-d8ca7f5bface","Type":"ContainerStarted","Data":"8e56ec5330069c627b0eb26355803056cb05be8228820b03c1ff19f50a6ab3c7"} Jul 06 23:57:02.721800 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:02.721776 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" event={"ID":"ec257cdb-b4fa-417a-b3fb-c6712391b321","Type":"ContainerStarted","Data":"c3fd9f4a24d924faab954c08900f7dbff0ac09c8038e5461d676903e959fc4ee"} Jul 06 23:57:02.745996 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:02.745945 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qf9l5" podStartSLOduration=1.430935371 podStartE2EDuration="7.745928624s" totalImagesPullingTime="6.314993253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:02.745584523 +0000 UTC m=+43.958616266" watchObservedRunningTime="2026-07-06 23:57:02.745928624 +0000 UTC m=+43.958960346" Jul 06 23:57:05.733136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:05.733105 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pcz8g" event={"ID":"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf","Type":"ContainerStarted","Data":"a0f3ef20a60d0586a824faa66a4d848254ed1fe6a76ac44324ad4aca5ee5b019"} Jul 06 23:57:05.734791 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:05.734762 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" event={"ID":"ec257cdb-b4fa-417a-b3fb-c6712391b321","Type":"ContainerStarted","Data":"9e163e2b0e5b0637f1d87bf0698df42179285591eda67e46fece9a9c25d51225"} Jul 06 23:57:05.734889 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:05.734805 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" event={"ID":"ec257cdb-b4fa-417a-b3fb-c6712391b321","Type":"ContainerStarted","Data":"b177628fe145f2b5a82f16498d7e1ad200f8327fb62f8c273ab105f632949e5c"} Jul 06 23:57:05.755415 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:05.755377 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9tv2d" podStartSLOduration=4.435269584 podStartE2EDuration="10.755359416s" totalImagesPullingTime="6.320089832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:02.785761689 +0000 UTC m=+43.998793434" watchObservedRunningTime="2026-07-06 23:57:05.755359416 +0000 UTC m=+46.968391154" Jul 06 23:57:05.755703 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:05.755681 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-h7pdr" podStartSLOduration=5.554531282 podStartE2EDuration="8.755671914s" totalImagesPullingTime="3.201140632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:05.753991576 +0000 UTC m=+46.967023339" watchObservedRunningTime="2026-07-06 23:57:05.755671914 +0000 UTC m=+46.968703651" Jul 06 23:57:06.740452 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:06.740395 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79854b6fd8-6mlnx" event={"ID":"05edae08-e6c7-4ac9-b6e9-87a64a78376a","Type":"ContainerStarted","Data":"af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee"} Jul 06 23:57:06.758034 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:06.757947 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79854b6fd8-6mlnx" podStartSLOduration=6.279907316 podStartE2EDuration="9.757926993s" totalImagesPullingTime="3.478019677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:06.757600365 +0000 UTC m=+47.970632111" watchObservedRunningTime="2026-07-06 23:57:06.757926993 +0000 UTC m=+47.970958737" Jul 06 23:57:07.660350 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.660309 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:57:07.660536 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.660379 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:57:07.661879 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.661841 2568 patch_prober.go:28] interesting pod/console-79854b6fd8-6mlnx container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 06 23:57:07.662005 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.661918 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-79854b6fd8-6mlnx" podUID="05edae08-e6c7-4ac9-b6e9-87a64a78376a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 06 23:57:07.903758 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.903724 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr"] Jul 06 23:57:07.955663 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.955591 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr"] Jul 06 23:57:07.955663 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.955601 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:07.961816 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.961778 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fzrh5\"" Jul 06 23:57:07.962837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.962807 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 23:57:07.962837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.962830 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 23:57:07.990684 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:07.990654 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-chnzn"] Jul 06 23:57:08.006251 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.006226 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.010684 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.010659 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sslvj\"" Jul 06 23:57:08.010949 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.010932 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 23:57:08.011172 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.011153 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 23:57:08.011270 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.011175 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 23:57:08.141757 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.141719 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df7fe433-77a1-4f93-b040-32de0cfd88fd-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.141757 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.141753 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5b1a4cf-4a57-46da-a6cb-691ec124627b-root\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.141942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.141788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.141942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.141859 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchpw\" (UniqueName: \"kubernetes.io/projected/d5b1a4cf-4a57-46da-a6cb-691ec124627b-kube-api-access-lchpw\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.141942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.141895 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df7fe433-77a1-4f93-b040-32de0cfd88fd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.141942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.141917 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-tls\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.141942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.141938 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df7fe433-77a1-4f93-b040-32de0cfd88fd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.142255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.141999 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-textfile\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.142255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.142053 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-wtmp\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.142255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.142071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5b1a4cf-4a57-46da-a6cb-691ec124627b-sys\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.142255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.142090 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvlk\" (UniqueName: \"kubernetes.io/projected/df7fe433-77a1-4f93-b040-32de0cfd88fd-kube-api-access-9lvlk\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.142255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.142130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b1a4cf-4a57-46da-a6cb-691ec124627b-metrics-client-ca\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.142255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.142166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-accelerators-collector-config\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.243558 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243423 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchpw\" (UniqueName: \"kubernetes.io/projected/d5b1a4cf-4a57-46da-a6cb-691ec124627b-kube-api-access-lchpw\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.243558 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243484 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df7fe433-77a1-4f93-b040-32de0cfd88fd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.243558 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-tls\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.243558 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243547 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df7fe433-77a1-4f93-b040-32de0cfd88fd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.243782 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-textfile\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.243782 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-wtmp\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.243782 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5b1a4cf-4a57-46da-a6cb-691ec124627b-sys\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.243925 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243808 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvlk\" (UniqueName: \"kubernetes.io/projected/df7fe433-77a1-4f93-b040-32de0cfd88fd-kube-api-access-9lvlk\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.243925 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b1a4cf-4a57-46da-a6cb-691ec124627b-metrics-client-ca\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.243925 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243895 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-accelerators-collector-config\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.244064 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243923 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-wtmp\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.244064 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.243999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df7fe433-77a1-4f93-b040-32de0cfd88fd-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.244064 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.244027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5b1a4cf-4a57-46da-a6cb-691ec124627b-root\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.244206 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.244068 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.244298 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.244271 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5b1a4cf-4a57-46da-a6cb-691ec124627b-sys\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.244861 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.244520 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-accelerators-collector-config\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.244861 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.244593 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-textfile\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.244861 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.244661 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5b1a4cf-4a57-46da-a6cb-691ec124627b-root\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.245149 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.245006 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b1a4cf-4a57-46da-a6cb-691ec124627b-metrics-client-ca\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.245204 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.245182 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df7fe433-77a1-4f93-b040-32de0cfd88fd-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.246959 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.246932 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df7fe433-77a1-4f93-b040-32de0cfd88fd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.247122 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.247098 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-tls\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.247374 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.247357 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df7fe433-77a1-4f93-b040-32de0cfd88fd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.247427 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.247370 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5b1a4cf-4a57-46da-a6cb-691ec124627b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.256145 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.256112 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchpw\" (UniqueName: \"kubernetes.io/projected/d5b1a4cf-4a57-46da-a6cb-691ec124627b-kube-api-access-lchpw\") pod \"node-exporter-chnzn\" (UID: \"d5b1a4cf-4a57-46da-a6cb-691ec124627b\") " pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.256663 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.256639 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvlk\" (UniqueName: \"kubernetes.io/projected/df7fe433-77a1-4f93-b040-32de0cfd88fd-kube-api-access-9lvlk\") pod \"openshift-state-metrics-54cf96dfcf-r6ztr\" (UID: \"df7fe433-77a1-4f93-b040-32de0cfd88fd\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.264301 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.264279 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" Jul 06 23:57:08.315289 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.315265 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chnzn" Jul 06 23:57:08.335091 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:08.335055 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b1a4cf_4a57_46da_a6cb_691ec124627b.slice/crio-b4ec50bcd0b92526a7fad8e7fe20d1867e9d59e900aa3036d5b3b9d109d28f21 WatchSource:0}: Error finding container b4ec50bcd0b92526a7fad8e7fe20d1867e9d59e900aa3036d5b3b9d109d28f21: Status 404 returned error can't find the container with id b4ec50bcd0b92526a7fad8e7fe20d1867e9d59e900aa3036d5b3b9d109d28f21 Jul 06 23:57:08.408983 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.408946 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr"] Jul 06 23:57:08.411800 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:08.411773 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7fe433_77a1_4f93_b040_32de0cfd88fd.slice/crio-ead12e7361aaa2e614cca43ad44c6021bbb5638241efeeb94119d558ca0e3553 WatchSource:0}: Error finding container ead12e7361aaa2e614cca43ad44c6021bbb5638241efeeb94119d558ca0e3553: Status 404 returned error can't find the container with id ead12e7361aaa2e614cca43ad44c6021bbb5638241efeeb94119d558ca0e3553 Jul 06 23:57:08.750098 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.750008 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pcz8g" event={"ID":"3d0f8f96-cc8e-43f2-9325-bd89a90b61bf","Type":"ContainerStarted","Data":"2a93bef05157d849e2578a6a6eb4e144ac039a5722dc5819b52871390332e318"} Jul 06 23:57:08.751266 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.751245 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chnzn" event={"ID":"d5b1a4cf-4a57-46da-a6cb-691ec124627b","Type":"ContainerStarted","Data":"b4ec50bcd0b92526a7fad8e7fe20d1867e9d59e900aa3036d5b3b9d109d28f21"} Jul 06 23:57:08.752819 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.752793 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" event={"ID":"df7fe433-77a1-4f93-b040-32de0cfd88fd","Type":"ContainerStarted","Data":"2e311b5620bb5ad46ba1a8ea2ecddc42de43b817e5bb50a704e35574abb1be4a"} Jul 06 23:57:08.752938 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.752843 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" event={"ID":"df7fe433-77a1-4f93-b040-32de0cfd88fd","Type":"ContainerStarted","Data":"26c18309fe1b3528d07eceb8579a073154de8313fea4049c7b1a9cb3c331e59d"} Jul 06 23:57:08.752938 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.752875 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" event={"ID":"df7fe433-77a1-4f93-b040-32de0cfd88fd","Type":"ContainerStarted","Data":"ead12e7361aaa2e614cca43ad44c6021bbb5638241efeeb94119d558ca0e3553"} Jul 06 23:57:08.772095 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.772045 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pcz8g" podStartSLOduration=1.612071434 podStartE2EDuration="13.77203074s" totalImagesPullingTime="12.159959306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:08.771266848 +0000 UTC m=+49.984298592" watchObservedRunningTime="2026-07-06 23:57:08.77203074 +0000 UTC m=+49.985062489" Jul 06 23:57:08.928526 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.928386 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 23:57:08.952015 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.951299 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 23:57:08.952015 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.951497 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:08.955232 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.955203 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 23:57:08.955232 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.955227 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 23:57:08.955473 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.955249 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zlb4l\"" Jul 06 23:57:08.955473 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.955282 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 23:57:08.955473 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.955213 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 23:57:08.955473 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.955227 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 23:57:08.955473 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.955283 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 23:57:08.955775 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.955744 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 23:57:08.956934 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.956903 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 23:57:08.956934 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:08.956931 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 23:57:09.051143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.050866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.050905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-config-out\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.050922 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.050941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-web-config\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.050960 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.050988 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.051013 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-config-volume\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.051097 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.051149 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051691 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.051167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051691 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.051186 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051691 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.051243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.051691 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.051265 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjqw\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-kube-api-access-fbjqw\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.152521 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152473 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.152681 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjqw\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-kube-api-access-fbjqw\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.152681 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152604 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.152681 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152663 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-config-out\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.152850 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152705 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.152850 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152735 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-web-config\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.152850 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.152850 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.152850 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152839 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-config-volume\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.153090 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.153090 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.153090 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.153090 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.152992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.155558 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:57:09.154555 2568 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 23:57:09.155558 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:57:09.154652 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-main-tls podName:6061f99d-d211-4b81-8327-eec0e7805340 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:09.654621588 +0000 UTC m=+50.867653330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6061f99d-d211-4b81-8327-eec0e7805340") : secret "alertmanager-main-tls" not found Jul 06 23:57:09.155558 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.155476 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.155803 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.155775 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.156980 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.156950 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.157152 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.157103 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.157233 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.157191 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-web-config\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.157302 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.157279 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.157589 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.157562 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.157704 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.157664 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.161029 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.159536 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.162774 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.162755 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-config-out\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.163634 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.163616 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-config-volume\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.170921 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.170899 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjqw\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-kube-api-access-fbjqw\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.658920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.658875 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.661638 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.661615 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:09.861155 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:09.861121 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:10.258827 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:10.258752 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 23:57:10.260720 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:10.260690 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6061f99d_d211_4b81_8327_eec0e7805340.slice/crio-45f945e1e350bebf4ce4f0a147531c6f6d5155629b797fd9524a220da78c2c8b WatchSource:0}: Error finding container 45f945e1e350bebf4ce4f0a147531c6f6d5155629b797fd9524a220da78c2c8b: Status 404 returned error can't find the container with id 45f945e1e350bebf4ce4f0a147531c6f6d5155629b797fd9524a220da78c2c8b Jul 06 23:57:10.762082 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:10.762021 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" event={"ID":"df7fe433-77a1-4f93-b040-32de0cfd88fd","Type":"ContainerStarted","Data":"a1d3e162aa987c84a5fafa446d6095d30254ca8833a5b8bbf886b65da39d5695"} Jul 06 23:57:10.763685 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:10.763653 2568 generic.go:356] "Generic (PLEG): container finished" podID="d5b1a4cf-4a57-46da-a6cb-691ec124627b" containerID="8d7d6f9c4dc02fce2802e3734362ed6a2ab9e6cac9e08bae6a8228ab497b3b30" exitCode=0 Jul 06 23:57:10.763823 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:10.763687 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chnzn" event={"ID":"d5b1a4cf-4a57-46da-a6cb-691ec124627b","Type":"ContainerDied","Data":"8d7d6f9c4dc02fce2802e3734362ed6a2ab9e6cac9e08bae6a8228ab497b3b30"} Jul 06 23:57:10.764874 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:10.764856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerStarted","Data":"45f945e1e350bebf4ce4f0a147531c6f6d5155629b797fd9524a220da78c2c8b"} Jul 06 23:57:10.784101 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:10.784053 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-r6ztr" podStartSLOduration=2.2837843749999998 podStartE2EDuration="3.784039019s" totalImagesPullingTime="1.500254644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:10.783600801 +0000 UTC m=+51.996632545" watchObservedRunningTime="2026-07-06 23:57:10.784039019 +0000 UTC m=+51.997070762" Jul 06 23:57:11.771863 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:11.771815 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chnzn" event={"ID":"d5b1a4cf-4a57-46da-a6cb-691ec124627b","Type":"ContainerStarted","Data":"729036862b3a7b5f30540d7200e4c5d4bb08dea72982b141beb4653c1d1c5fa8"} Jul 06 23:57:11.772413 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:11.771881 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chnzn" event={"ID":"d5b1a4cf-4a57-46da-a6cb-691ec124627b","Type":"ContainerStarted","Data":"0c66b28dfcbb6607d24335069ebbcc6a31cecd1d19cca5e4a23bfc07b74edbcc"} Jul 06 23:57:11.795383 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:11.795331 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-chnzn" podStartSLOduration=2.93462945 podStartE2EDuration="4.795315639s" totalImagesPullingTime="1.772461281s" totalInitContainerRuntime="88.224908ms" isStatefulPod=true podCreationTimestamp="2026-07-06 23:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:11.794759786 +0000 UTC m=+53.007791529" watchObservedRunningTime="2026-07-06 23:57:11.795315639 +0000 UTC m=+53.008347382" Jul 06 23:57:12.726871 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.726844 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qf9l5" Jul 06 23:57:12.765383 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.765351 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g"] Jul 06 23:57:12.767293 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.767278 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:12.770775 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.770757 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 23:57:12.770901 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.770876 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wmknt\"" Jul 06 23:57:12.778092 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.778064 2568 generic.go:356] "Generic (PLEG): container finished" podID="6061f99d-d211-4b81-8327-eec0e7805340" containerID="51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac" exitCode=0 Jul 06 23:57:12.780534 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.779465 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerDied","Data":"51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac"} Jul 06 23:57:12.787913 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.787890 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7wgz\" (UniqueName: \"kubernetes.io/projected/1fe23294-e2eb-49ff-92fd-ae27f1cbedac-kube-api-access-p7wgz\") pod \"monitoring-plugin-d5dcff7cf-wjk6g\" (UID: \"1fe23294-e2eb-49ff-92fd-ae27f1cbedac\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:12.788020 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.787963 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe23294-e2eb-49ff-92fd-ae27f1cbedac-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wjk6g\" (UID: \"1fe23294-e2eb-49ff-92fd-ae27f1cbedac\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:12.790367 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.790344 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g"] Jul 06 23:57:12.888597 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.888559 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7wgz\" (UniqueName: \"kubernetes.io/projected/1fe23294-e2eb-49ff-92fd-ae27f1cbedac-kube-api-access-p7wgz\") pod \"monitoring-plugin-d5dcff7cf-wjk6g\" (UID: \"1fe23294-e2eb-49ff-92fd-ae27f1cbedac\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:12.888792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.888657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe23294-e2eb-49ff-92fd-ae27f1cbedac-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wjk6g\" (UID: \"1fe23294-e2eb-49ff-92fd-ae27f1cbedac\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:12.888859 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:57:12.888784 2568 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 23:57:12.888859 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:57:12.888853 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe23294-e2eb-49ff-92fd-ae27f1cbedac-monitoring-plugin-cert podName:1fe23294-e2eb-49ff-92fd-ae27f1cbedac nodeName:}" failed. No retries permitted until 2026-07-06 23:57:13.388832582 +0000 UTC m=+54.601864319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe23294-e2eb-49ff-92fd-ae27f1cbedac-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-wjk6g" (UID: "1fe23294-e2eb-49ff-92fd-ae27f1cbedac") : secret "monitoring-plugin-cert" not found Jul 06 23:57:12.903985 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.903953 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7wgz\" (UniqueName: \"kubernetes.io/projected/1fe23294-e2eb-49ff-92fd-ae27f1cbedac-kube-api-access-p7wgz\") pod \"monitoring-plugin-d5dcff7cf-wjk6g\" (UID: \"1fe23294-e2eb-49ff-92fd-ae27f1cbedac\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:12.914027 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.913991 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79854b6fd8-6mlnx"] Jul 06 23:57:12.981264 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.981187 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c49c858b8-whsj7"] Jul 06 23:57:12.983444 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:12.983418 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.013210 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.013172 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c49c858b8-whsj7"] Jul 06 23:57:13.089993 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.089953 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-oauth-config\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.089993 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.089994 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-trusted-ca-bundle\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.090257 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.090121 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-config\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.090257 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.090170 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpr4\" (UniqueName: \"kubernetes.io/projected/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-kube-api-access-4mpr4\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.090257 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.090229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-service-ca\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.090401 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.090255 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-serving-cert\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.090401 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.090315 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-oauth-serving-cert\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.191168 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.191129 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpr4\" (UniqueName: \"kubernetes.io/projected/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-kube-api-access-4mpr4\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.191168 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.191177 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-service-ca\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.191466 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.191196 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-serving-cert\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.191466 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.191236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-oauth-serving-cert\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.191466 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.191267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-oauth-config\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.191466 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.191292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-trusted-ca-bundle\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.191656 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.191502 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-config\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.192179 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.191990 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-service-ca\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.192179 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.192040 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-oauth-serving-cert\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.192179 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.192109 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-config\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.192179 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.192142 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-trusted-ca-bundle\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.193981 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.193953 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-oauth-config\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.194069 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.193957 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-serving-cert\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.203835 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.203807 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpr4\" (UniqueName: \"kubernetes.io/projected/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-kube-api-access-4mpr4\") pod \"console-7c49c858b8-whsj7\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.292030 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.291933 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:13.394598 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.394213 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe23294-e2eb-49ff-92fd-ae27f1cbedac-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wjk6g\" (UID: \"1fe23294-e2eb-49ff-92fd-ae27f1cbedac\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:13.397346 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.397318 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe23294-e2eb-49ff-92fd-ae27f1cbedac-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wjk6g\" (UID: \"1fe23294-e2eb-49ff-92fd-ae27f1cbedac\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:13.432648 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.432472 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c49c858b8-whsj7"] Jul 06 23:57:13.435692 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:13.435661 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2057ad8f_ffdf_46a1_ab60_00e4abf92aea.slice/crio-5c7924fc52f7dade1b40a8866904a14daa231dfdbfc697cd8a95cf64efa91815 WatchSource:0}: Error finding container 5c7924fc52f7dade1b40a8866904a14daa231dfdbfc697cd8a95cf64efa91815: Status 404 returned error can't find the container with id 5c7924fc52f7dade1b40a8866904a14daa231dfdbfc697cd8a95cf64efa91815 Jul 06 23:57:13.676698 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.676660 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:13.785345 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.785307 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c49c858b8-whsj7" event={"ID":"2057ad8f-ffdf-46a1-ab60-00e4abf92aea","Type":"ContainerStarted","Data":"92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049"} Jul 06 23:57:13.785723 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.785374 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c49c858b8-whsj7" event={"ID":"2057ad8f-ffdf-46a1-ab60-00e4abf92aea","Type":"ContainerStarted","Data":"5c7924fc52f7dade1b40a8866904a14daa231dfdbfc697cd8a95cf64efa91815"} Jul 06 23:57:13.811225 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.811175 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c49c858b8-whsj7" podStartSLOduration=1.811157837 podStartE2EDuration="1.811157837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:13.809735527 +0000 UTC m=+55.022767272" watchObservedRunningTime="2026-07-06 23:57:13.811157837 +0000 UTC m=+55.024189580" Jul 06 23:57:13.822187 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:13.822152 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g"] Jul 06 23:57:13.826598 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:13.826561 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe23294_e2eb_49ff_92fd_ae27f1cbedac.slice/crio-3bd97a4d55ea38d040f18febe6693a2b6f36f8d5b75e687570976fa2e5d6d3bb WatchSource:0}: Error finding container 3bd97a4d55ea38d040f18febe6693a2b6f36f8d5b75e687570976fa2e5d6d3bb: Status 404 returned error can't find the container with id 3bd97a4d55ea38d040f18febe6693a2b6f36f8d5b75e687570976fa2e5d6d3bb Jul 06 23:57:14.184195 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.184154 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:57:14.192269 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.192238 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.202624 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.202587 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 23:57:14.202769 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.202636 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 23:57:14.203536 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.203513 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 23:57:14.204224 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.204205 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 23:57:14.204350 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.204230 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 23:57:14.204350 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.204235 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 23:57:14.206559 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.206373 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 23:57:14.207537 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.207360 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 23:57:14.208920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.208470 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 23:57:14.208920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.208564 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 23:57:14.208920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.208577 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 23:57:14.208920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.208720 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fm64k\"" Jul 06 23:57:14.208920 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.208823 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ekrb9d21bcba4\"" Jul 06 23:57:14.209649 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.209631 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 23:57:14.212567 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.212545 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 23:57:14.213748 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.213729 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 23:57:14.218569 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.218546 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 23:57:14.223780 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.223758 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:57:14.302747 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.302707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.302942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.302758 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.302942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.302795 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.302942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.302819 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.302942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.302851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.302942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.302880 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.302942 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.302937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-web-config\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.302981 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9tw\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-kube-api-access-hf9tw\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303009 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303036 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303092 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303127 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303144 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303491 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303491 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303214 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303491 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303236 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config-out\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.303491 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.303251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404224 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404244 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404285 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-web-config\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404328 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9tw\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-kube-api-access-hf9tw\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404380 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404464 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404488 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404520 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404598 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404616 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.406151 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config-out\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.407136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404663 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.407136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.407136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.404711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.407136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.405610 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.407136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.405735 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.407136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.405811 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.407136 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.406961 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.407517 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.407324 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.409948 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.409922 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.410544 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.410518 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.411233 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.411208 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.412862 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.412831 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.413182 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.413156 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.413660 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.413632 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config-out\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.414371 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.414321 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.414489 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.414409 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-web-config\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.415592 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.414748 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.415592 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.415016 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.415592 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.415553 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.416291 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.416266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.416291 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.416275 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.420143 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.420119 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.423264 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.423242 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9tw\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-kube-api-access-hf9tw\") pod \"prometheus-k8s-0\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.505293 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.505178 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:14.716547 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.716396 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:57:14.797800 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.797763 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerStarted","Data":"bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e"} Jul 06 23:57:14.798181 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.797853 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerStarted","Data":"ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf"} Jul 06 23:57:14.798181 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.797896 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerStarted","Data":"9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705"} Jul 06 23:57:14.800620 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.800588 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerStarted","Data":"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052"} Jul 06 23:57:14.800737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.800669 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerStarted","Data":"aa1caf25355f212c8b3ae7e809f753ea7f8c18201791aa441e75e9ce98557367"} Jul 06 23:57:14.802283 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:14.802238 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" event={"ID":"1fe23294-e2eb-49ff-92fd-ae27f1cbedac","Type":"ContainerStarted","Data":"3bd97a4d55ea38d040f18febe6693a2b6f36f8d5b75e687570976fa2e5d6d3bb"} Jul 06 23:57:15.814302 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:15.814260 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerStarted","Data":"06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0"} Jul 06 23:57:15.814866 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:15.814387 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerStarted","Data":"ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2"} Jul 06 23:57:15.816024 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:15.815997 2568 generic.go:356] "Generic (PLEG): container finished" podID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerID="3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052" exitCode=0 Jul 06 23:57:15.816160 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:15.816081 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerDied","Data":"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052"} Jul 06 23:57:15.817874 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:15.817845 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" event={"ID":"1fe23294-e2eb-49ff-92fd-ae27f1cbedac","Type":"ContainerStarted","Data":"0dc0212319a989c0fc0cfe613d5fdb9650d92bb3a217bb71f49667b03b0fb9bb"} Jul 06 23:57:15.818293 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:15.818275 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:15.823818 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:15.823793 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" Jul 06 23:57:15.883379 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:15.883331 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wjk6g" podStartSLOduration=2.216584897 podStartE2EDuration="3.883313762s" totalImagesPullingTime="1.666728865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:15.88192633 +0000 UTC m=+57.094958077" watchObservedRunningTime="2026-07-06 23:57:15.883313762 +0000 UTC m=+57.096345505" Jul 06 23:57:16.825994 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:16.825898 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerStarted","Data":"263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec"} Jul 06 23:57:19.843037 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:19.843007 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerStarted","Data":"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d"} Jul 06 23:57:19.843389 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:19.843070 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerStarted","Data":"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8"} Jul 06 23:57:20.665015 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:20.664985 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvndj" Jul 06 23:57:20.729224 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:20.729174 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.398514754 podStartE2EDuration="12.729157076s" totalImagesPullingTime="4.311749774s" totalInitContainerRuntime="18.892548ms" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:16.862497714 +0000 UTC m=+58.075529463" watchObservedRunningTime="2026-07-06 23:57:20.729157076 +0000 UTC m=+61.942188859" Jul 06 23:57:21.855019 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:21.854924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerStarted","Data":"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763"} Jul 06 23:57:21.855019 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:21.854985 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerStarted","Data":"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa"} Jul 06 23:57:21.855019 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:21.855012 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerStarted","Data":"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c"} Jul 06 23:57:21.855572 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:21.855037 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerStarted","Data":"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf"} Jul 06 23:57:21.889466 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:21.889392 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.374832606 podStartE2EDuration="7.889374137s" totalImagesPullingTime="5.489672605s" totalInitContainerRuntime="24.868926ms" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:21.887701692 +0000 UTC m=+63.100733488" watchObservedRunningTime="2026-07-06 23:57:21.889374137 +0000 UTC m=+63.102405882" Jul 06 23:57:23.293024 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:23.292987 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:23.293402 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:23.293086 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:23.297903 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:23.297879 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:23.870822 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:23.870795 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:57:24.505644 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:24.505601 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:25.109749 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.109718 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:57:25.109925 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.109758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:57:25.112664 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.112643 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:57:25.112664 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.112654 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:57:25.122204 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.122178 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faeb3a9-7522-4d7a-a47f-8e65ad2809b1-metrics-certs\") pod \"network-metrics-daemon-gdjjf\" (UID: \"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1\") " pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:57:25.122299 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.122216 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7dcfee5-cbb6-40da-9c87-6eb081c7d885-original-pull-secret\") pod \"global-pull-secret-syncer-v592j\" (UID: \"f7dcfee5-cbb6-40da-9c87-6eb081c7d885\") " pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:57:25.210855 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.210818 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:57:25.213878 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.213857 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:57:25.224819 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.224796 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:57:25.234846 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.234827 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzt6\" (UniqueName: \"kubernetes.io/projected/9aafc90b-4ee0-4eda-afa3-ecb0faa2110c-kube-api-access-zdzt6\") pod \"network-check-target-tdphw\" (UID: \"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c\") " pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:57:25.319934 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.319906 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v592j" Jul 06 23:57:25.327105 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.327085 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dplr9\"" Jul 06 23:57:25.334375 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.334359 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdjjf" Jul 06 23:57:25.425002 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.424972 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-czj4k\"" Jul 06 23:57:25.432681 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.432621 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:57:25.497020 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.496988 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v592j"] Jul 06 23:57:25.500620 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:25.500537 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dcfee5_cbb6_40da_9c87_6eb081c7d885.slice/crio-64edf01dce1610f44d79fe480245f03a14658c7e0fad5531e968b5c72e982d1f WatchSource:0}: Error finding container 64edf01dce1610f44d79fe480245f03a14658c7e0fad5531e968b5c72e982d1f: Status 404 returned error can't find the container with id 64edf01dce1610f44d79fe480245f03a14658c7e0fad5531e968b5c72e982d1f Jul 06 23:57:25.537475 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.535001 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gdjjf"] Jul 06 23:57:25.537475 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:25.535906 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faeb3a9_7522_4d7a_a47f_8e65ad2809b1.slice/crio-06d47f003c689a04090abf5c62483afbe3bca094ddfd3570457cd43354c00ced WatchSource:0}: Error finding container 06d47f003c689a04090abf5c62483afbe3bca094ddfd3570457cd43354c00ced: Status 404 returned error can't find the container with id 06d47f003c689a04090abf5c62483afbe3bca094ddfd3570457cd43354c00ced Jul 06 23:57:25.616645 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.616618 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tdphw"] Jul 06 23:57:25.618647 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:57:25.618620 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aafc90b_4ee0_4eda_afa3_ecb0faa2110c.slice/crio-d092e02c4eded8462a2706d02384a78d280dbfae098449f5b3a361087cc1ed77 WatchSource:0}: Error finding container d092e02c4eded8462a2706d02384a78d280dbfae098449f5b3a361087cc1ed77: Status 404 returned error can't find the container with id d092e02c4eded8462a2706d02384a78d280dbfae098449f5b3a361087cc1ed77 Jul 06 23:57:25.876400 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.876311 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tdphw" event={"ID":"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c","Type":"ContainerStarted","Data":"d092e02c4eded8462a2706d02384a78d280dbfae098449f5b3a361087cc1ed77"} Jul 06 23:57:25.877466 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.877417 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdjjf" event={"ID":"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1","Type":"ContainerStarted","Data":"06d47f003c689a04090abf5c62483afbe3bca094ddfd3570457cd43354c00ced"} Jul 06 23:57:25.878511 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:25.878487 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v592j" event={"ID":"f7dcfee5-cbb6-40da-9c87-6eb081c7d885","Type":"ContainerStarted","Data":"64edf01dce1610f44d79fe480245f03a14658c7e0fad5531e968b5c72e982d1f"} Jul 06 23:57:26.887366 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:26.887285 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdjjf" event={"ID":"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1","Type":"ContainerStarted","Data":"fcef0b176063ba3dcd7f88ce553f4470ef845c7500eafaca5c806e9646e4b7db"} Jul 06 23:57:27.895336 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:27.895257 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdjjf" event={"ID":"5faeb3a9-7522-4d7a-a47f-8e65ad2809b1","Type":"ContainerStarted","Data":"30eeb8dcb37bdf289d6cff4c0365019e2d952ec45344b01ed2d0da8c5cfdb605"} Jul 06 23:57:27.915262 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:27.915148 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gdjjf" podStartSLOduration=66.785580908 podStartE2EDuration="1m7.91512988s" totalImagesPullingTime="1.129548972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:27.91249042 +0000 UTC m=+69.125522165" watchObservedRunningTime="2026-07-06 23:57:27.91512988 +0000 UTC m=+69.128161625" Jul 06 23:57:30.910582 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:30.910544 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tdphw" event={"ID":"9aafc90b-4ee0-4eda-afa3-ecb0faa2110c","Type":"ContainerStarted","Data":"301390bc33e1ed39563546e5747a635bb64021cdd36fb280a7eb31f606992190"} Jul 06 23:57:30.911037 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:30.910663 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:57:30.912003 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:30.911976 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v592j" event={"ID":"f7dcfee5-cbb6-40da-9c87-6eb081c7d885","Type":"ContainerStarted","Data":"8d240d491741e828bac3104a27b9f5a516e1e82ed97075d05cc01c900c800113"} Jul 06 23:57:30.927525 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:30.927478 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tdphw" podStartSLOduration=66.412941989 podStartE2EDuration="1m10.927463982s" totalImagesPullingTime="4.514521993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:30.926224582 +0000 UTC m=+72.139256347" watchObservedRunningTime="2026-07-06 23:57:30.927463982 +0000 UTC m=+72.140495727" Jul 06 23:57:30.941937 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:30.941884 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v592j" podStartSLOduration=66.306191507 podStartE2EDuration="1m10.941854774s" totalImagesPullingTime="4.635663267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:30.941810453 +0000 UTC m=+72.154842198" watchObservedRunningTime="2026-07-06 23:57:30.941854774 +0000 UTC m=+72.154886513" Jul 06 23:57:37.935095 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:37.935025 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-79854b6fd8-6mlnx" podUID="05edae08-e6c7-4ac9-b6e9-87a64a78376a" containerName="console" containerID="cri-o://af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee" gracePeriod=15 Jul 06 23:57:38.201724 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.201652 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:57:38.333739 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.333703 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-config\") pod \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " Jul 06 23:57:38.333739 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.333746 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-serving-cert\") pod \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " Jul 06 23:57:38.333991 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.333772 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-oauth-serving-cert\") pod \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " Jul 06 23:57:38.333991 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.333811 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-service-ca\") pod \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " Jul 06 23:57:38.333991 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.333854 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-trusted-ca-bundle\") pod \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " Jul 06 23:57:38.333991 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.333872 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-oauth-config\") pod \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " Jul 06 23:57:38.333991 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.333930 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5k9z\" (UniqueName: \"kubernetes.io/projected/05edae08-e6c7-4ac9-b6e9-87a64a78376a-kube-api-access-l5k9z\") pod \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\" (UID: \"05edae08-e6c7-4ac9-b6e9-87a64a78376a\") " Jul 06 23:57:38.334298 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.334228 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-config" (OuterVolumeSpecName: "console-config") pod "05edae08-e6c7-4ac9-b6e9-87a64a78376a" (UID: "05edae08-e6c7-4ac9-b6e9-87a64a78376a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:57:38.334298 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.334252 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "05edae08-e6c7-4ac9-b6e9-87a64a78376a" (UID: "05edae08-e6c7-4ac9-b6e9-87a64a78376a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:57:38.334509 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.334308 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "05edae08-e6c7-4ac9-b6e9-87a64a78376a" (UID: "05edae08-e6c7-4ac9-b6e9-87a64a78376a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:57:38.334509 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.334345 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-service-ca" (OuterVolumeSpecName: "service-ca") pod "05edae08-e6c7-4ac9-b6e9-87a64a78376a" (UID: "05edae08-e6c7-4ac9-b6e9-87a64a78376a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:57:38.336378 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.336345 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "05edae08-e6c7-4ac9-b6e9-87a64a78376a" (UID: "05edae08-e6c7-4ac9-b6e9-87a64a78376a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:38.336515 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.336421 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05edae08-e6c7-4ac9-b6e9-87a64a78376a-kube-api-access-l5k9z" (OuterVolumeSpecName: "kube-api-access-l5k9z") pod "05edae08-e6c7-4ac9-b6e9-87a64a78376a" (UID: "05edae08-e6c7-4ac9-b6e9-87a64a78376a"). InnerVolumeSpecName "kube-api-access-l5k9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:57:38.336515 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.336421 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "05edae08-e6c7-4ac9-b6e9-87a64a78376a" (UID: "05edae08-e6c7-4ac9-b6e9-87a64a78376a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:38.434876 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.434843 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5k9z\" (UniqueName: \"kubernetes.io/projected/05edae08-e6c7-4ac9-b6e9-87a64a78376a-kube-api-access-l5k9z\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:38.434876 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.434870 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:38.434876 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.434879 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-serving-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:38.434876 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.434887 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-oauth-serving-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:38.435129 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.434896 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:38.435129 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.434903 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05edae08-e6c7-4ac9-b6e9-87a64a78376a-trusted-ca-bundle\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:38.435129 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.434912 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05edae08-e6c7-4ac9-b6e9-87a64a78376a-console-oauth-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:38.946117 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.946082 2568 generic.go:356] "Generic (PLEG): container finished" podID="05edae08-e6c7-4ac9-b6e9-87a64a78376a" containerID="af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee" exitCode=0 Jul 06 23:57:38.946787 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.946135 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79854b6fd8-6mlnx" Jul 06 23:57:38.946787 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.946165 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79854b6fd8-6mlnx" event={"ID":"05edae08-e6c7-4ac9-b6e9-87a64a78376a","Type":"ContainerDied","Data":"af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee"} Jul 06 23:57:38.946787 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.946214 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79854b6fd8-6mlnx" event={"ID":"05edae08-e6c7-4ac9-b6e9-87a64a78376a","Type":"ContainerDied","Data":"c4d764ef0953dd5b9b0992d8c2ae86318e422d87e600c42cdb1ef6e99c9a3fdd"} Jul 06 23:57:38.946787 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.946233 2568 scope.go:117] "RemoveContainer" containerID="af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee" Jul 06 23:57:38.961309 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.961292 2568 scope.go:117] "RemoveContainer" containerID="af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee" Jul 06 23:57:38.961613 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:57:38.961593 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee\": container with ID starting with af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee not found: ID does not exist" containerID="af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee" Jul 06 23:57:38.961669 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.961620 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee"} err="failed to get container status \"af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee\": rpc error: code = NotFound desc = could not find container \"af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee\": container with ID starting with af0882ff5274478dd1e54bcea664ec35f80a56b3f75265031cb62f4714ec02ee not found: ID does not exist" Jul 06 23:57:38.969127 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.969104 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79854b6fd8-6mlnx"] Jul 06 23:57:38.973912 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:38.973891 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79854b6fd8-6mlnx"] Jul 06 23:57:39.314417 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:39.314338 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05edae08-e6c7-4ac9-b6e9-87a64a78376a" path="/var/lib/kubelet/pods/05edae08-e6c7-4ac9-b6e9-87a64a78376a/volumes" Jul 06 23:57:48.680660 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:48.680621 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:48.699817 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:48.699784 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:49.003021 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:49.002927 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:57:58.328648 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:58.328613 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 23:57:58.329017 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:58.328944 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="alertmanager" containerID="cri-o://9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705" gracePeriod=120 Jul 06 23:57:58.329091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:58.329026 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy-metric" containerID="cri-o://06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0" gracePeriod=120 Jul 06 23:57:58.329091 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:58.329047 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy-web" containerID="cri-o://bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e" gracePeriod=120 Jul 06 23:57:58.329193 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:58.329086 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy" containerID="cri-o://ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2" gracePeriod=120 Jul 06 23:57:58.329193 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:58.329116 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="config-reloader" containerID="cri-o://ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf" gracePeriod=120 Jul 06 23:57:58.329289 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:58.329255 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="prom-label-proxy" containerID="cri-o://263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec" gracePeriod=120 Jul 06 23:57:59.034580 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034546 2568 generic.go:356] "Generic (PLEG): container finished" podID="6061f99d-d211-4b81-8327-eec0e7805340" containerID="263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec" exitCode=0 Jul 06 23:57:59.034580 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034574 2568 generic.go:356] "Generic (PLEG): container finished" podID="6061f99d-d211-4b81-8327-eec0e7805340" containerID="06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0" exitCode=0 Jul 06 23:57:59.034580 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034584 2568 generic.go:356] "Generic (PLEG): container finished" podID="6061f99d-d211-4b81-8327-eec0e7805340" containerID="ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2" exitCode=0 Jul 06 23:57:59.034580 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034579 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerDied","Data":"263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec"} Jul 06 23:57:59.034855 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034656 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerDied","Data":"06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0"} Jul 06 23:57:59.034855 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034594 2568 generic.go:356] "Generic (PLEG): container finished" podID="6061f99d-d211-4b81-8327-eec0e7805340" containerID="ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf" exitCode=0 Jul 06 23:57:59.034855 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034673 2568 generic.go:356] "Generic (PLEG): container finished" podID="6061f99d-d211-4b81-8327-eec0e7805340" containerID="9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705" exitCode=0 Jul 06 23:57:59.034855 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034686 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerDied","Data":"ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2"} Jul 06 23:57:59.034855 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerDied","Data":"ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf"} Jul 06 23:57:59.034855 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.034735 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerDied","Data":"9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705"} Jul 06 23:57:59.408694 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:57:59.408663 2568 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6061f99d_d211_4b81_8327_eec0e7805340.slice/crio-bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e.scope\": RecentStats: unable to find data in memory cache]" Jul 06 23:57:59.602886 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.602858 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:57:59.713865 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.713831 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714030 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.713881 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-config-out\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714030 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.713909 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-main-tls\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714030 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.713944 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-cluster-tls-config\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714030 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.713986 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-metrics-client-ca\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714216 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714173 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-config-volume\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714270 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714224 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbjqw\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-kube-api-access-fbjqw\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714321 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714280 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-trusted-ca-bundle\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714375 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714319 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-web\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714375 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714324 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:57:59.714375 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714345 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-tls-assets\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714579 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714386 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714579 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714420 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-main-db\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714579 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714475 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-web-config\") pod \"6061f99d-d211-4b81-8327-eec0e7805340\" (UID: \"6061f99d-d211-4b81-8327-eec0e7805340\") " Jul 06 23:57:59.714981 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.714732 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-metrics-client-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.715588 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.715555 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:57:59.716535 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.716469 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:57:59.717661 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.717620 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:59.718778 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.718749 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:59.718971 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.718758 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-kube-api-access-fbjqw" (OuterVolumeSpecName: "kube-api-access-fbjqw") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "kube-api-access-fbjqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:57:59.719323 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.719104 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-config-out" (OuterVolumeSpecName: "config-out") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:57:59.719323 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.719168 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:57:59.719323 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.719199 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:59.719749 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.719724 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:59.720087 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.720053 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-config-volume" (OuterVolumeSpecName: "config-volume") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:59.723010 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.722975 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:59.729737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.729582 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-web-config" (OuterVolumeSpecName: "web-config") pod "6061f99d-d211-4b81-8327-eec0e7805340" (UID: "6061f99d-d211-4b81-8327-eec0e7805340"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:59.816011 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.815975 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816011 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816007 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-main-db\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816011 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816017 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-web-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816027 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816037 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6061f99d-d211-4b81-8327-eec0e7805340-config-out\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816045 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-main-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816054 2568 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-cluster-tls-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816062 2568 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-config-volume\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816071 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjqw\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-kube-api-access-fbjqw\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816078 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6061f99d-d211-4b81-8327-eec0e7805340-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816087 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6061f99d-d211-4b81-8327-eec0e7805340-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:57:59.816226 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:57:59.816095 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6061f99d-d211-4b81-8327-eec0e7805340-tls-assets\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:00.043284 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.043202 2568 generic.go:356] "Generic (PLEG): container finished" podID="6061f99d-d211-4b81-8327-eec0e7805340" containerID="bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e" exitCode=0 Jul 06 23:58:00.043427 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.043283 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.043427 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.043290 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerDied","Data":"bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e"} Jul 06 23:58:00.043427 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.043349 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6061f99d-d211-4b81-8327-eec0e7805340","Type":"ContainerDied","Data":"45f945e1e350bebf4ce4f0a147531c6f6d5155629b797fd9524a220da78c2c8b"} Jul 06 23:58:00.043427 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.043381 2568 scope.go:117] "RemoveContainer" containerID="263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec" Jul 06 23:58:00.052264 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.052244 2568 scope.go:117] "RemoveContainer" containerID="06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0" Jul 06 23:58:00.061277 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.061147 2568 scope.go:117] "RemoveContainer" containerID="ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2" Jul 06 23:58:00.068954 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.068936 2568 scope.go:117] "RemoveContainer" containerID="bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e" Jul 06 23:58:00.075090 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.075063 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 23:58:00.076358 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.076343 2568 scope.go:117] "RemoveContainer" containerID="ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf" Jul 06 23:58:00.080272 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.080234 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 23:58:00.083663 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.083641 2568 scope.go:117] "RemoveContainer" containerID="9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705" Jul 06 23:58:00.090357 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.090341 2568 scope.go:117] "RemoveContainer" containerID="51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac" Jul 06 23:58:00.097102 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.097082 2568 scope.go:117] "RemoveContainer" containerID="263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec" Jul 06 23:58:00.097378 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:00.097359 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec\": container with ID starting with 263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec not found: ID does not exist" containerID="263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec" Jul 06 23:58:00.097481 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.097393 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec"} err="failed to get container status \"263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec\": rpc error: code = NotFound desc = could not find container \"263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec\": container with ID starting with 263a051a91c31faac41d956a2ade7c34b92190563197d5f4f0dcec55d88fffec not found: ID does not exist" Jul 06 23:58:00.097481 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.097418 2568 scope.go:117] "RemoveContainer" containerID="06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0" Jul 06 23:58:00.097707 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:00.097687 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0\": container with ID starting with 06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0 not found: ID does not exist" containerID="06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0" Jul 06 23:58:00.097749 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.097716 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0"} err="failed to get container status \"06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0\": rpc error: code = NotFound desc = could not find container \"06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0\": container with ID starting with 06c452e0ff787e4705473b1a14552ff8de3a12cd0669ae883226b353e7af66c0 not found: ID does not exist" Jul 06 23:58:00.097749 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.097732 2568 scope.go:117] "RemoveContainer" containerID="ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2" Jul 06 23:58:00.097943 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:00.097926 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2\": container with ID starting with ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2 not found: ID does not exist" containerID="ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2" Jul 06 23:58:00.098002 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.097951 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2"} err="failed to get container status \"ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2\": rpc error: code = NotFound desc = could not find container \"ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2\": container with ID starting with ecb13cae2ec8c89954ad434a4dac7cec7908ade0a9ecffb0cf7cb7d2fef7aae2 not found: ID does not exist" Jul 06 23:58:00.098002 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.097971 2568 scope.go:117] "RemoveContainer" containerID="bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e" Jul 06 23:58:00.098200 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:00.098184 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e\": container with ID starting with bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e not found: ID does not exist" containerID="bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e" Jul 06 23:58:00.098233 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.098214 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e"} err="failed to get container status \"bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e\": rpc error: code = NotFound desc = could not find container \"bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e\": container with ID starting with bcac779bab51ab23ac8310e09c0f3772919112e6a68022fe28c4ac0326115c6e not found: ID does not exist" Jul 06 23:58:00.098233 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.098228 2568 scope.go:117] "RemoveContainer" containerID="ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf" Jul 06 23:58:00.098450 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:00.098415 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf\": container with ID starting with ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf not found: ID does not exist" containerID="ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf" Jul 06 23:58:00.098533 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.098518 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf"} err="failed to get container status \"ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf\": rpc error: code = NotFound desc = could not find container \"ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf\": container with ID starting with ed25f32098032884815c9ba7b1728195cc39b6365b74cc15edecdd7d4b606adf not found: ID does not exist" Jul 06 23:58:00.098568 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.098534 2568 scope.go:117] "RemoveContainer" containerID="9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705" Jul 06 23:58:00.098731 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:00.098718 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705\": container with ID starting with 9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705 not found: ID does not exist" containerID="9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705" Jul 06 23:58:00.098771 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.098735 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705"} err="failed to get container status \"9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705\": rpc error: code = NotFound desc = could not find container \"9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705\": container with ID starting with 9b6669f73ccffaa5d96d57d1c4e91d0ccefb6242dd107612d0ca9ad4d4ff3705 not found: ID does not exist" Jul 06 23:58:00.098771 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.098745 2568 scope.go:117] "RemoveContainer" containerID="51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac" Jul 06 23:58:00.098936 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:00.098917 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac\": container with ID starting with 51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac not found: ID does not exist" containerID="51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac" Jul 06 23:58:00.098985 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.098943 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac"} err="failed to get container status \"51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac\": rpc error: code = NotFound desc = could not find container \"51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac\": container with ID starting with 51890a08ce2cb61f83025cec95aa24e311b8875036684e115dfcab7ff2346fac not found: ID does not exist" Jul 06 23:58:00.120467 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120412 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 23:58:00.120778 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120762 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="prom-label-proxy" Jul 06 23:58:00.120832 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120781 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="prom-label-proxy" Jul 06 23:58:00.120832 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120791 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy-metric" Jul 06 23:58:00.120832 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120797 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy-metric" Jul 06 23:58:00.120832 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120807 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05edae08-e6c7-4ac9-b6e9-87a64a78376a" containerName="console" Jul 06 23:58:00.120832 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120814 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="05edae08-e6c7-4ac9-b6e9-87a64a78376a" containerName="console" Jul 06 23:58:00.120832 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120825 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy" Jul 06 23:58:00.120832 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120831 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120838 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="init-config-reloader" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120843 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="init-config-reloader" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120853 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="alertmanager" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120858 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="alertmanager" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120863 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy-web" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120868 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy-web" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120878 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="config-reloader" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120883 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="config-reloader" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120928 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120935 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="alertmanager" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120941 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy-metric" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120947 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="prom-label-proxy" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120952 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="kube-rbac-proxy-web" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120958 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="05edae08-e6c7-4ac9-b6e9-87a64a78376a" containerName="console" Jul 06 23:58:00.121040 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.120964 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6061f99d-d211-4b81-8327-eec0e7805340" containerName="config-reloader" Jul 06 23:58:00.126452 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.126415 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.129588 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.129569 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 23:58:00.129699 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.129571 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 23:58:00.129926 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.129894 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 23:58:00.130112 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.130095 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zlb4l\"" Jul 06 23:58:00.130186 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.130110 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 23:58:00.130231 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.130220 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 23:58:00.130339 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.130313 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 23:58:00.130492 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.130477 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 23:58:00.130569 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.130521 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 23:58:00.138545 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.138526 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 23:58:00.141697 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.141678 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 23:58:00.219966 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.219933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.219966 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.219970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220185 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.219994 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220185 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-web-config\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220185 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220043 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220185 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220057 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jsd5\" (UniqueName: \"kubernetes.io/projected/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-kube-api-access-8jsd5\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220185 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220076 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220185 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220122 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220371 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220187 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220371 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220218 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220371 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220371 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.220371 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.220310 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-config-out\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.321748 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.321748 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-config-out\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.321748 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321723 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.321748 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321738 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-web-config\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321942 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jsd5\" (UniqueName: \"kubernetes.io/projected/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-kube-api-access-8jsd5\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.321994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.322017 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.322042 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322519 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.322075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.322740 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.322576 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.324376 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.323482 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.324376 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.323729 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.325097 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.325066 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-config-out\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.325207 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.325079 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-web-config\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.325207 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.325116 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.325207 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.325144 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.325207 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.325190 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.325207 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.325200 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.325508 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.325339 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.326691 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.326670 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.327033 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.327016 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.333765 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.333748 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jsd5\" (UniqueName: \"kubernetes.io/projected/ffcdd7a8-5bb6-4871-8835-b712ce7c44f2-kube-api-access-8jsd5\") pod \"alertmanager-main-0\" (UID: \"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.436484 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.436453 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 23:58:00.584189 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:00.584116 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 23:58:01.049954 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:01.049924 2568 generic.go:356] "Generic (PLEG): container finished" podID="ffcdd7a8-5bb6-4871-8835-b712ce7c44f2" containerID="94edfd30c05c1e4b114a27fadda2fd03f310b35fe0cc02b58b54c2ca312d7d4d" exitCode=0 Jul 06 23:58:01.050107 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:01.049984 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2","Type":"ContainerDied","Data":"94edfd30c05c1e4b114a27fadda2fd03f310b35fe0cc02b58b54c2ca312d7d4d"} Jul 06 23:58:01.050107 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:01.050036 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2","Type":"ContainerStarted","Data":"1e3d61bed8666931a02b435fcb7e207c72526201fdaad0dca05a43e87b0b5f05"} Jul 06 23:58:01.315582 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:01.315554 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6061f99d-d211-4b81-8327-eec0e7805340" path="/var/lib/kubelet/pods/6061f99d-d211-4b81-8327-eec0e7805340/volumes" Jul 06 23:58:01.918620 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:01.918583 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tdphw" Jul 06 23:58:02.057596 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.057559 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2","Type":"ContainerStarted","Data":"106c5ac1e9ed079f0c38077f73f554297014fc1b5b247d2467d1927d840962ba"} Jul 06 23:58:02.057771 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.057644 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2","Type":"ContainerStarted","Data":"217a31847b48082831da28c3693e301a1fe9952f79c5fbe23fe2f39055ef8bca"} Jul 06 23:58:02.057771 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.057683 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2","Type":"ContainerStarted","Data":"f2ba0da66686992f67c897b9c47db79f4aecdbc42079b6291afa901752e94621"} Jul 06 23:58:02.057771 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.057720 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2","Type":"ContainerStarted","Data":"a795df5dd1852eb181a01ad1e38cdef75b8cfba8038b3db5c1078d081bf451d0"} Jul 06 23:58:02.057771 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.057756 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2","Type":"ContainerStarted","Data":"4cce91fb5e9edd0a0a9eb2f894276fef235ca9e0daa351460b914851cee0eb9e"} Jul 06 23:58:02.057893 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.057787 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffcdd7a8-5bb6-4871-8835-b712ce7c44f2","Type":"ContainerStarted","Data":"dd12c064fc228f5acb95d91c8c356b865fac5ff2daf413176c90ab39e09a97cb"} Jul 06 23:58:02.094529 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.094480 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.079395803 podStartE2EDuration="2.094460603s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.0648ms" isStatefulPod=false podCreationTimestamp="2026-07-06 23:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:02.093451792 +0000 UTC m=+103.306483528" watchObservedRunningTime="2026-07-06 23:58:02.094460603 +0000 UTC m=+103.307492343" Jul 06 23:58:02.688415 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.688372 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:58:02.688792 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.688757 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="prometheus" containerID="cri-o://7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" gracePeriod=600 Jul 06 23:58:02.689177 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.689146 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" gracePeriod=600 Jul 06 23:58:02.689289 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.689270 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy" containerID="cri-o://c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" gracePeriod=600 Jul 06 23:58:02.689366 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.689349 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy-web" containerID="cri-o://672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" gracePeriod=600 Jul 06 23:58:02.689458 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.689421 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="thanos-sidecar" containerID="cri-o://e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" gracePeriod=600 Jul 06 23:58:02.689533 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.689517 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="config-reloader" containerID="cri-o://eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" gracePeriod=600 Jul 06 23:58:02.966349 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:02.966320 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.047717 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.047774 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-metrics-client-ca\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.047824 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-trusted-ca-bundle\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.047848 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-thanos-prometheus-http-client-file\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.047871 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-web-config\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.047894 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-2\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.047938 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.047965 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-kubelet-serving-ca-bundle\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.047994 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-metrics-client-certs\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048025 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf9tw\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-kube-api-access-hf9tw\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048056 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-tls-assets\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048095 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-kube-rbac-proxy\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048127 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-tls\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048173 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-1\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048199 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config-out\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048229 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-grpc-tls\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048250 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.050405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048275 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-db\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.051392 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048301 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-serving-certs-ca-bundle\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.051392 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048333 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-0\") pod \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\" (UID: \"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c\") " Jul 06 23:58:03.051392 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048515 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:03.051392 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048643 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-metrics-client-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.051392 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.048904 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:03.051392 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.049055 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:03.051874 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.051413 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:03.052685 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.052653 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config-out" (OuterVolumeSpecName: "config-out") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:03.053108 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.053083 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:03.054538 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.054319 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:03.054653 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.054605 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:03.055139 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.055102 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:58:03.055139 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.055111 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:03.055139 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.055118 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:03.055328 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.055251 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:03.055410 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.055386 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:03.055589 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.055552 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:03.055692 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.055650 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:03.057046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.057022 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config" (OuterVolumeSpecName: "config") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:03.057374 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.057350 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-kube-api-access-hf9tw" (OuterVolumeSpecName: "kube-api-access-hf9tw") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "kube-api-access-hf9tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:58:03.057494 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.057473 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:03.058553 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.058525 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:03.065991 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.065965 2568 generic.go:356] "Generic (PLEG): container finished" podID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerID="1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" exitCode=0 Jul 06 23:58:03.065991 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.065989 2568 generic.go:356] "Generic (PLEG): container finished" podID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerID="c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" exitCode=0 Jul 06 23:58:03.065991 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.065997 2568 generic.go:356] "Generic (PLEG): container finished" podID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerID="672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" exitCode=0 Jul 06 23:58:03.066227 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066002 2568 generic.go:356] "Generic (PLEG): container finished" podID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerID="e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" exitCode=0 Jul 06 23:58:03.066227 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066009 2568 generic.go:356] "Generic (PLEG): container finished" podID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerID="eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" exitCode=0 Jul 06 23:58:03.066227 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066014 2568 generic.go:356] "Generic (PLEG): container finished" podID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerID="7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" exitCode=0 Jul 06 23:58:03.066227 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066048 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.066227 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066044 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerDied","Data":"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763"} Jul 06 23:58:03.066227 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066180 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerDied","Data":"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa"} Jul 06 23:58:03.066227 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066212 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerDied","Data":"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c"} Jul 06 23:58:03.066922 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066238 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerDied","Data":"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf"} Jul 06 23:58:03.066922 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066265 2568 scope.go:117] "RemoveContainer" containerID="1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" Jul 06 23:58:03.066922 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066277 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerDied","Data":"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d"} Jul 06 23:58:03.066922 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066316 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerDied","Data":"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8"} Jul 06 23:58:03.066922 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.066353 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c","Type":"ContainerDied","Data":"aa1caf25355f212c8b3ae7e809f753ea7f8c18201791aa441e75e9ce98557367"} Jul 06 23:58:03.067111 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.067087 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-web-config" (OuterVolumeSpecName: "web-config") pod "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" (UID: "e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:03.074306 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.074287 2568 scope.go:117] "RemoveContainer" containerID="c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" Jul 06 23:58:03.081636 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.081617 2568 scope.go:117] "RemoveContainer" containerID="672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" Jul 06 23:58:03.088552 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.088534 2568 scope.go:117] "RemoveContainer" containerID="e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" Jul 06 23:58:03.095815 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.095795 2568 scope.go:117] "RemoveContainer" containerID="eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" Jul 06 23:58:03.102799 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.102781 2568 scope.go:117] "RemoveContainer" containerID="7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" Jul 06 23:58:03.110038 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.110019 2568 scope.go:117] "RemoveContainer" containerID="3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052" Jul 06 23:58:03.116907 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.116892 2568 scope.go:117] "RemoveContainer" containerID="1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" Jul 06 23:58:03.117164 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:03.117147 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": container with ID starting with 1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763 not found: ID does not exist" containerID="1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" Jul 06 23:58:03.117217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.117174 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763"} err="failed to get container status \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": rpc error: code = NotFound desc = could not find container \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": container with ID starting with 1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763 not found: ID does not exist" Jul 06 23:58:03.117217 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.117191 2568 scope.go:117] "RemoveContainer" containerID="c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" Jul 06 23:58:03.117423 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:03.117403 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": container with ID starting with c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa not found: ID does not exist" containerID="c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" Jul 06 23:58:03.117486 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.117428 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa"} err="failed to get container status \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": rpc error: code = NotFound desc = could not find container \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": container with ID starting with c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa not found: ID does not exist" Jul 06 23:58:03.117486 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.117466 2568 scope.go:117] "RemoveContainer" containerID="672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" Jul 06 23:58:03.117729 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:03.117713 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": container with ID starting with 672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c not found: ID does not exist" containerID="672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" Jul 06 23:58:03.117773 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.117735 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c"} err="failed to get container status \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": rpc error: code = NotFound desc = could not find container \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": container with ID starting with 672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c not found: ID does not exist" Jul 06 23:58:03.117773 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.117748 2568 scope.go:117] "RemoveContainer" containerID="e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" Jul 06 23:58:03.118022 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:03.118002 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": container with ID starting with e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf not found: ID does not exist" containerID="e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" Jul 06 23:58:03.118065 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.118027 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf"} err="failed to get container status \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": rpc error: code = NotFound desc = could not find container \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": container with ID starting with e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf not found: ID does not exist" Jul 06 23:58:03.118065 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.118049 2568 scope.go:117] "RemoveContainer" containerID="eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" Jul 06 23:58:03.118279 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:03.118264 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": container with ID starting with eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d not found: ID does not exist" containerID="eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" Jul 06 23:58:03.118333 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.118281 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d"} err="failed to get container status \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": rpc error: code = NotFound desc = could not find container \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": container with ID starting with eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d not found: ID does not exist" Jul 06 23:58:03.118333 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.118304 2568 scope.go:117] "RemoveContainer" containerID="7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" Jul 06 23:58:03.122878 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:03.119066 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": container with ID starting with 7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8 not found: ID does not exist" containerID="7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" Jul 06 23:58:03.122878 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.119155 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8"} err="failed to get container status \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": rpc error: code = NotFound desc = could not find container \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": container with ID starting with 7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8 not found: ID does not exist" Jul 06 23:58:03.122878 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.119183 2568 scope.go:117] "RemoveContainer" containerID="3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052" Jul 06 23:58:03.123347 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:03.123321 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": container with ID starting with 3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052 not found: ID does not exist" containerID="3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052" Jul 06 23:58:03.123500 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.123353 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052"} err="failed to get container status \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": rpc error: code = NotFound desc = could not find container \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": container with ID starting with 3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052 not found: ID does not exist" Jul 06 23:58:03.123500 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.123373 2568 scope.go:117] "RemoveContainer" containerID="1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" Jul 06 23:58:03.123688 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.123662 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763"} err="failed to get container status \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": rpc error: code = NotFound desc = could not find container \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": container with ID starting with 1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763 not found: ID does not exist" Jul 06 23:58:03.123732 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.123692 2568 scope.go:117] "RemoveContainer" containerID="c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" Jul 06 23:58:03.123937 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.123915 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa"} err="failed to get container status \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": rpc error: code = NotFound desc = could not find container \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": container with ID starting with c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa not found: ID does not exist" Jul 06 23:58:03.123937 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.123934 2568 scope.go:117] "RemoveContainer" containerID="672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" Jul 06 23:58:03.124179 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.124147 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c"} err="failed to get container status \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": rpc error: code = NotFound desc = could not find container \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": container with ID starting with 672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c not found: ID does not exist" Jul 06 23:58:03.124179 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.124172 2568 scope.go:117] "RemoveContainer" containerID="e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" Jul 06 23:58:03.124416 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.124396 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf"} err="failed to get container status \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": rpc error: code = NotFound desc = could not find container \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": container with ID starting with e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf not found: ID does not exist" Jul 06 23:58:03.124416 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.124416 2568 scope.go:117] "RemoveContainer" containerID="eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" Jul 06 23:58:03.124670 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.124652 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d"} err="failed to get container status \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": rpc error: code = NotFound desc = could not find container \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": container with ID starting with eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d not found: ID does not exist" Jul 06 23:58:03.124715 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.124673 2568 scope.go:117] "RemoveContainer" containerID="7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" Jul 06 23:58:03.124906 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.124880 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8"} err="failed to get container status \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": rpc error: code = NotFound desc = could not find container \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": container with ID starting with 7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8 not found: ID does not exist" Jul 06 23:58:03.124906 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.124899 2568 scope.go:117] "RemoveContainer" containerID="3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052" Jul 06 23:58:03.125138 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125120 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052"} err="failed to get container status \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": rpc error: code = NotFound desc = could not find container \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": container with ID starting with 3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052 not found: ID does not exist" Jul 06 23:58:03.125174 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125139 2568 scope.go:117] "RemoveContainer" containerID="1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" Jul 06 23:58:03.125362 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125341 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763"} err="failed to get container status \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": rpc error: code = NotFound desc = could not find container \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": container with ID starting with 1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763 not found: ID does not exist" Jul 06 23:58:03.125403 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125363 2568 scope.go:117] "RemoveContainer" containerID="c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" Jul 06 23:58:03.125570 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125555 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa"} err="failed to get container status \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": rpc error: code = NotFound desc = could not find container \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": container with ID starting with c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa not found: ID does not exist" Jul 06 23:58:03.125640 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125571 2568 scope.go:117] "RemoveContainer" containerID="672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" Jul 06 23:58:03.125794 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125778 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c"} err="failed to get container status \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": rpc error: code = NotFound desc = could not find container \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": container with ID starting with 672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c not found: ID does not exist" Jul 06 23:58:03.125844 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125794 2568 scope.go:117] "RemoveContainer" containerID="e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" Jul 06 23:58:03.125979 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125964 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf"} err="failed to get container status \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": rpc error: code = NotFound desc = could not find container \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": container with ID starting with e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf not found: ID does not exist" Jul 06 23:58:03.126022 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.125979 2568 scope.go:117] "RemoveContainer" containerID="eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" Jul 06 23:58:03.126164 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126148 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d"} err="failed to get container status \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": rpc error: code = NotFound desc = could not find container \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": container with ID starting with eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d not found: ID does not exist" Jul 06 23:58:03.126215 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126165 2568 scope.go:117] "RemoveContainer" containerID="7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" Jul 06 23:58:03.126357 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126341 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8"} err="failed to get container status \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": rpc error: code = NotFound desc = could not find container \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": container with ID starting with 7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8 not found: ID does not exist" Jul 06 23:58:03.126403 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126358 2568 scope.go:117] "RemoveContainer" containerID="3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052" Jul 06 23:58:03.126576 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126555 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052"} err="failed to get container status \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": rpc error: code = NotFound desc = could not find container \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": container with ID starting with 3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052 not found: ID does not exist" Jul 06 23:58:03.126622 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126577 2568 scope.go:117] "RemoveContainer" containerID="1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" Jul 06 23:58:03.126757 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126740 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763"} err="failed to get container status \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": rpc error: code = NotFound desc = could not find container \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": container with ID starting with 1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763 not found: ID does not exist" Jul 06 23:58:03.126796 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126758 2568 scope.go:117] "RemoveContainer" containerID="c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" Jul 06 23:58:03.126960 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126941 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa"} err="failed to get container status \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": rpc error: code = NotFound desc = could not find container \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": container with ID starting with c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa not found: ID does not exist" Jul 06 23:58:03.127000 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.126962 2568 scope.go:117] "RemoveContainer" containerID="672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" Jul 06 23:58:03.127149 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.127131 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c"} err="failed to get container status \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": rpc error: code = NotFound desc = could not find container \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": container with ID starting with 672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c not found: ID does not exist" Jul 06 23:58:03.127195 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.127150 2568 scope.go:117] "RemoveContainer" containerID="e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" Jul 06 23:58:03.127336 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.127321 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf"} err="failed to get container status \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": rpc error: code = NotFound desc = could not find container \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": container with ID starting with e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf not found: ID does not exist" Jul 06 23:58:03.127336 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.127334 2568 scope.go:117] "RemoveContainer" containerID="eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" Jul 06 23:58:03.127565 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.127542 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d"} err="failed to get container status \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": rpc error: code = NotFound desc = could not find container \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": container with ID starting with eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d not found: ID does not exist" Jul 06 23:58:03.127649 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.127574 2568 scope.go:117] "RemoveContainer" containerID="7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" Jul 06 23:58:03.127781 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.127766 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8"} err="failed to get container status \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": rpc error: code = NotFound desc = could not find container \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": container with ID starting with 7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8 not found: ID does not exist" Jul 06 23:58:03.127824 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.127780 2568 scope.go:117] "RemoveContainer" containerID="3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052" Jul 06 23:58:03.128001 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.127983 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052"} err="failed to get container status \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": rpc error: code = NotFound desc = could not find container \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": container with ID starting with 3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052 not found: ID does not exist" Jul 06 23:58:03.128043 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.128001 2568 scope.go:117] "RemoveContainer" containerID="1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" Jul 06 23:58:03.128213 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.128197 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763"} err="failed to get container status \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": rpc error: code = NotFound desc = could not find container \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": container with ID starting with 1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763 not found: ID does not exist" Jul 06 23:58:03.128213 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.128212 2568 scope.go:117] "RemoveContainer" containerID="c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" Jul 06 23:58:03.128462 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.128423 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa"} err="failed to get container status \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": rpc error: code = NotFound desc = could not find container \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": container with ID starting with c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa not found: ID does not exist" Jul 06 23:58:03.128462 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.128463 2568 scope.go:117] "RemoveContainer" containerID="672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" Jul 06 23:58:03.128678 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.128659 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c"} err="failed to get container status \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": rpc error: code = NotFound desc = could not find container \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": container with ID starting with 672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c not found: ID does not exist" Jul 06 23:58:03.128720 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.128680 2568 scope.go:117] "RemoveContainer" containerID="e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" Jul 06 23:58:03.128895 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.128878 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf"} err="failed to get container status \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": rpc error: code = NotFound desc = could not find container \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": container with ID starting with e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf not found: ID does not exist" Jul 06 23:58:03.128895 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.128895 2568 scope.go:117] "RemoveContainer" containerID="eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" Jul 06 23:58:03.129101 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.129084 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d"} err="failed to get container status \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": rpc error: code = NotFound desc = could not find container \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": container with ID starting with eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d not found: ID does not exist" Jul 06 23:58:03.129101 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.129101 2568 scope.go:117] "RemoveContainer" containerID="7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" Jul 06 23:58:03.129292 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.129277 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8"} err="failed to get container status \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": rpc error: code = NotFound desc = could not find container \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": container with ID starting with 7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8 not found: ID does not exist" Jul 06 23:58:03.129330 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.129293 2568 scope.go:117] "RemoveContainer" containerID="3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052" Jul 06 23:58:03.129678 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.129661 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052"} err="failed to get container status \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": rpc error: code = NotFound desc = could not find container \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": container with ID starting with 3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052 not found: ID does not exist" Jul 06 23:58:03.129747 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.129679 2568 scope.go:117] "RemoveContainer" containerID="1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763" Jul 06 23:58:03.129918 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.129903 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763"} err="failed to get container status \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": rpc error: code = NotFound desc = could not find container \"1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763\": container with ID starting with 1d695d28a74e40c7eb56c27b76d4232fbe143030e7dad1325fd18ea59390f763 not found: ID does not exist" Jul 06 23:58:03.129965 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.129919 2568 scope.go:117] "RemoveContainer" containerID="c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa" Jul 06 23:58:03.130152 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.130137 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa"} err="failed to get container status \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": rpc error: code = NotFound desc = could not find container \"c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa\": container with ID starting with c2a60bffbfc5e238c0b6f4896e32f3a2b6c747d6130a96659764ed6f9d742dfa not found: ID does not exist" Jul 06 23:58:03.130195 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.130151 2568 scope.go:117] "RemoveContainer" containerID="672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c" Jul 06 23:58:03.130342 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.130322 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c"} err="failed to get container status \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": rpc error: code = NotFound desc = could not find container \"672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c\": container with ID starting with 672754a59dc79bdb62b928ac825aa96611e515e0c3d4193e73727fd5ef80374c not found: ID does not exist" Jul 06 23:58:03.130388 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.130343 2568 scope.go:117] "RemoveContainer" containerID="e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf" Jul 06 23:58:03.130615 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.130595 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf"} err="failed to get container status \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": rpc error: code = NotFound desc = could not find container \"e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf\": container with ID starting with e923c382659b37638102a777540f11862bdb40e6938f664e0f5fa62bf134bfaf not found: ID does not exist" Jul 06 23:58:03.130707 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.130615 2568 scope.go:117] "RemoveContainer" containerID="eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d" Jul 06 23:58:03.130875 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.130857 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d"} err="failed to get container status \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": rpc error: code = NotFound desc = could not find container \"eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d\": container with ID starting with eecac894f898a96be7b53ac9089cd65d656da1802eb966e0d9656883a5138c4d not found: ID does not exist" Jul 06 23:58:03.130927 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.130877 2568 scope.go:117] "RemoveContainer" containerID="7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8" Jul 06 23:58:03.131106 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.131082 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8"} err="failed to get container status \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": rpc error: code = NotFound desc = could not find container \"7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8\": container with ID starting with 7ccdeb3d1293a19f7f8a67a9f71d2ce48219d6c32802a54c69b3bd2267d75fb8 not found: ID does not exist" Jul 06 23:58:03.131106 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.131100 2568 scope.go:117] "RemoveContainer" containerID="3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052" Jul 06 23:58:03.131379 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.131362 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052"} err="failed to get container status \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": rpc error: code = NotFound desc = could not find container \"3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052\": container with ID starting with 3efb71bb9bae6068c7be74e3fbfde7e15c6d1832e28b26889601e04933dfb052 not found: ID does not exist" Jul 06 23:58:03.150209 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150007 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150209 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150099 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150209 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150135 2568 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150209 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150154 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-web-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150209 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150169 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150209 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150183 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150209 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150197 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150209 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150209 2568 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-metrics-client-certs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150209 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150221 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9tw\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-kube-api-access-hf9tw\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150234 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-tls-assets\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150245 2568 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-kube-rbac-proxy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150257 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150268 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150280 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config-out\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150291 2568 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-secret-grpc-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150302 2568 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150313 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-db\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150325 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.150623 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.150338 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:03.397360 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.397327 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:58:03.404845 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.404819 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:58:03.448147 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448116 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:58:03.448373 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448361 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="thanos-sidecar" Jul 06 23:58:03.448415 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448375 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="thanos-sidecar" Jul 06 23:58:03.448415 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448389 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="init-config-reloader" Jul 06 23:58:03.448415 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448395 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="init-config-reloader" Jul 06 23:58:03.448415 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448403 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy-web" Jul 06 23:58:03.448415 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448409 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy-web" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448419 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="prometheus" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448426 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="prometheus" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448448 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448453 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448460 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="config-reloader" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448465 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="config-reloader" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448470 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy-thanos" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448476 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy-thanos" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448519 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="thanos-sidecar" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448525 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy-web" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448530 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy-thanos" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448537 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="prometheus" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448541 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="kube-rbac-proxy" Jul 06 23:58:03.448593 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.448548 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" containerName="config-reloader" Jul 06 23:58:03.480885 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.480860 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:58:03.481038 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.481018 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.483888 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.483865 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 23:58:03.484085 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.484072 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 23:58:03.484255 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.484243 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fm64k\"" Jul 06 23:58:03.484836 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.484808 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 23:58:03.484836 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.484819 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 23:58:03.485013 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.484875 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 23:58:03.485253 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.485220 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 23:58:03.485379 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.485346 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 23:58:03.485990 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.485974 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 23:58:03.486132 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.485998 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 23:58:03.486132 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.486032 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 23:58:03.486264 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.486202 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 23:58:03.486690 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.486585 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ekrb9d21bcba4\"" Jul 06 23:58:03.486803 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.486753 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 23:58:03.486870 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.486808 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 23:58:03.487580 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.487559 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 23:58:03.489994 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.489974 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 23:58:03.554369 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554327 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554369 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554598 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554598 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2zp\" (UniqueName: \"kubernetes.io/projected/7c26c073-6f76-4931-b890-e637aa8cdb80-kube-api-access-ph2zp\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554598 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554522 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554598 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554598 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554564 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554598 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554589 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554768 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554627 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554768 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c26c073-6f76-4931-b890-e637aa8cdb80-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554768 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554665 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554768 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-config\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554768 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554711 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554768 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554768 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554739 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554768 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c26c073-6f76-4931-b890-e637aa8cdb80-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554985 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554789 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554985 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554985 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.554985 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.554883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.655983 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.655896 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.655983 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.655933 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.655983 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.655953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656188 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2zp\" (UniqueName: \"kubernetes.io/projected/7c26c073-6f76-4931-b890-e637aa8cdb80-kube-api-access-ph2zp\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656188 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656188 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656137 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656188 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656173 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656370 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656370 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656370 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c26c073-6f76-4931-b890-e637aa8cdb80-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656370 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656370 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656317 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-config\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656370 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656345 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656370 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656423 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c26c073-6f76-4931-b890-e637aa8cdb80-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656484 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656510 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.656737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656611 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.657193 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.656811 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.657376 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.657349 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.657555 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.657531 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.659625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.659495 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.659625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.659528 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.659625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.659578 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.659625 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.659598 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.659888 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.659788 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c26c073-6f76-4931-b890-e637aa8cdb80-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.659888 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.659805 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.659888 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.659860 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.660205 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.660032 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.660455 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.660407 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.660537 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.660518 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.660924 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.660900 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.661017 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.660934 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.661763 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.661742 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.661837 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.661822 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c26c073-6f76-4931-b890-e637aa8cdb80-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.662443 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.662412 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c26c073-6f76-4931-b890-e637aa8cdb80-config\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.663324 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.663307 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c26c073-6f76-4931-b890-e637aa8cdb80-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.665741 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.665725 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2zp\" (UniqueName: \"kubernetes.io/projected/7c26c073-6f76-4931-b890-e637aa8cdb80-kube-api-access-ph2zp\") pod \"prometheus-k8s-0\" (UID: \"7c26c073-6f76-4931-b890-e637aa8cdb80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.790861 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.790813 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:03.957658 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:03.957628 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:58:03.960537 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:58:03.960505 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c26c073_6f76_4931_b890_e637aa8cdb80.slice/crio-38f7fe9e5f5d139512dd3140ccc9ba1c7426af5e99943d474a05a689e9f308f5 WatchSource:0}: Error finding container 38f7fe9e5f5d139512dd3140ccc9ba1c7426af5e99943d474a05a689e9f308f5: Status 404 returned error can't find the container with id 38f7fe9e5f5d139512dd3140ccc9ba1c7426af5e99943d474a05a689e9f308f5 Jul 06 23:58:04.080139 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:04.080094 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c26c073-6f76-4931-b890-e637aa8cdb80","Type":"ContainerStarted","Data":"38f7fe9e5f5d139512dd3140ccc9ba1c7426af5e99943d474a05a689e9f308f5"} Jul 06 23:58:05.085998 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:05.085964 2568 generic.go:356] "Generic (PLEG): container finished" podID="7c26c073-6f76-4931-b890-e637aa8cdb80" containerID="aa0c999df2c6f80ea08e3c429421a60d6a04fd9ae96192cb8b434316f6f5f9b9" exitCode=0 Jul 06 23:58:05.086377 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:05.086059 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c26c073-6f76-4931-b890-e637aa8cdb80","Type":"ContainerDied","Data":"aa0c999df2c6f80ea08e3c429421a60d6a04fd9ae96192cb8b434316f6f5f9b9"} Jul 06 23:58:05.317556 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:05.317520 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c" path="/var/lib/kubelet/pods/e8a3a5e7-e3d1-4186-9d2f-c8e38e32195c/volumes" Jul 06 23:58:06.093756 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.093723 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c26c073-6f76-4931-b890-e637aa8cdb80","Type":"ContainerStarted","Data":"37ee06101acf959365ed2d7102855e367e89b928bcb4ac549094caf6bc02ba2f"} Jul 06 23:58:06.094173 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.093784 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c26c073-6f76-4931-b890-e637aa8cdb80","Type":"ContainerStarted","Data":"b538149a4566bc15ff481ca08e343918d8db6fcac6d2fc45dc3d6638a7566813"} Jul 06 23:58:06.094173 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.093809 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c26c073-6f76-4931-b890-e637aa8cdb80","Type":"ContainerStarted","Data":"f405cee1c0ed363b297650eeb37440a2f3cad083daf299caf17ae3e463ae4607"} Jul 06 23:58:06.094173 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.093838 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c26c073-6f76-4931-b890-e637aa8cdb80","Type":"ContainerStarted","Data":"1fa3b2d35d09f9cd34f2f48e4370283db8f5f80c310702ba952390404c254b2c"} Jul 06 23:58:06.094173 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.093861 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c26c073-6f76-4931-b890-e637aa8cdb80","Type":"ContainerStarted","Data":"0b70d4baacbeeb269bc86de577a5a8fdf1ce18ab192d67700bcfbf6301bdbb9a"} Jul 06 23:58:06.094173 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.093883 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c26c073-6f76-4931-b890-e637aa8cdb80","Type":"ContainerStarted","Data":"dc17d7223f07f2384010ed106d89cd0f5019ac63ff03c7cf314d9cc04a5e8b4d"} Jul 06 23:58:06.122670 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.122620 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.098792156 podStartE2EDuration="3.122604387s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.812231ms" isStatefulPod=false podCreationTimestamp="2026-07-06 23:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:06.122323339 +0000 UTC m=+107.335355084" watchObservedRunningTime="2026-07-06 23:58:06.122604387 +0000 UTC m=+107.335636130" Jul 06 23:58:06.952652 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.952617 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69bf855568-z2q2s"] Jul 06 23:58:06.955135 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.955117 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:06.970722 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:06.970692 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69bf855568-z2q2s"] Jul 06 23:58:07.088720 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.088681 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-service-ca\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.088919 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.088749 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-trusted-ca-bundle\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.088919 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.088776 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-oauth-config\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.088919 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.088797 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-config\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.088919 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.088817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-serving-cert\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.088919 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.088837 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-oauth-serving-cert\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.088919 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.088871 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfmkf\" (UniqueName: \"kubernetes.io/projected/9add24f0-3837-46b2-a4a6-db91933d0b7b-kube-api-access-qfmkf\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.189533 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.189495 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-oauth-serving-cert\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.190018 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.189544 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfmkf\" (UniqueName: \"kubernetes.io/projected/9add24f0-3837-46b2-a4a6-db91933d0b7b-kube-api-access-qfmkf\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.190018 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.189590 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-service-ca\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.190166 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.190054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-trusted-ca-bundle\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.190166 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.190157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-oauth-config\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.190279 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.190248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-config\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.190345 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.190324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-serving-cert\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.190345 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.190323 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-oauth-serving-cert\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.190478 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.190367 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-service-ca\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.191405 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.191376 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-config\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.191712 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.191691 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-trusted-ca-bundle\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.193029 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.193007 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-oauth-config\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.193262 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.193238 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-serving-cert\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.199503 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.199479 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfmkf\" (UniqueName: \"kubernetes.io/projected/9add24f0-3837-46b2-a4a6-db91933d0b7b-kube-api-access-qfmkf\") pod \"console-69bf855568-z2q2s\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.264760 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.264666 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:07.410915 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:07.410824 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69bf855568-z2q2s"] Jul 06 23:58:07.413390 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:58:07.413362 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9add24f0_3837_46b2_a4a6_db91933d0b7b.slice/crio-2eb0a165ec3ac0ad62566890efe4a069ca1c9f6d813db5c8cc37bb60b89b124c WatchSource:0}: Error finding container 2eb0a165ec3ac0ad62566890efe4a069ca1c9f6d813db5c8cc37bb60b89b124c: Status 404 returned error can't find the container with id 2eb0a165ec3ac0ad62566890efe4a069ca1c9f6d813db5c8cc37bb60b89b124c Jul 06 23:58:08.104553 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:08.104516 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69bf855568-z2q2s" event={"ID":"9add24f0-3837-46b2-a4a6-db91933d0b7b","Type":"ContainerStarted","Data":"9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878"} Jul 06 23:58:08.104724 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:08.104564 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69bf855568-z2q2s" event={"ID":"9add24f0-3837-46b2-a4a6-db91933d0b7b","Type":"ContainerStarted","Data":"2eb0a165ec3ac0ad62566890efe4a069ca1c9f6d813db5c8cc37bb60b89b124c"} Jul 06 23:58:08.124195 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:08.124142 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69bf855568-z2q2s" podStartSLOduration=2.124126044 podStartE2EDuration="2.124126044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:08.122737548 +0000 UTC m=+109.335769292" watchObservedRunningTime="2026-07-06 23:58:08.124126044 +0000 UTC m=+109.337157789" Jul 06 23:58:08.791413 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:08.791375 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:58:17.265287 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:17.265251 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:17.265765 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:17.265331 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:17.270130 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:17.270100 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:18.148575 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:18.148545 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:58:18.194104 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:18.194066 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c49c858b8-whsj7"] Jul 06 23:58:43.218325 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.218274 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7c49c858b8-whsj7" podUID="2057ad8f-ffdf-46a1-ab60-00e4abf92aea" containerName="console" containerID="cri-o://92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049" gracePeriod=15 Jul 06 23:58:43.469157 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.469101 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:58:43.497294 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.497268 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-config\") pod \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " Jul 06 23:58:43.497530 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.497306 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-serving-cert\") pod \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " Jul 06 23:58:43.497530 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.497330 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mpr4\" (UniqueName: \"kubernetes.io/projected/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-kube-api-access-4mpr4\") pod \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " Jul 06 23:58:43.497530 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.497399 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-oauth-config\") pod \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " Jul 06 23:58:43.497530 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.497463 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-service-ca\") pod \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " Jul 06 23:58:43.497530 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.497486 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-oauth-serving-cert\") pod \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " Jul 06 23:58:43.497863 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.497545 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-trusted-ca-bundle\") pod \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\" (UID: \"2057ad8f-ffdf-46a1-ab60-00e4abf92aea\") " Jul 06 23:58:43.497863 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.497802 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-config" (OuterVolumeSpecName: "console-config") pod "2057ad8f-ffdf-46a1-ab60-00e4abf92aea" (UID: "2057ad8f-ffdf-46a1-ab60-00e4abf92aea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:43.497938 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.497895 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-service-ca" (OuterVolumeSpecName: "service-ca") pod "2057ad8f-ffdf-46a1-ab60-00e4abf92aea" (UID: "2057ad8f-ffdf-46a1-ab60-00e4abf92aea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:43.498044 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.498013 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2057ad8f-ffdf-46a1-ab60-00e4abf92aea" (UID: "2057ad8f-ffdf-46a1-ab60-00e4abf92aea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:43.498184 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.498098 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2057ad8f-ffdf-46a1-ab60-00e4abf92aea" (UID: "2057ad8f-ffdf-46a1-ab60-00e4abf92aea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:43.500057 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.500023 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-kube-api-access-4mpr4" (OuterVolumeSpecName: "kube-api-access-4mpr4") pod "2057ad8f-ffdf-46a1-ab60-00e4abf92aea" (UID: "2057ad8f-ffdf-46a1-ab60-00e4abf92aea"). InnerVolumeSpecName "kube-api-access-4mpr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:58:43.500149 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.500109 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2057ad8f-ffdf-46a1-ab60-00e4abf92aea" (UID: "2057ad8f-ffdf-46a1-ab60-00e4abf92aea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:43.500637 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.500612 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2057ad8f-ffdf-46a1-ab60-00e4abf92aea" (UID: "2057ad8f-ffdf-46a1-ab60-00e4abf92aea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:43.599019 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.598983 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:43.599019 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.599010 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-serving-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:43.599019 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.599020 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpr4\" (UniqueName: \"kubernetes.io/projected/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-kube-api-access-4mpr4\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:43.599229 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.599029 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-console-oauth-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:43.599229 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.599039 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:43.599229 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.599047 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-oauth-serving-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:43.599229 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:43.599054 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2057ad8f-ffdf-46a1-ab60-00e4abf92aea-trusted-ca-bundle\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.239518 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:44.239487 2568 generic.go:356] "Generic (PLEG): container finished" podID="2057ad8f-ffdf-46a1-ab60-00e4abf92aea" containerID="92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049" exitCode=0 Jul 06 23:58:44.239910 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:44.239534 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c49c858b8-whsj7" Jul 06 23:58:44.239910 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:44.239586 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c49c858b8-whsj7" event={"ID":"2057ad8f-ffdf-46a1-ab60-00e4abf92aea","Type":"ContainerDied","Data":"92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049"} Jul 06 23:58:44.239910 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:44.239627 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c49c858b8-whsj7" event={"ID":"2057ad8f-ffdf-46a1-ab60-00e4abf92aea","Type":"ContainerDied","Data":"5c7924fc52f7dade1b40a8866904a14daa231dfdbfc697cd8a95cf64efa91815"} Jul 06 23:58:44.239910 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:44.239649 2568 scope.go:117] "RemoveContainer" containerID="92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049" Jul 06 23:58:44.252079 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:44.252060 2568 scope.go:117] "RemoveContainer" containerID="92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049" Jul 06 23:58:44.252385 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:58:44.252359 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049\": container with ID starting with 92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049 not found: ID does not exist" containerID="92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049" Jul 06 23:58:44.252449 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:44.252384 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049"} err="failed to get container status \"92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049\": rpc error: code = NotFound desc = could not find container \"92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049\": container with ID starting with 92fe234d424dcfa38fdb5d7eaa7ee2a6e85edd4a30b9e3aab2b5f16f21852049 not found: ID does not exist" Jul 06 23:58:44.267684 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:44.267650 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c49c858b8-whsj7"] Jul 06 23:58:44.272772 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:44.272743 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c49c858b8-whsj7"] Jul 06 23:58:45.314204 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:58:45.314166 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2057ad8f-ffdf-46a1-ab60-00e4abf92aea" path="/var/lib/kubelet/pods/2057ad8f-ffdf-46a1-ab60-00e4abf92aea/volumes" Jul 06 23:59:03.791992 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:03.791937 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:59:03.807768 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:03.807740 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:59:04.330540 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:04.330513 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:59:17.079071 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.078987 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d86646f66-x48nn"] Jul 06 23:59:17.079512 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.079307 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2057ad8f-ffdf-46a1-ab60-00e4abf92aea" containerName="console" Jul 06 23:59:17.079512 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.079320 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2057ad8f-ffdf-46a1-ab60-00e4abf92aea" containerName="console" Jul 06 23:59:17.079512 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.079373 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2057ad8f-ffdf-46a1-ab60-00e4abf92aea" containerName="console" Jul 06 23:59:17.082159 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.082143 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.115983 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.115954 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d86646f66-x48nn"] Jul 06 23:59:17.170793 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.170753 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-service-ca\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.170793 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.170793 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-oauth-config\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.170997 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.170914 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-serving-cert\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.170997 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.170949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxhhl\" (UniqueName: \"kubernetes.io/projected/cbec0012-d111-4450-9a32-8f0eed2de5b1-kube-api-access-nxhhl\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.170997 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.170985 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-trusted-ca-bundle\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.171095 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.171019 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-config\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.171095 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.171039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-oauth-serving-cert\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.271988 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.271956 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-serving-cert\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.272184 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.271996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxhhl\" (UniqueName: \"kubernetes.io/projected/cbec0012-d111-4450-9a32-8f0eed2de5b1-kube-api-access-nxhhl\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.272184 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.272171 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-trusted-ca-bundle\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.272306 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.272207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-config\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.272306 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.272239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-oauth-serving-cert\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.272306 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.272283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-service-ca\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.272480 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.272320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-oauth-config\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.273009 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.272981 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-oauth-serving-cert\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.273130 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.273010 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-config\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.273250 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.273230 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-trusted-ca-bundle\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.273340 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.273315 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-service-ca\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.274745 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.274721 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-serving-cert\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.274853 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.274836 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-oauth-config\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.281913 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.281890 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxhhl\" (UniqueName: \"kubernetes.io/projected/cbec0012-d111-4450-9a32-8f0eed2de5b1-kube-api-access-nxhhl\") pod \"console-5d86646f66-x48nn\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.390918 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.390841 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:17.530456 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:17.530407 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d86646f66-x48nn"] Jul 06 23:59:17.533592 ip-10-0-141-86 kubenswrapper[2568]: W0706 23:59:17.533566 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbec0012_d111_4450_9a32_8f0eed2de5b1.slice/crio-c5ee1472c9a2a1ff23e2d401521c5d1103159cebf02c19f7ac612cdeff97e7ec WatchSource:0}: Error finding container c5ee1472c9a2a1ff23e2d401521c5d1103159cebf02c19f7ac612cdeff97e7ec: Status 404 returned error can't find the container with id c5ee1472c9a2a1ff23e2d401521c5d1103159cebf02c19f7ac612cdeff97e7ec Jul 06 23:59:18.370260 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:18.370214 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d86646f66-x48nn" event={"ID":"cbec0012-d111-4450-9a32-8f0eed2de5b1","Type":"ContainerStarted","Data":"68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025"} Jul 06 23:59:18.370738 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:18.370300 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d86646f66-x48nn" event={"ID":"cbec0012-d111-4450-9a32-8f0eed2de5b1","Type":"ContainerStarted","Data":"c5ee1472c9a2a1ff23e2d401521c5d1103159cebf02c19f7ac612cdeff97e7ec"} Jul 06 23:59:18.395988 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:18.395932 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d86646f66-x48nn" podStartSLOduration=1.3959151699999999 podStartE2EDuration="1.39591517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:59:18.393899836 +0000 UTC m=+179.606931580" watchObservedRunningTime="2026-07-06 23:59:18.39591517 +0000 UTC m=+179.608946913" Jul 06 23:59:27.391267 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:27.391213 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:27.391712 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:27.391295 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:27.396025 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:27.396000 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:27.405290 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:27.405269 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d86646f66-x48nn" Jul 06 23:59:27.494337 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:27.494304 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69bf855568-z2q2s"] Jul 06 23:59:52.521483 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.521404 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-69bf855568-z2q2s" podUID="9add24f0-3837-46b2-a4a6-db91933d0b7b" containerName="console" containerID="cri-o://9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878" gracePeriod=15 Jul 06 23:59:52.766776 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.766752 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:59:52.868185 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868101 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfmkf\" (UniqueName: \"kubernetes.io/projected/9add24f0-3837-46b2-a4a6-db91933d0b7b-kube-api-access-qfmkf\") pod \"9add24f0-3837-46b2-a4a6-db91933d0b7b\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " Jul 06 23:59:52.868185 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868159 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-config\") pod \"9add24f0-3837-46b2-a4a6-db91933d0b7b\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " Jul 06 23:59:52.868185 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868186 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-oauth-config\") pod \"9add24f0-3837-46b2-a4a6-db91933d0b7b\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " Jul 06 23:59:52.868473 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868206 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-service-ca\") pod \"9add24f0-3837-46b2-a4a6-db91933d0b7b\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " Jul 06 23:59:52.868473 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868329 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-trusted-ca-bundle\") pod \"9add24f0-3837-46b2-a4a6-db91933d0b7b\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " Jul 06 23:59:52.868473 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868423 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-oauth-serving-cert\") pod \"9add24f0-3837-46b2-a4a6-db91933d0b7b\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " Jul 06 23:59:52.868627 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868487 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-serving-cert\") pod \"9add24f0-3837-46b2-a4a6-db91933d0b7b\" (UID: \"9add24f0-3837-46b2-a4a6-db91933d0b7b\") " Jul 06 23:59:52.868749 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868706 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-service-ca" (OuterVolumeSpecName: "service-ca") pod "9add24f0-3837-46b2-a4a6-db91933d0b7b" (UID: "9add24f0-3837-46b2-a4a6-db91933d0b7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:52.868749 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868735 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-config" (OuterVolumeSpecName: "console-config") pod "9add24f0-3837-46b2-a4a6-db91933d0b7b" (UID: "9add24f0-3837-46b2-a4a6-db91933d0b7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:52.868908 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.868866 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9add24f0-3837-46b2-a4a6-db91933d0b7b" (UID: "9add24f0-3837-46b2-a4a6-db91933d0b7b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:52.869288 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.869254 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9add24f0-3837-46b2-a4a6-db91933d0b7b" (UID: "9add24f0-3837-46b2-a4a6-db91933d0b7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:52.870601 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.870574 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9add24f0-3837-46b2-a4a6-db91933d0b7b" (UID: "9add24f0-3837-46b2-a4a6-db91933d0b7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:59:52.870684 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.870643 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9add24f0-3837-46b2-a4a6-db91933d0b7b-kube-api-access-qfmkf" (OuterVolumeSpecName: "kube-api-access-qfmkf") pod "9add24f0-3837-46b2-a4a6-db91933d0b7b" (UID: "9add24f0-3837-46b2-a4a6-db91933d0b7b"). InnerVolumeSpecName "kube-api-access-qfmkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:59:52.870871 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.870844 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9add24f0-3837-46b2-a4a6-db91933d0b7b" (UID: "9add24f0-3837-46b2-a4a6-db91933d0b7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:59:52.969488 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.969427 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-oauth-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:59:52.969488 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.969482 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:59:52.969488 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.969491 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-trusted-ca-bundle\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:59:52.969488 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.969500 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-oauth-serving-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:59:52.969737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.969508 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-serving-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:59:52.969737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.969518 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfmkf\" (UniqueName: \"kubernetes.io/projected/9add24f0-3837-46b2-a4a6-db91933d0b7b-kube-api-access-qfmkf\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:59:52.969737 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:52.969536 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9add24f0-3837-46b2-a4a6-db91933d0b7b-console-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 06 23:59:53.501366 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:53.501325 2568 generic.go:356] "Generic (PLEG): container finished" podID="9add24f0-3837-46b2-a4a6-db91933d0b7b" containerID="9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878" exitCode=0 Jul 06 23:59:53.501544 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:53.501374 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69bf855568-z2q2s" Jul 06 23:59:53.501544 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:53.501421 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69bf855568-z2q2s" event={"ID":"9add24f0-3837-46b2-a4a6-db91933d0b7b","Type":"ContainerDied","Data":"9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878"} Jul 06 23:59:53.501544 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:53.501502 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69bf855568-z2q2s" event={"ID":"9add24f0-3837-46b2-a4a6-db91933d0b7b","Type":"ContainerDied","Data":"2eb0a165ec3ac0ad62566890efe4a069ca1c9f6d813db5c8cc37bb60b89b124c"} Jul 06 23:59:53.501544 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:53.501523 2568 scope.go:117] "RemoveContainer" containerID="9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878" Jul 06 23:59:53.509695 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:53.509662 2568 scope.go:117] "RemoveContainer" containerID="9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878" Jul 06 23:59:53.509981 ip-10-0-141-86 kubenswrapper[2568]: E0706 23:59:53.509956 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878\": container with ID starting with 9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878 not found: ID does not exist" containerID="9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878" Jul 06 23:59:53.510046 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:53.509989 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878"} err="failed to get container status \"9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878\": rpc error: code = NotFound desc = could not find container \"9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878\": container with ID starting with 9985f63650256fc317d710daee100386895e151729bad0715d40bd7bd975a878 not found: ID does not exist" Jul 06 23:59:53.519335 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:53.519306 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69bf855568-z2q2s"] Jul 06 23:59:53.521643 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:53.521619 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69bf855568-z2q2s"] Jul 06 23:59:55.314305 ip-10-0-141-86 kubenswrapper[2568]: I0706 23:59:55.314257 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9add24f0-3837-46b2-a4a6-db91933d0b7b" path="/var/lib/kubelet/pods/9add24f0-3837-46b2-a4a6-db91933d0b7b/volumes" Jul 07 00:00:32.129490 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.129428 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xtj8b"] Jul 07 00:00:32.129974 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.129722 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9add24f0-3837-46b2-a4a6-db91933d0b7b" containerName="console" Jul 07 00:00:32.129974 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.129733 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9add24f0-3837-46b2-a4a6-db91933d0b7b" containerName="console" Jul 07 00:00:32.129974 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.129818 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9add24f0-3837-46b2-a4a6-db91933d0b7b" containerName="console" Jul 07 00:00:32.132789 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.132771 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xtj8b" Jul 07 00:00:32.135476 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.135453 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qvpcm\"" Jul 07 00:00:32.135476 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.135464 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 00:00:32.136339 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.136326 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 00:00:32.146288 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.146263 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xtj8b"] Jul 07 00:00:32.195684 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.195657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdc7\" (UniqueName: \"kubernetes.io/projected/b80aab9e-7c56-430c-a45f-4eeb45d0e807-kube-api-access-fcdc7\") pod \"cert-manager-55b4d954d5-xtj8b\" (UID: \"b80aab9e-7c56-430c-a45f-4eeb45d0e807\") " pod="cert-manager/cert-manager-55b4d954d5-xtj8b" Jul 07 00:00:32.195848 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.195696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b80aab9e-7c56-430c-a45f-4eeb45d0e807-bound-sa-token\") pod \"cert-manager-55b4d954d5-xtj8b\" (UID: \"b80aab9e-7c56-430c-a45f-4eeb45d0e807\") " pod="cert-manager/cert-manager-55b4d954d5-xtj8b" Jul 07 00:00:32.296312 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.296273 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdc7\" (UniqueName: \"kubernetes.io/projected/b80aab9e-7c56-430c-a45f-4eeb45d0e807-kube-api-access-fcdc7\") pod \"cert-manager-55b4d954d5-xtj8b\" (UID: \"b80aab9e-7c56-430c-a45f-4eeb45d0e807\") " pod="cert-manager/cert-manager-55b4d954d5-xtj8b" Jul 07 00:00:32.296534 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.296330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b80aab9e-7c56-430c-a45f-4eeb45d0e807-bound-sa-token\") pod \"cert-manager-55b4d954d5-xtj8b\" (UID: \"b80aab9e-7c56-430c-a45f-4eeb45d0e807\") " pod="cert-manager/cert-manager-55b4d954d5-xtj8b" Jul 07 00:00:32.308895 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.306602 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b80aab9e-7c56-430c-a45f-4eeb45d0e807-bound-sa-token\") pod \"cert-manager-55b4d954d5-xtj8b\" (UID: \"b80aab9e-7c56-430c-a45f-4eeb45d0e807\") " pod="cert-manager/cert-manager-55b4d954d5-xtj8b" Jul 07 00:00:32.308895 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.306781 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdc7\" (UniqueName: \"kubernetes.io/projected/b80aab9e-7c56-430c-a45f-4eeb45d0e807-kube-api-access-fcdc7\") pod \"cert-manager-55b4d954d5-xtj8b\" (UID: \"b80aab9e-7c56-430c-a45f-4eeb45d0e807\") " pod="cert-manager/cert-manager-55b4d954d5-xtj8b" Jul 07 00:00:32.451214 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.451181 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xtj8b" Jul 07 00:00:32.580239 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.580148 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xtj8b"] Jul 07 00:00:32.582990 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:00:32.582966 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb80aab9e_7c56_430c_a45f_4eeb45d0e807.slice/crio-0c12eded7a095fb5fedf58f573319340228b82110e54c1b63903eb1ec32806b9 WatchSource:0}: Error finding container 0c12eded7a095fb5fedf58f573319340228b82110e54c1b63903eb1ec32806b9: Status 404 returned error can't find the container with id 0c12eded7a095fb5fedf58f573319340228b82110e54c1b63903eb1ec32806b9 Jul 07 00:00:32.648577 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:32.648543 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xtj8b" event={"ID":"b80aab9e-7c56-430c-a45f-4eeb45d0e807","Type":"ContainerStarted","Data":"0c12eded7a095fb5fedf58f573319340228b82110e54c1b63903eb1ec32806b9"} Jul 07 00:00:36.666992 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:36.666958 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xtj8b" event={"ID":"b80aab9e-7c56-430c-a45f-4eeb45d0e807","Type":"ContainerStarted","Data":"8e69f7497911de71a43079afacbcf7efd137b4669bcd67740b99d98b29165791"} Jul 07 00:00:36.683362 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:36.683311 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xtj8b" podStartSLOduration=1.5503926030000001 podStartE2EDuration="4.683293582s" totalImagesPullingTime="3.132900979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:00:36.682580315 +0000 UTC m=+257.895612060" watchObservedRunningTime="2026-07-07 00:00:36.683293582 +0000 UTC m=+257.896325329" Jul 07 00:00:41.411676 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.411592 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2"] Jul 07 00:00:41.414288 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.414272 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" Jul 07 00:00:41.416745 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.416718 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 00:00:41.416867 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.416773 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7f8jg\"" Jul 07 00:00:41.417766 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.417749 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 00:00:41.423902 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.423874 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2"] Jul 07 00:00:41.470347 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.470307 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qwb\" (UniqueName: \"kubernetes.io/projected/3b5e5455-2da6-4277-978c-002c919531f4-kube-api-access-z6qwb\") pod \"openshift-lws-operator-fd8ccff4c-zltx2\" (UID: \"3b5e5455-2da6-4277-978c-002c919531f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" Jul 07 00:00:41.470534 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.470362 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5e5455-2da6-4277-978c-002c919531f4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zltx2\" (UID: \"3b5e5455-2da6-4277-978c-002c919531f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" Jul 07 00:00:41.570720 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.570688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qwb\" (UniqueName: \"kubernetes.io/projected/3b5e5455-2da6-4277-978c-002c919531f4-kube-api-access-z6qwb\") pod \"openshift-lws-operator-fd8ccff4c-zltx2\" (UID: \"3b5e5455-2da6-4277-978c-002c919531f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" Jul 07 00:00:41.570852 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.570748 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5e5455-2da6-4277-978c-002c919531f4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zltx2\" (UID: \"3b5e5455-2da6-4277-978c-002c919531f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" Jul 07 00:00:41.571142 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.571125 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5e5455-2da6-4277-978c-002c919531f4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zltx2\" (UID: \"3b5e5455-2da6-4277-978c-002c919531f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" Jul 07 00:00:41.579505 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.579473 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qwb\" (UniqueName: \"kubernetes.io/projected/3b5e5455-2da6-4277-978c-002c919531f4-kube-api-access-z6qwb\") pod \"openshift-lws-operator-fd8ccff4c-zltx2\" (UID: \"3b5e5455-2da6-4277-978c-002c919531f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" Jul 07 00:00:41.723199 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.723112 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" Jul 07 00:00:41.853171 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:41.853062 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2"] Jul 07 00:00:41.855820 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:00:41.855786 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b5e5455_2da6_4277_978c_002c919531f4.slice/crio-af3f7808f29e655a5e12160ad4e54343e91fd85a80bc18b96a674f485aec654c WatchSource:0}: Error finding container af3f7808f29e655a5e12160ad4e54343e91fd85a80bc18b96a674f485aec654c: Status 404 returned error can't find the container with id af3f7808f29e655a5e12160ad4e54343e91fd85a80bc18b96a674f485aec654c Jul 07 00:00:42.693105 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:42.693057 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" event={"ID":"3b5e5455-2da6-4277-978c-002c919531f4","Type":"ContainerStarted","Data":"af3f7808f29e655a5e12160ad4e54343e91fd85a80bc18b96a674f485aec654c"} Jul 07 00:00:44.702792 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:44.702753 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" event={"ID":"3b5e5455-2da6-4277-978c-002c919531f4","Type":"ContainerStarted","Data":"5c0cbe215c697b6fd9c9517e4f8ac4fda2b2e918918998854d49d9a8be31986a"} Jul 07 00:00:44.720755 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:00:44.720698 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zltx2" podStartSLOduration=1.4733137250000001 podStartE2EDuration="3.720677778s" totalImagesPullingTime="2.247364053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:00:44.718658812 +0000 UTC m=+265.931690560" watchObservedRunningTime="2026-07-07 00:00:44.720677778 +0000 UTC m=+265.933709527" Jul 07 00:01:00.540078 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.540037 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq"] Jul 07 00:01:00.543248 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.543231 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.550542 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.550522 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 00:01:00.550655 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.550521 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 00:01:00.550722 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.550596 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 00:01:00.551104 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.551079 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-swm29\"" Jul 07 00:01:00.551172 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.551157 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 00:01:00.566200 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.566165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq"] Jul 07 00:01:00.628037 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.627999 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-manifests\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.628210 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.628053 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.628210 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.628099 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9sn\" (UniqueName: \"kubernetes.io/projected/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-kube-api-access-7q9sn\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.628210 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.628169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-tmp\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.628354 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.628216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.728646 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.728614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-tmp\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.728849 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.728667 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.728849 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.728710 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-manifests\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.728849 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.728732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.728849 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.728751 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9sn\" (UniqueName: \"kubernetes.io/projected/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-kube-api-access-7q9sn\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.729171 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.729147 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-manifests\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.731175 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.731151 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-tmp\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.731302 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.731253 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.731302 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.731273 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.739003 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.738978 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9sn\" (UniqueName: \"kubernetes.io/projected/6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8-kube-api-access-7q9sn\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-56xfq\" (UID: \"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:00.853520 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:00.853407 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:01.007660 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:01.007630 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq"] Jul 07 00:01:01.011228 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:01:01.011189 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abdbb4c_5f07_42f7_a6e4_8696c76fd3a8.slice/crio-56399a586c4252b6c848f6e97b5db2043bd1693fdd3b50ed9008e8fd4c389842 WatchSource:0}: Error finding container 56399a586c4252b6c848f6e97b5db2043bd1693fdd3b50ed9008e8fd4c389842: Status 404 returned error can't find the container with id 56399a586c4252b6c848f6e97b5db2043bd1693fdd3b50ed9008e8fd4c389842 Jul 07 00:01:01.769522 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:01.769477 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" event={"ID":"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8","Type":"ContainerStarted","Data":"56399a586c4252b6c848f6e97b5db2043bd1693fdd3b50ed9008e8fd4c389842"} Jul 07 00:01:04.784413 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:04.784372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" event={"ID":"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8","Type":"ContainerStarted","Data":"bce6184124b42e0f04dabf6467c0bb52d6fae5761e4a9e7cb05b84910ba6e8ac"} Jul 07 00:01:05.790233 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:05.790202 2568 generic.go:356] "Generic (PLEG): container finished" podID="6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8" containerID="bce6184124b42e0f04dabf6467c0bb52d6fae5761e4a9e7cb05b84910ba6e8ac" exitCode=0 Jul 07 00:01:05.790676 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:05.790292 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" event={"ID":"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8","Type":"ContainerDied","Data":"bce6184124b42e0f04dabf6467c0bb52d6fae5761e4a9e7cb05b84910ba6e8ac"} Jul 07 00:01:06.797426 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:06.797390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" event={"ID":"6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8","Type":"ContainerStarted","Data":"40c49d0ed639c2a4026d9667fb01638fbd26be004fc6d8b5e51303ef09aa1fdb"} Jul 07 00:01:06.797841 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:06.797549 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:06.829576 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:06.829519 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" podStartSLOduration=2.47805153 podStartE2EDuration="6.829501339s" totalImagesPullingTime="3.255413849s" totalInitContainerRuntime="1.09603596s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:06.827121336 +0000 UTC m=+288.040153099" watchObservedRunningTime="2026-07-07 00:01:06.829501339 +0000 UTC m=+288.042533082" Jul 07 00:01:17.804075 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:17.804044 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-56xfq" Jul 07 00:01:19.162231 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:19.162201 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:01:19.162676 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:19.162410 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:01:19.167427 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:19.167406 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 00:01:20.378976 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.378941 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4"] Jul 07 00:01:20.380961 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.380941 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.383609 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.383588 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 00:01:20.384719 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.384697 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 00:01:20.384842 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.384730 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 00:01:20.384842 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.384697 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z9lxp\"" Jul 07 00:01:20.391405 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.391378 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4"] Jul 07 00:01:20.414168 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.414137 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1182c64-551f-485a-b95c-fec6ad91086c-metrics-cert\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.414326 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.414208 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwzf\" (UniqueName: \"kubernetes.io/projected/b1182c64-551f-485a-b95c-fec6ad91086c-kube-api-access-jtwzf\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.414326 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.414272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1182c64-551f-485a-b95c-fec6ad91086c-cert\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.414326 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.414300 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b1182c64-551f-485a-b95c-fec6ad91086c-manager-config\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.515558 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.515521 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1182c64-551f-485a-b95c-fec6ad91086c-cert\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.515558 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.515560 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b1182c64-551f-485a-b95c-fec6ad91086c-manager-config\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.515777 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.515582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1182c64-551f-485a-b95c-fec6ad91086c-metrics-cert\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.515777 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.515637 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwzf\" (UniqueName: \"kubernetes.io/projected/b1182c64-551f-485a-b95c-fec6ad91086c-kube-api-access-jtwzf\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.516335 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.516305 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b1182c64-551f-485a-b95c-fec6ad91086c-manager-config\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.518252 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.518233 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1182c64-551f-485a-b95c-fec6ad91086c-metrics-cert\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.518367 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.518350 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1182c64-551f-485a-b95c-fec6ad91086c-cert\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.524748 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.524723 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwzf\" (UniqueName: \"kubernetes.io/projected/b1182c64-551f-485a-b95c-fec6ad91086c-kube-api-access-jtwzf\") pod \"lws-controller-manager-6647df896d-zlmc4\" (UID: \"b1182c64-551f-485a-b95c-fec6ad91086c\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.690837 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.690795 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:20.871168 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:01:20.871132 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1182c64_551f_485a_b95c_fec6ad91086c.slice/crio-ea535edca4c2336e877629ff326c8e9200337bc8032a6b159a589a1698d960ab WatchSource:0}: Error finding container ea535edca4c2336e877629ff326c8e9200337bc8032a6b159a589a1698d960ab: Status 404 returned error can't find the container with id ea535edca4c2336e877629ff326c8e9200337bc8032a6b159a589a1698d960ab Jul 07 00:01:20.873887 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.873864 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:01:20.874881 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:20.874857 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4"] Jul 07 00:01:21.858899 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:21.858858 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" event={"ID":"b1182c64-551f-485a-b95c-fec6ad91086c","Type":"ContainerStarted","Data":"ea535edca4c2336e877629ff326c8e9200337bc8032a6b159a589a1698d960ab"} Jul 07 00:01:22.865543 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:22.865501 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" event={"ID":"b1182c64-551f-485a-b95c-fec6ad91086c","Type":"ContainerStarted","Data":"4c6d0529414022721638ee3d0915dfdeeac4218b89e0923d6908e59c6a29fddc"} Jul 07 00:01:22.865959 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:22.865634 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:22.917548 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:22.917491 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" podStartSLOduration=1.5066985050000001 podStartE2EDuration="2.91747365s" totalImagesPullingTime="1.410775145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:22.917295741 +0000 UTC m=+304.130327485" watchObservedRunningTime="2026-07-07 00:01:22.91747365 +0000 UTC m=+304.130505394" Jul 07 00:01:29.036302 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.036260 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk"] Jul 07 00:01:29.038734 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.038712 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.041936 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.041913 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nlr46\"" Jul 07 00:01:29.042272 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.042256 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 00:01:29.042469 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.042453 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 00:01:29.043469 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.043451 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 00:01:29.081733 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.081699 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk"] Jul 07 00:01:29.098127 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.098094 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.098127 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.098129 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2pf\" (UniqueName: \"kubernetes.io/projected/c3e9c576-87f4-4523-9b35-7d502774328f-kube-api-access-rj2pf\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.098306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.098156 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.098306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.098223 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e9c576-87f4-4523-9b35-7d502774328f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.098306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.098257 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.098306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.098289 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.098306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.098304 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e9c576-87f4-4523-9b35-7d502774328f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.098485 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.098347 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.098485 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.098371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e9c576-87f4-4523-9b35-7d502774328f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.199225 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199184 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.199225 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199222 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2pf\" (UniqueName: \"kubernetes.io/projected/c3e9c576-87f4-4523-9b35-7d502774328f-kube-api-access-rj2pf\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.199521 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199242 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.199521 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e9c576-87f4-4523-9b35-7d502774328f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.199521 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199293 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.199521 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199338 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.199521 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e9c576-87f4-4523-9b35-7d502774328f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.199521 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199395 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.199521 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199461 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e9c576-87f4-4523-9b35-7d502774328f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.200049 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.199988 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.200170 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.200045 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.200170 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.200064 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.200170 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.200113 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.200170 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.200153 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e9c576-87f4-4523-9b35-7d502774328f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.201970 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.201950 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c576-87f4-4523-9b35-7d502774328f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.204109 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.202502 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e9c576-87f4-4523-9b35-7d502774328f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.209315 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.209286 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2pf\" (UniqueName: \"kubernetes.io/projected/c3e9c576-87f4-4523-9b35-7d502774328f-kube-api-access-rj2pf\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.209734 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.209708 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e9c576-87f4-4523-9b35-7d502774328f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dpgtk\" (UID: \"c3e9c576-87f4-4523-9b35-7d502774328f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.349020 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.348935 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:29.530773 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.530743 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk"] Jul 07 00:01:29.532887 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:01:29.532859 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e9c576_87f4_4523_9b35_7d502774328f.slice/crio-b2fdc69b7a9f38a5cb86100ce74752fc7feb922f1f7944e141b3db6c8921ef70 WatchSource:0}: Error finding container b2fdc69b7a9f38a5cb86100ce74752fc7feb922f1f7944e141b3db6c8921ef70: Status 404 returned error can't find the container with id b2fdc69b7a9f38a5cb86100ce74752fc7feb922f1f7944e141b3db6c8921ef70 Jul 07 00:01:29.894652 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:29.894607 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" event={"ID":"c3e9c576-87f4-4523-9b35-7d502774328f","Type":"ContainerStarted","Data":"b2fdc69b7a9f38a5cb86100ce74752fc7feb922f1f7944e141b3db6c8921ef70"} Jul 07 00:01:31.914043 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:31.914007 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:01:31.914272 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:31.914086 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:01:31.914272 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:31.914115 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:01:32.909740 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:32.909703 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" event={"ID":"c3e9c576-87f4-4523-9b35-7d502774328f","Type":"ContainerStarted","Data":"54bb606937bac8a10527bc956943b6d7d4a9f0018acf3a4804fdef52f52d4682"} Jul 07 00:01:32.969623 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:32.969574 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" podStartSLOduration=1.590677069 podStartE2EDuration="3.969556234s" totalImagesPullingTime="2.378879165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:32.968005932 +0000 UTC m=+314.181037676" watchObservedRunningTime="2026-07-07 00:01:32.969556234 +0000 UTC m=+314.182587978" Jul 07 00:01:33.349750 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:33.349713 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:33.354543 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:33.354518 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:33.871708 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:33.871672 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6647df896d-zlmc4" Jul 07 00:01:33.914637 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:33.914605 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:33.915550 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:33.915530 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dpgtk" Jul 07 00:01:34.495455 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.493581 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x"] Jul 07 00:01:34.503158 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.503127 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.505947 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.505903 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7qhsd\"" Jul 07 00:01:34.506096 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.506037 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 00:01:34.506457 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.506423 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 00:01:34.506519 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.506425 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 00:01:34.529892 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.529862 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x"] Jul 07 00:01:34.645673 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.645636 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72d39ae9-8292-482d-8c63-6b1f8755f924-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.645857 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.645702 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.645857 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.645794 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.645857 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.645844 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.646011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.645870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkct\" (UniqueName: \"kubernetes.io/projected/72d39ae9-8292-482d-8c63-6b1f8755f924-kube-api-access-mxkct\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.646011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.645892 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.646011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.645912 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d39ae9-8292-482d-8c63-6b1f8755f924-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.746768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.746683 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.746768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.746733 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.746768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.746755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.747072 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.746779 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkct\" (UniqueName: \"kubernetes.io/projected/72d39ae9-8292-482d-8c63-6b1f8755f924-kube-api-access-mxkct\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.747072 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.746806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.747072 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.746926 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d39ae9-8292-482d-8c63-6b1f8755f924-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.747072 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.747009 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72d39ae9-8292-482d-8c63-6b1f8755f924-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.747602 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.747535 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.749466 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.749422 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.749560 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.749467 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72d39ae9-8292-482d-8c63-6b1f8755f924-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.749560 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.749468 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d39ae9-8292-482d-8c63-6b1f8755f924-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.749636 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.749574 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.758232 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.758208 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkct\" (UniqueName: \"kubernetes.io/projected/72d39ae9-8292-482d-8c63-6b1f8755f924-kube-api-access-mxkct\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.759124 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.759106 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d39ae9-8292-482d-8c63-6b1f8755f924-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4zb4x\" (UID: \"72d39ae9-8292-482d-8c63-6b1f8755f924\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:34.812202 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:34.812151 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:35.013191 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:35.013107 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x"] Jul 07 00:01:35.017935 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:01:35.017905 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d39ae9_8292_482d_8c63_6b1f8755f924.slice/crio-c2378abc8414b60ec29c57b3406c768621a26a90dca2f4ccb54ace0f441d9083 WatchSource:0}: Error finding container c2378abc8414b60ec29c57b3406c768621a26a90dca2f4ccb54ace0f441d9083: Status 404 returned error can't find the container with id c2378abc8414b60ec29c57b3406c768621a26a90dca2f4ccb54ace0f441d9083 Jul 07 00:01:35.926906 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:35.926835 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" event={"ID":"72d39ae9-8292-482d-8c63-6b1f8755f924","Type":"ContainerStarted","Data":"c2378abc8414b60ec29c57b3406c768621a26a90dca2f4ccb54ace0f441d9083"} Jul 07 00:01:37.372924 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:37.372879 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:01:37.373294 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:37.372958 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:01:37.938575 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:37.938530 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" event={"ID":"72d39ae9-8292-482d-8c63-6b1f8755f924","Type":"ContainerStarted","Data":"4f019fd16665398ee192ef3a5037316aed2f8fce941a5c9ffb4f60ad790a5ae2"} Jul 07 00:01:37.938742 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:37.938669 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:01:37.982745 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:37.982679 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" podStartSLOduration=1.629985536 podStartE2EDuration="3.982656697s" totalImagesPullingTime="2.352671161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:37.980762346 +0000 UTC m=+319.193794110" watchObservedRunningTime="2026-07-07 00:01:37.982656697 +0000 UTC m=+319.195688433" Jul 07 00:01:38.945141 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:01:38.945111 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4zb4x" Jul 07 00:03:08.768637 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.768606 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56f854bd96-9g642"] Jul 07 00:03:08.771356 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.771337 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.786391 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.786362 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f854bd96-9g642"] Jul 07 00:03:08.884089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.884044 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-console-config\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.884273 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.884118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dff2386-88af-468c-88c6-304323056720-console-serving-cert\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.884273 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.884163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z59k7\" (UniqueName: \"kubernetes.io/projected/8dff2386-88af-468c-88c6-304323056720-kube-api-access-z59k7\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.884273 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.884184 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dff2386-88af-468c-88c6-304323056720-console-oauth-config\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.884273 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.884205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-trusted-ca-bundle\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.884273 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.884236 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-service-ca\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.884469 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.884279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-oauth-serving-cert\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.985165 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.985120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z59k7\" (UniqueName: \"kubernetes.io/projected/8dff2386-88af-468c-88c6-304323056720-kube-api-access-z59k7\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.985165 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.985171 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dff2386-88af-468c-88c6-304323056720-console-oauth-config\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.985459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.985186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-trusted-ca-bundle\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.985459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.985212 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-service-ca\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.985459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.985267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-oauth-serving-cert\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.985459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.985310 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-console-config\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.985459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.985371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dff2386-88af-468c-88c6-304323056720-console-serving-cert\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.986138 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.986105 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-oauth-serving-cert\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.986269 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.986105 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-service-ca\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.986269 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.986218 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-trusted-ca-bundle\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.986357 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.986308 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dff2386-88af-468c-88c6-304323056720-console-config\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.987912 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.987884 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dff2386-88af-468c-88c6-304323056720-console-oauth-config\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.988002 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.987923 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dff2386-88af-468c-88c6-304323056720-console-serving-cert\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:08.995181 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:08.995156 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z59k7\" (UniqueName: \"kubernetes.io/projected/8dff2386-88af-468c-88c6-304323056720-kube-api-access-z59k7\") pod \"console-56f854bd96-9g642\" (UID: \"8dff2386-88af-468c-88c6-304323056720\") " pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:09.080225 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:09.080125 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:09.214631 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:09.214566 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f854bd96-9g642"] Jul 07 00:03:09.217218 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:03:09.217190 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dff2386_88af_468c_88c6_304323056720.slice/crio-d8f4cf21206a429edede798064b91848f218452a4b7a0a96a81bf9cb0e1b10a0 WatchSource:0}: Error finding container d8f4cf21206a429edede798064b91848f218452a4b7a0a96a81bf9cb0e1b10a0: Status 404 returned error can't find the container with id d8f4cf21206a429edede798064b91848f218452a4b7a0a96a81bf9cb0e1b10a0 Jul 07 00:03:09.331794 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:09.331696 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f854bd96-9g642" event={"ID":"8dff2386-88af-468c-88c6-304323056720","Type":"ContainerStarted","Data":"c6e5eb045d776047a4517a41e028d4c4c1e0492fd1e384b1514a3bf1c8c36841"} Jul 07 00:03:09.331794 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:09.331761 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f854bd96-9g642" event={"ID":"8dff2386-88af-468c-88c6-304323056720","Type":"ContainerStarted","Data":"d8f4cf21206a429edede798064b91848f218452a4b7a0a96a81bf9cb0e1b10a0"} Jul 07 00:03:09.354067 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:09.354004 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56f854bd96-9g642" podStartSLOduration=1.353985167 podStartE2EDuration="1.353985167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:03:09.35217333 +0000 UTC m=+410.565205073" watchObservedRunningTime="2026-07-07 00:03:09.353985167 +0000 UTC m=+410.567016911" Jul 07 00:03:19.080297 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:19.080253 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:19.080783 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:19.080311 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:19.085105 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:19.085084 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:19.378360 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:19.378272 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56f854bd96-9g642" Jul 07 00:03:19.452599 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:19.452560 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d86646f66-x48nn"] Jul 07 00:03:44.473672 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.473556 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5d86646f66-x48nn" podUID="cbec0012-d111-4450-9a32-8f0eed2de5b1" containerName="console" containerID="cri-o://68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025" gracePeriod=15 Jul 07 00:03:44.723944 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.723877 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d86646f66-x48nn" Jul 07 00:03:44.810145 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810115 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-oauth-config\") pod \"cbec0012-d111-4450-9a32-8f0eed2de5b1\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " Jul 07 00:03:44.810145 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810148 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxhhl\" (UniqueName: \"kubernetes.io/projected/cbec0012-d111-4450-9a32-8f0eed2de5b1-kube-api-access-nxhhl\") pod \"cbec0012-d111-4450-9a32-8f0eed2de5b1\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " Jul 07 00:03:44.810349 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810166 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-config\") pod \"cbec0012-d111-4450-9a32-8f0eed2de5b1\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " Jul 07 00:03:44.810349 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810211 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-service-ca\") pod \"cbec0012-d111-4450-9a32-8f0eed2de5b1\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " Jul 07 00:03:44.810475 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810341 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-oauth-serving-cert\") pod \"cbec0012-d111-4450-9a32-8f0eed2de5b1\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " Jul 07 00:03:44.810642 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810582 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "cbec0012-d111-4450-9a32-8f0eed2de5b1" (UID: "cbec0012-d111-4450-9a32-8f0eed2de5b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:03:44.810642 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810428 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-serving-cert\") pod \"cbec0012-d111-4450-9a32-8f0eed2de5b1\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " Jul 07 00:03:44.810832 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810649 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-config" (OuterVolumeSpecName: "console-config") pod "cbec0012-d111-4450-9a32-8f0eed2de5b1" (UID: "cbec0012-d111-4450-9a32-8f0eed2de5b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:03:44.810832 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810725 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-trusted-ca-bundle\") pod \"cbec0012-d111-4450-9a32-8f0eed2de5b1\" (UID: \"cbec0012-d111-4450-9a32-8f0eed2de5b1\") " Jul 07 00:03:44.810832 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810769 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cbec0012-d111-4450-9a32-8f0eed2de5b1" (UID: "cbec0012-d111-4450-9a32-8f0eed2de5b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:03:44.811306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810951 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:03:44.811306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810970 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:03:44.811306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.810983 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-oauth-serving-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:03:44.811499 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.811394 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cbec0012-d111-4450-9a32-8f0eed2de5b1" (UID: "cbec0012-d111-4450-9a32-8f0eed2de5b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:03:44.812621 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.812599 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cbec0012-d111-4450-9a32-8f0eed2de5b1" (UID: "cbec0012-d111-4450-9a32-8f0eed2de5b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:03:44.812710 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.812634 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cbec0012-d111-4450-9a32-8f0eed2de5b1" (UID: "cbec0012-d111-4450-9a32-8f0eed2de5b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:03:44.812710 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.812665 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbec0012-d111-4450-9a32-8f0eed2de5b1-kube-api-access-nxhhl" (OuterVolumeSpecName: "kube-api-access-nxhhl") pod "cbec0012-d111-4450-9a32-8f0eed2de5b1" (UID: "cbec0012-d111-4450-9a32-8f0eed2de5b1"). InnerVolumeSpecName "kube-api-access-nxhhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:03:44.911570 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.911535 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-serving-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:03:44.911570 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.911563 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbec0012-d111-4450-9a32-8f0eed2de5b1-trusted-ca-bundle\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:03:44.911570 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.911572 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbec0012-d111-4450-9a32-8f0eed2de5b1-console-oauth-config\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:03:44.911570 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:44.911582 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxhhl\" (UniqueName: \"kubernetes.io/projected/cbec0012-d111-4450-9a32-8f0eed2de5b1-kube-api-access-nxhhl\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:03:45.496852 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:45.496816 2568 generic.go:356] "Generic (PLEG): container finished" podID="cbec0012-d111-4450-9a32-8f0eed2de5b1" containerID="68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025" exitCode=0 Jul 07 00:03:45.497334 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:45.496869 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d86646f66-x48nn" Jul 07 00:03:45.497334 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:45.496858 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d86646f66-x48nn" event={"ID":"cbec0012-d111-4450-9a32-8f0eed2de5b1","Type":"ContainerDied","Data":"68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025"} Jul 07 00:03:45.497334 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:45.497007 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d86646f66-x48nn" event={"ID":"cbec0012-d111-4450-9a32-8f0eed2de5b1","Type":"ContainerDied","Data":"c5ee1472c9a2a1ff23e2d401521c5d1103159cebf02c19f7ac612cdeff97e7ec"} Jul 07 00:03:45.497334 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:45.497048 2568 scope.go:117] "RemoveContainer" containerID="68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025" Jul 07 00:03:45.505419 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:45.505397 2568 scope.go:117] "RemoveContainer" containerID="68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025" Jul 07 00:03:45.505752 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:03:45.505729 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025\": container with ID starting with 68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025 not found: ID does not exist" containerID="68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025" Jul 07 00:03:45.505832 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:45.505767 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025"} err="failed to get container status \"68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025\": rpc error: code = NotFound desc = could not find container \"68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025\": container with ID starting with 68bbadf0fee9679254fe2dbc66aaba6c9d28f28506843117dba2b8c984ff4025 not found: ID does not exist" Jul 07 00:03:45.538492 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:45.538456 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d86646f66-x48nn"] Jul 07 00:03:45.549550 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:45.549517 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d86646f66-x48nn"] Jul 07 00:03:47.315233 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:47.315203 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbec0012-d111-4450-9a32-8f0eed2de5b1" path="/var/lib/kubelet/pods/cbec0012-d111-4450-9a32-8f0eed2de5b1/volumes" Jul 07 00:03:53.359468 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.359419 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj"] Jul 07 00:03:53.359901 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.359742 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbec0012-d111-4450-9a32-8f0eed2de5b1" containerName="console" Jul 07 00:03:53.359901 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.359754 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbec0012-d111-4450-9a32-8f0eed2de5b1" containerName="console" Jul 07 00:03:53.359901 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.359812 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbec0012-d111-4450-9a32-8f0eed2de5b1" containerName="console" Jul 07 00:03:53.362491 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.362474 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.365378 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.365355 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c26zl\"" Jul 07 00:03:53.383167 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.383136 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj"] Jul 07 00:03:53.490840 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.490802 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f2db2c-526b-427e-a103-215869094379-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.491011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.490844 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.491011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.490921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.491011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.490937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f2db2c-526b-427e-a103-215869094379-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.491011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.490958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f2db2c-526b-427e-a103-215869094379-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.491149 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.491037 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.491149 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.491066 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.491149 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.491103 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsflm\" (UniqueName: \"kubernetes.io/projected/91f2db2c-526b-427e-a103-215869094379-kube-api-access-bsflm\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.491149 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.491130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592247 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592212 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f2db2c-526b-427e-a103-215869094379-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592475 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592475 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592475 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsflm\" (UniqueName: \"kubernetes.io/projected/91f2db2c-526b-427e-a103-215869094379-kube-api-access-bsflm\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592475 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592475 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f2db2c-526b-427e-a103-215869094379-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592475 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592475 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592426 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592475 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592476 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f2db2c-526b-427e-a103-215869094379-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592896 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592822 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.592896 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592878 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.593011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.592926 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.593063 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.593035 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f2db2c-526b-427e-a103-215869094379-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.593229 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.593208 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.594768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.594749 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f2db2c-526b-427e-a103-215869094379-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.595092 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.595075 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f2db2c-526b-427e-a103-215869094379-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.601888 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.601865 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f2db2c-526b-427e-a103-215869094379-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.602318 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.602299 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsflm\" (UniqueName: \"kubernetes.io/projected/91f2db2c-526b-427e-a103-215869094379-kube-api-access-bsflm\") pod \"maas-default-gateway-openshift-default-56654dcf7-khdfj\" (UID: \"91f2db2c-526b-427e-a103-215869094379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.672605 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.672564 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:53.815370 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.815343 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj"] Jul 07 00:03:53.817202 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:03:53.817171 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f2db2c_526b_427e_a103_215869094379.slice/crio-8a204aa230982ec8e4f0b1f1d72a89415c8dc3b6c03cc7fc4be5f1a89176bc09 WatchSource:0}: Error finding container 8a204aa230982ec8e4f0b1f1d72a89415c8dc3b6c03cc7fc4be5f1a89176bc09: Status 404 returned error can't find the container with id 8a204aa230982ec8e4f0b1f1d72a89415c8dc3b6c03cc7fc4be5f1a89176bc09 Jul 07 00:03:53.819317 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.819289 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:03:53.819371 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.819356 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:03:53.819410 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:53.819385 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:03:54.538547 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:54.538510 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" event={"ID":"91f2db2c-526b-427e-a103-215869094379","Type":"ContainerStarted","Data":"e88b66196c212b67ec04f9f72458b641c2290cd907850fb8d33a40da7085cfdc"} Jul 07 00:03:54.539038 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:54.538560 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" event={"ID":"91f2db2c-526b-427e-a103-215869094379","Type":"ContainerStarted","Data":"8a204aa230982ec8e4f0b1f1d72a89415c8dc3b6c03cc7fc4be5f1a89176bc09"} Jul 07 00:03:54.563034 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:54.562980 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" podStartSLOduration=1.5629616610000001 podStartE2EDuration="1.562961661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:03:54.561290886 +0000 UTC m=+455.774322642" watchObservedRunningTime="2026-07-07 00:03:54.562961661 +0000 UTC m=+455.775993404" Jul 07 00:03:54.672812 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:54.672779 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:54.678298 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:54.678268 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:55.543711 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:55.543673 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:03:55.544608 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:03:55.544588 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:04:36.563759 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.563725 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f546649f9-2tzwr"] Jul 07 00:04:36.568447 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.568412 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:36.572468 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.572424 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 00:04:36.572576 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.572551 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 00:04:36.573886 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.573866 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5jls4\"" Jul 07 00:04:36.579342 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.579322 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f546649f9-2tzwr"] Jul 07 00:04:36.685150 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.685104 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trmw\" (UniqueName: \"kubernetes.io/projected/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-kube-api-access-4trmw\") pod \"maas-controller-f546649f9-2tzwr\" (UID: \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\") " pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:36.685348 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.685247 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-webhook-cert\") pod \"maas-controller-f546649f9-2tzwr\" (UID: \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\") " pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:36.752343 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.752306 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8498ff5b75-zplp5"] Jul 07 00:04:36.755609 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.755590 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:36.767608 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.767583 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498ff5b75-zplp5"] Jul 07 00:04:36.786291 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.786264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trmw\" (UniqueName: \"kubernetes.io/projected/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-kube-api-access-4trmw\") pod \"maas-controller-f546649f9-2tzwr\" (UID: \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\") " pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:36.786410 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.786380 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-webhook-cert\") pod \"maas-controller-f546649f9-2tzwr\" (UID: \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\") " pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:36.788911 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.788885 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-webhook-cert\") pod \"maas-controller-f546649f9-2tzwr\" (UID: \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\") " pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:36.794817 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.794797 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trmw\" (UniqueName: \"kubernetes.io/projected/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-kube-api-access-4trmw\") pod \"maas-controller-f546649f9-2tzwr\" (UID: \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\") " pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:36.878647 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.878553 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:36.886790 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.886763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjgp\" (UniqueName: \"kubernetes.io/projected/8db05622-5575-48eb-a43c-d4bbce25e9d5-kube-api-access-bqjgp\") pod \"maas-controller-8498ff5b75-zplp5\" (UID: \"8db05622-5575-48eb-a43c-d4bbce25e9d5\") " pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:36.886892 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.886827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8db05622-5575-48eb-a43c-d4bbce25e9d5-webhook-cert\") pod \"maas-controller-8498ff5b75-zplp5\" (UID: \"8db05622-5575-48eb-a43c-d4bbce25e9d5\") " pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:36.988283 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.988244 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjgp\" (UniqueName: \"kubernetes.io/projected/8db05622-5575-48eb-a43c-d4bbce25e9d5-kube-api-access-bqjgp\") pod \"maas-controller-8498ff5b75-zplp5\" (UID: \"8db05622-5575-48eb-a43c-d4bbce25e9d5\") " pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:36.988482 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.988344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8db05622-5575-48eb-a43c-d4bbce25e9d5-webhook-cert\") pod \"maas-controller-8498ff5b75-zplp5\" (UID: \"8db05622-5575-48eb-a43c-d4bbce25e9d5\") " pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:36.990969 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.990946 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8db05622-5575-48eb-a43c-d4bbce25e9d5-webhook-cert\") pod \"maas-controller-8498ff5b75-zplp5\" (UID: \"8db05622-5575-48eb-a43c-d4bbce25e9d5\") " pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:36.997296 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:36.997264 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjgp\" (UniqueName: \"kubernetes.io/projected/8db05622-5575-48eb-a43c-d4bbce25e9d5-kube-api-access-bqjgp\") pod \"maas-controller-8498ff5b75-zplp5\" (UID: \"8db05622-5575-48eb-a43c-d4bbce25e9d5\") " pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:37.020783 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:37.020755 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f546649f9-2tzwr"] Jul 07 00:04:37.022446 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:04:37.022407 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc83a5c_1140_4180_97fd_66f9e9dc4aad.slice/crio-dbd92407b8103ba28bf5bb955fd759838c62af917f0e529f1b9490f4e9e2618c WatchSource:0}: Error finding container dbd92407b8103ba28bf5bb955fd759838c62af917f0e529f1b9490f4e9e2618c: Status 404 returned error can't find the container with id dbd92407b8103ba28bf5bb955fd759838c62af917f0e529f1b9490f4e9e2618c Jul 07 00:04:37.066081 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:37.066048 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:37.203699 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:37.203637 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498ff5b75-zplp5"] Jul 07 00:04:37.207276 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:04:37.207228 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db05622_5575_48eb_a43c_d4bbce25e9d5.slice/crio-2f704b3631342d8a3c9fdb7317bf202e0140fce712df09f578141f7e3d1fea2d WatchSource:0}: Error finding container 2f704b3631342d8a3c9fdb7317bf202e0140fce712df09f578141f7e3d1fea2d: Status 404 returned error can't find the container with id 2f704b3631342d8a3c9fdb7317bf202e0140fce712df09f578141f7e3d1fea2d Jul 07 00:04:37.726955 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:37.726915 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498ff5b75-zplp5" event={"ID":"8db05622-5575-48eb-a43c-d4bbce25e9d5","Type":"ContainerStarted","Data":"2f704b3631342d8a3c9fdb7317bf202e0140fce712df09f578141f7e3d1fea2d"} Jul 07 00:04:37.728328 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:37.728284 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f546649f9-2tzwr" event={"ID":"3bc83a5c-1140-4180-97fd-66f9e9dc4aad","Type":"ContainerStarted","Data":"dbd92407b8103ba28bf5bb955fd759838c62af917f0e529f1b9490f4e9e2618c"} Jul 07 00:04:39.746651 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:39.746614 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f546649f9-2tzwr" event={"ID":"3bc83a5c-1140-4180-97fd-66f9e9dc4aad","Type":"ContainerStarted","Data":"26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876"} Jul 07 00:04:39.747111 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:39.746675 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:39.748031 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:39.748008 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498ff5b75-zplp5" event={"ID":"8db05622-5575-48eb-a43c-d4bbce25e9d5","Type":"ContainerStarted","Data":"52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2"} Jul 07 00:04:39.748171 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:39.748150 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:39.776450 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:39.776379 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f546649f9-2tzwr" podStartSLOduration=1.39162032 podStartE2EDuration="3.776363666s" totalImagesPullingTime="2.384743346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:04:39.776028779 +0000 UTC m=+500.989060524" watchObservedRunningTime="2026-07-07 00:04:39.776363666 +0000 UTC m=+500.989395409" Jul 07 00:04:39.811682 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:39.811632 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8498ff5b75-zplp5" podStartSLOduration=1.618701668 podStartE2EDuration="3.811613466s" totalImagesPullingTime="2.192911798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:04:39.810831431 +0000 UTC m=+501.023863175" watchObservedRunningTime="2026-07-07 00:04:39.811613466 +0000 UTC m=+501.024645210" Jul 07 00:04:50.758864 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:50.758832 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:04:50.759366 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:50.758902 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:50.820557 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:50.820519 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f546649f9-2tzwr"] Jul 07 00:04:50.820763 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:50.820738 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-f546649f9-2tzwr" podUID="3bc83a5c-1140-4180-97fd-66f9e9dc4aad" containerName="manager" containerID="cri-o://26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876" gracePeriod=30 Jul 07 00:04:51.059913 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.059886 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:51.217805 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.217766 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4trmw\" (UniqueName: \"kubernetes.io/projected/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-kube-api-access-4trmw\") pod \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\" (UID: \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\") " Jul 07 00:04:51.217970 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.217837 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-webhook-cert\") pod \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\" (UID: \"3bc83a5c-1140-4180-97fd-66f9e9dc4aad\") " Jul 07 00:04:51.220263 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.220226 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-kube-api-access-4trmw" (OuterVolumeSpecName: "kube-api-access-4trmw") pod "3bc83a5c-1140-4180-97fd-66f9e9dc4aad" (UID: "3bc83a5c-1140-4180-97fd-66f9e9dc4aad"). InnerVolumeSpecName "kube-api-access-4trmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:04:51.220263 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.220226 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3bc83a5c-1140-4180-97fd-66f9e9dc4aad" (UID: "3bc83a5c-1140-4180-97fd-66f9e9dc4aad"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:04:51.318487 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.318398 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4trmw\" (UniqueName: \"kubernetes.io/projected/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-kube-api-access-4trmw\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:04:51.318487 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.318422 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bc83a5c-1140-4180-97fd-66f9e9dc4aad-webhook-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:04:51.802987 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.802953 2568 generic.go:356] "Generic (PLEG): container finished" podID="3bc83a5c-1140-4180-97fd-66f9e9dc4aad" containerID="26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876" exitCode=0 Jul 07 00:04:51.803420 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.803006 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f546649f9-2tzwr" Jul 07 00:04:51.803420 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.803016 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f546649f9-2tzwr" event={"ID":"3bc83a5c-1140-4180-97fd-66f9e9dc4aad","Type":"ContainerDied","Data":"26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876"} Jul 07 00:04:51.803420 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.803056 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f546649f9-2tzwr" event={"ID":"3bc83a5c-1140-4180-97fd-66f9e9dc4aad","Type":"ContainerDied","Data":"dbd92407b8103ba28bf5bb955fd759838c62af917f0e529f1b9490f4e9e2618c"} Jul 07 00:04:51.803420 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.803078 2568 scope.go:117] "RemoveContainer" containerID="26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876" Jul 07 00:04:51.811363 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.811344 2568 scope.go:117] "RemoveContainer" containerID="26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876" Jul 07 00:04:51.811646 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:04:51.811627 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876\": container with ID starting with 26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876 not found: ID does not exist" containerID="26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876" Jul 07 00:04:51.811694 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.811655 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876"} err="failed to get container status \"26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876\": rpc error: code = NotFound desc = could not find container \"26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876\": container with ID starting with 26d946fbc8e4a9776ddb0a4c79784a70b59dd1f08846b3558c047fbe14619876 not found: ID does not exist" Jul 07 00:04:51.822365 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.822340 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f546649f9-2tzwr"] Jul 07 00:04:51.826002 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:51.825974 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f546649f9-2tzwr"] Jul 07 00:04:53.315647 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:04:53.315615 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc83a5c-1140-4180-97fd-66f9e9dc4aad" path="/var/lib/kubelet/pods/3bc83a5c-1140-4180-97fd-66f9e9dc4aad/volumes" Jul 07 00:05:12.558936 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.558855 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-674c64d4c7-mhn92"] Jul 07 00:05:12.559324 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.559192 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc83a5c-1140-4180-97fd-66f9e9dc4aad" containerName="manager" Jul 07 00:05:12.559324 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.559204 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc83a5c-1140-4180-97fd-66f9e9dc4aad" containerName="manager" Jul 07 00:05:12.559324 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.559252 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc83a5c-1140-4180-97fd-66f9e9dc4aad" containerName="manager" Jul 07 00:05:12.565093 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.565071 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:12.568444 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.568406 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-drsk8\"" Jul 07 00:05:12.568444 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.568443 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 00:05:12.572644 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.572625 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-674c64d4c7-mhn92"] Jul 07 00:05:12.617901 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.617860 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d3e48c-6d78-4e38-9799-b2205dcd7725-openshift-service-ca\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:12.617901 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.617906 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d3e48c-6d78-4e38-9799-b2205dcd7725-maas-api-tls\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:12.618103 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.617925 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cz7r\" (UniqueName: \"kubernetes.io/projected/95d3e48c-6d78-4e38-9799-b2205dcd7725-kube-api-access-9cz7r\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:12.719387 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.719341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d3e48c-6d78-4e38-9799-b2205dcd7725-openshift-service-ca\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:12.719387 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.719394 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d3e48c-6d78-4e38-9799-b2205dcd7725-maas-api-tls\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:12.719687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.719410 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cz7r\" (UniqueName: \"kubernetes.io/projected/95d3e48c-6d78-4e38-9799-b2205dcd7725-kube-api-access-9cz7r\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:12.719687 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:05:12.719544 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 07 00:05:12.719687 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:05:12.719622 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d3e48c-6d78-4e38-9799-b2205dcd7725-maas-api-tls podName:95d3e48c-6d78-4e38-9799-b2205dcd7725 nodeName:}" failed. No retries permitted until 2026-07-07 00:05:13.219601515 +0000 UTC m=+534.432633251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95d3e48c-6d78-4e38-9799-b2205dcd7725-maas-api-tls") pod "maas-api-674c64d4c7-mhn92" (UID: "95d3e48c-6d78-4e38-9799-b2205dcd7725") : secret "maas-api-serving-cert" not found Jul 07 00:05:12.720025 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.720003 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d3e48c-6d78-4e38-9799-b2205dcd7725-openshift-service-ca\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:12.730798 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:12.730762 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cz7r\" (UniqueName: \"kubernetes.io/projected/95d3e48c-6d78-4e38-9799-b2205dcd7725-kube-api-access-9cz7r\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:13.077121 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.077081 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-xrvfm"] Jul 07 00:05:13.081580 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.081561 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:13.084267 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.084242 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 07 00:05:13.084402 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.084385 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-d7zq9\"" Jul 07 00:05:13.092225 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.092199 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-xrvfm"] Jul 07 00:05:13.124225 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.124190 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64985bdd-cb48-4b4a-9b1b-57a17912eef7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-xrvfm\" (UID: \"64985bdd-cb48-4b4a-9b1b-57a17912eef7\") " pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:13.124415 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.124384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswmv\" (UniqueName: \"kubernetes.io/projected/64985bdd-cb48-4b4a-9b1b-57a17912eef7-kube-api-access-bswmv\") pod \"payload-pre-processing-7964457696-xrvfm\" (UID: \"64985bdd-cb48-4b4a-9b1b-57a17912eef7\") " pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:13.225218 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.225176 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswmv\" (UniqueName: \"kubernetes.io/projected/64985bdd-cb48-4b4a-9b1b-57a17912eef7-kube-api-access-bswmv\") pod \"payload-pre-processing-7964457696-xrvfm\" (UID: \"64985bdd-cb48-4b4a-9b1b-57a17912eef7\") " pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:13.225415 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.225245 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64985bdd-cb48-4b4a-9b1b-57a17912eef7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-xrvfm\" (UID: \"64985bdd-cb48-4b4a-9b1b-57a17912eef7\") " pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:13.225415 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.225326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d3e48c-6d78-4e38-9799-b2205dcd7725-maas-api-tls\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:13.225959 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.225934 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64985bdd-cb48-4b4a-9b1b-57a17912eef7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-xrvfm\" (UID: \"64985bdd-cb48-4b4a-9b1b-57a17912eef7\") " pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:13.227980 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.227952 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d3e48c-6d78-4e38-9799-b2205dcd7725-maas-api-tls\") pod \"maas-api-674c64d4c7-mhn92\" (UID: \"95d3e48c-6d78-4e38-9799-b2205dcd7725\") " pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:13.234743 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.234723 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswmv\" (UniqueName: \"kubernetes.io/projected/64985bdd-cb48-4b4a-9b1b-57a17912eef7-kube-api-access-bswmv\") pod \"payload-pre-processing-7964457696-xrvfm\" (UID: \"64985bdd-cb48-4b4a-9b1b-57a17912eef7\") " pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:13.394028 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.393929 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:13.476490 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.476455 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:13.536586 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.536489 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-xrvfm"] Jul 07 00:05:13.540604 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:05:13.540575 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64985bdd_cb48_4b4a_9b1b_57a17912eef7.slice/crio-9e06682bbce5c1834e36fa8b26078c84a96de689ba9cba6cc7f76fdaa6441fd3 WatchSource:0}: Error finding container 9e06682bbce5c1834e36fa8b26078c84a96de689ba9cba6cc7f76fdaa6441fd3: Status 404 returned error can't find the container with id 9e06682bbce5c1834e36fa8b26078c84a96de689ba9cba6cc7f76fdaa6441fd3 Jul 07 00:05:13.660129 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.660008 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-674c64d4c7-mhn92"] Jul 07 00:05:13.663684 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:05:13.663656 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d3e48c_6d78_4e38_9799_b2205dcd7725.slice/crio-8e2833246b5154729ddb8280f8d9f3e4cba2a49f58597e3fbd7e02a0fca08a0d WatchSource:0}: Error finding container 8e2833246b5154729ddb8280f8d9f3e4cba2a49f58597e3fbd7e02a0fca08a0d: Status 404 returned error can't find the container with id 8e2833246b5154729ddb8280f8d9f3e4cba2a49f58597e3fbd7e02a0fca08a0d Jul 07 00:05:13.904360 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.904320 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" event={"ID":"64985bdd-cb48-4b4a-9b1b-57a17912eef7","Type":"ContainerStarted","Data":"9e06682bbce5c1834e36fa8b26078c84a96de689ba9cba6cc7f76fdaa6441fd3"} Jul 07 00:05:13.905493 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:13.905466 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-674c64d4c7-mhn92" event={"ID":"95d3e48c-6d78-4e38-9799-b2205dcd7725","Type":"ContainerStarted","Data":"8e2833246b5154729ddb8280f8d9f3e4cba2a49f58597e3fbd7e02a0fca08a0d"} Jul 07 00:05:15.919111 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:15.918172 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" event={"ID":"64985bdd-cb48-4b4a-9b1b-57a17912eef7","Type":"ContainerStarted","Data":"df26f9bc7718dbe9f065df1e4ea7ed8a85cbc0787d43ef949e6c0fd11c794bae"} Jul 07 00:05:15.919111 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:15.919041 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:15.921262 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:15.921235 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-674c64d4c7-mhn92" event={"ID":"95d3e48c-6d78-4e38-9799-b2205dcd7725","Type":"ContainerStarted","Data":"c81df042b3179760a1892c1e7278635cbdd5750274bd92f177b1cb9b922bec7a"} Jul 07 00:05:15.921779 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:15.921731 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:15.937641 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:15.937583 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" podStartSLOduration=1.3397297400000001 podStartE2EDuration="2.937564498s" totalImagesPullingTime="1.597834758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:05:15.935564314 +0000 UTC m=+537.148596069" watchObservedRunningTime="2026-07-07 00:05:15.937564498 +0000 UTC m=+537.150596242" Jul 07 00:05:15.953135 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:15.953078 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-674c64d4c7-mhn92" podStartSLOduration=1.969210833 podStartE2EDuration="3.953059177s" totalImagesPullingTime="1.983848344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:05:15.95260338 +0000 UTC m=+537.165635126" watchObservedRunningTime="2026-07-07 00:05:15.953059177 +0000 UTC m=+537.166090922" Jul 07 00:05:20.949336 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:20.949301 2568 generic.go:356] "Generic (PLEG): container finished" podID="91f2db2c-526b-427e-a103-215869094379" containerID="e88b66196c212b67ec04f9f72458b641c2290cd907850fb8d33a40da7085cfdc" exitCode=0 Jul 07 00:05:20.949792 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:20.949359 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" event={"ID":"91f2db2c-526b-427e-a103-215869094379","Type":"ContainerDied","Data":"e88b66196c212b67ec04f9f72458b641c2290cd907850fb8d33a40da7085cfdc"} Jul 07 00:05:20.949792 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:20.949783 2568 scope.go:117] "RemoveContainer" containerID="e88b66196c212b67ec04f9f72458b641c2290cd907850fb8d33a40da7085cfdc" Jul 07 00:05:20.950768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:20.950741 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:05:20.950827 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:20.950815 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:05:20.950865 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:20.950849 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:05:21.940625 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:21.940589 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-674c64d4c7-mhn92" Jul 07 00:05:21.956000 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:21.955971 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" event={"ID":"91f2db2c-526b-427e-a103-215869094379","Type":"ContainerStarted","Data":"e313a5926cb0146d3a12edea7b148efecf2bb11df841fac9c8e441954f4a3bd8"} Jul 07 00:05:22.673327 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:22.673275 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:05:22.674379 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:22.674349 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-khdfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 07 00:05:22.674551 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:22.674404 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" podUID="91f2db2c-526b-427e-a103-215869094379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 07 00:05:23.673880 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:23.673841 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-khdfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 07 00:05:23.674266 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:23.673901 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" podUID="91f2db2c-526b-427e-a103-215869094379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 07 00:05:24.673302 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:24.673259 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-khdfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 07 00:05:24.673535 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:24.673321 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" podUID="91f2db2c-526b-427e-a103-215869094379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 07 00:05:25.544605 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:25.544559 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:05:25.673476 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:25.673423 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-khdfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 07 00:05:25.673650 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:25.673497 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" podUID="91f2db2c-526b-427e-a103-215869094379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 07 00:05:26.677535 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:26.677504 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:05:26.678529 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:26.678508 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-khdfj" Jul 07 00:05:27.934901 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:27.934865 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-xrvfm" Jul 07 00:05:30.568288 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.568256 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l"] Jul 07 00:05:30.572053 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.572032 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.574623 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.574597 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-st4mb\"" Jul 07 00:05:30.574762 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.574597 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 00:05:30.575633 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.575616 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 00:05:30.575735 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.575687 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 00:05:30.582566 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.582541 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l"] Jul 07 00:05:30.716240 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.716203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5584c707-83c2-460d-9f22-0df56a72dab1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.716240 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.716238 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.716485 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.716266 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7jzb\" (UniqueName: \"kubernetes.io/projected/5584c707-83c2-460d-9f22-0df56a72dab1-kube-api-access-q7jzb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.716485 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.716310 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.716485 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.716330 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.716485 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.716413 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.817823 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.817781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.818013 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.817867 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5584c707-83c2-460d-9f22-0df56a72dab1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.818013 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.817895 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.818013 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.817925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7jzb\" (UniqueName: \"kubernetes.io/projected/5584c707-83c2-460d-9f22-0df56a72dab1-kube-api-access-q7jzb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.818013 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.817964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.818013 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.817992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.818282 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.818204 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.818362 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.818292 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.818362 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.818350 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.820555 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.820533 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5584c707-83c2-460d-9f22-0df56a72dab1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.820695 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.820614 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5584c707-83c2-460d-9f22-0df56a72dab1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.827335 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.827311 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7jzb\" (UniqueName: \"kubernetes.io/projected/5584c707-83c2-460d-9f22-0df56a72dab1-kube-api-access-q7jzb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l\" (UID: \"5584c707-83c2-460d-9f22-0df56a72dab1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:30.882373 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:30.882336 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:31.022142 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:31.022114 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l"] Jul 07 00:05:31.024477 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:05:31.024449 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5584c707_83c2_460d_9f22_0df56a72dab1.slice/crio-961d93722aeaccdfb86dfb086a1b6689ec22e96eee4336271b405c6f890da1c2 WatchSource:0}: Error finding container 961d93722aeaccdfb86dfb086a1b6689ec22e96eee4336271b405c6f890da1c2: Status 404 returned error can't find the container with id 961d93722aeaccdfb86dfb086a1b6689ec22e96eee4336271b405c6f890da1c2 Jul 07 00:05:32.007091 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:32.007045 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" event={"ID":"5584c707-83c2-460d-9f22-0df56a72dab1","Type":"ContainerStarted","Data":"961d93722aeaccdfb86dfb086a1b6689ec22e96eee4336271b405c6f890da1c2"} Jul 07 00:05:33.012714 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:33.012675 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" event={"ID":"5584c707-83c2-460d-9f22-0df56a72dab1","Type":"ContainerStarted","Data":"077e568307472ef63246282b0021959a7c9e6935223faa294fc567b7484983da"} Jul 07 00:05:33.029970 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:33.029901 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" podStartSLOduration=1.8790107040000001 podStartE2EDuration="3.029886446s" totalImagesPullingTime="1.150875742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:05:33.029499907 +0000 UTC m=+554.242531848" watchObservedRunningTime="2026-07-07 00:05:33.029886446 +0000 UTC m=+554.242918191" Jul 07 00:05:40.883330 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:40.883289 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:40.883782 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:40.883379 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:40.895987 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:40.895953 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:05:41.061570 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:05:41.061536 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l" Jul 07 00:06:05.165696 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.165666 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc"] Jul 07 00:06:05.169414 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.169393 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.172728 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.172709 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 00:06:05.181809 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.181787 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc"] Jul 07 00:06:05.242848 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.242809 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d91174-900e-47f3-a539-ce9e6e0cfeba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.243022 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.242911 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2r5\" (UniqueName: \"kubernetes.io/projected/b9d91174-900e-47f3-a539-ce9e6e0cfeba-kube-api-access-vl2r5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.243022 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.242949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.243022 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.242975 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.243022 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.242990 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.243022 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.243012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.343979 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.343930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.343979 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.343983 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.344190 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.344106 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.344190 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.344162 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.344287 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.344200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d91174-900e-47f3-a539-ce9e6e0cfeba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.344344 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.344316 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2r5\" (UniqueName: \"kubernetes.io/projected/b9d91174-900e-47f3-a539-ce9e6e0cfeba-kube-api-access-vl2r5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.344408 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.344337 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.344408 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.344379 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.344604 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.344583 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.346503 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.346480 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d91174-900e-47f3-a539-ce9e6e0cfeba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.346751 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.346731 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d91174-900e-47f3-a539-ce9e6e0cfeba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.353414 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.353376 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2r5\" (UniqueName: \"kubernetes.io/projected/b9d91174-900e-47f3-a539-ce9e6e0cfeba-kube-api-access-vl2r5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc\" (UID: \"b9d91174-900e-47f3-a539-ce9e6e0cfeba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.479083 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.478982 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:05.634512 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:05.634450 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc"] Jul 07 00:06:05.636758 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:06:05.636731 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d91174_900e_47f3_a539_ce9e6e0cfeba.slice/crio-36984ce67435802d65a572e46186a1d5f90769420679aaad9661e27095d931e6 WatchSource:0}: Error finding container 36984ce67435802d65a572e46186a1d5f90769420679aaad9661e27095d931e6: Status 404 returned error can't find the container with id 36984ce67435802d65a572e46186a1d5f90769420679aaad9661e27095d931e6 Jul 07 00:06:06.163418 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:06.163380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" event={"ID":"b9d91174-900e-47f3-a539-ce9e6e0cfeba","Type":"ContainerStarted","Data":"75a1adb2bee1f13fdcf2491f4f978347393ef25cc5377771a424a233973c8182"} Jul 07 00:06:06.163623 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:06.163460 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" event={"ID":"b9d91174-900e-47f3-a539-ce9e6e0cfeba","Type":"ContainerStarted","Data":"36984ce67435802d65a572e46186a1d5f90769420679aaad9661e27095d931e6"} Jul 07 00:06:06.198972 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:06.198916 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" podStartSLOduration=0.999717872 podStartE2EDuration="1.198899827s" totalImagesPullingTime="199.181955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:06:06.197735233 +0000 UTC m=+587.410766976" watchObservedRunningTime="2026-07-07 00:06:06.198899827 +0000 UTC m=+587.411931580" Jul 07 00:06:15.479731 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:15.479683 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:15.480137 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:15.479769 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:15.492214 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:15.492187 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:16.220001 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:16.219972 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc" Jul 07 00:06:19.189708 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:19.189666 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:06:19.193074 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:06:19.193052 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:08:25.502594 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.502507 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8498ff5b75-zplp5"] Jul 07 00:08:25.503543 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.502696 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-8498ff5b75-zplp5" podUID="8db05622-5575-48eb-a43c-d4bbce25e9d5" containerName="manager" containerID="cri-o://52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2" gracePeriod=30 Jul 07 00:08:25.746843 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.746816 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:08:25.823798 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.823719 2568 generic.go:356] "Generic (PLEG): container finished" podID="8db05622-5575-48eb-a43c-d4bbce25e9d5" containerID="52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2" exitCode=0 Jul 07 00:08:25.823798 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.823760 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498ff5b75-zplp5" Jul 07 00:08:25.823991 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.823799 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498ff5b75-zplp5" event={"ID":"8db05622-5575-48eb-a43c-d4bbce25e9d5","Type":"ContainerDied","Data":"52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2"} Jul 07 00:08:25.823991 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.823856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498ff5b75-zplp5" event={"ID":"8db05622-5575-48eb-a43c-d4bbce25e9d5","Type":"ContainerDied","Data":"2f704b3631342d8a3c9fdb7317bf202e0140fce712df09f578141f7e3d1fea2d"} Jul 07 00:08:25.823991 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.823879 2568 scope.go:117] "RemoveContainer" containerID="52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2" Jul 07 00:08:25.832115 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.832086 2568 scope.go:117] "RemoveContainer" containerID="52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2" Jul 07 00:08:25.832349 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:08:25.832331 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2\": container with ID starting with 52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2 not found: ID does not exist" containerID="52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2" Jul 07 00:08:25.832402 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.832363 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2"} err="failed to get container status \"52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2\": rpc error: code = NotFound desc = could not find container \"52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2\": container with ID starting with 52294b7ddb437c268d9e737be096659b479ef31cd08e4883c36b366d5d1812e2 not found: ID does not exist" Jul 07 00:08:25.839680 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.839656 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqjgp\" (UniqueName: \"kubernetes.io/projected/8db05622-5575-48eb-a43c-d4bbce25e9d5-kube-api-access-bqjgp\") pod \"8db05622-5575-48eb-a43c-d4bbce25e9d5\" (UID: \"8db05622-5575-48eb-a43c-d4bbce25e9d5\") " Jul 07 00:08:25.839793 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.839711 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8db05622-5575-48eb-a43c-d4bbce25e9d5-webhook-cert\") pod \"8db05622-5575-48eb-a43c-d4bbce25e9d5\" (UID: \"8db05622-5575-48eb-a43c-d4bbce25e9d5\") " Jul 07 00:08:25.841964 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.841929 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db05622-5575-48eb-a43c-d4bbce25e9d5-kube-api-access-bqjgp" (OuterVolumeSpecName: "kube-api-access-bqjgp") pod "8db05622-5575-48eb-a43c-d4bbce25e9d5" (UID: "8db05622-5575-48eb-a43c-d4bbce25e9d5"). InnerVolumeSpecName "kube-api-access-bqjgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:08:25.841964 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.841941 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db05622-5575-48eb-a43c-d4bbce25e9d5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8db05622-5575-48eb-a43c-d4bbce25e9d5" (UID: "8db05622-5575-48eb-a43c-d4bbce25e9d5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:08:25.940648 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.940605 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8db05622-5575-48eb-a43c-d4bbce25e9d5-webhook-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:08:25.940648 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:25.940643 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqjgp\" (UniqueName: \"kubernetes.io/projected/8db05622-5575-48eb-a43c-d4bbce25e9d5-kube-api-access-bqjgp\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:08:26.146012 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.145980 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8498ff5b75-zplp5"] Jul 07 00:08:26.149478 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.149425 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8498ff5b75-zplp5"] Jul 07 00:08:26.938172 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.938136 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8498ff5b75-hqp5j"] Jul 07 00:08:26.938559 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.938496 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db05622-5575-48eb-a43c-d4bbce25e9d5" containerName="manager" Jul 07 00:08:26.938559 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.938509 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db05622-5575-48eb-a43c-d4bbce25e9d5" containerName="manager" Jul 07 00:08:26.938627 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.938569 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db05622-5575-48eb-a43c-d4bbce25e9d5" containerName="manager" Jul 07 00:08:26.941170 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.941154 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:08:26.944833 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.944809 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 00:08:26.944953 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.944843 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5jls4\"" Jul 07 00:08:26.944953 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.944809 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 00:08:26.950404 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:26.950379 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498ff5b75-hqp5j"] Jul 07 00:08:27.048937 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.048902 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4m4l\" (UniqueName: \"kubernetes.io/projected/8859ca0d-3189-4094-b50b-4ea9ed76032a-kube-api-access-d4m4l\") pod \"maas-controller-8498ff5b75-hqp5j\" (UID: \"8859ca0d-3189-4094-b50b-4ea9ed76032a\") " pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:08:27.049153 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.048975 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8859ca0d-3189-4094-b50b-4ea9ed76032a-webhook-cert\") pod \"maas-controller-8498ff5b75-hqp5j\" (UID: \"8859ca0d-3189-4094-b50b-4ea9ed76032a\") " pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:08:27.150474 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.150398 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4m4l\" (UniqueName: \"kubernetes.io/projected/8859ca0d-3189-4094-b50b-4ea9ed76032a-kube-api-access-d4m4l\") pod \"maas-controller-8498ff5b75-hqp5j\" (UID: \"8859ca0d-3189-4094-b50b-4ea9ed76032a\") " pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:08:27.150651 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.150515 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8859ca0d-3189-4094-b50b-4ea9ed76032a-webhook-cert\") pod \"maas-controller-8498ff5b75-hqp5j\" (UID: \"8859ca0d-3189-4094-b50b-4ea9ed76032a\") " pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:08:27.153134 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.153102 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8859ca0d-3189-4094-b50b-4ea9ed76032a-webhook-cert\") pod \"maas-controller-8498ff5b75-hqp5j\" (UID: \"8859ca0d-3189-4094-b50b-4ea9ed76032a\") " pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:08:27.166042 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.166011 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4m4l\" (UniqueName: \"kubernetes.io/projected/8859ca0d-3189-4094-b50b-4ea9ed76032a-kube-api-access-d4m4l\") pod \"maas-controller-8498ff5b75-hqp5j\" (UID: \"8859ca0d-3189-4094-b50b-4ea9ed76032a\") " pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:08:27.251474 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.251334 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:08:27.322276 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.320934 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db05622-5575-48eb-a43c-d4bbce25e9d5" path="/var/lib/kubelet/pods/8db05622-5575-48eb-a43c-d4bbce25e9d5/volumes" Jul 07 00:08:27.391237 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.391207 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498ff5b75-hqp5j"] Jul 07 00:08:27.393549 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:08:27.393520 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8859ca0d_3189_4094_b50b_4ea9ed76032a.slice/crio-7e6ad847ae6b5e514b841e03ed0006d001d4cad875f56459e0400ee440a4346f WatchSource:0}: Error finding container 7e6ad847ae6b5e514b841e03ed0006d001d4cad875f56459e0400ee440a4346f: Status 404 returned error can't find the container with id 7e6ad847ae6b5e514b841e03ed0006d001d4cad875f56459e0400ee440a4346f Jul 07 00:08:27.394793 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.394773 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:08:27.834790 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:27.834753 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498ff5b75-hqp5j" event={"ID":"8859ca0d-3189-4094-b50b-4ea9ed76032a","Type":"ContainerStarted","Data":"7e6ad847ae6b5e514b841e03ed0006d001d4cad875f56459e0400ee440a4346f"} Jul 07 00:08:28.840628 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:28.840591 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498ff5b75-hqp5j" event={"ID":"8859ca0d-3189-4094-b50b-4ea9ed76032a","Type":"ContainerStarted","Data":"1d0fdcf41be9df3200cb8210723a89ce48d2bfa442779da35f9fcaa6ae02d74d"} Jul 07 00:08:28.841036 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:28.840714 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:08:28.861335 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:28.861274 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8498ff5b75-hqp5j" podStartSLOduration=2.373062706 podStartE2EDuration="2.861253806s" totalImagesPullingTime="488.1911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:08:28.860692437 +0000 UTC m=+730.073724205" watchObservedRunningTime="2026-07-07 00:08:28.861253806 +0000 UTC m=+730.074285552" Jul 07 00:08:39.851058 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:08:39.851027 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8498ff5b75-hqp5j" Jul 07 00:11:19.217463 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:11:19.217364 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:11:19.222329 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:11:19.222306 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:16:19.249294 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:16:19.249209 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:16:19.254164 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:16:19.254139 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:20:27.563682 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.563598 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s"] Jul 07 00:20:27.567373 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.567349 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.570342 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.570315 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a0d805de-openshift-default-dockercfg-dj62t\"" Jul 07 00:20:27.587237 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.587205 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s"] Jul 07 00:20:27.670809 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.670772 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-data\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.671010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.670832 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-socket\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.671010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.670881 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-certs\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.671010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.670918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-token\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.671010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.670986 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-envoy\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.671234 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.671023 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-credential-socket\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.671234 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.671058 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istiod-ca-cert\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.671234 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.671107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-podinfo\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.671234 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.671133 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5277\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-kube-api-access-t5277\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772240 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-socket\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772240 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-certs\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772548 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772256 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-token\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772548 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-envoy\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772548 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772310 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-credential-socket\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772548 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istiod-ca-cert\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772548 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772360 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-podinfo\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772548 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5277\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-kube-api-access-t5277\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772548 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-data\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772924 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772720 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-socket\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.772924 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772791 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-certs\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.773043 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.772993 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-credential-socket\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.773109 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.773087 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-data\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.773186 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.773165 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istiod-ca-cert\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.774845 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.774815 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-envoy\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.775155 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.775136 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-podinfo\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.781757 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.781729 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-token\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.781885 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.781868 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5277\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-kube-api-access-t5277\") pod \"e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:27.881710 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:27.881599 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:28.021968 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.021940 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s"] Jul 07 00:20:28.023665 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:20:28.023634 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852677b7_0e1a_4f15_9d71_7ee957da5ee2.slice/crio-645227f2afbf724eb101cfba040537c0e8f6980d9efc519bc2742e4ebc670ba8 WatchSource:0}: Error finding container 645227f2afbf724eb101cfba040537c0e8f6980d9efc519bc2742e4ebc670ba8: Status 404 returned error can't find the container with id 645227f2afbf724eb101cfba040537c0e8f6980d9efc519bc2742e4ebc670ba8 Jul 07 00:20:28.025931 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.025902 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:28.025993 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.025970 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:28.026026 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.025996 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:28.160823 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.160787 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" event={"ID":"852677b7-0e1a-4f15-9d71-7ee957da5ee2","Type":"ContainerStarted","Data":"9c2b301384ff2938809bbf2d3f91b19c9ba003278e9b291ec3e9ce8f45cf4c7f"} Jul 07 00:20:28.161017 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.160848 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" event={"ID":"852677b7-0e1a-4f15-9d71-7ee957da5ee2","Type":"ContainerStarted","Data":"645227f2afbf724eb101cfba040537c0e8f6980d9efc519bc2742e4ebc670ba8"} Jul 07 00:20:28.183419 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.183358 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" podStartSLOduration=1.1833349 podStartE2EDuration="1.1833349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:20:28.18133972 +0000 UTC m=+1449.394371486" watchObservedRunningTime="2026-07-07 00:20:28.1833349 +0000 UTC m=+1449.396366645" Jul 07 00:20:28.870408 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.870369 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg"] Jul 07 00:20:28.874617 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.874588 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:28.877119 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.877087 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-a0d805de-serving-cert\"" Jul 07 00:20:28.881696 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.881662 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:28.887471 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.887427 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg"] Jul 07 00:20:28.983362 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.983322 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-openshift-service-ca\") pod \"maas-api-e2e-ait-a0d805de-544666b65f-w74pg\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:28.983551 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.983367 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-maas-api-tls\") pod \"maas-api-e2e-ait-a0d805de-544666b65f-w74pg\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:28.983551 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:28.983494 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxg7m\" (UniqueName: \"kubernetes.io/projected/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-kube-api-access-lxg7m\") pod \"maas-api-e2e-ait-a0d805de-544666b65f-w74pg\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:29.084484 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.084449 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-openshift-service-ca\") pod \"maas-api-e2e-ait-a0d805de-544666b65f-w74pg\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:29.084688 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.084500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-maas-api-tls\") pod \"maas-api-e2e-ait-a0d805de-544666b65f-w74pg\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:29.084688 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.084546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxg7m\" (UniqueName: \"kubernetes.io/projected/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-kube-api-access-lxg7m\") pod \"maas-api-e2e-ait-a0d805de-544666b65f-w74pg\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:29.085141 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.085115 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-openshift-service-ca\") pod \"maas-api-e2e-ait-a0d805de-544666b65f-w74pg\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:29.087079 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.087059 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-maas-api-tls\") pod \"maas-api-e2e-ait-a0d805de-544666b65f-w74pg\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:29.097718 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.097689 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxg7m\" (UniqueName: \"kubernetes.io/projected/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-kube-api-access-lxg7m\") pod \"maas-api-e2e-ait-a0d805de-544666b65f-w74pg\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:29.186486 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.186410 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:29.333205 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.333175 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg"] Jul 07 00:20:29.335389 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:20:29.335361 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef168a2e_2ac6_4bf9_910d_3fd66084e0e0.slice/crio-abde39d5d80e27e6b3bf3cd34c57c87f78c3af06c178a1b62cd871aa55c668dc WatchSource:0}: Error finding container abde39d5d80e27e6b3bf3cd34c57c87f78c3af06c178a1b62cd871aa55c668dc: Status 404 returned error can't find the container with id abde39d5d80e27e6b3bf3cd34c57c87f78c3af06c178a1b62cd871aa55c668dc Jul 07 00:20:29.337038 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.337018 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:20:29.887761 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:29.887726 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:30.174023 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:30.173982 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" event={"ID":"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0","Type":"ContainerStarted","Data":"3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686"} Jul 07 00:20:30.174236 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:30.174049 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" event={"ID":"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0","Type":"ContainerStarted","Data":"abde39d5d80e27e6b3bf3cd34c57c87f78c3af06c178a1b62cd871aa55c668dc"} Jul 07 00:20:30.174306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:30.174279 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:30.174363 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:30.174324 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:30.175391 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:30.175369 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:30.190981 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:30.190927 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" podStartSLOduration=1.545522478 podStartE2EDuration="2.190911388s" totalImagesPullingTime="645.38891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:20:30.190215894 +0000 UTC m=+1451.403247638" watchObservedRunningTime="2026-07-07 00:20:30.190911388 +0000 UTC m=+1451.403943148" Jul 07 00:20:33.978309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:33.978270 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s"] Jul 07 00:20:33.978835 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:33.978473 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" podUID="852677b7-0e1a-4f15-9d71-7ee957da5ee2" containerName="istio-proxy" containerID="cri-o://9c2b301384ff2938809bbf2d3f91b19c9ba003278e9b291ec3e9ce8f45cf4c7f" gracePeriod=30 Jul 07 00:20:34.335040 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.334941 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg"] Jul 07 00:20:34.337997 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.335209 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" podUID="ef168a2e-2ac6-4bf9-910d-3fd66084e0e0" containerName="maas-api" containerID="cri-o://3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686" gracePeriod=30 Jul 07 00:20:34.607884 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.607857 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:34.750369 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.750334 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxg7m\" (UniqueName: \"kubernetes.io/projected/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-kube-api-access-lxg7m\") pod \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " Jul 07 00:20:34.750604 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.750406 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-openshift-service-ca\") pod \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " Jul 07 00:20:34.750604 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.750472 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-maas-api-tls\") pod \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\" (UID: \"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0\") " Jul 07 00:20:34.750778 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.750751 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef168a2e-2ac6-4bf9-910d-3fd66084e0e0" (UID: "ef168a2e-2ac6-4bf9-910d-3fd66084e0e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:20:34.752664 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.752640 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef168a2e-2ac6-4bf9-910d-3fd66084e0e0" (UID: "ef168a2e-2ac6-4bf9-910d-3fd66084e0e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:20:34.752664 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.752654 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-kube-api-access-lxg7m" (OuterVolumeSpecName: "kube-api-access-lxg7m") pod "ef168a2e-2ac6-4bf9-910d-3fd66084e0e0" (UID: "ef168a2e-2ac6-4bf9-910d-3fd66084e0e0"). InnerVolumeSpecName "kube-api-access-lxg7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:34.851422 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.851327 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxg7m\" (UniqueName: \"kubernetes.io/projected/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-kube-api-access-lxg7m\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:34.851422 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.851359 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:34.851422 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:34.851370 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:35.199365 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.199333 2568 generic.go:356] "Generic (PLEG): container finished" podID="ef168a2e-2ac6-4bf9-910d-3fd66084e0e0" containerID="3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686" exitCode=0 Jul 07 00:20:35.199876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.199380 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" Jul 07 00:20:35.199876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.199423 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" event={"ID":"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0","Type":"ContainerDied","Data":"3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686"} Jul 07 00:20:35.199876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.199500 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg" event={"ID":"ef168a2e-2ac6-4bf9-910d-3fd66084e0e0","Type":"ContainerDied","Data":"abde39d5d80e27e6b3bf3cd34c57c87f78c3af06c178a1b62cd871aa55c668dc"} Jul 07 00:20:35.199876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.199533 2568 scope.go:117] "RemoveContainer" containerID="3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686" Jul 07 00:20:35.210171 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.210154 2568 scope.go:117] "RemoveContainer" containerID="3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686" Jul 07 00:20:35.210422 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:20:35.210401 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686\": container with ID starting with 3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686 not found: ID does not exist" containerID="3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686" Jul 07 00:20:35.210520 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.210454 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686"} err="failed to get container status \"3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686\": rpc error: code = NotFound desc = could not find container \"3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686\": container with ID starting with 3e0947808950844763955a2727dbe85aaa18c8781749c2d76e65638c85e48686 not found: ID does not exist" Jul 07 00:20:35.223458 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.222670 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg"] Jul 07 00:20:35.226815 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.226784 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0d805de-544666b65f-w74pg"] Jul 07 00:20:35.316250 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:35.316202 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef168a2e-2ac6-4bf9-910d-3fd66084e0e0" path="/var/lib/kubelet/pods/ef168a2e-2ac6-4bf9-910d-3fd66084e0e0/volumes" Jul 07 00:20:39.890463 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:39.890408 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4"] Jul 07 00:20:39.891057 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:39.891037 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef168a2e-2ac6-4bf9-910d-3fd66084e0e0" containerName="maas-api" Jul 07 00:20:39.891200 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:39.891062 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef168a2e-2ac6-4bf9-910d-3fd66084e0e0" containerName="maas-api" Jul 07 00:20:39.891200 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:39.891162 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef168a2e-2ac6-4bf9-910d-3fd66084e0e0" containerName="maas-api" Jul 07 00:20:39.896679 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:39.896654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:39.899578 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:39.899551 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7ac6e78c-openshift-default-dockercfg-xdlrb\"" Jul 07 00:20:39.907819 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:39.907791 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4"] Jul 07 00:20:40.000109 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.000073 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-envoy\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.000109 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.000112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istiod-ca-cert\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.001289 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.000132 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-certs\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.001289 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.000225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-podinfo\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.001289 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.000293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-credential-socket\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.001289 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.000313 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-socket\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.001289 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.000343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-token\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.001289 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.000405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5ng\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-kube-api-access-rl5ng\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.001289 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.000479 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-data\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.101956 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.101929 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-credential-socket\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.101963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-socket\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.101990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-token\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5ng\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-kube-api-access-rl5ng\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-data\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102285 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102104 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-envoy\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102285 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istiod-ca-cert\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102285 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-certs\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102285 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-podinfo\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102426 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102353 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-credential-socket\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102426 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102383 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-socket\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102723 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102702 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-data\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.102929 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.102762 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-certs\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.103088 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.103065 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istiod-ca-cert\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.104604 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.104587 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-envoy\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.104808 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.104791 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-podinfo\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.112329 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.112299 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-token\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.112885 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.112867 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5ng\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-kube-api-access-rl5ng\") pod \"e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.210303 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.210261 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:40.225012 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.224961 2568 generic.go:356] "Generic (PLEG): container finished" podID="852677b7-0e1a-4f15-9d71-7ee957da5ee2" containerID="9c2b301384ff2938809bbf2d3f91b19c9ba003278e9b291ec3e9ce8f45cf4c7f" exitCode=0 Jul 07 00:20:40.225095 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.225031 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" event={"ID":"852677b7-0e1a-4f15-9d71-7ee957da5ee2","Type":"ContainerDied","Data":"9c2b301384ff2938809bbf2d3f91b19c9ba003278e9b291ec3e9ce8f45cf4c7f"} Jul 07 00:20:40.242869 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.242844 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:40.303908 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.303881 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-socket\") pod \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " Jul 07 00:20:40.304063 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.303962 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-token\") pod \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " Jul 07 00:20:40.304063 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.303983 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istiod-ca-cert\") pod \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " Jul 07 00:20:40.304181 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.304116 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5277\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-kube-api-access-t5277\") pod \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " Jul 07 00:20:40.304237 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.304194 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-credential-socket\") pod \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " Jul 07 00:20:40.304292 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.304247 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-podinfo\") pod \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " Jul 07 00:20:40.304347 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.304313 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-data\") pod \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " Jul 07 00:20:40.304347 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.304337 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-envoy\") pod \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " Jul 07 00:20:40.304470 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.304368 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-certs\") pod \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\" (UID: \"852677b7-0e1a-4f15-9d71-7ee957da5ee2\") " Jul 07 00:20:40.305028 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.304188 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "852677b7-0e1a-4f15-9d71-7ee957da5ee2" (UID: "852677b7-0e1a-4f15-9d71-7ee957da5ee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:40.305130 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.304389 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "852677b7-0e1a-4f15-9d71-7ee957da5ee2" (UID: "852677b7-0e1a-4f15-9d71-7ee957da5ee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:20:40.305130 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.304702 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "852677b7-0e1a-4f15-9d71-7ee957da5ee2" (UID: "852677b7-0e1a-4f15-9d71-7ee957da5ee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:40.305130 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.305087 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "852677b7-0e1a-4f15-9d71-7ee957da5ee2" (UID: "852677b7-0e1a-4f15-9d71-7ee957da5ee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:40.305523 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.305486 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-data" (OuterVolumeSpecName: "istio-data") pod "852677b7-0e1a-4f15-9d71-7ee957da5ee2" (UID: "852677b7-0e1a-4f15-9d71-7ee957da5ee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:40.308449 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.308389 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-token" (OuterVolumeSpecName: "istio-token") pod "852677b7-0e1a-4f15-9d71-7ee957da5ee2" (UID: "852677b7-0e1a-4f15-9d71-7ee957da5ee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:40.308569 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.308421 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-kube-api-access-t5277" (OuterVolumeSpecName: "kube-api-access-t5277") pod "852677b7-0e1a-4f15-9d71-7ee957da5ee2" (UID: "852677b7-0e1a-4f15-9d71-7ee957da5ee2"). InnerVolumeSpecName "kube-api-access-t5277". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:40.308569 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.308487 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "852677b7-0e1a-4f15-9d71-7ee957da5ee2" (UID: "852677b7-0e1a-4f15-9d71-7ee957da5ee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:20:40.308967 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.308941 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "852677b7-0e1a-4f15-9d71-7ee957da5ee2" (UID: "852677b7-0e1a-4f15-9d71-7ee957da5ee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:40.386989 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.386964 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4"] Jul 07 00:20:40.388470 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:20:40.388415 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaea7b20_eaea_4a56_a844_a4f2331d4e1e.slice/crio-5014263ced307847dc8f1a8b05b856219991d748d9a7ebf85ca0281445aca05e WatchSource:0}: Error finding container 5014263ced307847dc8f1a8b05b856219991d748d9a7ebf85ca0281445aca05e: Status 404 returned error can't find the container with id 5014263ced307847dc8f1a8b05b856219991d748d9a7ebf85ca0281445aca05e Jul 07 00:20:40.390666 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.390634 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:40.390749 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.390698 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:40.390749 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.390725 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:40.406211 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.406190 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-credential-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:40.406309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.406213 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-podinfo\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:40.406309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.406224 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-data\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:40.406309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.406232 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-envoy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:40.406309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.406239 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-certs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:40.406309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.406247 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852677b7-0e1a-4f15-9d71-7ee957da5ee2-workload-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:40.406309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.406255 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istio-token\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:40.406309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.406264 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852677b7-0e1a-4f15-9d71-7ee957da5ee2-istiod-ca-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:40.406309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:40.406272 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5277\" (UniqueName: \"kubernetes.io/projected/852677b7-0e1a-4f15-9d71-7ee957da5ee2-kube-api-access-t5277\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:41.093899 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.093868 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k"] Jul 07 00:20:41.094324 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.094302 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="852677b7-0e1a-4f15-9d71-7ee957da5ee2" containerName="istio-proxy" Jul 07 00:20:41.094472 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.094327 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="852677b7-0e1a-4f15-9d71-7ee957da5ee2" containerName="istio-proxy" Jul 07 00:20:41.094472 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.094424 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="852677b7-0e1a-4f15-9d71-7ee957da5ee2" containerName="istio-proxy" Jul 07 00:20:41.103926 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.102863 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.105612 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.105582 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-7ac6e78c-serving-cert\"" Jul 07 00:20:41.110157 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.110079 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k"] Jul 07 00:20:41.113518 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.113160 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a630b0-fc79-4249-96bc-654b277c23b2-openshift-service-ca\") pod \"maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.113957 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.113243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v978k\" (UniqueName: \"kubernetes.io/projected/d3a630b0-fc79-4249-96bc-654b277c23b2-kube-api-access-v978k\") pod \"maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.113957 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.113912 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a630b0-fc79-4249-96bc-654b277c23b2-maas-api-tls\") pod \"maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.215477 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.215408 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a630b0-fc79-4249-96bc-654b277c23b2-openshift-service-ca\") pod \"maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.215653 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.215494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v978k\" (UniqueName: \"kubernetes.io/projected/d3a630b0-fc79-4249-96bc-654b277c23b2-kube-api-access-v978k\") pod \"maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.215653 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.215557 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a630b0-fc79-4249-96bc-654b277c23b2-maas-api-tls\") pod \"maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.216072 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.216053 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a630b0-fc79-4249-96bc-654b277c23b2-openshift-service-ca\") pod \"maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.218294 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.218264 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a630b0-fc79-4249-96bc-654b277c23b2-maas-api-tls\") pod \"maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.224317 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.224293 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v978k\" (UniqueName: \"kubernetes.io/projected/d3a630b0-fc79-4249-96bc-654b277c23b2-kube-api-access-v978k\") pod \"maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.231533 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.231509 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" Jul 07 00:20:41.231681 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.231525 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s" event={"ID":"852677b7-0e1a-4f15-9d71-7ee957da5ee2","Type":"ContainerDied","Data":"645227f2afbf724eb101cfba040537c0e8f6980d9efc519bc2742e4ebc670ba8"} Jul 07 00:20:41.231681 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.231598 2568 scope.go:117] "RemoveContainer" containerID="9c2b301384ff2938809bbf2d3f91b19c9ba003278e9b291ec3e9ce8f45cf4c7f" Jul 07 00:20:41.233428 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.233399 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" event={"ID":"faea7b20-eaea-4a56-a844-a4f2331d4e1e","Type":"ContainerStarted","Data":"b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0"} Jul 07 00:20:41.233428 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.233494 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" event={"ID":"faea7b20-eaea-4a56-a844-a4f2331d4e1e","Type":"ContainerStarted","Data":"5014263ced307847dc8f1a8b05b856219991d748d9a7ebf85ca0281445aca05e"} Jul 07 00:20:41.258676 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.258620 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" podStartSLOduration=2.258602469 podStartE2EDuration="2.258602469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:20:41.25753958 +0000 UTC m=+1462.470571327" watchObservedRunningTime="2026-07-07 00:20:41.258602469 +0000 UTC m=+1462.471634213" Jul 07 00:20:41.275357 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.275320 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s"] Jul 07 00:20:41.279735 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.279704 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a0d805de-openshift-default-68c555ffbf-pnc4s"] Jul 07 00:20:41.316384 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.316340 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="852677b7-0e1a-4f15-9d71-7ee957da5ee2" path="/var/lib/kubelet/pods/852677b7-0e1a-4f15-9d71-7ee957da5ee2/volumes" Jul 07 00:20:41.418159 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.418129 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:41.549450 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:41.549406 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k"] Jul 07 00:20:41.550567 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:20:41.550540 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a630b0_fc79_4249_96bc_654b277c23b2.slice/crio-444f100480b291336b67f1aeaf4179b764eddd198df256d8282272e6bec16742 WatchSource:0}: Error finding container 444f100480b291336b67f1aeaf4179b764eddd198df256d8282272e6bec16742: Status 404 returned error can't find the container with id 444f100480b291336b67f1aeaf4179b764eddd198df256d8282272e6bec16742 Jul 07 00:20:42.210612 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:42.210577 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:42.215704 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:42.215668 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:42.241057 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:42.241016 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" event={"ID":"d3a630b0-fc79-4249-96bc-654b277c23b2","Type":"ContainerStarted","Data":"4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0"} Jul 07 00:20:42.241235 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:42.241195 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:42.241235 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:42.241229 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" event={"ID":"d3a630b0-fc79-4249-96bc-654b277c23b2","Type":"ContainerStarted","Data":"444f100480b291336b67f1aeaf4179b764eddd198df256d8282272e6bec16742"} Jul 07 00:20:42.242260 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:42.242238 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:42.243156 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:42.243138 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:42.257774 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:42.257720 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" podStartSLOduration=0.778864854 podStartE2EDuration="1.257704752s" totalImagesPullingTime="478.839898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:20:42.25639611 +0000 UTC m=+1463.469427874" watchObservedRunningTime="2026-07-07 00:20:42.257704752 +0000 UTC m=+1463.470736496" Jul 07 00:20:46.666822 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:46.666789 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k"] Jul 07 00:20:46.667260 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:46.667003 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" podUID="d3a630b0-fc79-4249-96bc-654b277c23b2" containerName="maas-api" containerID="cri-o://4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0" gracePeriod=30 Jul 07 00:20:46.949978 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:46.949950 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:47.073526 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.073494 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a630b0-fc79-4249-96bc-654b277c23b2-maas-api-tls\") pod \"d3a630b0-fc79-4249-96bc-654b277c23b2\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " Jul 07 00:20:47.073729 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.073578 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a630b0-fc79-4249-96bc-654b277c23b2-openshift-service-ca\") pod \"d3a630b0-fc79-4249-96bc-654b277c23b2\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " Jul 07 00:20:47.073729 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.073661 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v978k\" (UniqueName: \"kubernetes.io/projected/d3a630b0-fc79-4249-96bc-654b277c23b2-kube-api-access-v978k\") pod \"d3a630b0-fc79-4249-96bc-654b277c23b2\" (UID: \"d3a630b0-fc79-4249-96bc-654b277c23b2\") " Jul 07 00:20:47.073952 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.073922 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a630b0-fc79-4249-96bc-654b277c23b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3a630b0-fc79-4249-96bc-654b277c23b2" (UID: "d3a630b0-fc79-4249-96bc-654b277c23b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:20:47.075861 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.075824 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a630b0-fc79-4249-96bc-654b277c23b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3a630b0-fc79-4249-96bc-654b277c23b2" (UID: "d3a630b0-fc79-4249-96bc-654b277c23b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:20:47.075967 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.075890 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a630b0-fc79-4249-96bc-654b277c23b2-kube-api-access-v978k" (OuterVolumeSpecName: "kube-api-access-v978k") pod "d3a630b0-fc79-4249-96bc-654b277c23b2" (UID: "d3a630b0-fc79-4249-96bc-654b277c23b2"). InnerVolumeSpecName "kube-api-access-v978k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:47.174889 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.174853 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v978k\" (UniqueName: \"kubernetes.io/projected/d3a630b0-fc79-4249-96bc-654b277c23b2-kube-api-access-v978k\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:47.174889 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.174882 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a630b0-fc79-4249-96bc-654b277c23b2-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:47.174889 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.174893 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a630b0-fc79-4249-96bc-654b277c23b2-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:47.272074 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.271981 2568 generic.go:356] "Generic (PLEG): container finished" podID="d3a630b0-fc79-4249-96bc-654b277c23b2" containerID="4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0" exitCode=0 Jul 07 00:20:47.272074 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.272027 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" Jul 07 00:20:47.272074 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.272059 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" event={"ID":"d3a630b0-fc79-4249-96bc-654b277c23b2","Type":"ContainerDied","Data":"4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0"} Jul 07 00:20:47.272355 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.272113 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k" event={"ID":"d3a630b0-fc79-4249-96bc-654b277c23b2","Type":"ContainerDied","Data":"444f100480b291336b67f1aeaf4179b764eddd198df256d8282272e6bec16742"} Jul 07 00:20:47.272355 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.272136 2568 scope.go:117] "RemoveContainer" containerID="4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0" Jul 07 00:20:47.281977 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.281956 2568 scope.go:117] "RemoveContainer" containerID="4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0" Jul 07 00:20:47.282295 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:20:47.282277 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0\": container with ID starting with 4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0 not found: ID does not exist" containerID="4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0" Jul 07 00:20:47.282343 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.282305 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0"} err="failed to get container status \"4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0\": rpc error: code = NotFound desc = could not find container \"4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0\": container with ID starting with 4926947b05b0ea9fd778321ddff7dbf5c83efa7084816c20399e65dd9840fbf0 not found: ID does not exist" Jul 07 00:20:47.294777 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.294746 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k"] Jul 07 00:20:47.299642 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.299616 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ac6e78c-774748cddd-smp7k"] Jul 07 00:20:47.316130 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:47.316089 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a630b0-fc79-4249-96bc-654b277c23b2" path="/var/lib/kubelet/pods/d3a630b0-fc79-4249-96bc-654b277c23b2/volumes" Jul 07 00:20:51.293606 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:51.293524 2568 generic.go:356] "Generic (PLEG): container finished" podID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" containerID="b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0" exitCode=0 Jul 07 00:20:51.294008 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:51.293598 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" event={"ID":"faea7b20-eaea-4a56-a844-a4f2331d4e1e","Type":"ContainerDied","Data":"b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0"} Jul 07 00:20:51.294068 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:51.294023 2568 scope.go:117] "RemoveContainer" containerID="b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0" Jul 07 00:20:51.295077 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:51.295051 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:51.295150 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:51.295134 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:51.295199 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:51.295184 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:52.197123 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:52.197078 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4"] Jul 07 00:20:52.300187 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:52.300137 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" event={"ID":"faea7b20-eaea-4a56-a844-a4f2331d4e1e","Type":"ContainerStarted","Data":"950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf"} Jul 07 00:20:52.300635 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:52.300189 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" podUID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" containerName="istio-proxy" containerID="cri-o://950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf" gracePeriod=30 Jul 07 00:20:57.243122 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.243083 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:57.568655 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.568629 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:57.668682 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.668640 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-envoy\") pod \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " Jul 07 00:20:57.668682 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.668683 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-token\") pod \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " Jul 07 00:20:57.668946 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.668723 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-podinfo\") pod \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " Jul 07 00:20:57.668946 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.668752 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-socket\") pod \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " Jul 07 00:20:57.668946 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.668846 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-credential-socket\") pod \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " Jul 07 00:20:57.668946 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.668907 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl5ng\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-kube-api-access-rl5ng\") pod \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " Jul 07 00:20:57.668946 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.668945 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istiod-ca-cert\") pod \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " Jul 07 00:20:57.669229 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.668972 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-data\") pod \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " Jul 07 00:20:57.669229 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.669008 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-certs\") pod \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\" (UID: \"faea7b20-eaea-4a56-a844-a4f2331d4e1e\") " Jul 07 00:20:57.669229 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.669135 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "faea7b20-eaea-4a56-a844-a4f2331d4e1e" (UID: "faea7b20-eaea-4a56-a844-a4f2331d4e1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:57.669229 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.669163 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "faea7b20-eaea-4a56-a844-a4f2331d4e1e" (UID: "faea7b20-eaea-4a56-a844-a4f2331d4e1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:57.669647 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.669350 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "faea7b20-eaea-4a56-a844-a4f2331d4e1e" (UID: "faea7b20-eaea-4a56-a844-a4f2331d4e1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:57.669647 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.669360 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-credential-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:57.669647 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.669379 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:57.669815 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.669777 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "faea7b20-eaea-4a56-a844-a4f2331d4e1e" (UID: "faea7b20-eaea-4a56-a844-a4f2331d4e1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:20:57.669875 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.669827 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "faea7b20-eaea-4a56-a844-a4f2331d4e1e" (UID: "faea7b20-eaea-4a56-a844-a4f2331d4e1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:57.671157 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.671129 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "faea7b20-eaea-4a56-a844-a4f2331d4e1e" (UID: "faea7b20-eaea-4a56-a844-a4f2331d4e1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:57.671157 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.671140 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "faea7b20-eaea-4a56-a844-a4f2331d4e1e" (UID: "faea7b20-eaea-4a56-a844-a4f2331d4e1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:57.671157 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.671156 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "faea7b20-eaea-4a56-a844-a4f2331d4e1e" (UID: "faea7b20-eaea-4a56-a844-a4f2331d4e1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:20:57.671460 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.671418 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-kube-api-access-rl5ng" (OuterVolumeSpecName: "kube-api-access-rl5ng") pod "faea7b20-eaea-4a56-a844-a4f2331d4e1e" (UID: "faea7b20-eaea-4a56-a844-a4f2331d4e1e"). InnerVolumeSpecName "kube-api-access-rl5ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:57.770102 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.770058 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5ng\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-kube-api-access-rl5ng\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:57.770102 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.770095 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istiod-ca-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:57.770102 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.770107 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-data\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:57.770102 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.770116 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-workload-certs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:57.770382 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.770124 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-envoy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:57.770382 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.770131 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-token\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:57.770382 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:57.770138 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea7b20-eaea-4a56-a844-a4f2331d4e1e-istio-podinfo\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:20:58.329595 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.329563 2568 generic.go:356] "Generic (PLEG): container finished" podID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" containerID="950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf" exitCode=0 Jul 07 00:20:58.330021 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.329609 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" Jul 07 00:20:58.330021 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.329644 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" event={"ID":"faea7b20-eaea-4a56-a844-a4f2331d4e1e","Type":"ContainerDied","Data":"950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf"} Jul 07 00:20:58.330021 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.329695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4" event={"ID":"faea7b20-eaea-4a56-a844-a4f2331d4e1e","Type":"ContainerDied","Data":"5014263ced307847dc8f1a8b05b856219991d748d9a7ebf85ca0281445aca05e"} Jul 07 00:20:58.330021 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.329716 2568 scope.go:117] "RemoveContainer" containerID="950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf" Jul 07 00:20:58.339330 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.339297 2568 scope.go:117] "RemoveContainer" containerID="b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0" Jul 07 00:20:58.347658 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.347636 2568 scope.go:117] "RemoveContainer" containerID="950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf" Jul 07 00:20:58.347879 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:20:58.347861 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf\": container with ID starting with 950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf not found: ID does not exist" containerID="950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf" Jul 07 00:20:58.347928 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.347888 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf"} err="failed to get container status \"950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf\": rpc error: code = NotFound desc = could not find container \"950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf\": container with ID starting with 950125897a20bf3c5d830941d377e6c52a25d0c81e394ddd6213f1f97ed86edf not found: ID does not exist" Jul 07 00:20:58.347928 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.347910 2568 scope.go:117] "RemoveContainer" containerID="b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0" Jul 07 00:20:58.348157 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:20:58.348141 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0\": container with ID starting with b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0 not found: ID does not exist" containerID="b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0" Jul 07 00:20:58.348202 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.348160 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0"} err="failed to get container status \"b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0\": rpc error: code = NotFound desc = could not find container \"b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0\": container with ID starting with b338249ba422c09d7ae9e58ed3c23d7ff4929710dfdfb60af3a2c8b5f6d105c0 not found: ID does not exist" Jul 07 00:20:58.353207 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.353178 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4"] Jul 07 00:20:58.361494 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.361467 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7ac6e78c-openshift-default-774fb7dc5-wrkj4"] Jul 07 00:20:58.417941 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.417905 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2"] Jul 07 00:20:58.418484 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.418467 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" containerName="istio-proxy" Jul 07 00:20:58.418533 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.418491 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" containerName="istio-proxy" Jul 07 00:20:58.418533 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.418517 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a630b0-fc79-4249-96bc-654b277c23b2" containerName="maas-api" Jul 07 00:20:58.418533 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.418526 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a630b0-fc79-4249-96bc-654b277c23b2" containerName="maas-api" Jul 07 00:20:58.418634 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.418539 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" containerName="istio-proxy" Jul 07 00:20:58.418634 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.418548 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" containerName="istio-proxy" Jul 07 00:20:58.418700 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.418645 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a630b0-fc79-4249-96bc-654b277c23b2" containerName="maas-api" Jul 07 00:20:58.418700 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.418656 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" containerName="istio-proxy" Jul 07 00:20:58.423560 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.423536 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.425997 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.425976 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5d0fa173-openshift-default-dockercfg-nw9t4\"" Jul 07 00:20:58.432530 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.432501 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2"] Jul 07 00:20:58.476075 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.476038 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-envoy\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.476267 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.476083 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a364809e-3eb2-4091-9328-31cbf9150bdb-istiod-ca-cert\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.476267 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.476110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-podinfo\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.476267 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.476168 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9rs\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-kube-api-access-nm9rs\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.476267 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.476259 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-certs\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.476515 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.476321 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-data\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.476515 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.476355 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-socket\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.476515 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.476409 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-credential-socket\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.476515 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.476482 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-token\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.577421 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577372 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-data\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.577421 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-socket\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.577716 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-credential-socket\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.577716 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577521 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-token\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.577716 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-envoy\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.577716 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a364809e-3eb2-4091-9328-31cbf9150bdb-istiod-ca-cert\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.577716 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577679 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-podinfo\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.577716 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577707 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9rs\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-kube-api-access-nm9rs\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.578022 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577756 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-certs\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.578022 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577848 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-data\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.578022 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577910 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-credential-socket\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.578022 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.577983 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-socket\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.578226 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.578158 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-certs\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.578504 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.578482 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a364809e-3eb2-4091-9328-31cbf9150bdb-istiod-ca-cert\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.580090 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.580035 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-envoy\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.580185 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.580172 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-podinfo\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.585737 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.585712 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-token\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.585856 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.585837 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9rs\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-kube-api-access-nm9rs\") pod \"e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.737499 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.737455 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:58.879561 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.879470 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2"] Jul 07 00:20:58.884010 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:20:58.883977 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda364809e_3eb2_4091_9328_31cbf9150bdb.slice/crio-6010145d5fd360570d29b8211f5e625981be0e1b60057f3f839560b73cecccaa WatchSource:0}: Error finding container 6010145d5fd360570d29b8211f5e625981be0e1b60057f3f839560b73cecccaa: Status 404 returned error can't find the container with id 6010145d5fd360570d29b8211f5e625981be0e1b60057f3f839560b73cecccaa Jul 07 00:20:58.886496 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.886460 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:58.886613 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.886542 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:58.886613 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:58.886585 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:20:59.316640 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.316608 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" path="/var/lib/kubelet/pods/faea7b20-eaea-4a56-a844-a4f2331d4e1e/volumes" Jul 07 00:20:59.338574 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.338541 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" event={"ID":"a364809e-3eb2-4091-9328-31cbf9150bdb","Type":"ContainerStarted","Data":"9fe7ef1fb4e9a7d555d065d66f100cd06a6bfe89337a387e6ca9254e46f80394"} Jul 07 00:20:59.338996 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.338587 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" event={"ID":"a364809e-3eb2-4091-9328-31cbf9150bdb","Type":"ContainerStarted","Data":"6010145d5fd360570d29b8211f5e625981be0e1b60057f3f839560b73cecccaa"} Jul 07 00:20:59.358592 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.358537 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" podStartSLOduration=1.358516797 podStartE2EDuration="1.358516797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:20:59.35742706 +0000 UTC m=+1480.570458818" watchObservedRunningTime="2026-07-07 00:20:59.358516797 +0000 UTC m=+1480.571548542" Jul 07 00:20:59.727695 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.727646 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn"] Jul 07 00:20:59.728190 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.728162 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="faea7b20-eaea-4a56-a844-a4f2331d4e1e" containerName="istio-proxy" Jul 07 00:20:59.731524 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.731504 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:20:59.733899 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.733866 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-5d0fa173-serving-cert\"" Jul 07 00:20:59.737624 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.737599 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:59.742774 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.742743 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn"] Jul 07 00:20:59.744155 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.744131 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:20:59.791946 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.791907 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcpdp\" (UniqueName: \"kubernetes.io/projected/83986621-a5c0-435d-8903-c6f9ef1ba7ff-kube-api-access-pcpdp\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:20:59.792152 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.791990 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83986621-a5c0-435d-8903-c6f9ef1ba7ff-openshift-service-ca\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:20:59.792152 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.792030 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:20:59.892982 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.892946 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcpdp\" (UniqueName: \"kubernetes.io/projected/83986621-a5c0-435d-8903-c6f9ef1ba7ff-kube-api-access-pcpdp\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:20:59.893187 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.893020 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83986621-a5c0-435d-8903-c6f9ef1ba7ff-openshift-service-ca\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:20:59.893187 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.893056 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:20:59.893187 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:20:59.893161 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-5d0fa173-serving-cert: secret "maas-api-e2e-derive-5d0fa173-serving-cert" not found Jul 07 00:20:59.893298 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:20:59.893217 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls podName:83986621-a5c0-435d-8903-c6f9ef1ba7ff nodeName:}" failed. No retries permitted until 2026-07-07 00:21:00.393198192 +0000 UTC m=+1481.606229913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls") pod "maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" (UID: "83986621-a5c0-435d-8903-c6f9ef1ba7ff") : secret "maas-api-e2e-derive-5d0fa173-serving-cert" not found Jul 07 00:20:59.893770 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.893751 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83986621-a5c0-435d-8903-c6f9ef1ba7ff-openshift-service-ca\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:20:59.901932 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:20:59.901902 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcpdp\" (UniqueName: \"kubernetes.io/projected/83986621-a5c0-435d-8903-c6f9ef1ba7ff-kube-api-access-pcpdp\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:21:00.343151 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:00.343116 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:21:00.344080 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:00.344063 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:21:00.397640 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:00.397489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:21:00.400296 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:00.400266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls\") pod \"maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:21:00.647971 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:00.647880 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:21:00.783561 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:00.783536 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn"] Jul 07 00:21:00.785836 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:21:00.785803 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83986621_a5c0_435d_8903_c6f9ef1ba7ff.slice/crio-9f94fe187a750acfcdf04d04af715ac0b4bbd3e10f1a7b3f25a8f38adcdc197c WatchSource:0}: Error finding container 9f94fe187a750acfcdf04d04af715ac0b4bbd3e10f1a7b3f25a8f38adcdc197c: Status 404 returned error can't find the container with id 9f94fe187a750acfcdf04d04af715ac0b4bbd3e10f1a7b3f25a8f38adcdc197c Jul 07 00:21:01.350192 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:01.350159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" event={"ID":"83986621-a5c0-435d-8903-c6f9ef1ba7ff","Type":"ContainerStarted","Data":"9f94fe187a750acfcdf04d04af715ac0b4bbd3e10f1a7b3f25a8f38adcdc197c"} Jul 07 00:21:02.356885 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:02.356846 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" event={"ID":"83986621-a5c0-435d-8903-c6f9ef1ba7ff","Type":"ContainerStarted","Data":"3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca"} Jul 07 00:21:02.384048 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:02.383994 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" podStartSLOduration=2.862969905 podStartE2EDuration="3.38397757s" totalImagesPullingTime="521.007665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:21:02.382262883 +0000 UTC m=+1483.595294627" watchObservedRunningTime="2026-07-07 00:21:02.38397757 +0000 UTC m=+1483.597009314" Jul 07 00:21:03.368777 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:03.368727 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:21:04.189337 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:04.189302 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2"] Jul 07 00:21:04.189575 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:04.189552 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" podUID="a364809e-3eb2-4091-9328-31cbf9150bdb" containerName="istio-proxy" containerID="cri-o://9fe7ef1fb4e9a7d555d065d66f100cd06a6bfe89337a387e6ca9254e46f80394" gracePeriod=30 Jul 07 00:21:04.393033 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:04.392987 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn"] Jul 07 00:21:04.437279 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:21:04.437244 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-5d0fa173-serving-cert: secret "maas-api-e2e-derive-5d0fa173-serving-cert" not found Jul 07 00:21:04.437475 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:21:04.437321 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls podName:83986621-a5c0-435d-8903-c6f9ef1ba7ff nodeName:}" failed. No retries permitted until 2026-07-07 00:21:04.93730165 +0000 UTC m=+1486.150333397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls") pod "maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" (UID: "83986621-a5c0-435d-8903-c6f9ef1ba7ff") : secret "maas-api-e2e-derive-5d0fa173-serving-cert" not found Jul 07 00:21:04.941323 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:21:04.941281 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-5d0fa173-serving-cert: secret "maas-api-e2e-derive-5d0fa173-serving-cert" not found Jul 07 00:21:04.941660 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:21:04.941362 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls podName:83986621-a5c0-435d-8903-c6f9ef1ba7ff nodeName:}" failed. No retries permitted until 2026-07-07 00:21:05.941343684 +0000 UTC m=+1487.154375405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls") pod "maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" (UID: "83986621-a5c0-435d-8903-c6f9ef1ba7ff") : secret "maas-api-e2e-derive-5d0fa173-serving-cert" not found Jul 07 00:21:05.377561 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.377457 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" podUID="83986621-a5c0-435d-8903-c6f9ef1ba7ff" containerName="maas-api" containerID="cri-o://3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca" gracePeriod=30 Jul 07 00:21:05.652592 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.652563 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:21:05.749810 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.749776 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83986621-a5c0-435d-8903-c6f9ef1ba7ff-openshift-service-ca\") pod \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " Jul 07 00:21:05.749810 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.749815 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls\") pod \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " Jul 07 00:21:05.750076 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.749952 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcpdp\" (UniqueName: \"kubernetes.io/projected/83986621-a5c0-435d-8903-c6f9ef1ba7ff-kube-api-access-pcpdp\") pod \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\" (UID: \"83986621-a5c0-435d-8903-c6f9ef1ba7ff\") " Jul 07 00:21:05.750203 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.750176 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83986621-a5c0-435d-8903-c6f9ef1ba7ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83986621-a5c0-435d-8903-c6f9ef1ba7ff" (UID: "83986621-a5c0-435d-8903-c6f9ef1ba7ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:21:05.750329 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.750313 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83986621-a5c0-435d-8903-c6f9ef1ba7ff-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:05.752122 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.752091 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83986621-a5c0-435d-8903-c6f9ef1ba7ff" (UID: "83986621-a5c0-435d-8903-c6f9ef1ba7ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:21:05.752243 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.752124 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83986621-a5c0-435d-8903-c6f9ef1ba7ff-kube-api-access-pcpdp" (OuterVolumeSpecName: "kube-api-access-pcpdp") pod "83986621-a5c0-435d-8903-c6f9ef1ba7ff" (UID: "83986621-a5c0-435d-8903-c6f9ef1ba7ff"). InnerVolumeSpecName "kube-api-access-pcpdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:21:05.851789 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.851733 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83986621-a5c0-435d-8903-c6f9ef1ba7ff-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:05.851789 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:05.851782 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcpdp\" (UniqueName: \"kubernetes.io/projected/83986621-a5c0-435d-8903-c6f9ef1ba7ff-kube-api-access-pcpdp\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:06.383398 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:06.383362 2568 generic.go:356] "Generic (PLEG): container finished" podID="83986621-a5c0-435d-8903-c6f9ef1ba7ff" containerID="3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca" exitCode=0 Jul 07 00:21:06.383599 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:06.383409 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" Jul 07 00:21:06.383599 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:06.383461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" event={"ID":"83986621-a5c0-435d-8903-c6f9ef1ba7ff","Type":"ContainerDied","Data":"3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca"} Jul 07 00:21:06.383599 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:06.383503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn" event={"ID":"83986621-a5c0-435d-8903-c6f9ef1ba7ff","Type":"ContainerDied","Data":"9f94fe187a750acfcdf04d04af715ac0b4bbd3e10f1a7b3f25a8f38adcdc197c"} Jul 07 00:21:06.383599 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:06.383523 2568 scope.go:117] "RemoveContainer" containerID="3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca" Jul 07 00:21:06.392708 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:06.392691 2568 scope.go:117] "RemoveContainer" containerID="3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca" Jul 07 00:21:06.392955 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:21:06.392936 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca\": container with ID starting with 3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca not found: ID does not exist" containerID="3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca" Jul 07 00:21:06.393021 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:06.392961 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca"} err="failed to get container status \"3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca\": rpc error: code = NotFound desc = could not find container \"3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca\": container with ID starting with 3459d0f0331d998c1849c17a87c7a2fb6e428a5c954a95351bec1634f41d02ca not found: ID does not exist" Jul 07 00:21:06.408759 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:06.408720 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn"] Jul 07 00:21:06.412869 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:06.412845 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-5d0fa173-5db8f667bb-7n8cn"] Jul 07 00:21:07.316271 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:07.316242 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83986621-a5c0-435d-8903-c6f9ef1ba7ff" path="/var/lib/kubelet/pods/83986621-a5c0-435d-8903-c6f9ef1ba7ff/volumes" Jul 07 00:21:09.400196 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.400166 2568 generic.go:356] "Generic (PLEG): container finished" podID="a364809e-3eb2-4091-9328-31cbf9150bdb" containerID="9fe7ef1fb4e9a7d555d065d66f100cd06a6bfe89337a387e6ca9254e46f80394" exitCode=0 Jul 07 00:21:09.400576 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.400220 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" event={"ID":"a364809e-3eb2-4091-9328-31cbf9150bdb","Type":"ContainerDied","Data":"9fe7ef1fb4e9a7d555d065d66f100cd06a6bfe89337a387e6ca9254e46f80394"} Jul 07 00:21:09.449391 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.449326 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:21:09.589257 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589219 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-podinfo\") pod \"a364809e-3eb2-4091-9328-31cbf9150bdb\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " Jul 07 00:21:09.589257 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589259 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-certs\") pod \"a364809e-3eb2-4091-9328-31cbf9150bdb\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " Jul 07 00:21:09.589540 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589307 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-data\") pod \"a364809e-3eb2-4091-9328-31cbf9150bdb\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " Jul 07 00:21:09.589540 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589329 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-credential-socket\") pod \"a364809e-3eb2-4091-9328-31cbf9150bdb\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " Jul 07 00:21:09.589540 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589357 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-envoy\") pod \"a364809e-3eb2-4091-9328-31cbf9150bdb\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " Jul 07 00:21:09.589540 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589402 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-socket\") pod \"a364809e-3eb2-4091-9328-31cbf9150bdb\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " Jul 07 00:21:09.589540 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589484 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-token\") pod \"a364809e-3eb2-4091-9328-31cbf9150bdb\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " Jul 07 00:21:09.589540 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589519 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm9rs\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-kube-api-access-nm9rs\") pod \"a364809e-3eb2-4091-9328-31cbf9150bdb\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " Jul 07 00:21:09.589843 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589577 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a364809e-3eb2-4091-9328-31cbf9150bdb-istiod-ca-cert\") pod \"a364809e-3eb2-4091-9328-31cbf9150bdb\" (UID: \"a364809e-3eb2-4091-9328-31cbf9150bdb\") " Jul 07 00:21:09.589843 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589702 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a364809e-3eb2-4091-9328-31cbf9150bdb" (UID: "a364809e-3eb2-4091-9328-31cbf9150bdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:09.589843 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589717 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "a364809e-3eb2-4091-9328-31cbf9150bdb" (UID: "a364809e-3eb2-4091-9328-31cbf9150bdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:09.589843 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.589773 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a364809e-3eb2-4091-9328-31cbf9150bdb" (UID: "a364809e-3eb2-4091-9328-31cbf9150bdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:09.590036 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.590011 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-certs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:09.590085 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.590023 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a364809e-3eb2-4091-9328-31cbf9150bdb" (UID: "a364809e-3eb2-4091-9328-31cbf9150bdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:09.590085 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.590033 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-data\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:09.590193 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.590085 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-workload-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:09.590250 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.590212 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a364809e-3eb2-4091-9328-31cbf9150bdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a364809e-3eb2-4091-9328-31cbf9150bdb" (UID: "a364809e-3eb2-4091-9328-31cbf9150bdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:21:09.591736 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.591706 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a364809e-3eb2-4091-9328-31cbf9150bdb" (UID: "a364809e-3eb2-4091-9328-31cbf9150bdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:09.591851 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.591715 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a364809e-3eb2-4091-9328-31cbf9150bdb" (UID: "a364809e-3eb2-4091-9328-31cbf9150bdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:21:09.591851 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.591822 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a364809e-3eb2-4091-9328-31cbf9150bdb" (UID: "a364809e-3eb2-4091-9328-31cbf9150bdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:21:09.591928 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.591888 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-kube-api-access-nm9rs" (OuterVolumeSpecName: "kube-api-access-nm9rs") pod "a364809e-3eb2-4091-9328-31cbf9150bdb" (UID: "a364809e-3eb2-4091-9328-31cbf9150bdb"). InnerVolumeSpecName "kube-api-access-nm9rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:21:09.690803 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.690757 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-credential-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:09.690803 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.690793 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-envoy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:09.690803 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.690803 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-token\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:09.690803 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.690811 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9rs\" (UniqueName: \"kubernetes.io/projected/a364809e-3eb2-4091-9328-31cbf9150bdb-kube-api-access-nm9rs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:09.690803 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.690820 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a364809e-3eb2-4091-9328-31cbf9150bdb-istiod-ca-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:09.691118 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:09.690828 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a364809e-3eb2-4091-9328-31cbf9150bdb-istio-podinfo\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:21:10.408948 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:10.408909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" event={"ID":"a364809e-3eb2-4091-9328-31cbf9150bdb","Type":"ContainerDied","Data":"6010145d5fd360570d29b8211f5e625981be0e1b60057f3f839560b73cecccaa"} Jul 07 00:21:10.409393 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:10.408976 2568 scope.go:117] "RemoveContainer" containerID="9fe7ef1fb4e9a7d555d065d66f100cd06a6bfe89337a387e6ca9254e46f80394" Jul 07 00:21:10.409393 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:10.408925 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2" Jul 07 00:21:10.432843 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:10.432811 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2"] Jul 07 00:21:10.438812 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:10.438785 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5d0fa173-openshift-default-6b788dd55-r27v2"] Jul 07 00:21:11.315822 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:11.315791 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a364809e-3eb2-4091-9328-31cbf9150bdb" path="/var/lib/kubelet/pods/a364809e-3eb2-4091-9328-31cbf9150bdb/volumes" Jul 07 00:21:19.278817 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:19.278784 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:21:19.285039 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:21:19.285018 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:22:45.912591 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.912552 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv"] Jul 07 00:22:45.913170 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.913149 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a364809e-3eb2-4091-9328-31cbf9150bdb" containerName="istio-proxy" Jul 07 00:22:45.913270 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.913173 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a364809e-3eb2-4091-9328-31cbf9150bdb" containerName="istio-proxy" Jul 07 00:22:45.913270 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.913211 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83986621-a5c0-435d-8903-c6f9ef1ba7ff" containerName="maas-api" Jul 07 00:22:45.913270 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.913221 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="83986621-a5c0-435d-8903-c6f9ef1ba7ff" containerName="maas-api" Jul 07 00:22:45.913448 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.913308 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a364809e-3eb2-4091-9328-31cbf9150bdb" containerName="istio-proxy" Jul 07 00:22:45.913448 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.913323 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="83986621-a5c0-435d-8903-c6f9ef1ba7ff" containerName="maas-api" Jul 07 00:22:45.916701 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.916679 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:45.920254 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.920235 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-60cbb180-openshift-default-dockercfg-b45pd\"" Jul 07 00:22:45.942096 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:45.942059 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv"] Jul 07 00:22:46.017203 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.017161 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ktf\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-kube-api-access-64ktf\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.017415 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.017269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-envoy\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.017415 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.017391 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad66316-5f57-495f-a95c-dd3a2493e9b8-istiod-ca-cert\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.017567 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.017460 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-data\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.017567 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.017490 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-podinfo\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.017716 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.017556 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-credential-socket\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.017716 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.017593 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-socket\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.017716 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.017617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-certs\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.017857 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.017711 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-token\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119021 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.118982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-token\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119021 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119035 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ktf\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-kube-api-access-64ktf\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119361 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119063 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-envoy\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119361 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad66316-5f57-495f-a95c-dd3a2493e9b8-istiod-ca-cert\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119361 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119146 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-data\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119361 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-podinfo\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119361 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-credential-socket\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119361 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-socket\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119361 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119253 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-certs\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119775 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119629 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-data\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119775 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119708 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-credential-socket\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.119882 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119782 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-socket\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.120002 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.119978 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad66316-5f57-495f-a95c-dd3a2493e9b8-istiod-ca-cert\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.120142 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.120123 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-certs\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.121598 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.121579 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-envoy\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.121858 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.121842 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-podinfo\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.133075 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.133044 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ktf\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-kube-api-access-64ktf\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.133250 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.133225 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-token\") pod \"e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.226399 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.226305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:46.378772 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.378736 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv"] Jul 07 00:22:46.381813 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:22:46.381786 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad66316_5f57_495f_a95c_dd3a2493e9b8.slice/crio-0ba6908c1174b1ef9421567b7ae371ba39992221ae787b4c07d7110346da31d8 WatchSource:0}: Error finding container 0ba6908c1174b1ef9421567b7ae371ba39992221ae787b4c07d7110346da31d8: Status 404 returned error can't find the container with id 0ba6908c1174b1ef9421567b7ae371ba39992221ae787b4c07d7110346da31d8 Jul 07 00:22:46.383863 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.383828 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:22:46.383959 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.383910 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:22:46.383959 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.383937 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:22:46.850007 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.849968 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" event={"ID":"aad66316-5f57-495f-a95c-dd3a2493e9b8","Type":"ContainerStarted","Data":"9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2"} Jul 07 00:22:46.850184 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.850034 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" event={"ID":"aad66316-5f57-495f-a95c-dd3a2493e9b8","Type":"ContainerStarted","Data":"0ba6908c1174b1ef9421567b7ae371ba39992221ae787b4c07d7110346da31d8"} Jul 07 00:22:46.879284 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:46.879233 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" podStartSLOduration=1.879216714 podStartE2EDuration="1.879216714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:22:46.877181322 +0000 UTC m=+1588.090213065" watchObservedRunningTime="2026-07-07 00:22:46.879216714 +0000 UTC m=+1588.092248519" Jul 07 00:22:47.226611 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:47.226572 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:47.232712 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:47.232687 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:47.854872 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:47.854843 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:22:47.855767 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:22:47.855748 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:23:01.923167 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:01.923129 2568 generic.go:356] "Generic (PLEG): container finished" podID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerID="9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2" exitCode=0 Jul 07 00:23:01.923635 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:01.923203 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" event={"ID":"aad66316-5f57-495f-a95c-dd3a2493e9b8","Type":"ContainerDied","Data":"9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2"} Jul 07 00:23:01.923635 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:01.923631 2568 scope.go:117] "RemoveContainer" containerID="9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2" Jul 07 00:23:01.924659 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:01.924630 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:23:01.924778 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:01.924714 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:23:01.924778 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:01.924760 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:23:02.854957 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:02.854905 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:23:02.933277 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:02.933231 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" event={"ID":"aad66316-5f57-495f-a95c-dd3a2493e9b8","Type":"ContainerStarted","Data":"4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075"} Jul 07 00:23:03.228294 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:03.228262 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:23:03.228663 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:03.228641 2568 patch_prober.go:28] interesting pod/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 00:23:03.228820 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:03.228793 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 00:23:04.227655 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:04.227609 2568 patch_prober.go:28] interesting pod/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 00:23:04.228171 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:04.227688 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 00:23:05.227229 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:05.227175 2568 patch_prober.go:28] interesting pod/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 00:23:05.227479 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:05.227260 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 00:23:06.227121 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:06.226677 2568 patch_prober.go:28] interesting pod/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 00:23:06.227121 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:06.226749 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 00:23:07.231047 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:07.231015 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:23:07.231515 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:07.231304 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:23:07.232575 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:07.232552 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:23:07.328110 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:07.328076 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv"] Jul 07 00:23:07.959686 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:07.959651 2568 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" secret="" err="secret \"e2e-mt-60cbb180-openshift-default-dockercfg-b45pd\" not found" Jul 07 00:23:08.963833 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:08.963797 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" containerID="cri-o://4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075" gracePeriod=30 Jul 07 00:23:14.221064 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.221036 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:23:14.410831 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.410805 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-envoy\") pod \"aad66316-5f57-495f-a95c-dd3a2493e9b8\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " Jul 07 00:23:14.410986 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.410838 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-certs\") pod \"aad66316-5f57-495f-a95c-dd3a2493e9b8\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " Jul 07 00:23:14.410986 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.410957 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64ktf\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-kube-api-access-64ktf\") pod \"aad66316-5f57-495f-a95c-dd3a2493e9b8\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " Jul 07 00:23:14.411059 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.410990 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad66316-5f57-495f-a95c-dd3a2493e9b8-istiod-ca-cert\") pod \"aad66316-5f57-495f-a95c-dd3a2493e9b8\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " Jul 07 00:23:14.411059 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411037 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-credential-socket\") pod \"aad66316-5f57-495f-a95c-dd3a2493e9b8\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " Jul 07 00:23:14.411139 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411091 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-token\") pod \"aad66316-5f57-495f-a95c-dd3a2493e9b8\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " Jul 07 00:23:14.411139 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411098 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aad66316-5f57-495f-a95c-dd3a2493e9b8" (UID: "aad66316-5f57-495f-a95c-dd3a2493e9b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:14.411229 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411153 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-podinfo\") pod \"aad66316-5f57-495f-a95c-dd3a2493e9b8\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " Jul 07 00:23:14.411229 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411213 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-data\") pod \"aad66316-5f57-495f-a95c-dd3a2493e9b8\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " Jul 07 00:23:14.411340 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411238 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-socket\") pod \"aad66316-5f57-495f-a95c-dd3a2493e9b8\" (UID: \"aad66316-5f57-495f-a95c-dd3a2493e9b8\") " Jul 07 00:23:14.411397 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411334 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aad66316-5f57-495f-a95c-dd3a2493e9b8" (UID: "aad66316-5f57-495f-a95c-dd3a2493e9b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:14.411496 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411423 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad66316-5f57-495f-a95c-dd3a2493e9b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aad66316-5f57-495f-a95c-dd3a2493e9b8" (UID: "aad66316-5f57-495f-a95c-dd3a2493e9b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:23:14.411662 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411595 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aad66316-5f57-495f-a95c-dd3a2493e9b8" (UID: "aad66316-5f57-495f-a95c-dd3a2493e9b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:14.411912 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411893 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:23:14.411912 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411919 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-workload-certs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:23:14.412055 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411933 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad66316-5f57-495f-a95c-dd3a2493e9b8-istiod-ca-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:23:14.412055 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.411947 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-credential-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:23:14.412283 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.412262 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "aad66316-5f57-495f-a95c-dd3a2493e9b8" (UID: "aad66316-5f57-495f-a95c-dd3a2493e9b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:14.413296 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.413276 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aad66316-5f57-495f-a95c-dd3a2493e9b8" (UID: "aad66316-5f57-495f-a95c-dd3a2493e9b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:14.413464 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.413424 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aad66316-5f57-495f-a95c-dd3a2493e9b8" (UID: "aad66316-5f57-495f-a95c-dd3a2493e9b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:23:14.413739 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.413719 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "aad66316-5f57-495f-a95c-dd3a2493e9b8" (UID: "aad66316-5f57-495f-a95c-dd3a2493e9b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:23:14.413739 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.413730 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-kube-api-access-64ktf" (OuterVolumeSpecName: "kube-api-access-64ktf") pod "aad66316-5f57-495f-a95c-dd3a2493e9b8" (UID: "aad66316-5f57-495f-a95c-dd3a2493e9b8"). InnerVolumeSpecName "kube-api-access-64ktf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:23:14.512967 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.512923 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-token\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:23:14.512967 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.512957 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-podinfo\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:23:14.512967 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.512965 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-data\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:23:14.512967 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.512973 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad66316-5f57-495f-a95c-dd3a2493e9b8-istio-envoy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:23:14.513256 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.512991 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ktf\" (UniqueName: \"kubernetes.io/projected/aad66316-5f57-495f-a95c-dd3a2493e9b8-kube-api-access-64ktf\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:23:14.993424 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.993389 2568 generic.go:356] "Generic (PLEG): container finished" podID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerID="4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075" exitCode=0 Jul 07 00:23:14.993624 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.993465 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" Jul 07 00:23:14.993624 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.993465 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" event={"ID":"aad66316-5f57-495f-a95c-dd3a2493e9b8","Type":"ContainerDied","Data":"4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075"} Jul 07 00:23:14.993624 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.993535 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv" event={"ID":"aad66316-5f57-495f-a95c-dd3a2493e9b8","Type":"ContainerDied","Data":"0ba6908c1174b1ef9421567b7ae371ba39992221ae787b4c07d7110346da31d8"} Jul 07 00:23:14.993624 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:14.993569 2568 scope.go:117] "RemoveContainer" containerID="4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075" Jul 07 00:23:15.003099 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:15.003082 2568 scope.go:117] "RemoveContainer" containerID="9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2" Jul 07 00:23:15.011756 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:15.011734 2568 scope.go:117] "RemoveContainer" containerID="4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075" Jul 07 00:23:15.012056 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:23:15.012031 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075\": container with ID starting with 4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075 not found: ID does not exist" containerID="4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075" Jul 07 00:23:15.012128 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:15.012065 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075"} err="failed to get container status \"4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075\": rpc error: code = NotFound desc = could not find container \"4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075\": container with ID starting with 4a95a0db56bc5fdd75590cbc4150d1eaff9ef9ed4d0c5db9a1ba0105ee026075 not found: ID does not exist" Jul 07 00:23:15.012128 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:15.012083 2568 scope.go:117] "RemoveContainer" containerID="9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2" Jul 07 00:23:15.012323 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:23:15.012304 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2\": container with ID starting with 9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2 not found: ID does not exist" containerID="9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2" Jul 07 00:23:15.012496 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:15.012326 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2"} err="failed to get container status \"9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2\": rpc error: code = NotFound desc = could not find container \"9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2\": container with ID starting with 9566e5c9baba29c55712dfd2d9d65aeaaf7e09a65de30ccff18c0dc0185429e2 not found: ID does not exist" Jul 07 00:23:15.014729 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:15.014708 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv"] Jul 07 00:23:15.019622 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:15.019590 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-60cbb180-openshift-default-7b76667475-4gwsv"] Jul 07 00:23:15.315914 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:15.315822 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" path="/var/lib/kubelet/pods/aad66316-5f57-495f-a95c-dd3a2493e9b8/volumes" Jul 07 00:23:26.981640 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:26.981603 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz"] Jul 07 00:23:26.982262 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:26.982141 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" Jul 07 00:23:26.982262 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:26.982160 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" Jul 07 00:23:26.982262 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:26.982175 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" Jul 07 00:23:26.982262 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:26.982183 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" Jul 07 00:23:26.982507 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:26.982291 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" Jul 07 00:23:26.986501 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:26.986482 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:26.989845 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:26.989827 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7d49bf96-openshift-default-dockercfg-v4w8q\"" Jul 07 00:23:26.997969 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:26.997938 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz"] Jul 07 00:23:27.038206 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.038167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-data\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.038377 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.038227 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-envoy\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.038377 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.038246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-socket\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.038377 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.038263 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-podinfo\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.038377 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.038284 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmrv\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-kube-api-access-kzmrv\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.038377 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.038303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-certs\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.038578 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.038381 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-credential-socket\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.038578 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.038424 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-token\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.038578 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.038478 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530c4314-fff5-4125-8a91-9e6ca2e65c74-istiod-ca-cert\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.139677 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.139614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-data\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.139876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.139704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-envoy\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.139876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.139735 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-socket\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.139876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.139760 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-podinfo\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.139876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.139795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmrv\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-kube-api-access-kzmrv\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.139876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.139822 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-certs\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.139876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.139874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-credential-socket\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.140212 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.140009 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-token\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.140212 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.140054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530c4314-fff5-4125-8a91-9e6ca2e65c74-istiod-ca-cert\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.140212 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.140085 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-data\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.140666 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.140636 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-credential-socket\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.140926 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.140897 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-certs\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.141089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.140939 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530c4314-fff5-4125-8a91-9e6ca2e65c74-istiod-ca-cert\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.141089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.141084 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-socket\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.143277 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.142925 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-envoy\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.143277 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.143101 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-podinfo\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.148876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.148845 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-token\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.149653 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.149627 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmrv\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-kube-api-access-kzmrv\") pod \"e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.297650 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.297548 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:27.443848 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.443804 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz"] Jul 07 00:23:27.451321 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:23:27.451293 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530c4314_fff5_4125_8a91_9e6ca2e65c74.slice/crio-83a3682c677e377c1d15ed2b036fd88b22ec03cbb0576a79f8cf1c14895a56b9 WatchSource:0}: Error finding container 83a3682c677e377c1d15ed2b036fd88b22ec03cbb0576a79f8cf1c14895a56b9: Status 404 returned error can't find the container with id 83a3682c677e377c1d15ed2b036fd88b22ec03cbb0576a79f8cf1c14895a56b9 Jul 07 00:23:27.453408 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.453371 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:23:27.453507 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.453484 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:23:27.453559 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:27.453527 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:23:28.059322 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:28.059276 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" event={"ID":"530c4314-fff5-4125-8a91-9e6ca2e65c74","Type":"ContainerStarted","Data":"54390ecabb955e6c56bde805440ec1cd6146efb1cc38ec18a046e272ebc37b37"} Jul 07 00:23:28.059766 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:28.059346 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" event={"ID":"530c4314-fff5-4125-8a91-9e6ca2e65c74","Type":"ContainerStarted","Data":"83a3682c677e377c1d15ed2b036fd88b22ec03cbb0576a79f8cf1c14895a56b9"} Jul 07 00:23:28.083757 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:28.083702 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" podStartSLOduration=2.083684221 podStartE2EDuration="2.083684221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:23:28.080740968 +0000 UTC m=+1629.293772737" watchObservedRunningTime="2026-07-07 00:23:28.083684221 +0000 UTC m=+1629.296715965" Jul 07 00:23:28.298361 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:28.298317 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:28.303608 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:28.303581 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:29.064368 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:29.064331 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:29.065543 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:29.065523 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:30.092201 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.092165 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz"] Jul 07 00:23:30.092680 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.092593 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad66316-5f57-495f-a95c-dd3a2493e9b8" containerName="istio-proxy" Jul 07 00:23:30.095581 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.095565 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.098697 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.098681 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-7d49bf96-serving-cert\"" Jul 07 00:23:30.136139 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.136103 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz"] Jul 07 00:23:30.172595 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.172560 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgszz\" (UniqueName: \"kubernetes.io/projected/d3034a52-920d-49e2-b6b4-2eb36c447aab-kube-api-access-fgszz\") pod \"maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.172774 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.172602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3034a52-920d-49e2-b6b4-2eb36c447aab-openshift-service-ca\") pod \"maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.172774 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.172701 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3034a52-920d-49e2-b6b4-2eb36c447aab-maas-api-tls\") pod \"maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.274114 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.274071 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3034a52-920d-49e2-b6b4-2eb36c447aab-maas-api-tls\") pod \"maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.274322 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.274218 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgszz\" (UniqueName: \"kubernetes.io/projected/d3034a52-920d-49e2-b6b4-2eb36c447aab-kube-api-access-fgszz\") pod \"maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.274322 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.274248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3034a52-920d-49e2-b6b4-2eb36c447aab-openshift-service-ca\") pod \"maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.274833 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.274814 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3034a52-920d-49e2-b6b4-2eb36c447aab-openshift-service-ca\") pod \"maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.276637 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.276620 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3034a52-920d-49e2-b6b4-2eb36c447aab-maas-api-tls\") pod \"maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.310838 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.310808 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgszz\" (UniqueName: \"kubernetes.io/projected/d3034a52-920d-49e2-b6b4-2eb36c447aab-kube-api-access-fgszz\") pod \"maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.406223 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.406132 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:30.543250 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:30.543224 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz"] Jul 07 00:23:30.545261 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:23:30.545224 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3034a52_920d_49e2_b6b4_2eb36c447aab.slice/crio-6f3ecb14852ae8023127723a44965fd1c50fc45bc38922ab506dcfe947c55d6b WatchSource:0}: Error finding container 6f3ecb14852ae8023127723a44965fd1c50fc45bc38922ab506dcfe947c55d6b: Status 404 returned error can't find the container with id 6f3ecb14852ae8023127723a44965fd1c50fc45bc38922ab506dcfe947c55d6b Jul 07 00:23:31.075984 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:31.075950 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" event={"ID":"d3034a52-920d-49e2-b6b4-2eb36c447aab","Type":"ContainerStarted","Data":"fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a"} Jul 07 00:23:31.076159 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:31.076012 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" event={"ID":"d3034a52-920d-49e2-b6b4-2eb36c447aab","Type":"ContainerStarted","Data":"6f3ecb14852ae8023127723a44965fd1c50fc45bc38922ab506dcfe947c55d6b"} Jul 07 00:23:31.076242 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:31.076223 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:31.095003 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:31.094957 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" podStartSLOduration=0.705577966 podStartE2EDuration="1.094940845s" totalImagesPullingTime="389.362879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:23:31.093417748 +0000 UTC m=+1632.306449519" watchObservedRunningTime="2026-07-07 00:23:31.094940845 +0000 UTC m=+1632.307972588" Jul 07 00:23:37.086550 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:37.086507 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:23:44.066488 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:44.066425 2568 patch_prober.go:28] interesting pod/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 07 00:23:44.066923 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:44.066511 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 07 00:23:44.145277 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:44.145243 2568 generic.go:356] "Generic (PLEG): container finished" podID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerID="54390ecabb955e6c56bde805440ec1cd6146efb1cc38ec18a046e272ebc37b37" exitCode=0 Jul 07 00:23:44.145500 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:44.145320 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" event={"ID":"530c4314-fff5-4125-8a91-9e6ca2e65c74","Type":"ContainerDied","Data":"54390ecabb955e6c56bde805440ec1cd6146efb1cc38ec18a046e272ebc37b37"} Jul 07 00:23:44.145836 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:44.145818 2568 scope.go:117] "RemoveContainer" containerID="54390ecabb955e6c56bde805440ec1cd6146efb1cc38ec18a046e272ebc37b37" Jul 07 00:23:44.146920 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:44.146890 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:23:44.147011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:44.146960 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:23:44.147011 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:44.146992 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:23:45.152552 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:45.152516 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" event={"ID":"530c4314-fff5-4125-8a91-9e6ca2e65c74","Type":"ContainerStarted","Data":"d2a1dda988d1f5f281ba1afde579d9d9ae414e6709656dbeadfb4a8b618c85bc"} Jul 07 00:23:45.298055 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:45.298015 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:45.298238 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:45.298214 2568 patch_prober.go:28] interesting pod/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 07 00:23:45.298286 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:45.298265 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 07 00:23:46.298278 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:46.298233 2568 patch_prober.go:28] interesting pod/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 07 00:23:46.298800 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:46.298317 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 07 00:23:47.298041 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:47.297999 2568 patch_prober.go:28] interesting pod/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 07 00:23:47.298235 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:47.298073 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 07 00:23:48.298522 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:48.298485 2568 patch_prober.go:28] interesting pod/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 07 00:23:48.298903 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:48.298556 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 07 00:23:49.319155 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:49.319127 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:49.319633 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:49.319400 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:23:49.320166 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:23:49.320147 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:24:17.605488 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:17.605422 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz"] Jul 07 00:24:17.606097 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:17.605743 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" podUID="d3034a52-920d-49e2-b6b4-2eb36c447aab" containerName="maas-api" containerID="cri-o://fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a" gracePeriod=30 Jul 07 00:24:17.862948 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:17.862876 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:24:17.943848 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:17.943809 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3034a52-920d-49e2-b6b4-2eb36c447aab-maas-api-tls\") pod \"d3034a52-920d-49e2-b6b4-2eb36c447aab\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " Jul 07 00:24:17.944047 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:17.943914 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3034a52-920d-49e2-b6b4-2eb36c447aab-openshift-service-ca\") pod \"d3034a52-920d-49e2-b6b4-2eb36c447aab\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " Jul 07 00:24:17.944047 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:17.943995 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgszz\" (UniqueName: \"kubernetes.io/projected/d3034a52-920d-49e2-b6b4-2eb36c447aab-kube-api-access-fgszz\") pod \"d3034a52-920d-49e2-b6b4-2eb36c447aab\" (UID: \"d3034a52-920d-49e2-b6b4-2eb36c447aab\") " Jul 07 00:24:17.944312 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:17.944284 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3034a52-920d-49e2-b6b4-2eb36c447aab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3034a52-920d-49e2-b6b4-2eb36c447aab" (UID: "d3034a52-920d-49e2-b6b4-2eb36c447aab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:24:17.946199 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:17.946179 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3034a52-920d-49e2-b6b4-2eb36c447aab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3034a52-920d-49e2-b6b4-2eb36c447aab" (UID: "d3034a52-920d-49e2-b6b4-2eb36c447aab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:24:17.946304 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:17.946267 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3034a52-920d-49e2-b6b4-2eb36c447aab-kube-api-access-fgszz" (OuterVolumeSpecName: "kube-api-access-fgszz") pod "d3034a52-920d-49e2-b6b4-2eb36c447aab" (UID: "d3034a52-920d-49e2-b6b4-2eb36c447aab"). InnerVolumeSpecName "kube-api-access-fgszz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:24:18.045094 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.045056 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3034a52-920d-49e2-b6b4-2eb36c447aab-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:24:18.045094 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.045090 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3034a52-920d-49e2-b6b4-2eb36c447aab-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:24:18.045094 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.045103 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgszz\" (UniqueName: \"kubernetes.io/projected/d3034a52-920d-49e2-b6b4-2eb36c447aab-kube-api-access-fgszz\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:24:18.309557 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.309521 2568 generic.go:356] "Generic (PLEG): container finished" podID="d3034a52-920d-49e2-b6b4-2eb36c447aab" containerID="fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a" exitCode=0 Jul 07 00:24:18.309779 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.309552 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" event={"ID":"d3034a52-920d-49e2-b6b4-2eb36c447aab","Type":"ContainerDied","Data":"fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a"} Jul 07 00:24:18.309779 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.309608 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" event={"ID":"d3034a52-920d-49e2-b6b4-2eb36c447aab","Type":"ContainerDied","Data":"6f3ecb14852ae8023127723a44965fd1c50fc45bc38922ab506dcfe947c55d6b"} Jul 07 00:24:18.309779 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.309628 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz" Jul 07 00:24:18.309779 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.309637 2568 scope.go:117] "RemoveContainer" containerID="fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a" Jul 07 00:24:18.318686 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.318669 2568 scope.go:117] "RemoveContainer" containerID="fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a" Jul 07 00:24:18.318974 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:24:18.318958 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a\": container with ID starting with fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a not found: ID does not exist" containerID="fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a" Jul 07 00:24:18.319014 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.318982 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a"} err="failed to get container status \"fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a\": rpc error: code = NotFound desc = could not find container \"fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a\": container with ID starting with fc87074ede766ae7762599ba4a1e4f678b8cc4a959ad9f90cca0812cbc8c072a not found: ID does not exist" Jul 07 00:24:18.330461 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.330410 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz"] Jul 07 00:24:18.334908 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:18.334877 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-7d49bf96-6c7d5d9f84-2srqz"] Jul 07 00:24:19.316029 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:24:19.315987 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3034a52-920d-49e2-b6b4-2eb36c447aab" path="/var/lib/kubelet/pods/d3034a52-920d-49e2-b6b4-2eb36c447aab/volumes" Jul 07 00:25:11.444980 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:11.444876 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz"] Jul 07 00:25:11.445995 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:11.445223 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" containerID="cri-o://d2a1dda988d1f5f281ba1afde579d9d9ae414e6709656dbeadfb4a8b618c85bc" gracePeriod=30 Jul 07 00:25:16.606083 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.606046 2568 generic.go:356] "Generic (PLEG): container finished" podID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerID="d2a1dda988d1f5f281ba1afde579d9d9ae414e6709656dbeadfb4a8b618c85bc" exitCode=0 Jul 07 00:25:16.606600 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.606123 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" event={"ID":"530c4314-fff5-4125-8a91-9e6ca2e65c74","Type":"ContainerDied","Data":"d2a1dda988d1f5f281ba1afde579d9d9ae414e6709656dbeadfb4a8b618c85bc"} Jul 07 00:25:16.606600 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.606196 2568 scope.go:117] "RemoveContainer" containerID="54390ecabb955e6c56bde805440ec1cd6146efb1cc38ec18a046e272ebc37b37" Jul 07 00:25:16.710597 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.710529 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:25:16.803486 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803421 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-token\") pod \"530c4314-fff5-4125-8a91-9e6ca2e65c74\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " Jul 07 00:25:16.803486 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803500 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-podinfo\") pod \"530c4314-fff5-4125-8a91-9e6ca2e65c74\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " Jul 07 00:25:16.803758 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803526 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzmrv\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-kube-api-access-kzmrv\") pod \"530c4314-fff5-4125-8a91-9e6ca2e65c74\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " Jul 07 00:25:16.803758 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803547 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-envoy\") pod \"530c4314-fff5-4125-8a91-9e6ca2e65c74\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " Jul 07 00:25:16.803758 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803565 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530c4314-fff5-4125-8a91-9e6ca2e65c74-istiod-ca-cert\") pod \"530c4314-fff5-4125-8a91-9e6ca2e65c74\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " Jul 07 00:25:16.803758 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803641 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-socket\") pod \"530c4314-fff5-4125-8a91-9e6ca2e65c74\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " Jul 07 00:25:16.803758 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803701 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-data\") pod \"530c4314-fff5-4125-8a91-9e6ca2e65c74\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " Jul 07 00:25:16.803758 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803725 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-certs\") pod \"530c4314-fff5-4125-8a91-9e6ca2e65c74\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " Jul 07 00:25:16.804065 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803795 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-credential-socket\") pod \"530c4314-fff5-4125-8a91-9e6ca2e65c74\" (UID: \"530c4314-fff5-4125-8a91-9e6ca2e65c74\") " Jul 07 00:25:16.804065 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.803963 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530c4314-fff5-4125-8a91-9e6ca2e65c74-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "530c4314-fff5-4125-8a91-9e6ca2e65c74" (UID: "530c4314-fff5-4125-8a91-9e6ca2e65c74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:25:16.804186 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.804161 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "530c4314-fff5-4125-8a91-9e6ca2e65c74" (UID: "530c4314-fff5-4125-8a91-9e6ca2e65c74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:25:16.804295 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.804261 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-credential-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:25:16.804295 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.804279 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530c4314-fff5-4125-8a91-9e6ca2e65c74-istiod-ca-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:25:16.804534 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.804342 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "530c4314-fff5-4125-8a91-9e6ca2e65c74" (UID: "530c4314-fff5-4125-8a91-9e6ca2e65c74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:25:16.804534 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.804464 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "530c4314-fff5-4125-8a91-9e6ca2e65c74" (UID: "530c4314-fff5-4125-8a91-9e6ca2e65c74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:25:16.804913 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.804878 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-data" (OuterVolumeSpecName: "istio-data") pod "530c4314-fff5-4125-8a91-9e6ca2e65c74" (UID: "530c4314-fff5-4125-8a91-9e6ca2e65c74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:25:16.805992 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.805959 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-token" (OuterVolumeSpecName: "istio-token") pod "530c4314-fff5-4125-8a91-9e6ca2e65c74" (UID: "530c4314-fff5-4125-8a91-9e6ca2e65c74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:25:16.806344 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.806303 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-kube-api-access-kzmrv" (OuterVolumeSpecName: "kube-api-access-kzmrv") pod "530c4314-fff5-4125-8a91-9e6ca2e65c74" (UID: "530c4314-fff5-4125-8a91-9e6ca2e65c74"). InnerVolumeSpecName "kube-api-access-kzmrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:25:16.806463 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.806373 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "530c4314-fff5-4125-8a91-9e6ca2e65c74" (UID: "530c4314-fff5-4125-8a91-9e6ca2e65c74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:25:16.806463 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.806395 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "530c4314-fff5-4125-8a91-9e6ca2e65c74" (UID: "530c4314-fff5-4125-8a91-9e6ca2e65c74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:25:16.905720 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.905667 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-token\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:25:16.905720 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.905711 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-podinfo\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:25:16.905720 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.905724 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzmrv\" (UniqueName: \"kubernetes.io/projected/530c4314-fff5-4125-8a91-9e6ca2e65c74-kube-api-access-kzmrv\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:25:16.905720 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.905737 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-envoy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:25:16.906010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.905752 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:25:16.906010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.905763 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-istio-data\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:25:16.906010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:16.905773 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530c4314-fff5-4125-8a91-9e6ca2e65c74-workload-certs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:25:17.612360 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:17.612281 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" Jul 07 00:25:17.612849 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:17.612276 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz" event={"ID":"530c4314-fff5-4125-8a91-9e6ca2e65c74","Type":"ContainerDied","Data":"83a3682c677e377c1d15ed2b036fd88b22ec03cbb0576a79f8cf1c14895a56b9"} Jul 07 00:25:17.612849 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:17.612419 2568 scope.go:117] "RemoveContainer" containerID="d2a1dda988d1f5f281ba1afde579d9d9ae414e6709656dbeadfb4a8b618c85bc" Jul 07 00:25:17.635739 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:17.635699 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz"] Jul 07 00:25:17.638904 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:17.638873 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d49bf96-openshift-default-65998ddf7f-9jnrz"] Jul 07 00:25:19.315585 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:25:19.315551 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" path="/var/lib/kubelet/pods/530c4314-fff5-4125-8a91-9e6ca2e65c74/volumes" Jul 07 00:26:06.599161 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.599123 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd"] Jul 07 00:26:06.599776 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.599690 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3034a52-920d-49e2-b6b4-2eb36c447aab" containerName="maas-api" Jul 07 00:26:06.599776 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.599712 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3034a52-920d-49e2-b6b4-2eb36c447aab" containerName="maas-api" Jul 07 00:26:06.599776 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.599729 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" Jul 07 00:26:06.599776 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.599738 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" Jul 07 00:26:06.599776 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.599764 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" Jul 07 00:26:06.599776 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.599772 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" Jul 07 00:26:06.600075 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.599846 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" Jul 07 00:26:06.600075 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.599861 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3034a52-920d-49e2-b6b4-2eb36c447aab" containerName="maas-api" Jul 07 00:26:06.603128 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.603105 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.605967 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.605932 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cded59-openshift-default-dockercfg-2dfzn\"" Jul 07 00:26:06.615151 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.615125 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd"] Jul 07 00:26:06.664343 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.664307 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-envoy\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.664530 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.664363 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-podinfo\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.664530 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.664403 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-certs\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.664530 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.664426 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slscn\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-kube-api-access-slscn\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.664530 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.664499 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-data\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.664530 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.664524 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istiod-ca-cert\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.664703 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.664548 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-credential-socket\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.664703 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.664573 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-socket\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.664703 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.664634 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-token\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765147 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765104 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slscn\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-kube-api-access-slscn\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765355 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765163 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-data\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765355 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765337 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istiod-ca-cert\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765532 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-credential-socket\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765532 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765463 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-socket\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765532 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-token\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765532 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765518 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-data\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765742 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-envoy\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765742 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-podinfo\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765742 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-certs\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765893 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765756 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-credential-socket\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.765893 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.765840 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-socket\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.766025 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.766004 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-certs\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.766088 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.766069 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istiod-ca-cert\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.767935 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.767903 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-envoy\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.768132 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.768113 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-podinfo\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.788839 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.788808 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-token\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.789104 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.789083 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slscn\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-kube-api-access-slscn\") pod \"e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:06.914501 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:06.914466 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:07.075932 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:07.075905 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd"] Jul 07 00:26:07.077637 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:26:07.077606 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b8cd6d_a811_49de_9a32_49c3a216bcdf.slice/crio-710a7bf8bd9a6e8e7c39f2b1171c66921deb42496f498bf109de887e12b5b538 WatchSource:0}: Error finding container 710a7bf8bd9a6e8e7c39f2b1171c66921deb42496f498bf109de887e12b5b538: Status 404 returned error can't find the container with id 710a7bf8bd9a6e8e7c39f2b1171c66921deb42496f498bf109de887e12b5b538 Jul 07 00:26:07.079918 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:07.079886 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:26:07.079983 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:07.079957 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:26:07.080024 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:07.079983 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:26:07.850500 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:07.850458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" event={"ID":"55b8cd6d-a811-49de-9a32-49c3a216bcdf","Type":"ContainerStarted","Data":"986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd"} Jul 07 00:26:07.850876 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:07.850522 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" event={"ID":"55b8cd6d-a811-49de-9a32-49c3a216bcdf","Type":"ContainerStarted","Data":"710a7bf8bd9a6e8e7c39f2b1171c66921deb42496f498bf109de887e12b5b538"} Jul 07 00:26:07.877175 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:07.877120 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" podStartSLOduration=1.8771011020000001 podStartE2EDuration="1.877101102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:26:07.874551694 +0000 UTC m=+1789.087583435" watchObservedRunningTime="2026-07-07 00:26:07.877101102 +0000 UTC m=+1789.090132846" Jul 07 00:26:07.915134 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:07.915095 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:08.919823 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:08.919794 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:09.859301 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:09.859210 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:09.860250 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:09.860227 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:09.901384 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:09.901349 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv"] Jul 07 00:26:09.902202 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:09.902179 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="530c4314-fff5-4125-8a91-9e6ca2e65c74" containerName="istio-proxy" Jul 07 00:26:09.905306 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:09.905287 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:09.908540 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:09.908517 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-cded59-serving-cert\"" Jul 07 00:26:09.919026 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:09.919000 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv"] Jul 07 00:26:10.002503 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.002418 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fedd27-d78e-484d-8739-604a6b24fb87-maas-api-tls\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.003049 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.002547 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbfk\" (UniqueName: \"kubernetes.io/projected/d3fedd27-d78e-484d-8739-604a6b24fb87-kube-api-access-sdbfk\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.003049 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.002690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fedd27-d78e-484d-8739-604a6b24fb87-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.103995 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.103950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fedd27-d78e-484d-8739-604a6b24fb87-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.104211 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.104043 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fedd27-d78e-484d-8739-604a6b24fb87-maas-api-tls\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.104211 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.104092 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbfk\" (UniqueName: \"kubernetes.io/projected/d3fedd27-d78e-484d-8739-604a6b24fb87-kube-api-access-sdbfk\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.104344 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:26:10.104217 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-cded59-serving-cert: secret "maas-api-e2e-inf-a-cded59-serving-cert" not found Jul 07 00:26:10.104344 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:26:10.104313 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3fedd27-d78e-484d-8739-604a6b24fb87-maas-api-tls podName:d3fedd27-d78e-484d-8739-604a6b24fb87 nodeName:}" failed. No retries permitted until 2026-07-07 00:26:10.604288846 +0000 UTC m=+1791.817320583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3fedd27-d78e-484d-8739-604a6b24fb87-maas-api-tls") pod "maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" (UID: "d3fedd27-d78e-484d-8739-604a6b24fb87") : secret "maas-api-e2e-inf-a-cded59-serving-cert" not found Jul 07 00:26:10.104803 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.104782 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fedd27-d78e-484d-8739-604a6b24fb87-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.116378 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.116302 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbfk\" (UniqueName: \"kubernetes.io/projected/d3fedd27-d78e-484d-8739-604a6b24fb87-kube-api-access-sdbfk\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.609056 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.609014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fedd27-d78e-484d-8739-604a6b24fb87-maas-api-tls\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.611656 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.611632 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fedd27-d78e-484d-8739-604a6b24fb87-maas-api-tls\") pod \"maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.818730 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.818691 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:10.973330 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.973304 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv"] Jul 07 00:26:10.975419 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:26:10.975384 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fedd27_d78e_484d_8739_604a6b24fb87.slice/crio-21bb326f14bb2efc0bb965c9b93c10babfab8838664e185b001c2cfacedc3ce4 WatchSource:0}: Error finding container 21bb326f14bb2efc0bb965c9b93c10babfab8838664e185b001c2cfacedc3ce4: Status 404 returned error can't find the container with id 21bb326f14bb2efc0bb965c9b93c10babfab8838664e185b001c2cfacedc3ce4 Jul 07 00:26:10.976798 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:10.976780 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:26:11.870887 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:11.870773 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" event={"ID":"d3fedd27-d78e-484d-8739-604a6b24fb87","Type":"ContainerStarted","Data":"1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180"} Jul 07 00:26:11.870887 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:11.870840 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" event={"ID":"d3fedd27-d78e-484d-8739-604a6b24fb87","Type":"ContainerStarted","Data":"21bb326f14bb2efc0bb965c9b93c10babfab8838664e185b001c2cfacedc3ce4"} Jul 07 00:26:11.890517 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:11.890457 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" podStartSLOduration=2.450397049 podStartE2EDuration="2.89041518s" totalImagesPullingTime="440.018131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:26:11.889137492 +0000 UTC m=+1793.102169237" watchObservedRunningTime="2026-07-07 00:26:11.89041518 +0000 UTC m=+1793.103446925" Jul 07 00:26:12.875556 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:12.875521 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:18.886125 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:18.886092 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:26:19.308575 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:19.308546 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:26:19.342722 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:19.342698 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:26:21.101738 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.101697 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc"] Jul 07 00:26:21.105654 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.105633 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.109090 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.109065 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-0ec7c3-serving-cert\"" Jul 07 00:26:21.121657 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.121633 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc"] Jul 07 00:26:21.210742 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.210706 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.210921 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.210820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lj48\" (UniqueName: \"kubernetes.io/projected/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-kube-api-access-7lj48\") pod \"maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.210921 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.210851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.312501 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.312394 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lj48\" (UniqueName: \"kubernetes.io/projected/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-kube-api-access-7lj48\") pod \"maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.312674 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.312502 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.312674 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.312593 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.313404 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.313372 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.315613 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.315582 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.323846 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.323814 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lj48\" (UniqueName: \"kubernetes.io/projected/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-kube-api-access-7lj48\") pod \"maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.416243 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.416208 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:21.579059 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.578998 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc"] Jul 07 00:26:21.581876 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:26:21.581838 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b4bad4_9210_4ce8_8bb9_57ef56df74fc.slice/crio-0031cbbf74bb0885f86a686ef7fb7e9b4dd5c92cb05162d4e0017fd5186836e7 WatchSource:0}: Error finding container 0031cbbf74bb0885f86a686ef7fb7e9b4dd5c92cb05162d4e0017fd5186836e7: Status 404 returned error can't find the container with id 0031cbbf74bb0885f86a686ef7fb7e9b4dd5c92cb05162d4e0017fd5186836e7 Jul 07 00:26:21.922192 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:21.922156 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" event={"ID":"08b4bad4-9210-4ce8-8bb9-57ef56df74fc","Type":"ContainerStarted","Data":"0031cbbf74bb0885f86a686ef7fb7e9b4dd5c92cb05162d4e0017fd5186836e7"} Jul 07 00:26:22.929190 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:22.929144 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" event={"ID":"08b4bad4-9210-4ce8-8bb9-57ef56df74fc","Type":"ContainerStarted","Data":"793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485"} Jul 07 00:26:22.929588 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:22.929223 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:26:22.950002 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:22.949929 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" podStartSLOduration=1.426613587 podStartE2EDuration="1.949911508s" totalImagesPullingTime="523.297921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:26:22.948611082 +0000 UTC m=+1804.161642826" watchObservedRunningTime="2026-07-07 00:26:22.949911508 +0000 UTC m=+1804.162943273" Jul 07 00:26:23.935511 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:23.935472 2568 generic.go:356] "Generic (PLEG): container finished" podID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerID="986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd" exitCode=0 Jul 07 00:26:23.936012 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:23.935540 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" event={"ID":"55b8cd6d-a811-49de-9a32-49c3a216bcdf","Type":"ContainerDied","Data":"986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd"} Jul 07 00:26:23.936062 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:23.936012 2568 scope.go:117] "RemoveContainer" containerID="986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd" Jul 07 00:26:23.937075 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:23.937048 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:26:23.937150 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:23.937133 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:26:23.937187 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:23.937170 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:26:24.860270 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:24.860232 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:24.943243 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:24.943200 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" event={"ID":"55b8cd6d-a811-49de-9a32-49c3a216bcdf","Type":"ContainerStarted","Data":"0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d"} Jul 07 00:26:25.914885 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:25.914836 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:25.916076 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:25.916037 2568 patch_prober.go:28] interesting pod/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 00:26:25.916235 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:25.916097 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 00:26:26.915817 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:26.915776 2568 patch_prober.go:28] interesting pod/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 00:26:26.916465 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:26.916402 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 00:26:27.915560 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:27.915519 2568 patch_prober.go:28] interesting pod/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 00:26:27.915858 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:27.915831 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 00:26:28.919132 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:28.919095 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:28.919520 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:28.919387 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:28.920047 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:28.920033 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:26:28.942463 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:26:28.942414 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:27:15.355142 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.355107 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv"] Jul 07 00:27:15.355728 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.355320 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" podUID="d3fedd27-d78e-484d-8739-604a6b24fb87" containerName="maas-api" containerID="cri-o://1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180" gracePeriod=30 Jul 07 00:27:15.607305 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.607234 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:27:15.718244 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.718205 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fedd27-d78e-484d-8739-604a6b24fb87-openshift-service-ca\") pod \"d3fedd27-d78e-484d-8739-604a6b24fb87\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " Jul 07 00:27:15.718464 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.718290 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fedd27-d78e-484d-8739-604a6b24fb87-maas-api-tls\") pod \"d3fedd27-d78e-484d-8739-604a6b24fb87\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " Jul 07 00:27:15.718464 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.718354 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdbfk\" (UniqueName: \"kubernetes.io/projected/d3fedd27-d78e-484d-8739-604a6b24fb87-kube-api-access-sdbfk\") pod \"d3fedd27-d78e-484d-8739-604a6b24fb87\" (UID: \"d3fedd27-d78e-484d-8739-604a6b24fb87\") " Jul 07 00:27:15.718631 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.718605 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fedd27-d78e-484d-8739-604a6b24fb87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3fedd27-d78e-484d-8739-604a6b24fb87" (UID: "d3fedd27-d78e-484d-8739-604a6b24fb87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:27:15.718720 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.718702 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fedd27-d78e-484d-8739-604a6b24fb87-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:27:15.720652 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.720623 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fedd27-d78e-484d-8739-604a6b24fb87-kube-api-access-sdbfk" (OuterVolumeSpecName: "kube-api-access-sdbfk") pod "d3fedd27-d78e-484d-8739-604a6b24fb87" (UID: "d3fedd27-d78e-484d-8739-604a6b24fb87"). InnerVolumeSpecName "kube-api-access-sdbfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:27:15.720772 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.720749 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fedd27-d78e-484d-8739-604a6b24fb87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3fedd27-d78e-484d-8739-604a6b24fb87" (UID: "d3fedd27-d78e-484d-8739-604a6b24fb87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:27:15.819220 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.819184 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fedd27-d78e-484d-8739-604a6b24fb87-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:27:15.819220 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:15.819220 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbfk\" (UniqueName: \"kubernetes.io/projected/d3fedd27-d78e-484d-8739-604a6b24fb87-kube-api-access-sdbfk\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:27:16.190132 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:16.190095 2568 generic.go:356] "Generic (PLEG): container finished" podID="d3fedd27-d78e-484d-8739-604a6b24fb87" containerID="1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180" exitCode=0 Jul 07 00:27:16.190344 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:16.190145 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" Jul 07 00:27:16.190344 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:16.190148 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" event={"ID":"d3fedd27-d78e-484d-8739-604a6b24fb87","Type":"ContainerDied","Data":"1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180"} Jul 07 00:27:16.190344 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:16.190291 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv" event={"ID":"d3fedd27-d78e-484d-8739-604a6b24fb87","Type":"ContainerDied","Data":"21bb326f14bb2efc0bb965c9b93c10babfab8838664e185b001c2cfacedc3ce4"} Jul 07 00:27:16.190344 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:16.190317 2568 scope.go:117] "RemoveContainer" containerID="1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180" Jul 07 00:27:16.199321 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:16.199298 2568 scope.go:117] "RemoveContainer" containerID="1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180" Jul 07 00:27:16.199631 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:27:16.199613 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180\": container with ID starting with 1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180 not found: ID does not exist" containerID="1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180" Jul 07 00:27:16.199690 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:16.199641 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180"} err="failed to get container status \"1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180\": rpc error: code = NotFound desc = could not find container \"1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180\": container with ID starting with 1d3d2bbf139338b0774243d95cde8df629101e7b45555f83d13f0611ce8a9180 not found: ID does not exist" Jul 07 00:27:16.211863 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:16.211834 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv"] Jul 07 00:27:16.214197 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:16.214174 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cded59-bb4899b75-jfbdv"] Jul 07 00:27:17.315385 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:17.315347 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3fedd27-d78e-484d-8739-604a6b24fb87" path="/var/lib/kubelet/pods/d3fedd27-d78e-484d-8739-604a6b24fb87/volumes" Jul 07 00:27:57.411104 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:57.411013 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd"] Jul 07 00:27:57.411714 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:57.411271 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" containerID="cri-o://0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d" gracePeriod=30 Jul 07 00:27:58.595848 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.595805 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc"] Jul 07 00:27:58.596313 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.596110 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" podUID="08b4bad4-9210-4ce8-8bb9-57ef56df74fc" containerName="maas-api" containerID="cri-o://793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485" gracePeriod=30 Jul 07 00:27:58.840490 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.840460 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:27:58.921894 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.921861 2568 patch_prober.go:28] interesting pod/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 00:27:58.922059 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.921915 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 00:27:58.938500 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.938474 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-maas-api-tls\") pod \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " Jul 07 00:27:58.938574 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.938562 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lj48\" (UniqueName: \"kubernetes.io/projected/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-kube-api-access-7lj48\") pod \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " Jul 07 00:27:58.938612 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.938601 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-openshift-service-ca\") pod \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\" (UID: \"08b4bad4-9210-4ce8-8bb9-57ef56df74fc\") " Jul 07 00:27:58.938995 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.938971 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08b4bad4-9210-4ce8-8bb9-57ef56df74fc" (UID: "08b4bad4-9210-4ce8-8bb9-57ef56df74fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:27:58.940797 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.940772 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-kube-api-access-7lj48" (OuterVolumeSpecName: "kube-api-access-7lj48") pod "08b4bad4-9210-4ce8-8bb9-57ef56df74fc" (UID: "08b4bad4-9210-4ce8-8bb9-57ef56df74fc"). InnerVolumeSpecName "kube-api-access-7lj48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:27:58.940899 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:58.940855 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08b4bad4-9210-4ce8-8bb9-57ef56df74fc" (UID: "08b4bad4-9210-4ce8-8bb9-57ef56df74fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:27:59.039602 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.039565 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:27:59.039602 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.039595 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lj48\" (UniqueName: \"kubernetes.io/projected/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-kube-api-access-7lj48\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:27:59.039602 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.039607 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4bad4-9210-4ce8-8bb9-57ef56df74fc-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:27:59.393424 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.393386 2568 generic.go:356] "Generic (PLEG): container finished" podID="08b4bad4-9210-4ce8-8bb9-57ef56df74fc" containerID="793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485" exitCode=0 Jul 07 00:27:59.393424 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.393442 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" Jul 07 00:27:59.393678 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.393471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" event={"ID":"08b4bad4-9210-4ce8-8bb9-57ef56df74fc","Type":"ContainerDied","Data":"793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485"} Jul 07 00:27:59.393678 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.393526 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc" event={"ID":"08b4bad4-9210-4ce8-8bb9-57ef56df74fc","Type":"ContainerDied","Data":"0031cbbf74bb0885f86a686ef7fb7e9b4dd5c92cb05162d4e0017fd5186836e7"} Jul 07 00:27:59.393678 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.393551 2568 scope.go:117] "RemoveContainer" containerID="793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485" Jul 07 00:27:59.402155 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.402130 2568 scope.go:117] "RemoveContainer" containerID="793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485" Jul 07 00:27:59.402473 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:27:59.402453 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485\": container with ID starting with 793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485 not found: ID does not exist" containerID="793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485" Jul 07 00:27:59.402545 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.402485 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485"} err="failed to get container status \"793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485\": rpc error: code = NotFound desc = could not find container \"793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485\": container with ID starting with 793df2fa7e0bf60df87bcb3545ad13e8a97de0c760adeca1408c2e4f13624485 not found: ID does not exist" Jul 07 00:27:59.417307 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.417267 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc"] Jul 07 00:27:59.422554 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:27:59.422529 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-0ec7c3-7dbd66d6d7-7pkcc"] Jul 07 00:28:01.315486 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:01.315454 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b4bad4-9210-4ce8-8bb9-57ef56df74fc" path="/var/lib/kubelet/pods/08b4bad4-9210-4ce8-8bb9-57ef56df74fc/volumes" Jul 07 00:28:02.672019 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.671995 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:28:02.778155 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778049 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-data\") pod \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " Jul 07 00:28:02.778155 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778131 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-socket\") pod \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " Jul 07 00:28:02.778389 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778241 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-podinfo\") pod \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " Jul 07 00:28:02.778389 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778283 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slscn\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-kube-api-access-slscn\") pod \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " Jul 07 00:28:02.778389 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778316 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-envoy\") pod \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " Jul 07 00:28:02.778389 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778361 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-credential-socket\") pod \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " Jul 07 00:28:02.778603 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778392 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "55b8cd6d-a811-49de-9a32-49c3a216bcdf" (UID: "55b8cd6d-a811-49de-9a32-49c3a216bcdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:02.778603 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778473 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istiod-ca-cert\") pod \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " Jul 07 00:28:02.778603 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778542 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-token\") pod \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " Jul 07 00:28:02.778603 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778575 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-certs\") pod \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\" (UID: \"55b8cd6d-a811-49de-9a32-49c3a216bcdf\") " Jul 07 00:28:02.778813 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778651 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "55b8cd6d-a811-49de-9a32-49c3a216bcdf" (UID: "55b8cd6d-a811-49de-9a32-49c3a216bcdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:02.778945 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778897 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "55b8cd6d-a811-49de-9a32-49c3a216bcdf" (UID: "55b8cd6d-a811-49de-9a32-49c3a216bcdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:28:02.778945 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778921 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:02.778945 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.778942 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-credential-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:02.779212 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.779073 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "55b8cd6d-a811-49de-9a32-49c3a216bcdf" (UID: "55b8cd6d-a811-49de-9a32-49c3a216bcdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:02.779269 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.779245 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-data" (OuterVolumeSpecName: "istio-data") pod "55b8cd6d-a811-49de-9a32-49c3a216bcdf" (UID: "55b8cd6d-a811-49de-9a32-49c3a216bcdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:02.780717 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.780695 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "55b8cd6d-a811-49de-9a32-49c3a216bcdf" (UID: "55b8cd6d-a811-49de-9a32-49c3a216bcdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:28:02.780814 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.780795 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "55b8cd6d-a811-49de-9a32-49c3a216bcdf" (UID: "55b8cd6d-a811-49de-9a32-49c3a216bcdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:02.781218 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.781193 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-token" (OuterVolumeSpecName: "istio-token") pod "55b8cd6d-a811-49de-9a32-49c3a216bcdf" (UID: "55b8cd6d-a811-49de-9a32-49c3a216bcdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:28:02.781218 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.781193 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-kube-api-access-slscn" (OuterVolumeSpecName: "kube-api-access-slscn") pod "55b8cd6d-a811-49de-9a32-49c3a216bcdf" (UID: "55b8cd6d-a811-49de-9a32-49c3a216bcdf"). InnerVolumeSpecName "kube-api-access-slscn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:28:02.879973 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.879922 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-data\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:02.879973 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.879967 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-podinfo\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:02.879973 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.879976 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slscn\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-kube-api-access-slscn\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:02.879973 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.879986 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-envoy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:02.879973 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.879994 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istiod-ca-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:02.880290 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.880002 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b8cd6d-a811-49de-9a32-49c3a216bcdf-istio-token\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:02.880290 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:02.880009 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b8cd6d-a811-49de-9a32-49c3a216bcdf-workload-certs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:03.414489 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.414422 2568 generic.go:356] "Generic (PLEG): container finished" podID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerID="0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d" exitCode=0 Jul 07 00:28:03.414489 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.414471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" event={"ID":"55b8cd6d-a811-49de-9a32-49c3a216bcdf","Type":"ContainerDied","Data":"0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d"} Jul 07 00:28:03.414739 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.414520 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" Jul 07 00:28:03.414739 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.414530 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd" event={"ID":"55b8cd6d-a811-49de-9a32-49c3a216bcdf","Type":"ContainerDied","Data":"710a7bf8bd9a6e8e7c39f2b1171c66921deb42496f498bf109de887e12b5b538"} Jul 07 00:28:03.414739 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.414553 2568 scope.go:117] "RemoveContainer" containerID="0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d" Jul 07 00:28:03.423396 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.423377 2568 scope.go:117] "RemoveContainer" containerID="986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd" Jul 07 00:28:03.431668 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.431649 2568 scope.go:117] "RemoveContainer" containerID="0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d" Jul 07 00:28:03.431951 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:03.431931 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d\": container with ID starting with 0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d not found: ID does not exist" containerID="0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d" Jul 07 00:28:03.432010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.431961 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d"} err="failed to get container status \"0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d\": rpc error: code = NotFound desc = could not find container \"0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d\": container with ID starting with 0168c22ea966f4fe7f4e3d2899931136dcec86fb0d8de18bde8931a3efc8389d not found: ID does not exist" Jul 07 00:28:03.432010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.431980 2568 scope.go:117] "RemoveContainer" containerID="986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd" Jul 07 00:28:03.432222 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:03.432206 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd\": container with ID starting with 986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd not found: ID does not exist" containerID="986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd" Jul 07 00:28:03.432267 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.432229 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd"} err="failed to get container status \"986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd\": rpc error: code = NotFound desc = could not find container \"986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd\": container with ID starting with 986585ec7a81cd790761a7b628c57f9648b7c4201bc9cee62e27ded7c4b531dd not found: ID does not exist" Jul 07 00:28:03.432471 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.432453 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd"] Jul 07 00:28:03.438552 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:03.438529 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cded59-openshift-default-6d8d98dc49-kbvzd"] Jul 07 00:28:05.165340 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165311 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7"] Jul 07 00:28:05.165731 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165681 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" Jul 07 00:28:05.165731 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165693 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" Jul 07 00:28:05.165731 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165705 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b4bad4-9210-4ce8-8bb9-57ef56df74fc" containerName="maas-api" Jul 07 00:28:05.165731 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165711 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b4bad4-9210-4ce8-8bb9-57ef56df74fc" containerName="maas-api" Jul 07 00:28:05.165731 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165729 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3fedd27-d78e-484d-8739-604a6b24fb87" containerName="maas-api" Jul 07 00:28:05.165731 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165734 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3fedd27-d78e-484d-8739-604a6b24fb87" containerName="maas-api" Jul 07 00:28:05.165922 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165792 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3fedd27-d78e-484d-8739-604a6b24fb87" containerName="maas-api" Jul 07 00:28:05.165922 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165801 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" Jul 07 00:28:05.165922 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.165808 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b4bad4-9210-4ce8-8bb9-57ef56df74fc" containerName="maas-api" Jul 07 00:28:05.170760 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.170735 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.173369 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.173345 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e94a26-openshift-default-dockercfg-tll27\"" Jul 07 00:28:05.182172 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.182146 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7"] Jul 07 00:28:05.203562 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.203524 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-token\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.203562 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.203563 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-certs\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.203757 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.203594 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-data\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.203757 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.203677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-credential-socket\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.203757 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.203721 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istiod-ca-cert\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.203856 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.203787 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-envoy\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.203856 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.203831 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkgl9\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-kube-api-access-zkgl9\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.203920 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.203854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-socket\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.203962 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.203934 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-podinfo\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305282 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305238 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-envoy\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305282 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkgl9\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-kube-api-access-zkgl9\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305603 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305420 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-socket\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305603 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-podinfo\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305720 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305627 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-token\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305720 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-certs\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305720 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305689 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-data\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305884 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305736 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-credential-socket\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305884 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istiod-ca-cert\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.305884 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.305791 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-socket\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.306044 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.306020 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-data\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.306139 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.306117 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-credential-socket\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.306298 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.306277 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-certs\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.306771 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.306747 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istiod-ca-cert\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.307762 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.307743 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-envoy\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.308381 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.308358 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-podinfo\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.314247 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.314219 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-token\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.314355 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.314334 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkgl9\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-kube-api-access-zkgl9\") pod \"e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.318082 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.318053 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" path="/var/lib/kubelet/pods/55b8cd6d-a811-49de-9a32-49c3a216bcdf/volumes" Jul 07 00:28:05.481985 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.481887 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:05.633860 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.633827 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7"] Jul 07 00:28:05.635499 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:28:05.635468 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87824b6e_f7dd_4ebf_98a1_af9466b7936a.slice/crio-07a3328ba21c2ce92159542b11ec809f9cb57f1d691d0debde1ca0a93d7f9e53 WatchSource:0}: Error finding container 07a3328ba21c2ce92159542b11ec809f9cb57f1d691d0debde1ca0a93d7f9e53: Status 404 returned error can't find the container with id 07a3328ba21c2ce92159542b11ec809f9cb57f1d691d0debde1ca0a93d7f9e53 Jul 07 00:28:05.637795 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.637765 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:28:05.637867 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.637832 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:28:05.637867 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:05.637863 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:28:06.431419 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:06.431382 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" event={"ID":"87824b6e-f7dd-4ebf-98a1-af9466b7936a","Type":"ContainerStarted","Data":"e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7"} Jul 07 00:28:06.431923 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:06.431460 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" event={"ID":"87824b6e-f7dd-4ebf-98a1-af9466b7936a","Type":"ContainerStarted","Data":"07a3328ba21c2ce92159542b11ec809f9cb57f1d691d0debde1ca0a93d7f9e53"} Jul 07 00:28:06.454266 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:06.454207 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" podStartSLOduration=1.454190503 podStartE2EDuration="1.454190503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:28:06.45341052 +0000 UTC m=+1907.666442264" watchObservedRunningTime="2026-07-07 00:28:06.454190503 +0000 UTC m=+1907.667222247" Jul 07 00:28:06.483094 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:06.483054 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:06.484634 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:06.484607 2568 patch_prober.go:28] interesting pod/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 00:28:06.484750 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:06.484656 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 00:28:07.486399 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:07.486370 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:08.424178 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.424135 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7"] Jul 07 00:28:08.424576 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.424560 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" Jul 07 00:28:08.424647 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.424579 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" Jul 07 00:28:08.424647 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.424633 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b8cd6d-a811-49de-9a32-49c3a216bcdf" containerName="istio-proxy" Jul 07 00:28:08.427733 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.427714 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.431055 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.431031 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-e94a26-serving-cert\"" Jul 07 00:28:08.448622 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.448592 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7"] Jul 07 00:28:08.449986 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.449963 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:08.451027 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.451005 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:08.535125 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.535093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df274b1f-db26-4af0-b9be-e550a3261652-maas-api-tls\") pod \"maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.535525 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.535215 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhwd\" (UniqueName: \"kubernetes.io/projected/df274b1f-db26-4af0-b9be-e550a3261652-kube-api-access-qmhwd\") pod \"maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.535525 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.535290 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df274b1f-db26-4af0-b9be-e550a3261652-openshift-service-ca\") pod \"maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.636150 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.636114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df274b1f-db26-4af0-b9be-e550a3261652-maas-api-tls\") pod \"maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.636343 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.636213 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhwd\" (UniqueName: \"kubernetes.io/projected/df274b1f-db26-4af0-b9be-e550a3261652-kube-api-access-qmhwd\") pod \"maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.636343 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.636247 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df274b1f-db26-4af0-b9be-e550a3261652-openshift-service-ca\") pod \"maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.636948 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.636927 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df274b1f-db26-4af0-b9be-e550a3261652-openshift-service-ca\") pod \"maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.638729 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.638709 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df274b1f-db26-4af0-b9be-e550a3261652-maas-api-tls\") pod \"maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.645833 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.645809 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhwd\" (UniqueName: \"kubernetes.io/projected/df274b1f-db26-4af0-b9be-e550a3261652-kube-api-access-qmhwd\") pod \"maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.741691 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.741583 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:08.910227 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:08.910156 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7"] Jul 07 00:28:08.912567 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:28:08.912534 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf274b1f_db26_4af0_b9be_e550a3261652.slice/crio-9b075977da87cf903422cab4f9556d477673557b11d48557c28e65c1ff52476e WatchSource:0}: Error finding container 9b075977da87cf903422cab4f9556d477673557b11d48557c28e65c1ff52476e: Status 404 returned error can't find the container with id 9b075977da87cf903422cab4f9556d477673557b11d48557c28e65c1ff52476e Jul 07 00:28:09.455831 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:09.455795 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" event={"ID":"df274b1f-db26-4af0-b9be-e550a3261652","Type":"ContainerStarted","Data":"535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47"} Jul 07 00:28:09.456038 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:09.455856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" event={"ID":"df274b1f-db26-4af0-b9be-e550a3261652","Type":"ContainerStarted","Data":"9b075977da87cf903422cab4f9556d477673557b11d48557c28e65c1ff52476e"} Jul 07 00:28:09.456038 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:09.455947 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:09.488079 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:09.488027 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" podStartSLOduration=1.086309659 podStartE2EDuration="1.48800857s" totalImagesPullingTime="401.698911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:28:09.486402859 +0000 UTC m=+1910.699434603" watchObservedRunningTime="2026-07-07 00:28:09.48800857 +0000 UTC m=+1910.701040314" Jul 07 00:28:15.465913 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:15.465875 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:16.435570 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.435525 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76"] Jul 07 00:28:16.446455 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.445315 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.448138 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.448112 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-295217-serving-cert\"" Jul 07 00:28:16.453983 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.453949 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76"] Jul 07 00:28:16.526459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.521022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e72241-3c52-47f9-a26c-b23c60eedc4e-maas-api-tls\") pod \"maas-api-e2e-api-b-295217-6d4b6ccc75-77s76\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.526459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.521107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e72241-3c52-47f9-a26c-b23c60eedc4e-openshift-service-ca\") pod \"maas-api-e2e-api-b-295217-6d4b6ccc75-77s76\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.526459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.521217 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9r7w\" (UniqueName: \"kubernetes.io/projected/a1e72241-3c52-47f9-a26c-b23c60eedc4e-kube-api-access-k9r7w\") pod \"maas-api-e2e-api-b-295217-6d4b6ccc75-77s76\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.622353 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.622311 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9r7w\" (UniqueName: \"kubernetes.io/projected/a1e72241-3c52-47f9-a26c-b23c60eedc4e-kube-api-access-k9r7w\") pod \"maas-api-e2e-api-b-295217-6d4b6ccc75-77s76\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.622718 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.622693 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e72241-3c52-47f9-a26c-b23c60eedc4e-maas-api-tls\") pod \"maas-api-e2e-api-b-295217-6d4b6ccc75-77s76\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.622916 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.622892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e72241-3c52-47f9-a26c-b23c60eedc4e-openshift-service-ca\") pod \"maas-api-e2e-api-b-295217-6d4b6ccc75-77s76\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.623939 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.623914 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e72241-3c52-47f9-a26c-b23c60eedc4e-openshift-service-ca\") pod \"maas-api-e2e-api-b-295217-6d4b6ccc75-77s76\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.632080 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.632042 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e72241-3c52-47f9-a26c-b23c60eedc4e-maas-api-tls\") pod \"maas-api-e2e-api-b-295217-6d4b6ccc75-77s76\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.637173 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.637142 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9r7w\" (UniqueName: \"kubernetes.io/projected/a1e72241-3c52-47f9-a26c-b23c60eedc4e-kube-api-access-k9r7w\") pod \"maas-api-e2e-api-b-295217-6d4b6ccc75-77s76\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.760200 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.760103 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:16.940547 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:16.939721 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76"] Jul 07 00:28:16.945345 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:28:16.945309 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e72241_3c52_47f9_a26c_b23c60eedc4e.slice/crio-466a3633a3fc80a1000a0dfad9acf8235fdc0565242f213476cdfff388b61703 WatchSource:0}: Error finding container 466a3633a3fc80a1000a0dfad9acf8235fdc0565242f213476cdfff388b61703: Status 404 returned error can't find the container with id 466a3633a3fc80a1000a0dfad9acf8235fdc0565242f213476cdfff388b61703 Jul 07 00:28:17.499525 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:17.499487 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" event={"ID":"a1e72241-3c52-47f9-a26c-b23c60eedc4e","Type":"ContainerStarted","Data":"4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684"} Jul 07 00:28:17.499718 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:17.499561 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" event={"ID":"a1e72241-3c52-47f9-a26c-b23c60eedc4e","Type":"ContainerStarted","Data":"466a3633a3fc80a1000a0dfad9acf8235fdc0565242f213476cdfff388b61703"} Jul 07 00:28:17.499718 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:17.499591 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:17.520458 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:17.519086 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" podStartSLOduration=1.100298855 podStartE2EDuration="1.519067231s" totalImagesPullingTime="418.768376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:28:17.5188021 +0000 UTC m=+1918.731833845" watchObservedRunningTime="2026-07-07 00:28:17.519067231 +0000 UTC m=+1918.732098977" Jul 07 00:28:23.451878 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:23.451839 2568 patch_prober.go:28] interesting pod/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 00:28:23.452328 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:23.451912 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 00:28:23.509956 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:23.509928 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:23.531442 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:23.531391 2568 generic.go:356] "Generic (PLEG): container finished" podID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerID="e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7" exitCode=0 Jul 07 00:28:23.531619 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:23.531464 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" event={"ID":"87824b6e-f7dd-4ebf-98a1-af9466b7936a","Type":"ContainerDied","Data":"e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7"} Jul 07 00:28:23.531894 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:23.531880 2568 scope.go:117] "RemoveContainer" containerID="e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7" Jul 07 00:28:23.533010 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:23.532971 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:28:23.533102 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:23.533062 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:28:23.533102 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:23.533095 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:28:24.541204 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:24.541159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" event={"ID":"87824b6e-f7dd-4ebf-98a1-af9466b7936a","Type":"ContainerStarted","Data":"c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a"} Jul 07 00:28:25.482476 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:25.482414 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:25.482666 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:25.482588 2568 patch_prober.go:28] interesting pod/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 00:28:25.482666 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:25.482643 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 00:28:26.483367 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:26.483326 2568 patch_prober.go:28] interesting pod/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 00:28:26.483852 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:26.483389 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 00:28:27.482784 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:27.482741 2568 patch_prober.go:28] interesting pod/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 00:28:27.483025 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:27.482803 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 00:28:28.483027 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:28.482985 2568 patch_prober.go:28] interesting pod/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 00:28:28.483636 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:28.483045 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 00:28:29.486961 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:29.486928 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:29.487459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:29.487271 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:29.488039 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:29.488024 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:30.115351 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.115317 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7"] Jul 07 00:28:30.115666 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.115613 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" podUID="df274b1f-db26-4af0-b9be-e550a3261652" containerName="maas-api" containerID="cri-o://535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47" gracePeriod=30 Jul 07 00:28:30.367158 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.367092 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:30.473553 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.473518 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df274b1f-db26-4af0-b9be-e550a3261652-maas-api-tls\") pod \"df274b1f-db26-4af0-b9be-e550a3261652\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " Jul 07 00:28:30.473741 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.473614 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df274b1f-db26-4af0-b9be-e550a3261652-openshift-service-ca\") pod \"df274b1f-db26-4af0-b9be-e550a3261652\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " Jul 07 00:28:30.473741 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.473665 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmhwd\" (UniqueName: \"kubernetes.io/projected/df274b1f-db26-4af0-b9be-e550a3261652-kube-api-access-qmhwd\") pod \"df274b1f-db26-4af0-b9be-e550a3261652\" (UID: \"df274b1f-db26-4af0-b9be-e550a3261652\") " Jul 07 00:28:30.474027 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.473997 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df274b1f-db26-4af0-b9be-e550a3261652-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df274b1f-db26-4af0-b9be-e550a3261652" (UID: "df274b1f-db26-4af0-b9be-e550a3261652"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:28:30.475830 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.475793 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df274b1f-db26-4af0-b9be-e550a3261652-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df274b1f-db26-4af0-b9be-e550a3261652" (UID: "df274b1f-db26-4af0-b9be-e550a3261652"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:28:30.476035 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.476011 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df274b1f-db26-4af0-b9be-e550a3261652-kube-api-access-qmhwd" (OuterVolumeSpecName: "kube-api-access-qmhwd") pod "df274b1f-db26-4af0-b9be-e550a3261652" (UID: "df274b1f-db26-4af0-b9be-e550a3261652"). InnerVolumeSpecName "kube-api-access-qmhwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:28:30.571460 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.571389 2568 generic.go:356] "Generic (PLEG): container finished" podID="df274b1f-db26-4af0-b9be-e550a3261652" containerID="535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47" exitCode=0 Jul 07 00:28:30.571460 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.571464 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" Jul 07 00:28:30.571961 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.571480 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" event={"ID":"df274b1f-db26-4af0-b9be-e550a3261652","Type":"ContainerDied","Data":"535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47"} Jul 07 00:28:30.571961 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.571538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7" event={"ID":"df274b1f-db26-4af0-b9be-e550a3261652","Type":"ContainerDied","Data":"9b075977da87cf903422cab4f9556d477673557b11d48557c28e65c1ff52476e"} Jul 07 00:28:30.571961 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.571560 2568 scope.go:117] "RemoveContainer" containerID="535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47" Jul 07 00:28:30.574491 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.574469 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df274b1f-db26-4af0-b9be-e550a3261652-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:30.574586 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.574496 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df274b1f-db26-4af0-b9be-e550a3261652-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:30.574586 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.574512 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmhwd\" (UniqueName: \"kubernetes.io/projected/df274b1f-db26-4af0-b9be-e550a3261652-kube-api-access-qmhwd\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:30.580575 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.580551 2568 scope.go:117] "RemoveContainer" containerID="535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47" Jul 07 00:28:30.580823 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:30.580804 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47\": container with ID starting with 535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47 not found: ID does not exist" containerID="535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47" Jul 07 00:28:30.580872 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.580834 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47"} err="failed to get container status \"535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47\": rpc error: code = NotFound desc = could not find container \"535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47\": container with ID starting with 535125807d4af8a0dbc1aee26085ffb58b5c8a64c739a3eee021ef33e64c0b47 not found: ID does not exist" Jul 07 00:28:30.595470 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.595415 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7"] Jul 07 00:28:30.598891 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:30.598861 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-e94a26-5c4bff4bff-hzbn7"] Jul 07 00:28:31.315798 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:31.315767 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df274b1f-db26-4af0-b9be-e550a3261652" path="/var/lib/kubelet/pods/df274b1f-db26-4af0-b9be-e550a3261652/volumes" Jul 07 00:28:35.760459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:35.759902 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7"] Jul 07 00:28:35.760459 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:35.760159 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" containerID="cri-o://c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a" gracePeriod=30 Jul 07 00:28:37.159943 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.159909 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76"] Jul 07 00:28:37.160415 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.160162 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" podUID="a1e72241-3c52-47f9-a26c-b23c60eedc4e" containerName="maas-api" containerID="cri-o://4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684" gracePeriod=30 Jul 07 00:28:37.425152 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.425125 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:37.553624 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.553594 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9r7w\" (UniqueName: \"kubernetes.io/projected/a1e72241-3c52-47f9-a26c-b23c60eedc4e-kube-api-access-k9r7w\") pod \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " Jul 07 00:28:37.553718 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.553652 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e72241-3c52-47f9-a26c-b23c60eedc4e-maas-api-tls\") pod \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " Jul 07 00:28:37.553718 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.553681 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e72241-3c52-47f9-a26c-b23c60eedc4e-openshift-service-ca\") pod \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\" (UID: \"a1e72241-3c52-47f9-a26c-b23c60eedc4e\") " Jul 07 00:28:37.554116 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.554091 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e72241-3c52-47f9-a26c-b23c60eedc4e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1e72241-3c52-47f9-a26c-b23c60eedc4e" (UID: "a1e72241-3c52-47f9-a26c-b23c60eedc4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:28:37.555911 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.555879 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e72241-3c52-47f9-a26c-b23c60eedc4e-kube-api-access-k9r7w" (OuterVolumeSpecName: "kube-api-access-k9r7w") pod "a1e72241-3c52-47f9-a26c-b23c60eedc4e" (UID: "a1e72241-3c52-47f9-a26c-b23c60eedc4e"). InnerVolumeSpecName "kube-api-access-k9r7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:28:37.555911 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.555890 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e72241-3c52-47f9-a26c-b23c60eedc4e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1e72241-3c52-47f9-a26c-b23c60eedc4e" (UID: "a1e72241-3c52-47f9-a26c-b23c60eedc4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:28:37.606774 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.606738 2568 generic.go:356] "Generic (PLEG): container finished" podID="a1e72241-3c52-47f9-a26c-b23c60eedc4e" containerID="4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684" exitCode=0 Jul 07 00:28:37.606939 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.606790 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" Jul 07 00:28:37.606939 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.606807 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" event={"ID":"a1e72241-3c52-47f9-a26c-b23c60eedc4e","Type":"ContainerDied","Data":"4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684"} Jul 07 00:28:37.606939 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.606883 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76" event={"ID":"a1e72241-3c52-47f9-a26c-b23c60eedc4e","Type":"ContainerDied","Data":"466a3633a3fc80a1000a0dfad9acf8235fdc0565242f213476cdfff388b61703"} Jul 07 00:28:37.606939 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.606909 2568 scope.go:117] "RemoveContainer" containerID="4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684" Jul 07 00:28:37.616355 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.616336 2568 scope.go:117] "RemoveContainer" containerID="4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684" Jul 07 00:28:37.616621 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:37.616601 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684\": container with ID starting with 4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684 not found: ID does not exist" containerID="4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684" Jul 07 00:28:37.616692 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.616629 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684"} err="failed to get container status \"4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684\": rpc error: code = NotFound desc = could not find container \"4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684\": container with ID starting with 4142d8765370e42e32d25d9323b269b8aa82439c4e3b2f006ab2e9be16572684 not found: ID does not exist" Jul 07 00:28:37.639755 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.639723 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76"] Jul 07 00:28:37.647347 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.647325 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-295217-6d4b6ccc75-77s76"] Jul 07 00:28:37.654689 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.654658 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9r7w\" (UniqueName: \"kubernetes.io/projected/a1e72241-3c52-47f9-a26c-b23c60eedc4e-kube-api-access-k9r7w\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:37.654689 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.654683 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e72241-3c52-47f9-a26c-b23c60eedc4e-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:37.654689 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:37.654695 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e72241-3c52-47f9-a26c-b23c60eedc4e-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:39.316252 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:39.316218 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e72241-3c52-47f9-a26c-b23c60eedc4e" path="/var/lib/kubelet/pods/a1e72241-3c52-47f9-a26c-b23c60eedc4e/volumes" Jul 07 00:28:41.033085 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.033021 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:41.089455 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089403 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-credential-socket\") pod \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " Jul 07 00:28:41.089455 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089449 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-socket\") pod \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " Jul 07 00:28:41.089687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089493 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istiod-ca-cert\") pod \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " Jul 07 00:28:41.089687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089511 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-envoy\") pod \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " Jul 07 00:28:41.089687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089540 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-certs\") pod \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " Jul 07 00:28:41.089687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089563 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-token\") pod \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " Jul 07 00:28:41.089687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089604 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkgl9\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-kube-api-access-zkgl9\") pod \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " Jul 07 00:28:41.089687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089676 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-data\") pod \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " Jul 07 00:28:41.090000 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089694 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-podinfo\") pod \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\" (UID: \"87824b6e-f7dd-4ebf-98a1-af9466b7936a\") " Jul 07 00:28:41.090000 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089765 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87824b6e-f7dd-4ebf-98a1-af9466b7936a" (UID: "87824b6e-f7dd-4ebf-98a1-af9466b7936a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:41.090000 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089875 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87824b6e-f7dd-4ebf-98a1-af9466b7936a" (UID: "87824b6e-f7dd-4ebf-98a1-af9466b7936a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:41.090000 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.089949 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87824b6e-f7dd-4ebf-98a1-af9466b7936a" (UID: "87824b6e-f7dd-4ebf-98a1-af9466b7936a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:41.090208 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.090142 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-credential-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:41.090208 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.090164 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:41.090208 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.090178 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-workload-certs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:41.090360 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.090303 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87824b6e-f7dd-4ebf-98a1-af9466b7936a" (UID: "87824b6e-f7dd-4ebf-98a1-af9466b7936a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:28:41.090779 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.090754 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-data" (OuterVolumeSpecName: "istio-data") pod "87824b6e-f7dd-4ebf-98a1-af9466b7936a" (UID: "87824b6e-f7dd-4ebf-98a1-af9466b7936a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:41.092698 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.092665 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87824b6e-f7dd-4ebf-98a1-af9466b7936a" (UID: "87824b6e-f7dd-4ebf-98a1-af9466b7936a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:41.092698 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.092672 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-token" (OuterVolumeSpecName: "istio-token") pod "87824b6e-f7dd-4ebf-98a1-af9466b7936a" (UID: "87824b6e-f7dd-4ebf-98a1-af9466b7936a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:28:41.092835 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.092713 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87824b6e-f7dd-4ebf-98a1-af9466b7936a" (UID: "87824b6e-f7dd-4ebf-98a1-af9466b7936a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:28:41.092835 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.092723 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-kube-api-access-zkgl9" (OuterVolumeSpecName: "kube-api-access-zkgl9") pod "87824b6e-f7dd-4ebf-98a1-af9466b7936a" (UID: "87824b6e-f7dd-4ebf-98a1-af9466b7936a"). InnerVolumeSpecName "kube-api-access-zkgl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:28:41.191360 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.191321 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istiod-ca-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:41.191360 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.191355 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-envoy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:41.191360 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.191364 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-token\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:41.191360 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.191372 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkgl9\" (UniqueName: \"kubernetes.io/projected/87824b6e-f7dd-4ebf-98a1-af9466b7936a-kube-api-access-zkgl9\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:41.191661 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.191381 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-data\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:41.191661 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.191391 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87824b6e-f7dd-4ebf-98a1-af9466b7936a-istio-podinfo\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:28:41.629795 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.629688 2568 generic.go:356] "Generic (PLEG): container finished" podID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerID="c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a" exitCode=0 Jul 07 00:28:41.629795 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.629740 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" Jul 07 00:28:41.629795 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.629770 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" event={"ID":"87824b6e-f7dd-4ebf-98a1-af9466b7936a","Type":"ContainerDied","Data":"c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a"} Jul 07 00:28:41.630068 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.629829 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7" event={"ID":"87824b6e-f7dd-4ebf-98a1-af9466b7936a","Type":"ContainerDied","Data":"07a3328ba21c2ce92159542b11ec809f9cb57f1d691d0debde1ca0a93d7f9e53"} Jul 07 00:28:41.630068 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.629853 2568 scope.go:117] "RemoveContainer" containerID="c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a" Jul 07 00:28:41.638614 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.638588 2568 scope.go:117] "RemoveContainer" containerID="e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7" Jul 07 00:28:41.647393 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.647371 2568 scope.go:117] "RemoveContainer" containerID="c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a" Jul 07 00:28:41.647705 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:41.647684 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a\": container with ID starting with c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a not found: ID does not exist" containerID="c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a" Jul 07 00:28:41.647769 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.647714 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a"} err="failed to get container status \"c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a\": rpc error: code = NotFound desc = could not find container \"c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a\": container with ID starting with c2ecd46602d28eb0782bcabb93a6fb0d166300809b89ef100ec3fbc9d8bbbb7a not found: ID does not exist" Jul 07 00:28:41.647769 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.647732 2568 scope.go:117] "RemoveContainer" containerID="e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7" Jul 07 00:28:41.648019 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:41.647997 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7\": container with ID starting with e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7 not found: ID does not exist" containerID="e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7" Jul 07 00:28:41.648065 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.648027 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7"} err="failed to get container status \"e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7\": rpc error: code = NotFound desc = could not find container \"e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7\": container with ID starting with e678c21bb179f07696d157e25b4fec71f36893106cf9918c9954bbea07c320f7 not found: ID does not exist" Jul 07 00:28:41.649110 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.649091 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7"] Jul 07 00:28:41.653911 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:41.653886 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e94a26-openshift-default-5f64b76bf7-c2ld7"] Jul 07 00:28:43.321875 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.321840 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" path="/var/lib/kubelet/pods/87824b6e-f7dd-4ebf-98a1-af9466b7936a/volumes" Jul 07 00:28:43.526968 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.526935 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg"] Jul 07 00:28:43.527362 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527343 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" Jul 07 00:28:43.527471 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527365 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" Jul 07 00:28:43.527471 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527394 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e72241-3c52-47f9-a26c-b23c60eedc4e" containerName="maas-api" Jul 07 00:28:43.527471 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527404 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e72241-3c52-47f9-a26c-b23c60eedc4e" containerName="maas-api" Jul 07 00:28:43.527471 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527446 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df274b1f-db26-4af0-b9be-e550a3261652" containerName="maas-api" Jul 07 00:28:43.527471 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527456 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="df274b1f-db26-4af0-b9be-e550a3261652" containerName="maas-api" Jul 07 00:28:43.527730 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527547 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="df274b1f-db26-4af0-b9be-e550a3261652" containerName="maas-api" Jul 07 00:28:43.527730 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527561 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e72241-3c52-47f9-a26c-b23c60eedc4e" containerName="maas-api" Jul 07 00:28:43.527730 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527572 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" Jul 07 00:28:43.527730 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.527582 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" Jul 07 00:28:43.530863 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.530840 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.533479 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.533455 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-8714ac-openshift-default-dockercfg-zxfsp\"" Jul 07 00:28:43.544315 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.544287 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg"] Jul 07 00:28:43.615468 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.615349 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-certs\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.615468 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.615400 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-token\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.615687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.615526 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istiod-ca-cert\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.615687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.615566 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-podinfo\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.615687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.615591 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t54wz\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-kube-api-access-t54wz\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.615687 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.615667 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-envoy\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.615870 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.615713 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-data\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.615870 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.615752 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-socket\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.615870 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.615779 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-credential-socket\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.716932 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.716886 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-envoy\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717128 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.716943 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-data\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717128 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.716974 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-socket\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717128 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717001 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-credential-socket\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717128 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-certs\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717128 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717086 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-token\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717387 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istiod-ca-cert\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717387 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-podinfo\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717387 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t54wz\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-kube-api-access-t54wz\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717580 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717401 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-data\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717580 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717498 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-credential-socket\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717691 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717598 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-socket\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717814 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717789 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-certs\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.717915 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.717896 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istiod-ca-cert\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.719399 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.719380 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-envoy\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.719811 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.719786 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-podinfo\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.725652 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.725628 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-token\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.725783 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.725761 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t54wz\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-kube-api-access-t54wz\") pod \"e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.841663 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.841616 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:43.987065 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.987033 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg"] Jul 07 00:28:43.989628 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:28:43.989601 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9ae760_4c2f_4f3e_8c45_a57c5476bd1e.slice/crio-d9d8413e9004d3f35a6f322871e1a49a17a166ddefce688e81f75f1b089bf6c8 WatchSource:0}: Error finding container d9d8413e9004d3f35a6f322871e1a49a17a166ddefce688e81f75f1b089bf6c8: Status 404 returned error can't find the container with id d9d8413e9004d3f35a6f322871e1a49a17a166ddefce688e81f75f1b089bf6c8 Jul 07 00:28:43.992081 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.992048 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:28:43.992178 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.992127 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:28:43.992243 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:43.992198 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:28:44.647922 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:44.647886 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" event={"ID":"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e","Type":"ContainerStarted","Data":"4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8"} Jul 07 00:28:44.648324 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:44.647942 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" event={"ID":"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e","Type":"ContainerStarted","Data":"d9d8413e9004d3f35a6f322871e1a49a17a166ddefce688e81f75f1b089bf6c8"} Jul 07 00:28:44.683872 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:44.683813 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" podStartSLOduration=1.683797003 podStartE2EDuration="1.683797003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:28:44.677538376 +0000 UTC m=+1945.890570121" watchObservedRunningTime="2026-07-07 00:28:44.683797003 +0000 UTC m=+1945.896828746" Jul 07 00:28:44.842452 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:44.842391 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:45.846966 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:45.846933 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:46.658261 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.658221 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:46.659558 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.659526 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:28:46.835298 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.835263 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v"] Jul 07 00:28:46.835680 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.835665 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" Jul 07 00:28:46.835738 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.835685 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="87824b6e-f7dd-4ebf-98a1-af9466b7936a" containerName="istio-proxy" Jul 07 00:28:46.837892 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.837872 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:46.841107 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.841084 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-8714ac-serving-cert\"" Jul 07 00:28:46.871870 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.871833 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v"] Jul 07 00:28:46.952324 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.952246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-maas-api-tls\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:46.952641 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.952407 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5s66\" (UniqueName: \"kubernetes.io/projected/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-kube-api-access-x5s66\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:46.952641 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:46.952464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:47.053560 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:47.053526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-maas-api-tls\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:47.053741 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:47.053624 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5s66\" (UniqueName: \"kubernetes.io/projected/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-kube-api-access-x5s66\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:47.053741 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:47.053660 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:47.053741 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:47.053686 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-8714ac-serving-cert: secret "maas-api-e2e-shared-a-8714ac-serving-cert" not found Jul 07 00:28:47.053847 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:47.053749 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-maas-api-tls podName:c4fdecc1-f55e-4af5-8e01-acc472e2ca3d nodeName:}" failed. No retries permitted until 2026-07-07 00:28:47.553733538 +0000 UTC m=+1948.766765260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-maas-api-tls") pod "maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" (UID: "c4fdecc1-f55e-4af5-8e01-acc472e2ca3d") : secret "maas-api-e2e-shared-a-8714ac-serving-cert" not found Jul 07 00:28:47.054382 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:47.054364 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:47.066784 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:47.066761 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5s66\" (UniqueName: \"kubernetes.io/projected/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-kube-api-access-x5s66\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:47.558586 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:47.558544 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-maas-api-tls\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:47.561283 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:47.561248 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-maas-api-tls\") pod \"maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:47.749059 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:47.749017 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:47.882914 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:47.882882 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v"] Jul 07 00:28:47.884346 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:28:47.884322 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fdecc1_f55e_4af5_8e01_acc472e2ca3d.slice/crio-18ac7049d6e9515262e384b51b51adf67c6d8440084f38c384edce4cae9b5cd8 WatchSource:0}: Error finding container 18ac7049d6e9515262e384b51b51adf67c6d8440084f38c384edce4cae9b5cd8: Status 404 returned error can't find the container with id 18ac7049d6e9515262e384b51b51adf67c6d8440084f38c384edce4cae9b5cd8 Jul 07 00:28:48.670991 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:48.670956 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" event={"ID":"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d","Type":"ContainerStarted","Data":"c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405"} Jul 07 00:28:48.671169 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:48.671008 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" event={"ID":"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d","Type":"ContainerStarted","Data":"18ac7049d6e9515262e384b51b51adf67c6d8440084f38c384edce4cae9b5cd8"} Jul 07 00:28:48.689051 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:48.689004 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" podStartSLOduration=2.281440212 podStartE2EDuration="2.688980058s" totalImagesPullingTime="407.539846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:28:48.687864401 +0000 UTC m=+1949.900896147" watchObservedRunningTime="2026-07-07 00:28:48.688980058 +0000 UTC m=+1949.902011830" Jul 07 00:28:49.676089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:49.676058 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:54.609689 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.609646 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5"] Jul 07 00:28:54.613665 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.613636 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:54.619457 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.617841 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-4e3cf5-serving-cert\"" Jul 07 00:28:54.632460 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.627354 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5"] Jul 07 00:28:54.737057 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.737018 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7xh\" (UniqueName: \"kubernetes.io/projected/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-kube-api-access-st7xh\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:54.737723 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.737696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:54.737934 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.737912 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-maas-api-tls\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:54.839078 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.839033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7xh\" (UniqueName: \"kubernetes.io/projected/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-kube-api-access-st7xh\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:54.839395 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.839369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:54.839626 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.839591 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-maas-api-tls\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:54.839726 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:54.839701 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-4e3cf5-serving-cert: secret "maas-api-e2e-shared-b-4e3cf5-serving-cert" not found Jul 07 00:28:54.839802 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:28:54.839789 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-maas-api-tls podName:b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415 nodeName:}" failed. No retries permitted until 2026-07-07 00:28:55.339768042 +0000 UTC m=+1956.552799766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-maas-api-tls") pod "maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" (UID: "b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415") : secret "maas-api-e2e-shared-b-4e3cf5-serving-cert" not found Jul 07 00:28:54.840332 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.840306 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:54.860098 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:54.860031 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7xh\" (UniqueName: \"kubernetes.io/projected/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-kube-api-access-st7xh\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:55.345180 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:55.345138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-maas-api-tls\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:55.347852 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:55.347821 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-maas-api-tls\") pod \"maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:55.537281 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:55.537237 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:28:55.688303 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:55.688274 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:28:55.705503 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:55.705365 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5"] Jul 07 00:28:55.708397 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:28:55.708342 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bdc1fb_a1f0_4a67_b7aa_5a97824f1415.slice/crio-c2eb8275cf5b0f4644ba5dd26a6543db0b6b64ae3a7b53d005db421c0534789e WatchSource:0}: Error finding container c2eb8275cf5b0f4644ba5dd26a6543db0b6b64ae3a7b53d005db421c0534789e: Status 404 returned error can't find the container with id c2eb8275cf5b0f4644ba5dd26a6543db0b6b64ae3a7b53d005db421c0534789e Jul 07 00:28:56.711549 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:56.711505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" event={"ID":"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415","Type":"ContainerStarted","Data":"04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99"} Jul 07 00:28:56.711951 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:56.711568 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" event={"ID":"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415","Type":"ContainerStarted","Data":"c2eb8275cf5b0f4644ba5dd26a6543db0b6b64ae3a7b53d005db421c0534789e"} Jul 07 00:28:56.711951 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:28:56.711729 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:29:00.733598 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:00.733563 2568 generic.go:356] "Generic (PLEG): container finished" podID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerID="4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8" exitCode=0 Jul 07 00:29:00.734040 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:00.733645 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" event={"ID":"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e","Type":"ContainerDied","Data":"4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8"} Jul 07 00:29:00.734124 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:00.734097 2568 scope.go:117] "RemoveContainer" containerID="4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8" Jul 07 00:29:00.735228 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:00.735201 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:29:00.735301 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:00.735286 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:29:00.735371 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:00.735358 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:29:00.754931 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:00.754881 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" podStartSLOduration=6.40758427 podStartE2EDuration="6.754866246s" totalImagesPullingTime="347.281976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:28:56.730347626 +0000 UTC m=+1957.943379370" watchObservedRunningTime="2026-07-07 00:29:00.754866246 +0000 UTC m=+1961.967897990" Jul 07 00:29:01.658221 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:01.658162 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:29:01.743044 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:01.743004 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" event={"ID":"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e","Type":"ContainerStarted","Data":"29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef"} Jul 07 00:29:01.842161 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:01.842114 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:29:01.843224 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:01.843193 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 07 00:29:01.843378 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:01.843243 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 07 00:29:02.734738 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:02.734696 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:29:02.842323 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:02.842281 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 07 00:29:02.842777 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:02.842334 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 07 00:29:03.842842 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:03.842795 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 07 00:29:03.843242 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:03.842866 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 07 00:29:04.842289 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:04.842246 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 07 00:29:04.842504 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:04.842311 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 07 00:29:05.846532 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:05.846491 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:29:05.846972 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:05.846777 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:29:05.847423 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:29:05.847405 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:31:19.338969 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:31:19.338897 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:31:19.371831 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:31:19.371804 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:34:52.340117 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:52.340068 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v"] Jul 07 00:34:52.342685 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:52.340288 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" podUID="c4fdecc1-f55e-4af5-8e01-acc472e2ca3d" containerName="maas-api" containerID="cri-o://c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405" gracePeriod=30 Jul 07 00:34:52.782557 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:52.782530 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:34:52.948788 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:52.948754 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5s66\" (UniqueName: \"kubernetes.io/projected/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-kube-api-access-x5s66\") pod \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " Jul 07 00:34:52.948996 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:52.948826 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-maas-api-tls\") pod \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " Jul 07 00:34:52.948996 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:52.948939 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-openshift-service-ca\") pod \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\" (UID: \"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d\") " Jul 07 00:34:52.949309 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:52.949275 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4fdecc1-f55e-4af5-8e01-acc472e2ca3d" (UID: "c4fdecc1-f55e-4af5-8e01-acc472e2ca3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:34:52.951065 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:52.951031 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4fdecc1-f55e-4af5-8e01-acc472e2ca3d" (UID: "c4fdecc1-f55e-4af5-8e01-acc472e2ca3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:34:52.951169 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:52.951094 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-kube-api-access-x5s66" (OuterVolumeSpecName: "kube-api-access-x5s66") pod "c4fdecc1-f55e-4af5-8e01-acc472e2ca3d" (UID: "c4fdecc1-f55e-4af5-8e01-acc472e2ca3d"). InnerVolumeSpecName "kube-api-access-x5s66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:34:53.049927 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.049892 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:34:53.049927 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.049921 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5s66\" (UniqueName: \"kubernetes.io/projected/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-kube-api-access-x5s66\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:34:53.049927 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.049931 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:34:53.441353 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.441315 2568 generic.go:356] "Generic (PLEG): container finished" podID="c4fdecc1-f55e-4af5-8e01-acc472e2ca3d" containerID="c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405" exitCode=0 Jul 07 00:34:53.441353 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.441357 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" Jul 07 00:34:53.441889 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.441403 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" event={"ID":"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d","Type":"ContainerDied","Data":"c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405"} Jul 07 00:34:53.441889 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.441475 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v" event={"ID":"c4fdecc1-f55e-4af5-8e01-acc472e2ca3d","Type":"ContainerDied","Data":"18ac7049d6e9515262e384b51b51adf67c6d8440084f38c384edce4cae9b5cd8"} Jul 07 00:34:53.441889 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.441492 2568 scope.go:117] "RemoveContainer" containerID="c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405" Jul 07 00:34:53.450841 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.450823 2568 scope.go:117] "RemoveContainer" containerID="c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405" Jul 07 00:34:53.451098 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:34:53.451078 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405\": container with ID starting with c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405 not found: ID does not exist" containerID="c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405" Jul 07 00:34:53.451149 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.451106 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405"} err="failed to get container status \"c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405\": rpc error: code = NotFound desc = could not find container \"c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405\": container with ID starting with c8d3e8ebcc919f3ebb0dd1ee9efacfcbe1f2eb8ef8772afec5d437ba82beb405 not found: ID does not exist" Jul 07 00:34:53.458035 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.458010 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v"] Jul 07 00:34:53.464126 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:53.464099 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8714ac-5b5b8d4c75-k5b2v"] Jul 07 00:34:55.324120 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:34:55.324083 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4fdecc1-f55e-4af5-8e01-acc472e2ca3d" path="/var/lib/kubelet/pods/c4fdecc1-f55e-4af5-8e01-acc472e2ca3d/volumes" Jul 07 00:35:34.204076 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:34.204041 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg"] Jul 07 00:35:34.204690 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:34.204267 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" containerID="cri-o://29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef" gracePeriod=30 Jul 07 00:35:35.327769 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.327734 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5"] Jul 07 00:35:35.328226 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.327944 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" podUID="b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415" containerName="maas-api" containerID="cri-o://04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99" gracePeriod=30 Jul 07 00:35:35.584159 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.584091 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:35:35.650500 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.650464 2568 generic.go:356] "Generic (PLEG): container finished" podID="b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415" containerID="04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99" exitCode=0 Jul 07 00:35:35.650500 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.650500 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" Jul 07 00:35:35.650722 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.650518 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" event={"ID":"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415","Type":"ContainerDied","Data":"04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99"} Jul 07 00:35:35.650722 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.650568 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5" event={"ID":"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415","Type":"ContainerDied","Data":"c2eb8275cf5b0f4644ba5dd26a6543db0b6b64ae3a7b53d005db421c0534789e"} Jul 07 00:35:35.650722 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.650592 2568 scope.go:117] "RemoveContainer" containerID="04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99" Jul 07 00:35:35.659342 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.659322 2568 scope.go:117] "RemoveContainer" containerID="04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99" Jul 07 00:35:35.659656 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:35:35.659637 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99\": container with ID starting with 04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99 not found: ID does not exist" containerID="04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99" Jul 07 00:35:35.659714 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.659666 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99"} err="failed to get container status \"04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99\": rpc error: code = NotFound desc = could not find container \"04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99\": container with ID starting with 04d945b81ffe48ac2d76e56344711312207291e0f8dde685e2b416bd088a8f99 not found: ID does not exist" Jul 07 00:35:35.759843 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.759807 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-maas-api-tls\") pod \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " Jul 07 00:35:35.760020 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.759857 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-openshift-service-ca\") pod \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " Jul 07 00:35:35.760020 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.759901 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st7xh\" (UniqueName: \"kubernetes.io/projected/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-kube-api-access-st7xh\") pod \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\" (UID: \"b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415\") " Jul 07 00:35:35.760279 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.760256 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415" (UID: "b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:35:35.762146 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.762120 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415" (UID: "b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:35:35.762146 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.762120 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-kube-api-access-st7xh" (OuterVolumeSpecName: "kube-api-access-st7xh") pod "b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415" (UID: "b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415"). InnerVolumeSpecName "kube-api-access-st7xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:35:35.849813 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.849712 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 00:35:35.849813 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.849777 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 00:35:35.861014 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.860989 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-maas-api-tls\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:35.861014 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.861012 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-openshift-service-ca\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:35.861195 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.861022 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7xh\" (UniqueName: \"kubernetes.io/projected/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415-kube-api-access-st7xh\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:35.972059 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.972027 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5"] Jul 07 00:35:35.977894 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:35.977865 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-4e3cf5-b8766b8c8-7mhw5"] Jul 07 00:35:37.315882 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:37.315834 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415" path="/var/lib/kubelet/pods/b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415/volumes" Jul 07 00:35:39.466106 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.466041 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:35:39.595582 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.595550 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-data\") pod \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " Jul 07 00:35:39.595768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.595630 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-envoy\") pod \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " Jul 07 00:35:39.595768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.595657 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t54wz\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-kube-api-access-t54wz\") pod \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " Jul 07 00:35:39.595768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.595697 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istiod-ca-cert\") pod \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " Jul 07 00:35:39.595768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.595737 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-socket\") pod \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " Jul 07 00:35:39.595768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.595762 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-podinfo\") pod \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " Jul 07 00:35:39.596024 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.595780 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-certs\") pod \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " Jul 07 00:35:39.596024 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.595799 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-credential-socket\") pod \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " Jul 07 00:35:39.596024 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.595843 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-token\") pod \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\" (UID: \"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e\") " Jul 07 00:35:39.596179 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.596065 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" (UID: "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:39.596234 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.596214 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:39.596278 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.596250 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" (UID: "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:35:39.596322 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.596227 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" (UID: "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:39.596322 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.596303 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" (UID: "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:39.596874 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.596844 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" (UID: "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:39.598127 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.598096 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-kube-api-access-t54wz" (OuterVolumeSpecName: "kube-api-access-t54wz") pod "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" (UID: "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e"). InnerVolumeSpecName "kube-api-access-t54wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:35:39.598248 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.598143 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" (UID: "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:39.598248 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.598163 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" (UID: "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:35:39.598328 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.598241 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" (UID: "6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:35:39.673870 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.673839 2568 generic.go:356] "Generic (PLEG): container finished" podID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerID="29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef" exitCode=0 Jul 07 00:35:39.674037 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.673887 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" event={"ID":"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e","Type":"ContainerDied","Data":"29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef"} Jul 07 00:35:39.674037 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.673926 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" event={"ID":"6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e","Type":"ContainerDied","Data":"d9d8413e9004d3f35a6f322871e1a49a17a166ddefce688e81f75f1b089bf6c8"} Jul 07 00:35:39.674037 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.673946 2568 scope.go:117] "RemoveContainer" containerID="29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef" Jul 07 00:35:39.674037 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.673888 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg" Jul 07 00:35:39.685847 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.685827 2568 scope.go:117] "RemoveContainer" containerID="4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8" Jul 07 00:35:39.694033 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.694015 2568 scope.go:117] "RemoveContainer" containerID="29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef" Jul 07 00:35:39.694251 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:35:39.694232 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef\": container with ID starting with 29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef not found: ID does not exist" containerID="29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef" Jul 07 00:35:39.694325 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.694265 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef"} err="failed to get container status \"29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef\": rpc error: code = NotFound desc = could not find container \"29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef\": container with ID starting with 29897e154f5e79dd9cafee862b2b0b38458b0ed9f717283ad7d248bc805fc1ef not found: ID does not exist" Jul 07 00:35:39.694325 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.694290 2568 scope.go:117] "RemoveContainer" containerID="4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8" Jul 07 00:35:39.694535 ip-10-0-141-86 kubenswrapper[2568]: E0707 00:35:39.694519 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8\": container with ID starting with 4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8 not found: ID does not exist" containerID="4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8" Jul 07 00:35:39.694588 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.694540 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8"} err="failed to get container status \"4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8\": rpc error: code = NotFound desc = could not find container \"4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8\": container with ID starting with 4a3135bf7df7e2b6994df2c2d387268535ad331d7dd79a438581bfd80baa6ad8 not found: ID does not exist" Jul 07 00:35:39.697053 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.697033 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istiod-ca-cert\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:39.697131 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.697056 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-podinfo\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:39.697131 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.697069 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-workload-certs\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:39.697131 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.697082 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-credential-socket\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:39.697131 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.697097 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-token\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:39.697131 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.697109 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-data\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:39.697131 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.697120 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-istio-envoy\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:39.697328 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.697133 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t54wz\" (UniqueName: \"kubernetes.io/projected/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e-kube-api-access-t54wz\") on node \"ip-10-0-141-86.ec2.internal\" DevicePath \"\"" Jul 07 00:35:39.709905 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.709880 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg"] Jul 07 00:35:39.719668 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:39.719606 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-8714ac-openshift-default-c95d7c6cf-hh8wg"] Jul 07 00:35:41.315877 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:41.315838 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" path="/var/lib/kubelet/pods/6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e/volumes" Jul 07 00:35:46.727364 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:46.727336 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-674c64d4c7-mhn92_95d3e48c-6d78-4e38-9799-b2205dcd7725/maas-api/0.log" Jul 07 00:35:47.078368 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:47.078283 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8498ff5b75-hqp5j_8859ca0d-3189-4094-b50b-4ea9ed76032a/manager/0.log" Jul 07 00:35:47.434650 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:47.434622 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7bd8fcf-56xfq_6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8/manager/0.log" Jul 07 00:35:47.441260 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:47.441236 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7bd8fcf-56xfq_6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8/copy-manifests/0.log" Jul 07 00:35:50.303499 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:50.303466 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4dpgtk_c3e9c576-87f4-4523-9b35-7d502774328f/istio-proxy/0.log" Jul 07 00:35:50.422556 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:50.422520 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4zb4x_72d39ae9-8292-482d-8c63-6b1f8755f924/discovery/0.log" Jul 07 00:35:50.791962 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:50.791934 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-khdfj_91f2db2c-526b-427e-a103-215869094379/istio-proxy/1.log" Jul 07 00:35:50.913221 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:50.913182 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xrvfm_64985bdd-cb48-4b4a-9b1b-57a17912eef7/payload-pre-processing/0.log" Jul 07 00:35:51.481959 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:51.481933 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-vjc7l_5584c707-83c2-460d-9f22-0df56a72dab1/main/0.log" Jul 07 00:35:51.829854 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:51.829782 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnzpc_b9d91174-900e-47f3-a539-ce9e6e0cfeba/main/0.log" Jul 07 00:35:58.952328 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:58.952294 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v592j_f7dcfee5-cbb6-40da-9c87-6eb081c7d885/global-pull-secret-syncer/0.log" Jul 07 00:35:59.089115 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:59.089085 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z6ql9_b237712a-3bf7-4ae1-9eae-849409680965/konnectivity-agent/0.log" Jul 07 00:35:59.159758 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:35:59.159719 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-86.ec2.internal_f4c4d400fcc6b91a31ccccc5f9a95b03/haproxy/0.log" Jul 07 00:36:06.526561 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.526530 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ffcdd7a8-5bb6-4871-8835-b712ce7c44f2/alertmanager/0.log" Jul 07 00:36:06.548718 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.548693 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ffcdd7a8-5bb6-4871-8835-b712ce7c44f2/config-reloader/0.log" Jul 07 00:36:06.571287 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.571257 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ffcdd7a8-5bb6-4871-8835-b712ce7c44f2/kube-rbac-proxy-web/0.log" Jul 07 00:36:06.595976 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.595951 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ffcdd7a8-5bb6-4871-8835-b712ce7c44f2/kube-rbac-proxy/0.log" Jul 07 00:36:06.616226 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.616193 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ffcdd7a8-5bb6-4871-8835-b712ce7c44f2/kube-rbac-proxy-metric/0.log" Jul 07 00:36:06.637041 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.637006 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ffcdd7a8-5bb6-4871-8835-b712ce7c44f2/prom-label-proxy/0.log" Jul 07 00:36:06.657696 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.657667 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ffcdd7a8-5bb6-4871-8835-b712ce7c44f2/init-config-reloader/0.log" Jul 07 00:36:06.846572 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.846544 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-wjk6g_1fe23294-e2eb-49ff-92fd-ae27f1cbedac/monitoring-plugin/0.log" Jul 07 00:36:06.960774 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.960743 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chnzn_d5b1a4cf-4a57-46da-a6cb-691ec124627b/node-exporter/0.log" Jul 07 00:36:06.980007 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:06.979959 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chnzn_d5b1a4cf-4a57-46da-a6cb-691ec124627b/kube-rbac-proxy/0.log" Jul 07 00:36:07.002346 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.002322 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chnzn_d5b1a4cf-4a57-46da-a6cb-691ec124627b/init-textfile/0.log" Jul 07 00:36:07.119361 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.119274 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-r6ztr_df7fe433-77a1-4f93-b040-32de0cfd88fd/kube-rbac-proxy-main/0.log" Jul 07 00:36:07.145768 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.145742 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-r6ztr_df7fe433-77a1-4f93-b040-32de0cfd88fd/kube-rbac-proxy-self/0.log" Jul 07 00:36:07.169094 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.169062 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-r6ztr_df7fe433-77a1-4f93-b040-32de0cfd88fd/openshift-state-metrics/0.log" Jul 07 00:36:07.203214 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.203172 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c26c073-6f76-4931-b890-e637aa8cdb80/prometheus/0.log" Jul 07 00:36:07.223543 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.223515 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c26c073-6f76-4931-b890-e637aa8cdb80/config-reloader/0.log" Jul 07 00:36:07.248932 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.248902 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c26c073-6f76-4931-b890-e637aa8cdb80/thanos-sidecar/0.log" Jul 07 00:36:07.270125 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.270100 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c26c073-6f76-4931-b890-e637aa8cdb80/kube-rbac-proxy-web/0.log" Jul 07 00:36:07.294809 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.294778 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c26c073-6f76-4931-b890-e637aa8cdb80/kube-rbac-proxy/0.log" Jul 07 00:36:07.316365 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.316337 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c26c073-6f76-4931-b890-e637aa8cdb80/kube-rbac-proxy-thanos/0.log" Jul 07 00:36:07.337582 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.337557 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c26c073-6f76-4931-b890-e637aa8cdb80/init-config-reloader/0.log" Jul 07 00:36:07.363470 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.363423 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-h7pdr_ec257cdb-b4fa-417a-b3fb-c6712391b321/prometheus-operator/0.log" Jul 07 00:36:07.381249 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:07.381175 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-h7pdr_ec257cdb-b4fa-417a-b3fb-c6712391b321/kube-rbac-proxy/0.log" Jul 07 00:36:08.898220 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898177 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r"] Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898553 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898565 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898576 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415" containerName="maas-api" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898583 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415" containerName="maas-api" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898604 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898609 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898618 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4fdecc1-f55e-4af5-8e01-acc472e2ca3d" containerName="maas-api" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898623 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4fdecc1-f55e-4af5-8e01-acc472e2ca3d" containerName="maas-api" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898671 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f9ae760-4c2f-4f3e-8c45-a57c5476bd1e" containerName="istio-proxy" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898679 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4fdecc1-f55e-4af5-8e01-acc472e2ca3d" containerName="maas-api" Jul 07 00:36:08.898754 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.898688 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4bdc1fb-a1f0-4a67-b7aa-5a97824f1415" containerName="maas-api" Jul 07 00:36:08.901744 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.901722 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:08.904392 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.904365 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5sg4w\"/\"kube-root-ca.crt\"" Jul 07 00:36:08.904590 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.904574 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5sg4w\"/\"openshift-service-ca.crt\"" Jul 07 00:36:08.905580 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.905564 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5sg4w\"/\"default-dockercfg-h2ksb\"" Jul 07 00:36:08.912571 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.912548 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r"] Jul 07 00:36:08.971991 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.971949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-lib-modules\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:08.972196 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.972017 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-sys\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:08.972196 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.972067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-podres\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:08.972196 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.972159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrwv\" (UniqueName: \"kubernetes.io/projected/e8863f07-c662-46c1-a9d9-27063487b55c-kube-api-access-xmrwv\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:08.972345 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:08.972214 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-proc\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.072912 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.072867 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-lib-modules\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.073070 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.072937 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-sys\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.073070 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.072955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-podres\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.073070 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.072987 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrwv\" (UniqueName: \"kubernetes.io/projected/e8863f07-c662-46c1-a9d9-27063487b55c-kube-api-access-xmrwv\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.073070 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.073018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-proc\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.073250 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.073066 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-lib-modules\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.073250 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.073095 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-sys\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.073250 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.073135 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-proc\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.073250 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.073136 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8863f07-c662-46c1-a9d9-27063487b55c-podres\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.082252 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.082222 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrwv\" (UniqueName: \"kubernetes.io/projected/e8863f07-c662-46c1-a9d9-27063487b55c-kube-api-access-xmrwv\") pod \"perf-node-gather-daemonset-lxc4r\" (UID: \"e8863f07-c662-46c1-a9d9-27063487b55c\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.211934 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.211899 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.345163 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.345135 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r"] Jul 07 00:36:09.347580 ip-10-0-141-86 kubenswrapper[2568]: W0707 00:36:09.347551 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode8863f07_c662_46c1_a9d9_27063487b55c.slice/crio-7129ccdffae32e531329cf543bfd91c9cef8128f19b6b4fb303128944864eff4 WatchSource:0}: Error finding container 7129ccdffae32e531329cf543bfd91c9cef8128f19b6b4fb303128944864eff4: Status 404 returned error can't find the container with id 7129ccdffae32e531329cf543bfd91c9cef8128f19b6b4fb303128944864eff4 Jul 07 00:36:09.814382 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.814348 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" event={"ID":"e8863f07-c662-46c1-a9d9-27063487b55c","Type":"ContainerStarted","Data":"0a4e1d7451a62276c7a21774d881832f8dfdfe17919a0920058733cd60c3809c"} Jul 07 00:36:09.814628 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.814402 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" event={"ID":"e8863f07-c662-46c1-a9d9-27063487b55c","Type":"ContainerStarted","Data":"7129ccdffae32e531329cf543bfd91c9cef8128f19b6b4fb303128944864eff4"} Jul 07 00:36:09.814628 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.814472 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:09.848879 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.848830 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" podStartSLOduration=1.848814212 podStartE2EDuration="1.848814212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 00:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:36:09.847995666 +0000 UTC m=+2391.061027410" watchObservedRunningTime="2026-07-07 00:36:09.848814212 +0000 UTC m=+2391.061845955" Jul 07 00:36:09.974554 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:09.974519 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56f854bd96-9g642_8dff2386-88af-468c-88c6-304323056720/console/0.log" Jul 07 00:36:11.693551 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:11.693519 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qf9l5_4e1e414a-eea7-41d9-89a6-a7ca5825b7d3/dns/0.log" Jul 07 00:36:11.712591 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:11.712559 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qf9l5_4e1e414a-eea7-41d9-89a6-a7ca5825b7d3/kube-rbac-proxy/0.log" Jul 07 00:36:11.780477 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:11.780428 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f8h8b_bbe077f3-897c-4c32-b14a-98a8d3d04ffa/dns-node-resolver/0.log" Jul 07 00:36:12.403662 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:12.403628 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xl5vt_daaa7108-9765-49af-a9cf-187b26bcba6f/node-ca/0.log" Jul 07 00:36:13.314491 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:13.314458 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4dpgtk_c3e9c576-87f4-4523-9b35-7d502774328f/istio-proxy/0.log" Jul 07 00:36:13.424736 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:13.424700 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4zb4x_72d39ae9-8292-482d-8c63-6b1f8755f924/discovery/0.log" Jul 07 00:36:13.498644 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:13.498609 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-khdfj_91f2db2c-526b-427e-a103-215869094379/istio-proxy/0.log" Jul 07 00:36:13.525523 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:13.525489 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-khdfj_91f2db2c-526b-427e-a103-215869094379/istio-proxy/1.log" Jul 07 00:36:13.556089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:13.556057 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xrvfm_64985bdd-cb48-4b4a-9b1b-57a17912eef7/payload-pre-processing/0.log" Jul 07 00:36:14.317995 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:14.317963 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tv2d_556a6fe3-9d48-45db-b666-d8ca7f5bface/serve-healthcheck-canary/0.log" Jul 07 00:36:14.977623 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:14.977595 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pcz8g_3d0f8f96-cc8e-43f2-9325-bd89a90b61bf/kube-rbac-proxy/0.log" Jul 07 00:36:14.997883 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:14.997856 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pcz8g_3d0f8f96-cc8e-43f2-9325-bd89a90b61bf/exporter/0.log" Jul 07 00:36:15.019004 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:15.018974 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pcz8g_3d0f8f96-cc8e-43f2-9325-bd89a90b61bf/extractor/0.log" Jul 07 00:36:15.830374 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:15.830345 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-lxc4r" Jul 07 00:36:17.449181 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:17.449145 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-674c64d4c7-mhn92_95d3e48c-6d78-4e38-9799-b2205dcd7725/maas-api/0.log" Jul 07 00:36:17.606878 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:17.606845 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8498ff5b75-hqp5j_8859ca0d-3189-4094-b50b-4ea9ed76032a/manager/0.log" Jul 07 00:36:17.759616 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:17.759536 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7bd8fcf-56xfq_6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8/manager/0.log" Jul 07 00:36:17.781682 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:17.781652 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7bd8fcf-56xfq_6abdbb4c-5f07-42f7-a6e4-8696c76fd3a8/copy-manifests/0.log" Jul 07 00:36:19.165076 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:19.165050 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4dpgtk_c3e9c576-87f4-4523-9b35-7d502774328f/istio-proxy/0.log" Jul 07 00:36:19.283248 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:19.283209 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4zb4x_72d39ae9-8292-482d-8c63-6b1f8755f924/discovery/0.log" Jul 07 00:36:19.332536 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:19.332510 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-khdfj_91f2db2c-526b-427e-a103-215869094379/istio-proxy/0.log" Jul 07 00:36:19.363842 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:19.363813 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-khdfj_91f2db2c-526b-427e-a103-215869094379/istio-proxy/1.log" Jul 07 00:36:19.368246 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:19.368224 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:36:19.385117 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:19.385092 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xrvfm_64985bdd-cb48-4b4a-9b1b-57a17912eef7/payload-pre-processing/0.log" Jul 07 00:36:19.401457 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:19.401417 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lvndj_49f94231-7b6c-4397-8d6f-6b31880621c5/ovn-acl-logging/0.log" Jul 07 00:36:19.438182 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:19.438103 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6647df896d-zlmc4_b1182c64-551f-485a-b95c-fec6ad91086c/manager/0.log" Jul 07 00:36:19.462773 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:19.462746 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zltx2_3b5e5455-2da6-4277-978c-002c919531f4/openshift-lws-operator/0.log" Jul 07 00:36:20.729363 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:20.729334 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4dpgtk_c3e9c576-87f4-4523-9b35-7d502774328f/istio-proxy/0.log" Jul 07 00:36:20.839546 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:20.839460 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4zb4x_72d39ae9-8292-482d-8c63-6b1f8755f924/discovery/0.log" Jul 07 00:36:20.894005 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:20.893970 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-khdfj_91f2db2c-526b-427e-a103-215869094379/istio-proxy/0.log" Jul 07 00:36:20.924965 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:20.924936 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-khdfj_91f2db2c-526b-427e-a103-215869094379/istio-proxy/1.log" Jul 07 00:36:20.944777 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:20.944748 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xrvfm_64985bdd-cb48-4b4a-9b1b-57a17912eef7/payload-pre-processing/0.log" Jul 07 00:36:25.791015 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:25.790988 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncwhk_63306a16-c22f-423d-bb78-aef7a70665d2/kube-multus-additional-cni-plugins/0.log" Jul 07 00:36:25.810805 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:25.810775 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncwhk_63306a16-c22f-423d-bb78-aef7a70665d2/egress-router-binary-copy/0.log" Jul 07 00:36:25.830593 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:25.830565 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncwhk_63306a16-c22f-423d-bb78-aef7a70665d2/cni-plugins/0.log" Jul 07 00:36:25.850089 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:25.850061 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncwhk_63306a16-c22f-423d-bb78-aef7a70665d2/bond-cni-plugin/0.log" Jul 07 00:36:25.880950 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:25.880920 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncwhk_63306a16-c22f-423d-bb78-aef7a70665d2/routeoverride-cni/0.log" Jul 07 00:36:25.901246 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:25.901215 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncwhk_63306a16-c22f-423d-bb78-aef7a70665d2/whereabouts-cni-bincopy/0.log" Jul 07 00:36:25.922303 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:25.922275 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncwhk_63306a16-c22f-423d-bb78-aef7a70665d2/whereabouts-cni/0.log" Jul 07 00:36:26.026635 ip-10-0-141-86 kubenswrapper[2568]: I0707 00:36:26.026606 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p9zd2_36d56419-1664-4eff-aaac-26a7440264bd/kube-multus/0.log"